Cloud Storage: New API Functions
We recently wrote about our switch to a new platform, which has let us make big improvements to our Cloud Storage. We talked about how we developed our storage’s logic and architecture, as well as how we rewrote several components in Go, which makes our storage even faster and more stable than before.
These are far from all of our changes though: while working on the API, we added several new functions, which we hope you’ll find useful.
Authentication
Our storage fully supports the OpenStack Swift API, which can be used by a multitude of third-party clients.
To maximize compatibility with these clients, we’ve implemented Swift v1 authentication support and limited Keystone v2 and v3 options. This gives our API three authentication methods.
The first method hardly differs from what we used before (only the authorization URI was changed to better meet the standard):
$ curl -i https://api.selcdn.ru/auth/v1.0 -H "X-Auth-User:[username]" -H "X-Auth-Key:[password]
The second method (Keystone v2 authentication) uses a POST request instead of a GET request; the username and password are sent in a JSON array:
$ curl -i -X POST https://api.selcdn.ru/v2.0/tokens -H 'Content-type: application/json' -d '{"auth": {"passwordCredentials": {"username": [username], "password": [password]}}}'
The API sends a response in JSON format.
Keystone v3 looks like the following:
$ curl -i https://api.selcdn.ru/v3/auth/tokens -XPOST -d '{"auth": { "identity": { "methods": ["password"], "password": { "user": { "id": [username], "password": [password]}}}}}'
Please note that the Keystone API has not been fully integrated with our storage, only the login and password authentication functions. We do not support any Keystone protocol extensions.
The primary cloud storage address has also changed. It is now: http://api.selcdn.ru/v1/SEL_XXX (with your username in place of XXX).
Another change pertaining to the authentication system is the new temporary token mechanism. It’s now possible to generate additional tokens with time limits to provide access to a particular container. This is somewhat similar to the Amazon S3 Security Service:
$ curl -i -X GET https://api.selcdn.ru/v1/temptokens -H "X-Auth-Token: [main user’s token]" -H "X-Container:[container]" -H "X-ExpireAfter: 8600"
The main user’s valid token is sent in the X-Auth-Token header, the container that the token grants access to is given in the X-Container header, and the token’s time to live is given in seconds in the X-Expire-After header.
Working with Files
We’ve expanded the new API’s capabilities for working with files. Archives in .tar, .tar.gz, and gzip formats can be uncompressed in background mode after the file has been fully uploaded to storage. The old extraction method would uncompress archives on-the-fly, meaning a connection with our servers needed to be maintained until all of the create file requests from the uploaded archive were executed. If the connection had been interrupted, everything would have had to be uploaded all over again.
This has become much easier. Let’s say you have to upload an archive with several thousand files to storage. You can send a PUT request to the container address with the extract-archive-v2 query parameter. After the archive is sent, a 201 code is immediately returned and background extraction starts on the storage side:
curl -i -X PUT https://api.selcdn.ru/v1/SEL_XXX/[container name]/?extract-archive-v2=tar.bz2' -H "X-Auth-Token: [token]" -T "photos.tar.bz2" HTTP/1.1 100 Continue HTTP/1.1 201 Created Access-Control-Expose-Headers: X-Backend-Timestamp, Etag, Last-Modified, X-Object-Manifest, X-Timestamp Content-Length: 0 Content-Type: text/html Etag: a1adb438cb26e91228870158a2062ef2 Extract-Id: 6a62579d-9ee2-2a32-26a4-207d5a47af2a
Pay attention to the Extract-Id header in the response. It can be used to check the file extraction progress:
$ curl -i https://api.selcdn.ru/v1/extract-archive/[Extract-Id] -H "X-Auth-Token: [token]" HTTP/1.1 200 OK Number Files Created: 1185 Response Body: Response Status: 201 Created Errors:
In our case, a 200 code means all files were extracted successfully and the Number Files Created parameter shows us the total number of files that were extracted.
New Functions
Many new functions have been added to the API, especially for managing users, domains, certificates, and the CDN. We’ll talk more about this below.
Managing Users
Previously, users could only be added and deleted from the control panel. Now this can be done by sending standard HTTP requests to the API.
To view a list of users:
$ curl -i -XGET https://api.selcd.ru/v1/users[?format=json] -H "X-Auth-Token: [token]"
New users can be created with the following command:
$ curl -i -XPUT https://api.selcd.ru/v1/users/new_user -H "X-Auth-Token: [token]" -H "X-Auth-Key: 123456" -H "X-User-ACL-Containers-W: container1, container2, container3" -H "X-User-ACL-Containers-R: container4" -H "X-User-Store-Password: yes" -H "X-User-Active: on"
We had been asked multiple times to improve our user permission management. We’ve followed through on this, and it is evident in the example above: take a look at the X-User-ACL-Containers-W and X-User-ACL-Containers-R header. The former shows the containers where the new user can write information, and the latter shows containers with read-only access.
The X-Auth_key header shows the password for our new user. X-User-Store-Password is the same as the “Save password on Selectel servers” option in the control panel. The on value for the X-User-Active header means the new user will be active.
Settings for existing users can be changed as well:
$ curl -i -XPOST https://api.selcd.ru/v1/users/new_user -H "X-Auth-Token: [token]" -H "X-Auth-Key: 123456" -H "X-User-ACL-Containers-W: container1, container4" -H "X-User-ACL-Containers-R: container4" -H "X-User-Store-Password: yes" -H "X-User-Active: on"
Users can be deleted with a DELETE request:
$ curl -i -XDELETE https://selcdn.ru/users/[username] -H "X-Auth-Token: [token]"
The only thing you can change for the primary user is their password. This is done by sending a POST request with the X-Auth-Key header to the address https://api.selcd.ru/v1/users/[username].
Managing Domains
Previously, domains could only be managed from the GUI. Now all domain operations can be performed via API.
To view a list of domains attached to your containers:
$ curl -i https://api.selcdn.ru/v1/domains -H "X-Auth-Token: [token]"
Domains can be attached by sending POST requests with the X-Add-Container-Domains header to the the corresponding container:
$ curl -i https://api.selcdn.ru/v1/SEL_XXX/{container_name} -XPOST -H "X-Add-Container-Domains: domain1.ru" -H "X-Auth-Token: [token]"
Domains can similarly be deleted using the X-Remove-Container-Domains header:
$ curl -i https://api.selcdn.ru/v1/SEL_XXX/{container_name} -XPOST -H "X-Remove-Container-Domains: domain1.ru" -H "X-Auth-Token: [token]"
To completely change the list of domains bound to a container, we have to first send a POST request to the container’s address and then transfer the updated list in the X-Container-Domains header:
$ curl -i https://api.selcdn.ru/SEL_XXX/[container name] -H "X-Auth-Token: [token]" -XPOST -H "X-Сontainer-Domains: domain1.ru domain2.ru"
If several domains are attached to a container and they need to be deleted, we can send the X-Container-Domains header with the value 0.
$ curl -i https://ххххх.selcdn.ru/[container name] -H "X-Auth-Token: [token]" -XPOST -H "X-Сontainer-Domains: 0"
Requesting Logs
The API can be used to request access logs to user containers for a defined period of time:
$ curl -i -XPOST https://api.selcdn.ru/v1/logs -H "X-Auth-Token: [token]" -H "X-Start-Time: 2016-02-02 09:00:00" -H "X-End-Time: 2016-05-02 12:00:00" -H “X-Limit: 10000”
As you can probably guess, we enter the start and end time in the X-Start-Time and X-End-Time headers respectively.
With the optional X-Limit header, we can enter the maximum number of log entries to be written for the request (the default option is to send the whole log for the given interval).
Once the logs have been compiled, a private container labeled “logs” will be automatically created in storage, which is where your logs will be saved.
Managing SSL Certificates
Another new function is SSL certificate management. Now certificates can be managed both from the GUI and API.
To view a list of certificates attached to containers:
$ curl -i https://api.selcdn.ru/v1/ssl -XGET -H "X-Auth-Token: [token]"
To view certificate information:
$ curl -i https://api.selcdn.ru/v1/ssl/{cert_name} -XGET -H "X-Auth-Token: [token]"
To add a certificate:
$ curl -i https://api.selcdn.ru/v1/ssl/{cert_name} -XPUT -H "X-Auth-Token: [token]"
The certificate name ({cert_name}) is given in 001_cert1 format, where the number is the user login, and the second element is an arbitrary name. Certificate names cannot be repeated. The actual certificate and private key are sent in the body of the request as one file.
Certificates can be deleted with a DELETE request:
$ curl -I https://api.selcdn.ru/v1/ssl/001_cert1 -H "X-Auth-Token: [token]" -XDELETE
CDN Management
It is now possible to manage the CDN from the API. You can send requests to purge the cache (this used to be possible only from the GUI) and view the status of these requests.
The CDN can be purged using the PURGE request:
$ curl -i -X PURGE https://api.selcdn.ru/v1/cdn -H "X-Auth-Token: [token]" -d 'https://api.selcdn.ru/v1/SEL_XXX/container1/file1\nhttps://XXX.selcdn.ru/container1/file1'
We are now using Akamai’s CCU v3 with fast purge. After this request is sent, the cache should be cleane in about 5 seconds. The response will look like the following:
{"estimatedSeconds": 5, "purgeId": "0bb4d6fa-4ce4-11e6-b75e-9fdde8cfe544", "supportId": "17PY1468845301403318-210404544", "httpStatus": 201, "detail": "Request accepted"}
Conclusion
In this article, we looked at the new API functions for our Cloud Storage. We plan on making further improvements to the service. If you have any suggestions, please share them in the comments below. We’ll certainly consider your ideas.