summaryrefslogtreecommitdiffstats
path: root/doc/markdown/s3.md
blob: 086718eac5a148185a4bb510082a3fab7e4bb4b3 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
# Accessing GlusterFS over Amazon S3 API
This guide assumes that you have a GlusterFS cluster with volumes exported over Swift API using gluster-swift.

### Install and configure swift3 middleware
Install swift3 middleware from source or via whatever packaging system you may be using. Here is how you can install from source:

```sh
git clone https://github.com/openstack/swift3
cd swift3
python setup.py install
```

Modify `/etc/swift/proxy-server.conf` to add swift3 middleware in the pipeline to the left of auth middleware (tempauth/gswauth) and add swift3 filter section as illustrated below:

```ini
[pipeline:main]
pipeline = catch_errors cache swift3 tempauth proxy-server

[filter:swift3]
use = egg:swift3#swift3
```
If you are using gswauth, insert swift3 to the left of gswauth similar to the tempauth example above. You can also configure swift3 middleware to [work with keystone](https://github.com/openstack/swift3). You can also [configure tunables](https://github.com/openstack/swift3/blob/master/etc/proxy-server.conf-sample) in the swift3 middleware filter section.

Reload the proxy server process for the change in pipeline to take effect:

```sh
swift-init proxy reload
```

If you are not using any authentication filters, refer to [this section](#no_auth) for example.

### S3 API examples

The examples below use the [s3curl](https://aws.amazon.com/code/128) perl script which is a S3 authentication wrapper around curl command. Modify `s3curl.pl` to point to node running proxy server in your cluster.

```perl
# begin customizing here
my @endpoints = ( 'localhost');
```

These examples assume you have an account (GlusterFS volume) named `test` and an admin user named `tester` with password `testing`:

**Create a bucket**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --put /dev/null -- -k -v -s http://localhost:8080/bucket1
```

**Upload object to the bucket**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --put ./README -- -k -v -s http://localhost:8080/bucket1/a/b/c
```

**Download an object from the bucket**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --get -- -k -v -s http://localhost:8080/bucket1/a/b/c
```

**List buckets**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --get -- -k -v -s http://localhost:8080
```

**List objects in bucket**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --get -- -k -v -s http://localhost:8080/bucket1
```

**Delete object**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --del -- -k -v -s http://localhost:8080/bucket1/a/b/c
```

**Delete bucket (should be empty)**  
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --del -- -k -v -s http://localhost:8080/bucket1
```
**Multi-part upload an object to the bucket**  

  ***a. Initiate multi part***
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --post  --  "http://localhost:8080/bucket/mobject?uploads"
```

***b.upload the multi parts***
upload all the parts by changing partNumber. get the uploadId from the output of Initiate multipart
```sh
./s3curl.pl --id 'test:tester' --key 'testing' --put xaa -s 'http://localhost:8080/bucket/mobject?uploadId=<uploaIdFromPreviousStep>&partNumber=<partNumber>'
```

***c.complete the multi part***

  ****c.1 List the part numbers and Etags****
```sh
./s3curl.pl --id 'test:tester' --key 'testing' -- "http://localhost:8080/bucket/mobject?uploadId=<uploadId>"
```
  ****c.2 make a xml file with following format and do a POST request****
  
  example: cat ./complete_multi_tag
```xml. 
<CompleteMultipartUpload>
  <Part>
    <PartNumber>1</PartNumber>
    <ETag>"9dff188990a9831c5255d342895832df"</ETag>
  </Part>
  <Part>
    <PartNumber>2</PartNumber>
    <ETag>"72e117e4266579616ad295f399ae10a3"</ETag>
  </Part>
 <Part>
    <PartNumber>3</PartNumber>
    <ETag>"85c6e4487367f279926ad5da495a1d20"</ETag>
  </Part>
</CompleteMultipartUpload>
```
```sh
./s3curl.pl --id 'test:tester' --key 'testing'  --post ./complete_multi_tag -- http://localhost:8080/bucket/mobject?uploadId=<uploadId>
```

***d.download the multi part object***
```sh
./s3curl.pl --id 'test:tester' --key 'testing' -- "http://localhost:8080/bucket/mobject" > output_file
```

**Using boto module in python to access GlusterFS cluster over S3 API**
```python
#!/usr/bin/env python

from boto.s3.connection import S3Connection
from boto.s3.connection import OrdinaryCallingFormat
conn = S3Connection(aws_access_key_id='test:tester',
                    aws_secret_access_key='testing',
                    host='localhost', port=8080, is_secure=False,
                    calling_format=OrdinaryCallingFormat())

# List buckets example
buckets_list = conn.get_all_buckets()
for bucket in buckets_list:
     print(bucket.name)
```

### swift3 middleware and gswauth compatibility

Swift3 middleware can be easily used with gswauth when `auth_type` in gswauth is configured to be `Plaintext`.
The AWS S3 client uses password in plaintext to compute HMAC signature.

However, when `auth_type` in gswauth is configured to be `Sha1` or `Sha512`, gswauth can only use the stored hashed password to compute HMAC signature. This results in signature mismatch although the user credentials are correct. The only way for S3 clients to authenticate is by providing SHA1/SHA512 of password as input to it’s HMAC function. To generate hash of password, the S3 clients will have to know `auth_type` and `auth_type_salt` beforehand. Here is an example of gswauth configuration:

```ini
[filter:gswauth]
use = egg:gluster_swift#gswauth
set log_name = gswauth
super_admin_key = gswauthkey
metadata_volume = gsmetadata
token_life = 86400
max_token_life = 86400
auth_type = sha1
auth_type_salt = gswauthsalt
s3_support = on
```

Note that you have to explicitly set `s3_support` to on.

**Example: Create a bucket**

Generate key

```python
#!/usr/bin/env python
import hashlib

key = 'testing'
salt = 'gswauthsalt'

enc_key = '%s%s' % ('gswauthsalt', 'testing')
enc_val = hashlib.sha1(enc_key).hexdigest()
print(enc_val)  # f636f4ecc148efdce89ac471f805d7f51cec348f
```
This is not required when `auth_type` is set to `Plaintext`

```sh
./s3curl.pl --id 'test:tester' --key 'f636f4ecc148efdce89ac471f805d7f51cec348f' --put /dev/null -- -k -v -s http://localhost:8080/bucket1
```

<a name="no_auth" />
### No auth middleware in pipeline
In local and insecure deployments such as within an organization/department where no authentication middleware is in the proxy pipeline, you should just use glusterfs volume name as `id` or `AWSAccessKeyId`.

**Example: Create a bucket**

```sh
./s3curl.pl --id 'test' --key 'blah' --put /dev/null -- -k -v -s http://localhost:8080/bucket1
```
In this example, `test` is the name of glusterfs volume and key can be any string and is ignored.

#### References:
* [Configure Object Storage with the S3 API](http://docs.openstack.org/juno/config-reference/content/configuring-openstack-object-storage-with-s3_api.html)
* [S3 APIs on OpenStack Swift](http://www.buildcloudstorage.com/2011/11/s3-apis-on-openstack-swift.html)
* [swift3 middleware](https://github.com/openstack/swift3)
* [swauth and swift3 compatibility](https://github.com/openstack/swauth/blob/master/doc/source/index.rst#swift3-middleware-compatibility)