gluster-swift.git/test/unit/common, branch newton

Fix redundant stat in account and container server

2016-11-18T17:48:42+00:00

Multiple stat() calls were made while serving GET requests for container
and account. This removes those calls and can be easily verified using
strace.

There is room for further refactoring of code to simplify it. This will
be addressed as a separate change to keep things simple in this patch.

Change-Id: Ief457ff869c58519e9dbeb4ef13797185f536673
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/14543
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Use scandir if available

2016-11-18T16:15:52+00:00

scandir[1] is a directory iteration function like os.listdir(), which
can optimize os.walk() by avoiding unnecessary calls to os.stat()

Using scandir to avoid stat() calls requires GlusterFS to correctly
set d_type field of entries in readdir() responses[2].

[1] https://github.com/benhoyt/scandir
[2] http://review.gluster.org/#/c/14095/

Change-Id: Ibdb9a07d25708b5cd8fd663ac99669e7f1f7ba75
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/14460
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

s3: Make s3 support configurable

2016-09-19T01:39:03+00:00

Amazon S3 compatibility:
This change makes S3 support tunable using a config option and is
turned off by default.

This is a manual backport of this upstream swauth change:
https://review.openstack.org/#/c/326336/

Change-Id: I106e3274c6d68f4575c1bf1a9013f066e969cb17
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/15098
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Don't include salt in HMAC computation

2016-09-13T14:00:49+00:00

Currently, the input to HMAC function is the entire stored credential
in the format '$` but it should rather be only the hashed
key/password.

This is a minimal manual backport of this upstream swauth change:
https://review.openstack.org/#/c/292529/

Change-Id: Ib119522d36359f87579ff8e4ada7331643695634
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/15097
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Fix validation of marker dir objects

2016-09-12T17:14:52+00:00

For marker directory objects, validate_object() always returned False.
This was because st_size from stat was being compared to Content-Length
stored in metadata. Unlike files, for directories st_size is always
4096. Hence the comparison would always be '4096 == 0' which would
fail.

This patch makes the following changes:
* Do size comparison of st_size and Content-Length only for files.
* Get rid of _is_dir everywhere. This will simplify things.

Change-Id: Ib75e06c4e3bce36bab11ce7d029ff327f33c3146
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/14423
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Make swift's expirer compatible with gluster-swift

2016-04-20T19:08:03+00:00

Swift's object expirer in kilo series was incompatible with
gluster-swift. This change does the following:

* Optimizes crawl in account and container server for listing
  requests for containers and tracker objects in gsexpiring volume.

* Enables container server to delete tracker objects from gsexpiring
  volume. Swift's expirer sends request directly to container server
  to remove tracker object entry.

* delete_tracker_object() is now a common utility function that is
  invoked from container server and gluster-swift's object expirer.

* Run functional test to be run against both swift's object expirer
  and gluster-swift's object expirer

Change-Id: Ib5b7f7f08fe7dda574f6dd80be2f38bdfaee32bc
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/14038
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Enhance object expiration

2016-04-15T12:27:54+00:00

This change re-introduces object expiration feature with some internal
differences from earlier implementation such as:

* Creation of zero-byte tracker object is performed directly on the
  mount point. Earlier HTTP request was sent to object server
  for the same. This incurred overhead of metadata creation for
  these zero-byte files which isn't necessarry as all required
  information is encoded in the path itself.
* Crawling of zero-byte tracker objects is done by the object
  expirer daemon itself and not container server.
* Deletion of tracker object is performed by the object expiration daemon
  directly on mount point.

Deletion of actual data object is not carried out by object expiration
daemon directly. The object expirer sends a DELETE request to object
server which deletes the actual object. This behaviour is not changed.

There is no change in behaviour in comparison with older implementation.
This is asserted by re-enabling existing functional tests without
any changes.

Change-Id: I01dc77cc4db3be3147d54e3aa2a19ed182498900
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13913
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Don't fetch metadata for plain object listing

2016-04-05T15:52:57+00:00

Fetch metadata (xattr) for objects in a container ONLY when the
client asks for it (using content-type indicating JSON or XML response).
This avoids a lot of unnecessarry stat() and getxattr() calls whose
results would anyways be unused. The performance gain is obvious in
this case.

Change-Id: I4c8b0516dedd04553a5ed450bc855cafbfabada9
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13573
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Don't fetch metadata for plain container listing

2016-04-04T16:24:42+00:00

Fetch metadata (xattr) for containers in an account ONLY when the client
asks for it (using content-type indicating JSON or XML response).
This avoids a lot of unnecessarry stat() and getxattr() calls whose
results would anyways be unused. The performance gain is obvious in this
case.

This change is restricted to container listing. The same can be extended
to object listing as well (will be sent as a separate change)

Change-Id: Ibff1c5a90519f11053c0b651d8ea3385dda43a2f
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13497
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Fix changing of auth_type in existing deployments

2016-03-18T19:43:08+00:00

This changes does two things:
* Adds Sha512 as a supported auth_type.
* Fixes breakage when auth_type is changed in existing deployments.

If an existing gswauth deployment changes `auth_type` in conf file to a
different one (for example: sha1 to sha512), all attempts to authorize
existing/old users will fail because of change in encoder type.

With this change, the credentials match is done using an encoder with
which the password was initially encoded. This allows gswauth deployments
to change auth_type and old users will still be able to authorize.

A note on auth_type_salt:
There's still a distinction between how salt is managed in gswauth and
swauth: swauth will use a random salt if a salt is not set in conf file
where as gswauth will default to 'gswauthsalt' if a salt is not set in
conf file. This distinction is to ensure backward compatibility.

This change is derived from following upstream changes in swauth repo:
e14a7b3df86969d478090b314d9660b6d835afa7
https://review.openstack.org/#/c/285195/
https://review.openstack.org/#/c/285292/

Change-Id: I9a43adc4964d8e9f9f1faf73063a6dc1cd8ff354
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13654
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva