| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Final set of changes to remove the diffs carried to make UFO work with
OpenStack Swift. The code is now a complete layering on top of OpenStack Swift
where we either "monkey patch" or subclass as necessary.
See BZ 870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589).
There are a lot of changes here due for the most part to rearranging the
directory hierarchy to have create a proper python module hierarchy under the
"gluster" namespace. Plugin references have been removed. The differences that
used to be in the swift.diff file are now replaced with server implementations
for account, container, object, and proxy that subclass the swift versions.
Additionally, the plugins/conf directory has been moved to the "etc"
directory, and the plugins/bin directory promoted a level.
Unit tests pass.
A new setup.py file is provided so that the install process can use it for
creating all the necessary python install infrastructure (eggs and paste
support).
A new RPM spec file is provided which to properly install the new code, and
the sample configuration files have been modified to reference the new python
egg.
Change-Id: I4316c1b66dca80f847fe9b0d583174689c175599
BUG: 870589
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4180
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Refactor code to use the devices configuration file setting for account,
container, and object servers, dropping mount_path from the fs.conf file, and
constructing new account, container and object server rings.
This removes the next to last set of diffs in the swift diff file. See BZ
870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589).
The key to the change is the dropping of the pre-built account, container and
object rings, instead providing a script that will generate them for the user
given the gluster volume name in use. In addition, we override the Swift
check_mount() method and replace it with Gluster's which attempts to
"auto-mount" if it is not already mounted.
The following is an enumeration of the changes contained in this refactoring:
* The refactoring to override the Swift check_mount() involved condensing
a lot of code, removing redundancies and simplifying methods across a
number of modules
* Drop checking the mount point in the low level DiskDir, DiskAccount and
DiskFile objects now that Swift's normal mount checking is used, and
enable it by default in the template configuration file
* Add missing get_container_timestamp() method for DiskAccount objects
* Fix the plugin RPM spec file to provide the new ring builders, and while
we were at it, finally fix the over-writing of the configuration files
on install
* Bug fixes
* The monkey patched version of check_object_creation was not working
correctly due to a missing import of HTTPBadRequest and a bad
reference to validate_obj_name_component()
* Only have the utils module import the plugins constraints module for
the side effect of monkey patching if gluster is enabled
* Removed the db_file.db file in the tree since it is created on the
fly
* Updated 2011 copyright notices to 2012
Change-Id: I8f4454576b1423021c9bbf3c36176f8db51e62c0
BUG: 870589
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4179
Reviewed-by: Pete Zaitcev <zaitcev@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Change-Id: Ib3c671e693c2c332af98a593ca14c42c36f5ca76
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4175
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=870589
Remove the Glusterfs object, transforming it into a module providing module
data fields (like swift.common.constraints) and module methods for
mounting/unmounting and access the gluster volume information. As a result, we
can then remove the glusterfs filter from the pipeline since we no longer need
to provide the Glusterfs object through all the plugin code paths.
This is one more step closer to removing our dependency on modifying the Swift
code directly with these changes. See It is also the first step to acknowledging
that we are not a plugin, but a layering on top of Swift.
The major piece of work here is based on a recognition that the
plugins/Glusterfs.py module provided a Glusterfs class that instantiated
instances of an object that always contained the same data from the
configuration file. The fields of such an object were not being changed and
were treated as read-only in all cases. Since the object's data was the same
for all instantiations there was no need to pass the data from the glusterfs
filter all the way down into the bowels of the Gluster_DiskFile and DiskDir
objects.
Taking advantage of the nature of that data, we now just have those fields
read into module variables, and change the Glusterfs object methods into
module level functions. Much of the changes result from the consequence of
making that switch from object to module.
Here are a few other changes made along the way:
* Bump the release numbers in the spec files in recognition of these changes
* Create the plugins/fs_utils.py module so that the methods in the
plugins/Glusterfs.py module don't have to include plugins/utils.py, which
would create a circular dependency
* Note that this dependency comes from methods in plugins/utils.py
depending on the module level constructs in plugins/Glusterfs.py so that
we only store those values in one place
* Changed plugins/DiskDir.py:DiskDir class to not check for, and/or
optionally create, the /etc/swift/db_file.db at run time, just create it a
module init time
* Removed the duplicate strip_obj_storage_path() from plugins/DiskDir.py and
utils.py and move it to the Glusterfs module
* Used os.path.join in plugins/DiskDir.py where possible
* Renamed the .conf files to .conf-gluster so that we don't clobber existing
config files
* This is not a complete change, as the spec file also needs to be
modified to avoid the clobbering
* See also https://bugzilla.redhat.com/show_bug.cgi?id=865867
* Removed the redundant DIR_TYPE definition in plugins/utils.py
* Removed MOUNT_PATH from plugins/utils.py replacing references with that from
Glusterfs
* This actually fixes a bug if a user every used a different mount path
from the default in fs.conf
* Added ASYNCDIR definition to plugins/utils.py until such time as another
refactoring can rely on the one from swift.obj.server
* Renamed plugins/utils.py's plugin_enabled() function to Gluster_enabled()
* The diffs we carry for Swift are now a bit smaller in that we no longer
have to add the plugin() method, we don't have to keep a fs_object field
in these objects, and we can reference the Glusterfs module directly
* Unit tests were modified appropriately, but now need to be run in the
context of a Swift tree; this is unfortunate, but further refactoring will
address this
Change-Id: Id5d2510d56364761c03b3979bc71187dbe2f82fe
BUG: 870589
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4141
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Address BZ 868087: https://bugzilla.redhat.com/show_bug.cgi?id=868087
Store all of the data needed to generate the correct set of container and
account details in one object, respectively, rather than using three seperate
memcache keys.
Change-Id: I46bf60c405b37cdb22727965bfd67bc5c410e77c
BUG: 868087
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4139
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Just a couple of changes to mark a routine as internal to this module (leading
underscore, helps understand what is part of the utils API and what is not),
and use of the positional argument for memcache for consistency and reduction
in line count (doing this ahead of another refactoring to keep changes
concise).
Change-Id: I71581ad6ac4c383b1de787b767be568fc0a87eef
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4138
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This address BZ 868086, https://bugzilla.redhat.com/show_bug.cgi?id=868086
This refactoring consolidates the code a bit, and uses os.stat calls to get
all the information in one call when gathering the metadata for an object. The
five stat system calls (invoked via all the os.path.*() calls) have been
reduced to one.
We also added a unit test for the one new behavior where get_object_metadata()
will now throw an OSError exception if it has a problem stat'ing a file that
exists.
Change-Id: Iad5410c77938af68a47be757a3170abd201adeb0
BUG: 868086
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4112
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Tested-by: Anand Avati <avati@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
More refactoring towards reducing the number of extended attribute
reads/writes to the file system. See BZ 868120:
http://bugzilla.redhat.com/show_bug.cgi?id=868120
Basically the redundant routines restore_object, restore_account and
restore_container have been collapsed to one routine, restore_metadata, which
will only write out metadata if the new metadata is different from what was
originally read.
Along with these changes come a set of unit tests for all the functions
related to the routines that are in this module in the call tree where
restore_metadata is invoked.
Change-Id: I957ee2f8646cbe6df4d4420d3bdfb1f6ac62bdd2
BUG: 868120
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4111
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Anand Avati <avati@redhat.com>
|
|
|
Initial step towards addressing BZ 865619.
Prior to the patch, reading metadata for a given object or container involves
at minimum three getxattr system calls for objects that use only one xattr
key/value pair:
1. (via pyxattr) getxattr() to see if key exists and get its value
length (so it can allocate memory for second call below)
2. (from pyxattr) getxattr() to get actual value data
3. (via pyxattr) getxattr() to see if following key exists
For objects and containers that only have to use one xattr key/value pair,
this patch reduces the number system calls by one. This can be significant
given that almost every Swift API operation requires reads of the object or
containers metadata.
This patch is mostly a change to plugins.utils.read_metadata() to try to
unpickle the accumulated metadata as each key/value pair is read, rather than
trying to accumulate all the key/value pairs and unpickle at the end. Once we
get enough data to form the pickle, we no longer keep trying to get more keys,
even if those keys exist.
The extra keys can exist when the size of the stored metadata shrinks below a
key threshold. See https://bugzilla.redhat.com/show_bug.cgi?id=865619 for more
details.
See also https://bugzilla.redhat.com/show_bug.cgi?id=865858.
This patch also adds a unit test infrastructure, and uses it to test with full
coverage, the read_metadata, write_metadata and clean_metadata functions. As a
result, we found that an infinite loop would occur in clean_metadata() when an
unexpected IOError would occur trying to remove a key (this patch contains a
fix).
Change-Id: Ia1838c5e73af453b65360c1c525824231aa7c5d4
BUG: 865619
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4109
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Anand Avati <avati@redhat.com>
|
