summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
* object-storage: Restoring multi volume support in UFO.Mohammed Junaid2013-04-295-9/+166
| | | | | | | | | | | | | | | | | | | | | | | | * Currently, the users of UFO are restricted to use just one volume at any given point of time. This patch removes this limitation. * The usage of gluster-swift-gen-builders has also changed. With this commit the users should mention the list of volumes that they want to expose through UFO. So, only the volumes mentioned during the ring file generation can be accessed. Usage: gluster-swift-gen-builders <vol-name1> [<vol-name2>]... This is an intermediate fix until we remove the account, container and object server processes. Once we have this frame work running, it will completely eliminate the ring files. Change-Id: I9ad3808519fec9c7c60ad846c4f8b653117a8337 BUG: 909053 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4485 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com>
* object-storage: Use the wrapper functions provided by fs_utils.py to make ↵Mohammed Junaid2013-04-296-85/+409
| | | | | | | | | | | | | | | | | | | | | system calls. The set of changes: * Unit test cases for fs_utils.py * Replaced os.path with os_path * Implemented wrapper functions do_write, do_chmod, etc in fs_utils.py * Replaced os.<sys-call> with the wrapper functions. Change-Id: I770da878e83eda6b98e49d70193990406a2642a7 BUG: 887301 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4360 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: Store the lock file in /var/run/swift.Mohammed Junaid2013-04-292-8/+55
| | | | | | | | | | | | | | | * Openstack swift uses the /var/run/swift directory to store the pid files for all the servers. * Also, added a script that would unmount the gluster client on a volume stop. Change-Id: Ib5b9a2964987ca7696d9a2570f1f7af8490b2168 BUG: 861497 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4417 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: yield during directory tree walksPeter Portante2013-04-291-10/+15
| | | | | | | | | | | | | | | | | | | Have the co-routine (greenlet) yield during long running directory tree walks to avoid starving out other greenlets that might be available to run. A directory walk involves statting every file in the tree to get it size. For large numbers of files, this can get expensive. So we yield after every stat call, and after processing every directory. Change-Id: I07f1dfeef6a09b5817e0c237ecc748c491d52a31 BUG: 894674 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4380 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: use temp file optimizationMohammed Junaid2013-04-292-16/+32
| | | | | | | | | | | | | | | A file name '.<FILENAME>.<RANDOM>' will hash to the same GlusterFS node as a file named '<FILENAME>', thus avoiding creation/deletion of linkfiles on a rename. This is part of the work needed to address BZ 876660 (https://bugzilla.redhat.com/show_bug.cgi?id=876660). Change-Id: I6f18c14b8eaa7a35c96f7e455ef3a19bee7dbde5 BUG: 876660 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4325 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com>
* object-storage: add UFO unit testsPeter Portante2013-04-292-7/+3
| | | | | | | | | | | | | | | | | | Run the UFO unit tests following the main GlusterFS unit tests. The UFO unit tests run under the Nose unit test running harness, requiring the python-nose package to be installed. These unit tests run fast enough that we could even consider running them on a commit hook run by rfc.sh. Note that they are not functional tests, as they don't require a real Gluster file system to run. Change-Id: I55972890c24be16fc588a6950cb6bfd539287bbe Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4333 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: only open file when requestedPeter Portante2013-04-292-4/+8
| | | | | | | | | | | | | | | | | | | Only open a file when the caller of the constructor indicates that it expects to have a file pointer to access in the "fp" field. Found by Junaid during review http://review.gluster.org/4284, Patch set 6. We also bump the version number to mark a line in the set of changes we want to perform a mini release with to double check for performance regressions. Change-Id: I13cf336bb519088890192ee111f4ece85a5982c4 BUG: 886730 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4303 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: don't sort object list, not req'dPeter Portante2013-04-291-3/+0
| | | | | | | | | | | | | | The REST API for listing objects in a container does not require that the list of objects be sorted (the API for listing containers in an account does require it). Since we can have thousands and thousands of objects in a container, don't sort them when it is not required. Change-Id: I6939ef3fec3ea3814a49e3a3046273304889831c BUG: 887301 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4312 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: do not list directory for deletePeter Portante2013-04-291-4/+5
| | | | | | | | | | | | | | | | Do not list the entire directory contents of the directory containing the file before deleting it. We have already verified the files existence constructing the Gluster_DiskFile object. This fixes a customer issue. See 885281 (https://bugzilla.redhat.com/show_bug.cgi?id=885281). Change-Id: I4425c42af3e03624370c779d74b7f073e6eef024 BUG: 885281 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4302 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: apply upstream DiskFile refactorPeter Portante2013-04-292-73/+81
| | | | | | | | | | | | | Apply the upstream DiskFile refactoring in ahead of its use to easiliy apply the temp file optimization. Change-Id: I2708733eed3d87759c70eb3d9e6cd74ef91d0c7b BUG: 876660 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4288 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* object-storage: a set of cleanups to DiskFilePeter Portante2013-04-292-132/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Here are a set of cleanups for things noticed while working on the unit test coverage: * Remove unused constants, adding a constant for default disk chunk size * Document missing logger parameter from constructor * Add iter_hook paramater to constructor in anticipation of 1.7.4 support * Add back meta_file field even though it is not used in our version of the constructor for paranoid compatibility * Remove is_valid field as it is not referenced anywhere, and it is not a field of the super class * Rename fields only used internally by DiskFile with leading underscores * Convert to using os.path.join() instead of hard coded '/' references * Use data_file field where possible * Assert that put_metadata() will only work when the file exists * Remove update_object() method since it is identical to put_metadata() and a direct write will suffice * Create the directory object only if it does not exist when a marker directory is requested Change-Id: If207fed4c0f423e6bd3232e4507f87de877057c4 BUG: 876660 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4287 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* object-storage: Initial unittest of DiskFile classPeter Portante2013-04-295-49/+1080
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If we had this ahead of time, we could have avoided the errors that were encountered leading to the fix-account-mapping fix (see http://review.gluster.org/4222). This represents 100% coverage of the DiskFile module, but the coverage report says otherwise, unfortunately. That is because the put() method invokes eventlets during the test run, and coverage is not able to accurately track the coverage lines properly. If one comments out the "tpool.execute()" line in DiskFile.put() the coverage report then reports 100% for the DiskFile module. Additionally, we changed DiskFile.put() in four ways that should not change its behavior: 1. Comments were added to explain various code paths and mark potential issues / fixes 2. It no longer returns a boolean value, matching the behavior of swift.obj.server.DiskFile.put() 3. It no longer logs a message when it detects a directory that already exists, instead is raises an exception We believe this is okay because we cannot find a code path that would lead to his condition. As a result, it makes it easier to test all the code paths in that routine. 4. It no longer logs a message when create_dir_object() fails, since create_dir_object() raises an exception on failure only We also modified create_dir_object() to not return a boolean as a result of the above behavior. Note that by implementing these tests up to this point we found three code paths that would have failed if encountered due to missing imports. We also made changes to the DiskFile module to make it a bit easier to test, also eliminating an extra stat system call when deleting directory objects. Change-Id: I3286de83c1ec7c5e8d6cab9354e1c4397cee7497 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4284 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: unit tests to 100% cover utilsPeter Portante2013-04-291-12/+146
| | | | | | | | | | Change-Id: I2d540e16cc3d420b9bb88e6857e9795d56e34529 BUG: 884327 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4225 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: use temp file optimization for pklPeter Portante2013-04-293-10/+73
| | | | | | | | | | | | | | | | | | Override OpenStack Swift's swift.common.utils.write_pickle with our own implementation that uses the GlusterFS temporary file operation. A file name '.<FILENAME>.<RANDOM>' will hash to the same GlusterFS node as a file named '<FILENAME>', those avoiding a move of the file on a rename. This is part of the work needed to address BZ 876660 (https://bugzilla.redhat.com/show_bug.cgi?id=876660). Change-Id: I1cb9f97f289ab2ca76ec9221366df74de08268bb BUG: 876660 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4224 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: Use "-dev" in the version stringPeter Portante2013-04-292-2/+2
| | | | | | | | | | | | | | | | | | | Currently, we have not solved the account mapping story entirely (we don't have a way to support more than one account to gluster volume in the mapping). As such, do not declare this as the final version. Also update the default RHS 2.0 spec file to "-3" for the release number to refer to the fact that we now have a working account mapping (see http://review.gluster.org/4222). Change-Id: Ia1cd0d18c1d7a7b20f732cc0ff867fecda30a7af BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4283 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: enforce RPM package dependenciesPeter Portante2013-04-291-18/+10
| | | | | | | | | | | | | We now have access to openstack-swift-* packages that work with these sources so we can enforce the dependencies and what gets obsoleted. Change-Id: I83f442f3eb2424d176a2257f3498ed39eed32251 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4223 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* object-storage: fix account mappingPeter Portante2013-04-294-10/+8
| | | | | | | | | | | | | | | | | | | | | Fix the account mapping so that we always use what is provided by the request. With this fix, "/mnt/gluster-object/AUTH_ufo0" directory hierachies won't be created. But this fix does not restore the one-to-one account to gluster volume name mapping. When the user runs the /usr/bin/gluster-swift-gen-builders script, it still only allows one cluster volume, and that is the only volume that is used. So the account names are effectively ignored. A future set of changes will address that problem. Change-Id: I2df608de2f00fd356a09c500d49fe12cc1a0a574 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4222 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* Replace "ssh foo gluster" with "gluster --remote-host=foo"Pete Zaitcev2013-04-291-4/+4
| | | | | | | | | | | | | | | | | | | With the current lack of authentication, there is no security improvement with passwordless ssh replaced. However, it is more convenient to install UFO if you do not have to generate ssh keys every time. There is a downside the gluster-server RPM becomes required. BUG: 880050 Change-Id: I04f4ad666ac96713055723bd432131a4db325b4f Signed-off-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-on: http://review.gluster.org/4231 Reviewed-by: Peter Portante <pportant@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Niels de Vos <ndevos@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Add a sorted list comparison to testPete Zaitcev2013-04-291-2/+2
| | | | | | | | | | | | | | | | | | After the commit b0cb7aaf for bz#870589, which adds 2 tarballs with layout patterns, Swift tests sometimes fail (depending on the kind of filesystem they are running at and the phase of the moon). As pattern tarball is unpacked, the underlying filesystem is free to return directory listings in any order, and straightforward use of '==' operator fails. As it turns out, one of the two comparisons had set() applied to the list already. So, add it to the other one. BUG: 874390 Change-Id: I02de99593b9567a13076113d58e242b079fde002 Signed-off-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-on: http://review.gluster.org/4165 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* object-storage: del 1.4.8 ref from dir hierarchyPeter Portante2013-04-2937-2/+2
| | | | | | | | | | | | Remove the "swift/1.4.8" directory from the hierarchy, replacing it with "ufo". Change-Id: I60ba171182d7402a2193616c1c95c90cd2ead5bc BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4200 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: final changes to remove swift diffPeter Portante2013-04-2937-669/+311
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Final set of changes to remove the diffs carried to make UFO work with OpenStack Swift. The code is now a complete layering on top of OpenStack Swift where we either "monkey patch" or subclass as necessary. See BZ 870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589). There are a lot of changes here due for the most part to rearranging the directory hierarchy to have create a proper python module hierarchy under the "gluster" namespace. Plugin references have been removed. The differences that used to be in the swift.diff file are now replaced with server implementations for account, container, object, and proxy that subclass the swift versions. Additionally, the plugins/conf directory has been moved to the "etc" directory, and the plugins/bin directory promoted a level. Unit tests pass. A new setup.py file is provided so that the install process can use it for creating all the necessary python install infrastructure (eggs and paste support). A new RPM spec file is provided which to properly install the new code, and the sample configuration files have been modified to reference the new python egg. Change-Id: I4316c1b66dca80f847fe9b0d583174689c175599 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4180 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: refactor to use swift devicesPeter Portante2013-04-2923-243/+198
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactor code to use the devices configuration file setting for account, container, and object servers, dropping mount_path from the fs.conf file, and constructing new account, container and object server rings. This removes the next to last set of diffs in the swift diff file. See BZ 870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589). The key to the change is the dropping of the pre-built account, container and object rings, instead providing a script that will generate them for the user given the gluster volume name in use. In addition, we override the Swift check_mount() method and replace it with Gluster's which attempts to "auto-mount" if it is not already mounted. The following is an enumeration of the changes contained in this refactoring: * The refactoring to override the Swift check_mount() involved condensing a lot of code, removing redundancies and simplifying methods across a number of modules * Drop checking the mount point in the low level DiskDir, DiskAccount and DiskFile objects now that Swift's normal mount checking is used, and enable it by default in the template configuration file * Add missing get_container_timestamp() method for DiskAccount objects * Fix the plugin RPM spec file to provide the new ring builders, and while we were at it, finally fix the over-writing of the configuration files on install * Bug fixes * The monkey patched version of check_object_creation was not working correctly due to a missing import of HTTPBadRequest and a bad reference to validate_obj_name_component() * Only have the utils module import the plugins constraints module for the side effect of monkey patching if gluster is enabled * Removed the db_file.db file in the tree since it is created on the fly * Updated 2011 copyright notices to 2012 Change-Id: I8f4454576b1423021c9bbf3c36176f8db51e62c0 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4179 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: make proper refs to module varsPeter Portante2013-04-291-8/+4
| | | | | | | | | | | | Not sure how this worked before this, but mount_ip and mount_path do not exist, their uppercase versions do. Change-Id: I53e9b550991d17b5541ed4ada5f1bc9b3cb1b8b1 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4178 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: reduce diffs, bury metadata updatePeter Portante2013-04-294-48/+29
| | | | | | | | | | | | | | Further reduce the diffs we have to carry by moving the Gluster specific metadata stored when we create an object to the DiskFile class's put() method. Change-Id: I8cef5412627d3ddfdadbf78ef947e067db5bf620 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4177 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: swift-diff removal: hide obj onlyPeter Portante2013-04-294-72/+47
| | | | | | | | | | | | | | | | | | | | | | | Towards swift-diff removal: hide object_only behavior inside DiskDir class objects. See BZ 870589 (https://bugzilla.redhat.com/show_bug.cgi?id=870589) One of the last few reasons for having the swift diff is the need to update the account and container information from the file system when the configuration is not for object-only use (meaning, other updates to the file system can come from clients other than UFO, which means UFO currently has to generate the necessary metadata for each request instead of caching it). This change moves that logic into the DiskDir class methods so that the container and account server code does not have to track that. Change-Id: If0970c77c72245facd09c2035083e681d6d80303 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4176 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: fix trailing whitespacePeter Portante2013-04-293-3/+1
| | | | | | | | | Change-Id: Ib3c671e693c2c332af98a593ca14c42c36f5ca76 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4175 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: remove the device and part paramsPeter Portante2013-04-292-9/+6
| | | | | | | | | Change-Id: I60b4b1b976ad8359ba072d5d0ed6a8dd2087a738 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4174 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: only update metadata that changesPeter Portante2013-04-291-2/+7
| | | | | | | | | | | | See BZ 868120 (https://bugzilla.redhat.com/show_bug.cgi?id=868120) Change-Id: Ia7f5562cff32eee30c5ca18652d5d9a5a2b18bf4 BUG: 868120 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4173 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: DiskAccount subclassing of DiskDirPeter Portante2013-04-292-35/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Changes to make DiskAccount subclass DiskDir complete. This change helps reduce the diffs ahead of further refactoring. Note that DiskDir initializes all the same fields as DiskAccount, plus a few more, and ends up reading the metadata properly, while only adding a couple of fields not used for account purposes. This change also passes the account controller's logger for use by the DiskAccount class. Along the way we: * Remove the redundant update_account() method in the DiskCommon base class, since the update_container() method in DiskDir performs the exact same functionality * Remove the initialization of variables ahead of assignments in two cases, and also move the initialization of the list variables closer to where they are used * Collapse the imports from swift.plugins.utils into one statement * Use a temporary directory for the db_file.db when we find ourselves in a unit test environment Change-Id: I4dc59b416a66393adfe0ed32c7a62c79731d70c0 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4172 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Tested-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: fix bad refs to check_acct_existsPeter Portante2013-04-291-4/+6
| | | | | | | | | | | | | | | | | Fix bad references to check_account_exists: this is a module local method, not provided by Glusterfs. Not sure how we missed this. More unit tests would help. Also added the ability to run the Swift tree unit tests with gluster artificially enabled to test as many code paths as possible ahead of full functional tests. Change-Id: Iaf696d34db8f841c887f03569e65321fca8894d1 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4171 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: fix bad refs to REMOTE_CLUSTERPeter Portante2013-04-291-2/+2
| | | | | | | | | Change-Id: I07cd997ef9a98645a87792632f9dd612928678b7 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4170 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: rename self.name to self.containerPeter Portante2013-04-291-11/+12
| | | | | | | | | | | | | | | | | Minor refactoring ahead of further diffs to add clarity to the code. Rename the DiskDir field from "name" to "container", which is what it is. Along the way, add a note regarding the one-to-one mapping of account name to gluster volume name, and fix the get_info() container method documentation to list all the dictionary keys returned. Change-Id: I659e1a9a070c16671287f17a170f9d77198534e7 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4169 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: unused parameter cleanup & handl'nPeter Portante2013-04-291-6/+5
| | | | | | | | | | | | | | | | | | | Unsed parameter cleanup and handling changes. The device and partition parameters are not used. The account parameter is used in place of the device parameter. Rather than just making the device parameter reference the value of the account parameter, we just don't use the device parameter at all and make references to account. We also remove the device_path field from the class since it is unused. Additionally, we assert that we have a logger since the rest of the relies on that fact. Change-Id: Iea556319744ab4f729b1ce23d987650a164749db Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4168 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Tested-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: add missing docs for paramsPeter Portante2013-04-291-3/+3
| | | | | | | | | | | Add missing documentation for constructor parameters. Change-Id: I3a656cd22f6a2a61e0b4d563c75c2f7fe68d3c54 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4167 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: minor code cleanupsPeter Portante2013-04-291-5/+2
| | | | | | | | | | | | | | | | | | Apply a few minor code cleanups to help reduce future refactoring diffs. * Remove extra lines between methods so that all methods are consistently spaced by one line * Remove unused field meta_file * Add missing documentation for uid and gid parameters Change-Id: I0a38eb06680eb0709edfaa1828d09657823f778e Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4166 Reviewed-by: Pete Zaitcev <zaitcev@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: remove glusterfs filter requirementPeter Portante2013-04-2916-495/+429
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes https://bugzilla.redhat.com/show_bug.cgi?id=870589 Remove the Glusterfs object, transforming it into a module providing module data fields (like swift.common.constraints) and module methods for mounting/unmounting and access the gluster volume information. As a result, we can then remove the glusterfs filter from the pipeline since we no longer need to provide the Glusterfs object through all the plugin code paths. This is one more step closer to removing our dependency on modifying the Swift code directly with these changes. See It is also the first step to acknowledging that we are not a plugin, but a layering on top of Swift. The major piece of work here is based on a recognition that the plugins/Glusterfs.py module provided a Glusterfs class that instantiated instances of an object that always contained the same data from the configuration file. The fields of such an object were not being changed and were treated as read-only in all cases. Since the object's data was the same for all instantiations there was no need to pass the data from the glusterfs filter all the way down into the bowels of the Gluster_DiskFile and DiskDir objects. Taking advantage of the nature of that data, we now just have those fields read into module variables, and change the Glusterfs object methods into module level functions. Much of the changes result from the consequence of making that switch from object to module. Here are a few other changes made along the way: * Bump the release numbers in the spec files in recognition of these changes * Create the plugins/fs_utils.py module so that the methods in the plugins/Glusterfs.py module don't have to include plugins/utils.py, which would create a circular dependency * Note that this dependency comes from methods in plugins/utils.py depending on the module level constructs in plugins/Glusterfs.py so that we only store those values in one place * Changed plugins/DiskDir.py:DiskDir class to not check for, and/or optionally create, the /etc/swift/db_file.db at run time, just create it a module init time * Removed the duplicate strip_obj_storage_path() from plugins/DiskDir.py and utils.py and move it to the Glusterfs module * Used os.path.join in plugins/DiskDir.py where possible * Renamed the .conf files to .conf-gluster so that we don't clobber existing config files * This is not a complete change, as the spec file also needs to be modified to avoid the clobbering * See also https://bugzilla.redhat.com/show_bug.cgi?id=865867 * Removed the redundant DIR_TYPE definition in plugins/utils.py * Removed MOUNT_PATH from plugins/utils.py replacing references with that from Glusterfs * This actually fixes a bug if a user every used a different mount path from the default in fs.conf * Added ASYNCDIR definition to plugins/utils.py until such time as another refactoring can rely on the one from swift.obj.server * Renamed plugins/utils.py's plugin_enabled() function to Gluster_enabled() * The diffs we carry for Swift are now a bit smaller in that we no longer have to add the plugin() method, we don't have to keep a fs_object field in these objects, and we can reference the Glusterfs module directly * Unit tests were modified appropriately, but now need to be run in the context of a Swift tree; this is unfortunate, but further refactoring will address this Change-Id: Id5d2510d56364761c03b3979bc71187dbe2f82fe BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4141 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: use constants for directory namesPeter Portante2013-04-291-2/+4
| | | | | | | | | Change-Id: I6109ee55061eae53c4c6762ca31fb5c4b13f0071 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4140 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Vijay Bellur <vbellur@redhat.com>
* object-storeage: refactor to use one memcache keyPeter Portante2013-04-294-65/+394
| | | | | | | | | | | | | | | Address BZ 868087: https://bugzilla.redhat.com/show_bug.cgi?id=868087 Store all of the data needed to generate the correct set of container and account details in one object, respectively, rather than using three seperate memcache keys. Change-Id: I46bf60c405b37cdb22727965bfd67bc5c410e77c BUG: 868087 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4139 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* Minor refactorings to mark an internal routinePeter Portante2013-04-292-9/+8
| | | | | | | | | | | | | | | Just a couple of changes to mark a routine as internal to this module (leading underscore, helps understand what is part of the utils API and what is not), and use of the positional argument for memcache for consistency and reduction in line count (doing this ahead of another refactoring to keep changes concise). Change-Id: I71581ad6ac4c383b1de787b767be568fc0a87eef Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4138 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* Reduce the number of stat/fstat system calls madePeter Portante2013-04-292-39/+34
| | | | | | | | | | | | | | | | | | | | | This address BZ 868086, https://bugzilla.redhat.com/show_bug.cgi?id=868086 This refactoring consolidates the code a bit, and uses os.stat calls to get all the information in one call when gathering the metadata for an object. The five stat system calls (invoked via all the os.path.*() calls) have been reduced to one. We also added a unit test for the one new behavior where get_object_metadata() will now throw an OSError exception if it has a problem stat'ing a file that exists. Change-Id: Iad5410c77938af68a47be757a3170abd201adeb0 BUG: 868086 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4112 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* Further reduce extended attribute reads/writesPeter Portante2013-04-294-41/+281
| | | | | | | | | | | | | | | | | | | | | | | More refactoring towards reducing the number of extended attribute reads/writes to the file system. See BZ 868120: http://bugzilla.redhat.com/show_bug.cgi?id=868120 Basically the redundant routines restore_object, restore_account and restore_container have been collapsed to one routine, restore_metadata, which will only write out metadata if the new metadata is different from what was originally read. Along with these changes come a set of unit tests for all the functions related to the routines that are in this module in the call tree where restore_metadata is invoked. Change-Id: I957ee2f8646cbe6df4d4420d3bdfb1f6ac62bdd2 BUG: 868120 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4111 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* Move _add_timestamp() closer to where it is usedPeter Portante2013-04-291-11/+12
| | | | | | | | | | | | Simple refactoring to move _add_timestamp() closer to where it is used ahead of further refactoring changes. Change-Id: I175c5672202f2d961c72d4e2f4a441b47de1ba28 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4110 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: reduce the number of getxattr system calls by onePeter Portante2013-04-296-13/+281
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Initial step towards addressing BZ 865619. Prior to the patch, reading metadata for a given object or container involves at minimum three getxattr system calls for objects that use only one xattr key/value pair: 1. (via pyxattr) getxattr() to see if key exists and get its value length (so it can allocate memory for second call below) 2. (from pyxattr) getxattr() to get actual value data 3. (via pyxattr) getxattr() to see if following key exists For objects and containers that only have to use one xattr key/value pair, this patch reduces the number system calls by one. This can be significant given that almost every Swift API operation requires reads of the object or containers metadata. This patch is mostly a change to plugins.utils.read_metadata() to try to unpickle the accumulated metadata as each key/value pair is read, rather than trying to accumulate all the key/value pairs and unpickle at the end. Once we get enough data to form the pickle, we no longer keep trying to get more keys, even if those keys exist. The extra keys can exist when the size of the stored metadata shrinks below a key threshold. See https://bugzilla.redhat.com/show_bug.cgi?id=865619 for more details. See also https://bugzilla.redhat.com/show_bug.cgi?id=865858. This patch also adds a unit test infrastructure, and uses it to test with full coverage, the read_metadata, write_metadata and clean_metadata functions. As a result, we found that an infinite loop would occur in clean_metadata() when an unexpected IOError would occur trying to remove a key (this patch contains a fix). Change-Id: Ia1838c5e73af453b65360c1c525824231aa7c5d4 BUG: 865619 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4109 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Bump size of metadata stored per xattr keyPeter Portante2013-04-291-2/+3
| | | | | | | | | | | | | | | | | | | For Gluster, since we require XFS, and XFS has a max metadata value size of 64 KB, use the increased stored size to reduce the number of system calls, and how often we exit and enter the Python interpreter (via calls to pyxattr module). Today, with the hardcoded 254 byte limit per xattr key/value pair, adding a couple hundred bytes of user specified metadata can translate to up to three xattr key/value pairs (remember that the internal python metadata dictionary is pickled first and then stored in chunks in the keys). Change-Id: I6648106e8fac31f973ce207a6fecbcdab11fa271 BUG: 865493 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4108 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Don't catch generic exceptions around xattr callsPeter Portante2013-04-291-2/+2
| | | | | | | | | | | | | We should really avoid catching generic Exception objects around any of our system calls. This simple fix brings this code in line with the other code wrapping xattr calls. Change-Id: I75492a61081fbfbec34c8d91a47214858c67e405 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4099 Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Refactor code to reduce Swift diffs carriedPeter Portante2013-04-2914-546/+379
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The upstream Swift code base contains the following commit which adds the ability to modify constraint values using the swift.conf file: https://github.com/openstack/swift/commit/a2ac5efaa64f57fbbe059066c6c4636dfd0715c2 These changes rely on the above commit being back-ported to Swift 1.4.8 (see https://github.com/portante/swift/commit/fc2421b04022ac6bbe9d5014362ec5f99f94c5e0). As a result, a good number of differences we carry can be removed, since the provided swift.conf file now contains the values we need. Along with these changes, we add middleware to get constraints loaded properly (via monkey patching) until subclassing is implemented in a future commit. We further simplify the diffs to a minimal set by storing the timestamp in file system metadata and moving new files to the plugins/middleware directory. Note that the original "gluster" middleware was in the swift.diff file, and has been renamed to "glusterfs" and moved to the new plugins/middleware directory. The "gluster" middleware is now a temporary way to get the constraints properly loaded (both of these middleware pieces should disappear in future commits when we refactor further to subclass the Swift objects instead of patching them). Change-Id: I9dc00d6b6cdd64e277896d75c2fb06431c3e69cb BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4093 Tested-by: Peter Portante <pportant@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: change logging of some errors to warningsPeter Portante2013-04-291-12/+12
| | | | | | | | | | | | | There are a number of errors logged which should probably be warnings, given that the code recovers on those paths. Change-Id: Ie643bbccb71e7b84ba76d68bac473d1dde663c44 BUG: 866557 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4089 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Remove the test subdirectory diffs ahead of initial refactoring.Peter Portante2013-04-291-136/+0
| | | | | | | | | | | | | | | | | | A subsequent commit will refactor the changes required of Swift to be much more minimal in nature, but does so by relying on an upstream Swift change to allow constraint values specified in the swift.conf file (see https://github.com/openstack/swift/commit/a2ac5efaa64f57fbbe059066c6c4636dfd0715c2). As a result, we can remove the test subdirectory diffs and assume that they will be present as a result of the upstream patch above. Change-Id: Ic450aca235d4c9f345e480f9b761bcb2857bd689 BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4082 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Reduce the number of gratuitous differences in constraints.Peter Portante2013-04-291-4/+6
| | | | | | | | | | | | | | Ahead of further refactoring to make things a bit easier to review, we have restored some of the code formatting as it was in the original constraints.py file from which this file as copied. Change-Id: Icd0b42b4790bc076c54dcbf08a9745648769acfd BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4081 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Fix BZ 865858: remove unnecessary system calls around xattr opsPeter Portante2013-04-291-73/+19
| | | | | | | | | | | | | | | | | The following is a simple refactoring of the read and write metadata methods to just make calls to get/set/remove xattr operations instead of trying to stat/open/close the file. For a single GET operation, this brings down the number of system calls made from 13 to 3 for a file object with no significantly sized metadata (meaning, only 1 xattr key/value pair is needed to store the metadata). Change-Id: I698f5886d63d5203f130c8eac1536a3ee4b0dbfb BUG: 865858 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4078 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
generated by cgit (git 2.34.1) at 2025-07-07 17:47:28 +0000