summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAgeFilesLines
* glusterfind: Speed up gfid lookup 100x by using an SQL indexNiklas Hambüchen2017-12-301-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #1529883. This fixes some bits of `glusterfind`'s horrible performance, making it 100x faster. Until now, glusterfind was, for each line in each CHANGELOG.* file, linearly reading the entire contents of the sqlite database in 4096-bytes-sized pread64() syscalls when executing the SELECT COUNT(1) FROM %s WHERE 1=1 AND gfid = ? query through the code path: get_changes() parse_changelog_to_db() when_data_meta() gfidpath_exists() _exists() In a quick benchmark on my laptop, doing one such `SELECT` query took ~75ms on a 10MB-sized sqlite DB, while doing the same query with an index took < 1ms. Change-Id: I8e7fe60f1f45a06c102f56b54d2ead9e0377794e BUG: 1529883 Signed-off-by: Niklas Hambüchen <mail@nh2.me>
* tools/setgfid2path: Tool to set GFID to Path xattr in brick backendAravinda VK2017-07-285-1/+202
| | | | | | | | | | | | | | | | | | | | Once storage/gfid2path feature is enabled using `gluster volume set <volname> storage.gfid2path enable`, it starts recording the gfid2path xattr on each files. But this feature will not add xattr to the existing files. This tool accepts the file path as argument and sets the necessary xattr required for this feature. Change-Id: I75ad82c86ce482950645e687ff2e33b413fa53da Updates: #139 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: https://review.gluster.org/17839 Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
* tools/glusterfind: Fix encoding to encode only space,newline and percent charsAravinda VK2017-07-216-50/+33
| | | | | | | | | | | | | | | | | libgfchangelog was encoding path using spec rfc3986, but encoding only required for SPACE, NEWLINE and PERCENT chars since the NEWLINE char is used as record separator and SPACE as field separator in the parsed changelogs output. Changed the encoding function to encode only SPACE, NEWLINE and PERCENT chars BUG: 1451724 Change-Id: Ic1dea824d23493dedcf3db45f353f90572f4e046 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: https://review.gluster.org/17788 CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com>
* tools/glusterfind: unquote DELETE path before further processingMilind Changire2017-06-301-2/+3
| | | | | | | | | | | | | | | | | | | Problem: DELETE path is quoted before it reaches glusterfind. This wasn't handled in the glusterfind code leading to double quoting of path separator '%2F' to '%252F' i.e. the '%' character in '%2F' itself was quoted to '%25' Solution: unquote the the deleted path before further processing Change-Id: I2dfbbd7792dc0f9da5c8e02093b0f1c031ff344a BUG: 1465024 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17629 Smoke: Gluster Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: initialize variable 'end'Milind Changire2017-06-191-0/+1
| | | | | | | | | | | | | | | | | Problem: syntax error due to uninitialized variable 'end' Solution: set end = -1 Change-Id: Ia8b500e9dd5304d0832e5f594abca0b5b2ab357e BUG: 1462241 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17557 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: add --end-time optionMilind Changire2017-06-123-24/+45
| | | | | | | | | | | | | | | | | | | | | | | | | Add optional --end-time argument to be used with --since-time when using the "query" command. "start" and "end" times are passed in the command-line to changelog.py only if --full has not been specified on command-line. -1 is passed to changelog.py as end time if user has not supplied --end-time on command-line. brickfind.py: Remove unused "start" command-line argument. Also: Minor indentation changes to keep flake8-2.7 happy. Change-Id: I063ef5459916f711503881ade5c4fc32374edad0 BUG: 1453151 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17439 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: add --field-separator optionMilind Changire2017-06-123-10/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Default field separator is a space character. This gets in the way if the file name itself has embedded spaces. Solution: Add --field-separator option to "pre" and "query" commands. The field separator string will be used to separate strings in the output lines that get written to the output file. eg. old output: NEW file1.txt RENAME file2 Copy.txt file3.txt with --field-separator as "===" new output: NEW===file1.txt RENAME===file2 Copy.txt===file3.txt Change-Id: I71e878fed58ba1113d97044ac9f6404ee66227c7 BUG: 1453151 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17481 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* build/packaging: Debian and Ubuntu don't have /usr/libexecKaleb S. KEITHLEY2017-03-133-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | GLUSTERFS_LIBEXECDIR is effectively hard-coded to /usr/libexec/glusterfs in configure(.ac) Debian-based distributions don't have a /usr/libexec/ directory This issues is partially mitigated by the use of $libexecdir in some of the Makefile.am files, but even so the incorrectly defined GLUSTERFS_LIBEXECDIR results in various things such as gsyncd, glusterfind, eventsd, etc., trying to invoke other scripts and programs from a location that doesn't exist. And once we correctly define GLUSTERFS_LIBEXECDIR, then we might as well use it appropriatedly. Change-Id: If5219cadc51ae316f7ba2e2831d739235c77902d BUG: 1430841 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: https://review.gluster.org/16880 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Joe Julian <me@joejulian.name> Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
* tools/glusterfind: delete temporary folderMilind Changire2017-01-301-2/+2
| | | | | | | | | | | | | | | | | | | Problem: Run specific temporary folder created under /usr/var/lib/misc/glusterfsd/glusterfind/<session>/<volume>/ remains undeleted. Solution: Delete the temporary folder from all nodes. Change-Id: I0edaf868aebb01b15c489434bbb26fe853351384 BUG: 1413526 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/16416 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: avoid deleting keys directoryMilind Changire2016-12-071-0/+4
| | | | | | | | | | | | | | | | | | Problem: gluster volume delete mistakenly deletes the .keys directory under /var/lib/glusterd/glusterfind. Solution: Check for ".keys" directory and avoid deleting it. Change-Id: Ia595c8bf3f423c1ad5d6faa183a29598c07a11f9 BUG: 1402369 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/16052 Reviewed-by: Aravinda VK <avishwan@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
* tools/glusterfind: xml parsing fix for tiered volumesMilind Changire2016-11-171-3/+12
| | | | | | | | | | | | | | | | | | | gluster volume info <vol> --xml for non-tiered volumes have 'bricks/brick' elements under the 'volInfo/volumes/volume' element. However, tiered volumes have a 'bricks/hotBricks/brick' and 'bricks/coldBricks/brick' elements under the 'volInfo/volumes/volume' element. Fix main.py::get_nodes() BUG: 1389481 Change-Id: I2f4465bfa8a55e7fa87917d3ec3e69b05d5241b9 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/15746 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: kill remote processes and separate run-time directoriesMilind Changire2016-10-254-16/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem #1: Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre has been initiated. Solution #1: Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned to remote process. When local process receives Keyboard Interrupt, SIGHUP is immediately conveyed to the remote terminal causing remote changelog.py process to terminate immediately. Problem #2: Concurrent glusterfind pre runs are not possible on the same glusterfind session in case of a runaway process. Solution #2: glusterfind pre runs now add random directory name to the working directory to store and manage temporary database and changelog processing. If KeyboardInterrupt is received, the function call run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename) cleans up the remote run specific directory. Patch: 7571380 cli/xml: Fix wrong XML format in volume get command broke "gluster volume get <vol> changelog.rollover-time --xml" Now fixed function utils.py::get_changelog_rollover_time() Fixed spurious trailing space getting written if second path is empty in main.py::write_output() Fixed repetitive changelog processing in changelog.py::get_changes() Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29 BUG: 1382236 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/15609 Smoke: Gluster Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* core: use readdir(3) with glibc, and associated cleanupKaleb S. KEITHLEY2016-07-181-20/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3) is marked as deprecated. Specifically the function decl in <dirent.h> has the deprecated attribute, and now warnings are thrown during the compile on Fedora 25 builds. The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say that glibc's readdir(3) is, and always has been, MT-SAFE as long as only one thread is accessing the directory object returned by opendir(). World+Dog also says there is a potential buffer overflow in readdir_r(). World+Dog suggests that it is preferable to simply use readdir(). There's an implication that eventually readdir_r(3) will be removed from glibc. POSIX has, apparently deprecated it in the standard, or even removed it entirely. Over and above that, our source near the various uses of readdir(_r)(3) has a few unsafe uses of strcpy()+strcat(). (AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see if the same is true on those platforms, and we can't be sure of MacOS even though we know it's based on *BSD.) Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568 BUG: 1356998 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/14838 Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* packaging: Remove ".py" extension from symlink targetAravinda VK2016-07-151-3/+3
| | | | | | | | | | | | | | | | | | | | | | S57glusterfind-delete-post.py* is packaged in $LIBEXEC/glusterfs/glusterfind directory, symlink created to /var/lib/glusterd/hooks directory ln -s $(libexecdir)/glusterfs/glusterfind/S57glusterfind-delete-post.py \ $(DESTDIR)$(GLUSTERD_WORKDIR)/hooks/1/delete/post/S57glusterfind-delete-post.py ".py" should be removed from symlink target so that build system will not try to compile the symlink files and generate .pyc/.pyo files. BUG: 1356868 Change-Id: I9155378aa72eb8559ba8af76d91a4985f69f38f9 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/14928 Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* scripts: bash-isms in scriptsKaleb S KEITHLEY2016-05-182-8/+9
| | | | | | | | | | | | | Change-Id: I05f8718fb73c16dbb37c1f7460bb7b55cc0ffe97 BUG: 1336853 Reported-by: Patrick Matthäi <pmatthaei@debian.org> Signed-off-by: Kaleb S KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/14401 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* tools/glusterfind: add --full option to query commandMilind Changire2016-03-303-35/+81
| | | | | | | | | | | | | | | | | The --full option will use brickfind.py to list all files in the volume. The output file will contain url-encoded file names prefixed with the tag string to indicate that all files should be considered as new. The default tag string for the --full option is "NEW". This can be changed with the --tag-for-full-find command-line option. Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792 BUG: 1286279 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/12779 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: Handling Unicode file namesAravinda VK2016-03-304-48/+34
| | | | | | | | | | | | | | | | | | | | Unicode filenames handled cleanly with this patch. Changelog files and output files are opened with utf-8 encoding using codecs.open. urllib.quote_plus and unquote_plus will not handle Unicode so, encode Unicode to 8-bit string version before calling unquote. urllib.quote_plus requires 8-bit string itself so do not decode to Unicode if we need to use quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set. BUG: 1319717 Change-Id: If5561c749ab5529445650d322c831eb4da22b65a Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/13798 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* tools/glusterfind: Do not fail cleanup script on volume deleteAravinda VK2016-03-081-1/+6
| | | | | | | | | | | | | | | If glusterfind dir is not present inside GLUSTERD_WORKDIR, then cleanup script Post Volume delete was failing. This patch safely handles the absence of glusterfind directory in GLUSTERD_WORKDIR BUG: 1289935 Change-Id: I9f7e6035e3dc6079927803df942b0f921218c79a Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/12923 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* tools/glusterfind: New option --no-encodeAravinda VK2016-02-269-42/+112
| | | | | | | | | | | | | | | | | | | | | | | | | New option added to skip encoding path in output file. Also handled Unicode strings. File paths can have newline characters, to differentiate between each path patch is encoded according to RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer applications have to decode the path before consuming it. With this option Paths are not encoded, can be directly consumed by applications. Unicode encoding is handled automatically BUG: 1310080 Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/13477 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* tools/glusterfind: add query command to list filesMilind Changire2015-11-242-38/+155
| | | | | | | | | | | | | | | | | | | | | | | When session information is maintained outside Gluster, there needs to be some mechanism to list files starting from a time-stamp. This patch implements the feature via the "query" command-line option. The only caveat is that the first time the query command is run for the volume, it will likely report that "historical changelogs are not available". This is due to the fact that changelogs had not been turned on for the volume so far. So the volume options need to be turned on outside glusterfind or the since_time need to be greater than the current time when the query command is run for the very first time for the volume. The query command turns on the required volume options for collecting changelogs. Change-Id: I6cb7a57a5ecd166210e2eb4deede06d40ccfa996 BUG: 1272006 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/12362 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: StrictHostKeyChecking=no for ssh/scp verificationAravinda VK2015-11-231-1/+5
| | | | | | | | | | | | | Also do not use scp command in case copy file from local node. Change-Id: Ie78c77eb0252945867173937391b82001f29c3b0 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1260918 Reviewed-on: http://review.gluster.org/12124 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: Handle Keyboard interruptAravinda VK2015-11-181-43/+53
| | | | | | | | | | | | | | Do not print Python traceback when glusterfind command is interrupted Change-Id: I67383534f965e410fef7ce09798e9d435ef738ae Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1243812 Reviewed-on: http://review.gluster.org/11698 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* build: MKDIR_P is not defined for Makefiles, use mkdir_p insteadNiels de Vos2015-11-101-2/+2
| | | | | | | | | | | Change-Id: Id6d5263eb7b1c53e72a7668e716e9cc4e34b82cd Reported-by: Milind Changire <mchangir@redhat.com> BUG: 1198849 Signed-off-by: Niels de Vos <ndevos@redhat.com> Reviewed-on: http://review.gluster.org/12553 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* core: use syscall wrappers instead of direct syscalls - miscellaneousKaleb S. KEITHLEY2015-10-281-1/+1
| | | | | | | | | | | | | | | various xlators and other components are invoking system calls directly instead of using the libglusterfs/syscall.[ch] wrappers. If not using the system call wrappers there should be a comment in the source explaining why the wrapper isn't used. Change-Id: I1f47820534c890a00b452fa61f7438eb2b3f667c BUG: 1267967 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/12276 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* tools/glusterfind: Do not show session corrupted if no status fileAravinda VK2015-08-271-1/+1
| | | | | | | | | | | | | | | | | When a glusterfind session is created it creates session directories in all the nodes which are part of the Volume. But session status file only present in initiated node. Show Session corrupted only if status file exists and invalid content. Change-Id: I8443c0335b872645b54c0aa77d9893dbe2589c92 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1243838 Reviewed-on: http://review.gluster.org/11699 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* tools/glusterfind: Prepend prefix in case of deleteSaravanakumar Arumugam2015-08-263-16/+23
| | | | | | | | | | | | | | | In case of delete operation alone, output prefix adding was not handled earlier. Output prefix is added now. Change-Id: Ia91444dddbff501b26a864f6185ca4c0aaf4c802 BUG: 1244144 Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-on: http://review.gluster.org/11712 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Milind Changire <mchangir@redhat.com>
* rpm: include required directory for glusterfindNiels de Vos2015-08-192-1/+71
| | | | | | | | | | | | | | | | | | | The directory was marked as %ghost, which causes the following installation failure: Error unpacking rpm package glusterfs-server-3.8dev-0.446.git45e13fe.el7.centos.x86_64 error: unpacking of archive failed on file /var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post.py;5581f20e: cpio: open Also, *all* Python files should be part of the RPM package. This includes generated .pyc and .pyo files. BUG: 1225465 Change-Id: Iee74905b101912c4a845257742c470c3fe42ce2a Signed-off-by: Niels de Vos <ndevos@redhat.com> Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/11298 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* tools/glusterfind: password prompts for peer nodes on deleteMilind Changire2015-08-041-0/+6
| | | | | | | | | | | | | | | | | SSH keys getting deleted on local node caused password prompts for peer nodes on a delete operation. This race for session directory cleanup on local node is now addressed by avoiding the delete path for local node in node_cmd() The session directory tree is deleted anyway in mode_delete() Change-Id: I51c4baf4f9c1ed3caa319d4163bef343bd621429 BUG: 1243785 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/11693 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: RENAME and MODIFY issuesAravinda VK2015-07-052-12/+30
| | | | | | | | | | | | | | | | | | | If Modification happens before RENAME, GFID to Path Conversion converts it into New Path. Delete Modify Entry and insert again So that MODIFY <NEW NAME> comes after RENAME. Default value of pgfids and basenames changed to "" instead of NULL Also fixed RENAME issue of displaying "RENAME <NEW NAME> <NEW NAME>". Also fixed RENAME followed by missing MODIFY Change-Id: I8202f6e6ec33f7bd921e71da38677f2ee2dab87a BUG: 1236270 Signed-off-by: Kotresh HR <khiremat@redhat.com> Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/11443 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
* build: Mac OS X build issues, no spinlock, need sys_lgetxattrKaleb S. KEITHLEY2015-07-052-15/+11
| | | | | | | | | | | | use regular locks, use our syscall wrappers in libglusterfs Change-Id: I7e0d00956366806af041b69b65d1f169aa0d2ae2 BUG: 1238793 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/11515 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Niels de Vos <ndevos@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
* tools/glusterfind: Fail glusterfind creation if volume is offlineKotresh HR2015-06-181-8/+17
| | | | | | | | | | | | | | | | | | | Following two fixes are done. 1. Fail glusterfind session creation if volume is not online even before session directories are created. This avoids 'glusterfind list' to pick the session directories and show status as 'Session Corrupted'. 2. Check of '!Started' instead of wether the volume is 'Stopped'. It covers all the cases. Change-Id: I3e9cb384d978ada28f508c07e37d6ceb2272a731 BUG: 1232729 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11278 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: print message for good casesMilind Changire2015-06-131-6/+16
| | | | | | | | | | | | Added messages for printing status of good or default cases for 'create', 'list', 'post' and 'delete' commands Change-Id: Ie983bff7f5a66f118de06c225166ff6bee870438 BUG: 1230017 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/11208 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tools/glusterfind: ignoring deleted filesMilind Changire2015-06-122-24/+43
| | | | | | | | | | | | | | | | | | | OSError and IOError exceptions were being thrown if files were deleted after session was created and a subsequent glusterfind pre was attmepted. glusterfind now detects this scenario and safely ignores these changes to the file-system. We also avoid recording deleted file paths into database in the case where gfid to path resolution cannot be performed for deleted files. Also, we now turn on volume option to capture delete paths. Change-Id: Iaf3883463f5e64643497bd77229ad80a9b2d5da5 BUG: 1228111 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/11194 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
* tools/glusterfind: Cleanup session dir after deleteAravinda VK2015-06-122-1/+20
| | | | | | | | | | | | | | | | | Sessions directories are created in /var/lib/glusterd/glusterfind/<SESSION>/<VOLUME>, only <VOLUME> dir is created since same session name can exists for two volumes. Now cleanup is added, <SESSION> dir will be removed if <SESSION> contains only one Volume. Change-Id: I236393659e63eb98342570e0255739f4adea4491 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1230111 Reviewed-on: http://review.gluster.org/11157 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: Fix GFID to Path conversion for dirAravinda VK2015-06-111-1/+1
| | | | | | | | | | | | | For Directories, GFID to Path conversion was fine, but wrong filter was used while updating to db. Change-Id: Id588adbb25ff5fd357f8cc2d5485bd2eb4a52b65 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1230127 Reviewed-on: http://review.gluster.org/11158 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: verifying volume is onlineMilind Changire2015-06-111-0/+14
| | | | | | | | | | | | Volume needs to be online (Started) to perform meaningful operations i.e. for "create" and "pre". This is now verified. Change-Id: I207cf1f5e30be0f67a776b866349b59d70a13984 BUG: 1225564 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/10955 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: verifying volume presenceMilind Changire2015-06-041-0/+10
| | | | | | | | | | | | | | An invalid volume name passed to "pre", "post" and "delete" operations throws up unfriendly errors. This patch presents friendly error messages. Change-Id: I233809653379b80b34c2e74c0b6f4fe502c8961e BUG: 1225424 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/10945 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: GFID to Path conversion using ChangelogAravinda VK2015-05-086-188/+683
| | | | | | | | | | | | | | | | | | | | | | | Records fop information collected from Changelogs in sqlite database. This is only working database, not required after processing. After post processing, output file is generated by reading these database files. This is applicable only in incremental run, When a changelog is parsed, all the details are saved in Db. GFID to Path is converted to those files for which information is available in Changelogs. For all the failed cases, it tries to convert to Path using Pgfid, if not found GFID to Path is done using find. BUG: 1201284 Change-Id: I53f168860dae15a0149004835e67f97aebd822be Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10463 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: Partial FindAravinda VK2015-05-088-289/+367
| | | | | | | | | | | | | | | | | | | | | | | | Enabled by default, if one node fails Glusterfind will not fail to return list of files from other nodes. This behavior can be changed using --disable-partial Now session is maintained in each nodes as well as in initiator node. Every pre command will pick the status file from respective node and start collecting list of changes happened after the status time. --reset-session-time, new option to force reset the session time. Next incremental run will start from this time. Change-detector argument is removed since Changelog mode is required to detect deletes and Renames. Change-Id: I0b83bc7c0e1b30b13de772b2d21fe968db4ff964 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1201289 Reviewed-on: http://review.gluster.org/10320 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: New option to pre --regenerate-outfileAravinda VK2015-05-041-0/+9
| | | | | | | | | | | | | When pre command is run twice, it overwrites the outfile. Now pre command will fail when executed twice. To force the regeneration use --regenerate-outfile Change-Id: I0cf7a139522812ece4decdfbcba667a05ce5c35e Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1207028 Reviewed-on: http://review.gluster.org/10418 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* tools/glusterfind: Do not fail to list if Session is invalidAravinda VK2015-04-231-1/+7
| | | | | | | | | | | If session is invalid, Show the session time as "Session Corrupted" BUG: 1209843 Change-Id: I10f7fb79f6a3a2cabab3f21bce86f2a2f424e8b6 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10164 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: NetBSD Build System
* Fix case mistake for MKDIR_P in MakefilesEmmanuel Dreyfus2015-04-201-1/+1
| | | | | | | | | | | | | | | | Some makefiles used $(mkdir_p) instead of the corectly defined $(MKDIR_P). The former is substituted as an empty string, leading to possible failures depending of the user shell tolerance. NetBSD's /bin/sh seems to choke more easily than Linux's /bin/bash, but if the later does not fail, it does not created the intended directories anyway. BUG: 1129939 Change-Id: I8caed4000f3c91cb3a685453848fb854793945ed Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org> Reviewed-on: http://review.gluster.org/10276 Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Niels de Vos <ndevos@redhat.com>
* tools/glusterfind: Prevent ssh public key overwrite issueAravinda VK2015-04-133-13/+40
| | | | | | | | | | | | | | | Same ssh key was used for all the sessions, when multiple sessions created in Cluster, public keys get overwritten by newest sessions. Moved ssh keys to respective session dir. BUG: 1206547 Change-Id: I3d8fac9b24bc7c71445c7b4deae83104693e7dab Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10150 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* tools/glusterfind: Check existence of SessionAravinda VK2015-04-091-0/+3
| | | | | | | | | | | | | | | Use glusterfind list command to check a session is valid or not glusterfind list --session <SESSION> --volume <VOLUME> Session exists if return code($?) is zero else invalid session Change-Id: I37faa09e4e1bf2f10725dd8e5b0cc36a36a3cad4 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1205057 Reviewed-on: http://review.gluster.org/10154 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tools/glusterfind: Validate Volume name before session CreateAravinda VK2015-04-011-0/+4
| | | | | | | | | | | | Do not create Session directories if Volume does not exists. Change-Id: I60c9107135c2bed2a7198582267b2328dec7e868 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1206065 Reviewed-on: http://review.gluster.org/10072 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* tools/glusterfind: Convert Output file path to absolute pathAravinda VK2015-04-011-2/+10
| | | | | | | | | | | | BUG: 1203656 Change-Id: Ie6c8f38894466246c453b2a863538a420acfe9f7 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10011 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Prashanth Pai <ppai@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* tools/glusterfind: Changelog_init before changelog_registerAravinda VK2015-03-302-0/+7
| | | | | | | | | | | | | | With the new Changelog RPC based approach, Changelog_init needs to be called before changelog_register BUG: 1206127 Change-Id: I817b84016d3f9ffdf2ea7a4e177953fe14ccc323 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10009 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Prashanth Pai <ppai@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* tools/glusterfind: Ignore .trashcan dir during Brick CrawlAravinda VK2015-03-304-5/+17
| | | | | | | | | | | BUG: 1203650 Change-Id: Icf81726ad34bd5cfec6d8e06763b02a159e84c2e Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/10010 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Prashanth Pai <ppai@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* feature/glusterfind: A tool to find incremental changesAravinda VK2015-03-1813-1/+1298
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Documentation is available in patch: http://review.gluster.org/#/c/9800/ A tool which helps to get list of modified files or list of all files in GlusterFS Volume using Changelog or find command. Usage ===== glusterfind --help Create: ------- glusterfind create --help The tool creates status file $GLUSTERD_WORKDIR/SESSION/VOLUME/status and records current timestamp to initiate the session. This timestamp will be used as start time for next runs. As part of create also generates ssh key and distributes to all peers. and enables build.pgfid and changelog using volume set command. Pre: ---- glusterfind pre --help This command is used to generate the list of files modified after session creation time or after last run. To get list of all files/dirs in Volume, run pre command with `--full` argument. The tool gets all nodes details using gluster volume info and runs node agent for each brick in respective nodes via ssh command. Once these node agents generate the output file, tool copies to local using scp. Merges all the output files to generate the final output file. Post: ----- glusterfind post --help After consuming the list, this sub command is called to update the session time based on pre command status file. List: ----- glusterfind list --help To view all the sessions Delete: ------- glusterfind delete --help Delete session. Known Issues ------------ 1. Deleted files will not get listed, since we can't convert GFID to Path if file/dir is deleted. 2. Only new name will get listed if Renamed. 3. All hardlinks will get listed. Change-Id: I82991feb0aea85cb6ec035fddbf80a2b276e86b0 BUG: 1193893 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/9682 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Prashanth Pai <ppai@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tools: Finds missing files in gluster volume given backend brickpathKotresh HR2015-03-156-0/+922
The tool finds the missing files in a geo-replication slave volume. The tool crawls backend .glusterfs of the brickpath, which is passed as a parameter and stats each entry on slave volume mount to check the presence of file. The mount used is aux-gfid-mount, hence no path conversion is required and is fast. The tool needs to be run on every node in cluster for each brickpath of geo-rep master volume to find missing files on slave volume. The tool is generic enough and can be used in non geo-replication context as well. Most of the crawler code is leverged from Avati's xfind and is modified to crawl only .glusterfs (https://github.com/avati/xsync) Thanks Aravinda for scripts to convert gfid to path. Change-Id: I84deaaaf638f7c571ff1319b67a3440fe27da810 BUG: 1187140 Signed-off-by: Aravinda VK <avishwan@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/9503 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
generated by cgit (git 2.34.1) at 2024-11-16 19:21:31 +0000