summaryrefslogtreecommitdiffstats
path: root/tools
Commit message (Collapse)AuthorAgeFilesLines
* tools/glusterfind: Remove an extra argumentShwetha K Acharya2020-01-141-1/+1
| | | | | | | | | | | | | Backport of: > Upstream Patch: https://review.gluster.org/#/c/glusterfs/+/24011/ >fixes: bz#1790748 >Change-Id: I1cb12c975142794139456d0f8e99fbdbb03c53a1 >Signed-off-by: Shwetha K Acharya <sacharya@redhat.com> >(cherry picked from commit d73872e764214f8071c8915536a75bdac1e5e685) fixes: bz#1790846 Change-Id: I1cb12c975142794139456d0f8e99fbdbb03c53a1 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* glusterfind: Fix py2/py3 issuesKotresh HR2020-01-142-9/+13
| | | | | | | | | | | | | | | | | | 1. In dictionary values(), returns list in py2 and not in py3. So explicitly convert it into list. 2. xattr module returns values in bytes. So explicitly convert them to str to work both with py2 and py3 Backport of: > fixes: bz#1789439 > Change-Id: I27a639cda4f7a4ece9744a97c3d16e247906bd94 > Signed-off-by: Kotresh HR <khiremat@redhat.com> > (cherry picked from commit 45894c39a4d05ed1f6a6f1bdbeafb5fe74ef29c3) Change-Id: I27a639cda4f7a4ece9744a97c3d16e247906bd94 Fixes: bz#1790423 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* glusterfind: python3 compatibilitySunny Kumar2020-01-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: While we delete gluster volume the hook script 'S57glusterfind-delete-post.py' is failed to execute and error message can be observed in glusterd log. Traceback: File "/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post", line 69, in <module> main() File "/var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post", line 39, in main glusterfind_dir = os.path.join(get_glusterd_workdir(), "glusterfind") File "/usr/lib64/python3.7/posixpath.py", line 94, in join genericpath._check_arg_types('join', a, *p) File "/usr/lib64/python3.7/genericpath.py", line 155, in _check_arg_types raise TypeError("Can't mix strings and bytes in path components") from None TypeError: Can't mix strings and bytes in path components Solution: Added the 'universal_newlines' flag to Popen to support backward compatibility. Backport of: > Change-Id: Ie5655b11b55535c5ad2338108d0448e6fdaacf4f > Fixes: bz#1789478 > Signed-off-by: Sunny Kumar <sunkumar@redhat.com> > (cherry picked from commit 33c3cbe71b67f523538b04334f1ef962953281ed) Change-Id: Ie5655b11b55535c5ad2338108d0448e6fdaacf4f Fixes: bz#1790438 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* tools/glusterfind: handle offline bricksMilind Changire2020-01-132-25/+61
| | | | | | | | | | | | | | | | | | | | | | | | | | Problem: glusterfind is unable to copy remote output file to local node when a remove-brick is in progress on the remote node. After copying remote files, in the --full output listing path, a "sort -u" command is run on the collected files. However, "sort" exits with an error code if it finds any file missing. Solution: Maintain a map of (pid, output file) when the node commands are started and remove the mapping for the pid for which the command returns an error. Use the list of files present in the map for the "sort" command. Backport of: > Change-Id: Ie6e019037379f4cb163f24b1c65eb382efc2fb3b > fixes: bz#1410439 > Signed-off-by: Milind Changire <mchangir@redhat.com> > Signed-off-by: Shwetha K Acharya <sacharya@redhat.com> > (cherry picked from commit 42c1605f42b89520d4d05806d7074e9e93b63640) Change-Id: Ie6e019037379f4cb163f24b1c65eb382efc2fb3b Fixes: bz#1790428 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* glusterfind: integrate with gfid2pathMilind Changire2019-06-201-4/+41
| | | | | | | | | | | | Integration with gfid2path helps avoid file-system crawl and saves precious time. Extended attributes starting with "trusted.gfid2path." are read and the <PGFID>/<BN> values are extracted and the <PGFID> is iteratively resolved from the brick backend to arrive at the full path. Change-Id: I593b02880e3413b77bfceed4a36b00d401f03bc0 fixes: #529 Signed-off-by: Milind Changire <mchangir@redhat.com> Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
* tools/glusterfind : validate session nameSaravanakumar Arumugam2019-04-261-8/+42
| | | | | | | | | | | | | Validate a session name(during create) for the following: 1. minimum 2 character length. 2. Maximum 256 characters. 3. No special characters apart from underscore, hyphen allowed. Also, validate volume(expect, while using glusterfind list). Change-Id: I1b1e64e218f93d0a531d3cf69fc2ce7e2ed11d01 BUG: 1241494 Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com>
* libgfchangelog : use find_library to locate shared librarySunny Kumar2019-04-151-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Issue: libgfchangelog.so: cannot open shared object file Due to hardcoded shared library name runtime loader looks for particular version of a shared library. Solution: Using find_library to locate shared library at runtime solves this issue. Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 323, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 82, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1261, in service_loop changelog_agent.init() File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: libgfchangelog.so: cannot open shared object file: No such file or directory Change-Id: I3dd013d701ed1cd99ba7ef20d1898f343e1db8f5 fixes: bz#1699394 Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
* glusterfind: revert shebangs to #!/usr/bin/python3Kaleb S. KEITHLEY2019-02-265-5/+5
| | | | | | | | | | | | | | | Using #!/usr/bin/env python is illegal in Fedora and RHEL packaging. (Debian and SUSE packaging will complain about it too but don't enforce it like Fedora packaging does. Yet.) We went through a giant exercise fixing these once already. The python needs to be python2/python3 clean. There is a build scriptlet that converts the shebangs back to /usr/bin/python on rhel7. Change-Id: If208a2557f7e5d727cda98c27905e249cf485d5b updates: bz#1193929 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* glusterfind: python2 to python3 compatShwetha K Acharya2019-02-029-25/+94
| | | | | | | | Made necessary modifications to ensure python3 compatibilty. fixes: bz#1658116 Change-Id: I5cf1d0447eaf3c44eb444245d1f67aadd60705c3 Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
* tools/glusterfind: option to display only files/directoriesShwetha K Acharya2019-01-234-7/+33
| | | | | | | | | | | Updated full find to filter for files and directories. --full --type f lists only the files, --full --type d lists only the directories, --full (by default) lists both files and directories. fixes: #579 Change-Id: If2c91a21a131667d5de34635c1846013e8fa20b7 Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
* tools: stack-buffer-overflow reported by asanHarpreet Kaur2018-12-121-1/+1
| | | | | | | | | | | This patch fixes buffer overflow in $SRC/tools/setgfid2path/src/main.c Memory access at offset 196 overflows "pgfid" variable SUMMARY: AddressSanitizer: stack-buffer-overflow (/lib64/libasan.so.5+0x4e935) updates: bz#1633930 Change-Id: Ib508c57e96c46b628f63c511437b853b39ae7955 Signed-off-by: Harpreet Kaur <hlalwani@redhat.com>
* libglusterfs: Move devel headers under glusterfs directoryShyamsundarR2018-12-052-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | libglusterfs devel package headers are referenced in code using include semantics for a program, this while it works can be better especially when dealing with out of tree xlator builds or in general out of tree devel package usage. Towards this, the following changes are done, - moved all devel headers under a glusterfs directory - Included these headers using system header notation <> in all code outside of libglusterfs - Included these headers using own program notation "" within libglusterfs This change although big, is just moving around the headers and making it correct when including these headers from other sources. This helps us correctly include libglusterfs includes without namespace conflicts. Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b Updates: bz#1193929 Signed-off-by: ShyamsundarR <srangana@redhat.com>
* glusterfind: add logs to identify parsing phasesMilind Changire2018-09-241-0/+10
| | | | | | | | Add logs to idenitfy start and finish of changelog parsing phases. Change-Id: Id250231f2af7829f887401d30ac98875ae1ae793 fixes: bz#1632236 Signed-off-by: Milind Changire <mchangir@redhat.com>
* Land part 2 of clang-format changesGluster Ant2018-09-122-483/+499
| | | | | Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4 Signed-off-by: Nigel Babu <nigelb@redhat.com>
* multiple: Fix python2 to python3 compatKotresh HR2018-09-071-1/+1
| | | | | | Change-Id: Iac241166d7a35dc7cc6cf07850f9f1bce38fe207 Updates: #411 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* core: python3Kaleb S. KEITHLEY2018-09-037-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/, https://review.gluster.org/#/c/19871/, https://review.gluster.org/#/c/19952/, https://review.gluster.org/#/c/20104/, https://review.gluster.org/#/c/20162/, https://review.gluster.org/#/c/20185/, https://review.gluster.org/#/c/20207/, https://review.gluster.org/#/c/20227/, https://review.gluster.org/#/c/20307/, https://review.gluster.org/#/c/20320/, https://review.gluster.org/#/c/20332/, https://review.gluster.org/#/c/20364/, https://review.gluster.org/#/c/20441/, and https://review.gluster.org/#/c/20484 shebangs changed from /usr/bin/python2 to /usr/bin/python3. (Reminder, various distribution packaging guidelines require use of explicit python version and don't allow '#!/usr/bin/env python', regardless of how handy that idiom may be.) glusterfs.spec(.in) package python{2,3}-gluster and python2 or python3 dependencies as appropriate. configure(.ac): + test for and use python2 or python3 as appropriate. If build machine has python2 and python3, use python3. Override by setting PYTHON=/usr/bin/python2 when running configure. + PYTHONDEV_CPPFLAGS from python[23]-config --includes is a better match to the original python sysconfig.get_python_inc(). All those other extraneous flags breaks the build. + Only change the shebangs once. Changing them over and over again, e.g., during a `make glusterrpms` in extras/LinuxRPM just sends make (is it really make that's looping?) into an infinite loop. If you figure out why, let me know. + Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie). Everything else has 2.7 or 3.x + logic from https://review.gluster.org/c/glusterfs/+/21050, which needs to be removed/merged after that patch is merged. Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the mysterious RHEL > 7. Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* multiple files: move from strlen() to sizeof()Yaniv Kaul2018-08-291-1/+1
| | | | | | | | | | | | | | | {glusterfsd|glusterfsd-mgmt|quota-common-utils|xlator|tier|stripe}.c tools/setgfid2path/src/main.c xlators/cluster/afr/src/afr-inode-read.c {glusterfs-acl|glusterfs}.h For const strings, just do compile time size calc instead of runtime. Compile-tested only! Change-Id: I303684b1ff29b05c10126fb1057f507e404ced07 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* glusterfind: Fix problems in python2 -> python3 compatKotresh HR2018-08-234-7/+7
| | | | | | | | | 1. Import configparser module correctly 2. Fix issues with urllib usage Updates: #411 Change-Id: I060af29f1ebac3a5e2aeeee2720b7372d4c31697 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* All: run codespell on the code and fix issues.Yaniv Kaul2018-07-221-1/+1
| | | | | | | | | | | | Please review, it's not always just the comments that were fixed. I've had to revert of course all calls to creat() that were changed to create() ... Only compile-tested! Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* core/various: python3 compat, prepare for python2 -> python3Kaleb S. KEITHLEY2018-06-044-4/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/, https://review.gluster.org/#/c/19871/, and https://review.gluster.org/#/c/19952/ This patch adds version agnostic imports for urllib, cpickle, socketserver, _thread, queue, etc., suggested by Aravinda in https://review.gluster.org/#/c/19767/1 Note: Fedora packaging guidelines require explicit shebangs, so popular practices like #!/usr/bin/env python and #!/usr/bin/python are not allowed; they must be #!/usr/bin/python2 or #!/usr/bin/python3 Note: Selected small fixes from 2to3 utility. Specifically apply, basestring, funcattrs, idioms, numliterals, set_literal, types, urllib, and zip have already been applied. Note: these 2to3 fixes report no changes are necessary: exec, execfile, exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne, next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr, standarderror, sys_exc, throw, tuple_params, xreadlines. Change-Id: I8d393064a1837874d8b4bc87c8ce05c679664642 updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* core/various: python3 compat, prepare for python2 -> python3Kaleb S. KEITHLEY2018-05-305-13/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/ and https://review.gluster.org/#/c/19871/ Selected small fixes from 2to3 utility. Specifically apply, basestring, funcattrs, idioms, numliterals, set_literal, types, urllib, zip Note: these 2to3 fixes report no changes are necessary: exec, execfile, exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne, next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr, standarderror, sys_exc, throw, tuple_params, xreadlines. Any 2to3 fixes not in the above two lists have more extensive changes which will follow in separate patches. most unicode changes suggested by 2to3 will need to be applied at the same time as changing the shebangs from python2 to python3. Prashanth notes that unicode strings in py2 need 'u' prefix; unicode strings in py3 3.0, 3.1, and 3.2 a 'u' prefix will throw an error, but in py3 3.3+ it is legal (or just ignored). All Linux dists we care about have 3.3 or later so we can leave 'u' prefixes on unicode strings. Change-Id: I49bba2f328b0ee24b9a8115a7183be979981563e updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* core/build/various: python3 compat, prepare for python2 -> python3Kaleb S. KEITHLEY2018-04-127-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note 1) we're not supposed to be using #!/usr/bin/env python, see https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines Note 2) we're also not supposed to be using "!/usr/bin/python, see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out The previous patch (https://review.gluster.org/19767) tried to do too much in one patch, so it was abandoned. This patch does two things: 1) minor cleanup of configure(.ac) to explicitly use python2 2) change all the shebang lines to #!/usr/bin/python2 and add them where they were missing based on warnings emitted during rpmbuild. In a follow-up patch python2 will eventually be changed to python3. Before that python2-isms (e.g. print, string.join(), etc.) need to be converted to python3. Some of those can be rewritten in version agnostic python. E.g. print statements become print() with "from __future_ import print_function". The python 2to3 utility will be used for some of those. Also Aravinda has given guidance in the comments to the first patch for changes. updates: #411 Change-Id: I471730962b2526022115a1fc33629fb078b74338 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* glusterfind: Log remote stderr on `node_cmd` error. Fixes #1559130Niklas Hambüchen2018-03-261-1/+16
| | | | | | | | | The problem of lost stderr was introduced in commit feea851fad4f89b48bfe89fe3b75250cc7bd6501. Change-Id: Ic98f9bc9682ae3bd9c3ebea3855667fc8ba2843d BUG: 1559130 Signed-off-by: Niklas Hambüchen <mail@nh2.me>
* python: Remove all uses of find_library. Fixes #1450593Niklas Hambüchen2018-03-241-2/+1
| | | | | | | | `find_library()` doesn't consider LD_LIBRARY_PATH on Python < 3.6. Change-Id: Iee26085cb5d14061001f19f032c2664d69a378a8 BUG: 1450593 Signed-off-by: Niklas Hambüchen <mail@nh2.me>
* glusterfind: Show C function when raising ChangelogExceptionNiklas Hambüchen2018-03-241-9/+10
| | | | | | | Fixes: #432 Change-Id: I9ab031e098aff717e619d9deb6410281b96de14a Signed-off-by: Niklas Hambüchen <mail@nh2.me>
* build: add --without-server optionNiels de Vos2018-02-194-1/+15
| | | | | | | | | | | | | | | | With Gluster 4.0 we will not provide the server components for EL6 and older. At one point Gluster 4.x will get GlusterD2, which requires Golang tools in the distribution. EL6 does not contain these at the moment. With this change, it is possible to `./configure --without-server` which prevents building glusterd and the xlators for the bricks. Building RPMs can pass `--without server` and the glusterfs-server sub-package will not be created. Change-Id: I97f5ccf9f2c76e60d9af83915fc59fae57ad6d25 BUG: 1074947 Signed-off-by: Niels de Vos <ndevos@redhat.com>
* glusterfind: Speed up gfid lookup 100x by using an SQL indexNiklas Hambüchen2017-12-301-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #1529883. This fixes some bits of `glusterfind`'s horrible performance, making it 100x faster. Until now, glusterfind was, for each line in each CHANGELOG.* file, linearly reading the entire contents of the sqlite database in 4096-bytes-sized pread64() syscalls when executing the SELECT COUNT(1) FROM %s WHERE 1=1 AND gfid = ? query through the code path: get_changes() parse_changelog_to_db() when_data_meta() gfidpath_exists() _exists() In a quick benchmark on my laptop, doing one such `SELECT` query took ~75ms on a 10MB-sized sqlite DB, while doing the same query with an index took < 1ms. Change-Id: I8e7fe60f1f45a06c102f56b54d2ead9e0377794e BUG: 1529883 Signed-off-by: Niklas Hambüchen <mail@nh2.me>
* tools/setgfid2path: Tool to set GFID to Path xattr in brick backendAravinda VK2017-07-285-1/+202
| | | | | | | | | | | | | | | | | | | | Once storage/gfid2path feature is enabled using `gluster volume set <volname> storage.gfid2path enable`, it starts recording the gfid2path xattr on each files. But this feature will not add xattr to the existing files. This tool accepts the file path as argument and sets the necessary xattr required for this feature. Change-Id: I75ad82c86ce482950645e687ff2e33b413fa53da Updates: #139 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: https://review.gluster.org/17839 Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
* tools/glusterfind: Fix encoding to encode only space,newline and percent charsAravinda VK2017-07-216-50/+33
| | | | | | | | | | | | | | | | | libgfchangelog was encoding path using spec rfc3986, but encoding only required for SPACE, NEWLINE and PERCENT chars since the NEWLINE char is used as record separator and SPACE as field separator in the parsed changelogs output. Changed the encoding function to encode only SPACE, NEWLINE and PERCENT chars BUG: 1451724 Change-Id: Ic1dea824d23493dedcf3db45f353f90572f4e046 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: https://review.gluster.org/17788 CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com>
* tools/glusterfind: unquote DELETE path before further processingMilind Changire2017-06-301-2/+3
| | | | | | | | | | | | | | | | | | | Problem: DELETE path is quoted before it reaches glusterfind. This wasn't handled in the glusterfind code leading to double quoting of path separator '%2F' to '%252F' i.e. the '%' character in '%2F' itself was quoted to '%25' Solution: unquote the the deleted path before further processing Change-Id: I2dfbbd7792dc0f9da5c8e02093b0f1c031ff344a BUG: 1465024 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17629 Smoke: Gluster Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: initialize variable 'end'Milind Changire2017-06-191-0/+1
| | | | | | | | | | | | | | | | | Problem: syntax error due to uninitialized variable 'end' Solution: set end = -1 Change-Id: Ia8b500e9dd5304d0832e5f594abca0b5b2ab357e BUG: 1462241 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17557 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: add --end-time optionMilind Changire2017-06-123-24/+45
| | | | | | | | | | | | | | | | | | | | | | | | | Add optional --end-time argument to be used with --since-time when using the "query" command. "start" and "end" times are passed in the command-line to changelog.py only if --full has not been specified on command-line. -1 is passed to changelog.py as end time if user has not supplied --end-time on command-line. brickfind.py: Remove unused "start" command-line argument. Also: Minor indentation changes to keep flake8-2.7 happy. Change-Id: I063ef5459916f711503881ade5c4fc32374edad0 BUG: 1453151 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17439 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: add --field-separator optionMilind Changire2017-06-123-10/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: Default field separator is a space character. This gets in the way if the file name itself has embedded spaces. Solution: Add --field-separator option to "pre" and "query" commands. The field separator string will be used to separate strings in the output lines that get written to the output file. eg. old output: NEW file1.txt RENAME file2 Copy.txt file3.txt with --field-separator as "===" new output: NEW===file1.txt RENAME===file2 Copy.txt===file3.txt Change-Id: I71e878fed58ba1113d97044ac9f6404ee66227c7 BUG: 1453151 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/17481 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* build/packaging: Debian and Ubuntu don't have /usr/libexecKaleb S. KEITHLEY2017-03-133-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | GLUSTERFS_LIBEXECDIR is effectively hard-coded to /usr/libexec/glusterfs in configure(.ac) Debian-based distributions don't have a /usr/libexec/ directory This issues is partially mitigated by the use of $libexecdir in some of the Makefile.am files, but even so the incorrectly defined GLUSTERFS_LIBEXECDIR results in various things such as gsyncd, glusterfind, eventsd, etc., trying to invoke other scripts and programs from a location that doesn't exist. And once we correctly define GLUSTERFS_LIBEXECDIR, then we might as well use it appropriatedly. Change-Id: If5219cadc51ae316f7ba2e2831d739235c77902d BUG: 1430841 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: https://review.gluster.org/16880 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Joe Julian <me@joejulian.name> Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
* tools/glusterfind: delete temporary folderMilind Changire2017-01-301-2/+2
| | | | | | | | | | | | | | | | | | | Problem: Run specific temporary folder created under /usr/var/lib/misc/glusterfsd/glusterfind/<session>/<volume>/ remains undeleted. Solution: Delete the temporary folder from all nodes. Change-Id: I0edaf868aebb01b15c489434bbb26fe853351384 BUG: 1413526 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: https://review.gluster.org/16416 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: avoid deleting keys directoryMilind Changire2016-12-071-0/+4
| | | | | | | | | | | | | | | | | | Problem: gluster volume delete mistakenly deletes the .keys directory under /var/lib/glusterd/glusterfind. Solution: Check for ".keys" directory and avoid deleting it. Change-Id: Ia595c8bf3f423c1ad5d6faa183a29598c07a11f9 BUG: 1402369 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/16052 Reviewed-by: Aravinda VK <avishwan@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
* tools/glusterfind: xml parsing fix for tiered volumesMilind Changire2016-11-171-3/+12
| | | | | | | | | | | | | | | | | | | gluster volume info <vol> --xml for non-tiered volumes have 'bricks/brick' elements under the 'volInfo/volumes/volume' element. However, tiered volumes have a 'bricks/hotBricks/brick' and 'bricks/coldBricks/brick' elements under the 'volInfo/volumes/volume' element. Fix main.py::get_nodes() BUG: 1389481 Change-Id: I2f4465bfa8a55e7fa87917d3ec3e69b05d5241b9 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/15746 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: kill remote processes and separate run-time directoriesMilind Changire2016-10-254-16/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem #1: Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre has been initiated. Solution #1: Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned to remote process. When local process receives Keyboard Interrupt, SIGHUP is immediately conveyed to the remote terminal causing remote changelog.py process to terminate immediately. Problem #2: Concurrent glusterfind pre runs are not possible on the same glusterfind session in case of a runaway process. Solution #2: glusterfind pre runs now add random directory name to the working directory to store and manage temporary database and changelog processing. If KeyboardInterrupt is received, the function call run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename) cleans up the remote run specific directory. Patch: 7571380 cli/xml: Fix wrong XML format in volume get command broke "gluster volume get <vol> changelog.rollover-time --xml" Now fixed function utils.py::get_changelog_rollover_time() Fixed spurious trailing space getting written if second path is empty in main.py::write_output() Fixed repetitive changelog processing in changelog.py::get_changes() Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29 BUG: 1382236 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/15609 Smoke: Gluster Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* core: use readdir(3) with glibc, and associated cleanupKaleb S. KEITHLEY2016-07-181-20/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3) is marked as deprecated. Specifically the function decl in <dirent.h> has the deprecated attribute, and now warnings are thrown during the compile on Fedora 25 builds. The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say that glibc's readdir(3) is, and always has been, MT-SAFE as long as only one thread is accessing the directory object returned by opendir(). World+Dog also says there is a potential buffer overflow in readdir_r(). World+Dog suggests that it is preferable to simply use readdir(). There's an implication that eventually readdir_r(3) will be removed from glibc. POSIX has, apparently deprecated it in the standard, or even removed it entirely. Over and above that, our source near the various uses of readdir(_r)(3) has a few unsafe uses of strcpy()+strcat(). (AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see if the same is true on those platforms, and we can't be sure of MacOS even though we know it's based on *BSD.) Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568 BUG: 1356998 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/14838 Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* packaging: Remove ".py" extension from symlink targetAravinda VK2016-07-151-3/+3
| | | | | | | | | | | | | | | | | | | | | | S57glusterfind-delete-post.py* is packaged in $LIBEXEC/glusterfs/glusterfind directory, symlink created to /var/lib/glusterd/hooks directory ln -s $(libexecdir)/glusterfs/glusterfind/S57glusterfind-delete-post.py \ $(DESTDIR)$(GLUSTERD_WORKDIR)/hooks/1/delete/post/S57glusterfind-delete-post.py ".py" should be removed from symlink target so that build system will not try to compile the symlink files and generate .pyc/.pyo files. BUG: 1356868 Change-Id: I9155378aa72eb8559ba8af76d91a4985f69f38f9 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/14928 Smoke: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* scripts: bash-isms in scriptsKaleb S KEITHLEY2016-05-182-8/+9
| | | | | | | | | | | | | Change-Id: I05f8718fb73c16dbb37c1f7460bb7b55cc0ffe97 BUG: 1336853 Reported-by: Patrick Matthäi <pmatthaei@debian.org> Signed-off-by: Kaleb S KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/14401 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* tools/glusterfind: add --full option to query commandMilind Changire2016-03-303-35/+81
| | | | | | | | | | | | | | | | | The --full option will use brickfind.py to list all files in the volume. The output file will contain url-encoded file names prefixed with the tag string to indicate that all files should be considered as new. The default tag string for the --full option is "NEW". This can be changed with the --tag-for-full-find command-line option. Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792 BUG: 1286279 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/12779 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: Handling Unicode file namesAravinda VK2016-03-304-48/+34
| | | | | | | | | | | | | | | | | | | | Unicode filenames handled cleanly with this patch. Changelog files and output files are opened with utf-8 encoding using codecs.open. urllib.quote_plus and unquote_plus will not handle Unicode so, encode Unicode to 8-bit string version before calling unquote. urllib.quote_plus requires 8-bit string itself so do not decode to Unicode if we need to use quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set. BUG: 1319717 Change-Id: If5561c749ab5529445650d322c831eb4da22b65a Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/13798 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* tools/glusterfind: Do not fail cleanup script on volume deleteAravinda VK2016-03-081-1/+6
| | | | | | | | | | | | | | | If glusterfind dir is not present inside GLUSTERD_WORKDIR, then cleanup script Post Volume delete was failing. This patch safely handles the absence of glusterfind directory in GLUSTERD_WORKDIR BUG: 1289935 Change-Id: I9f7e6035e3dc6079927803df942b0f921218c79a Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/12923 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* tools/glusterfind: New option --no-encodeAravinda VK2016-02-269-42/+112
| | | | | | | | | | | | | | | | | | | | | | | | | New option added to skip encoding path in output file. Also handled Unicode strings. File paths can have newline characters, to differentiate between each path patch is encoded according to RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer applications have to decode the path before consuming it. With this option Paths are not encoded, can be directly consumed by applications. Unicode encoding is handled automatically BUG: 1310080 Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/13477 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* tools/glusterfind: add query command to list filesMilind Changire2015-11-242-38/+155
| | | | | | | | | | | | | | | | | | | | | | | When session information is maintained outside Gluster, there needs to be some mechanism to list files starting from a time-stamp. This patch implements the feature via the "query" command-line option. The only caveat is that the first time the query command is run for the volume, it will likely report that "historical changelogs are not available". This is due to the fact that changelogs had not been turned on for the volume so far. So the volume options need to be turned on outside glusterfind or the since_time need to be greater than the current time when the query command is run for the very first time for the volume. The query command turns on the required volume options for collecting changelogs. Change-Id: I6cb7a57a5ecd166210e2eb4deede06d40ccfa996 BUG: 1272006 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/12362 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: StrictHostKeyChecking=no for ssh/scp verificationAravinda VK2015-11-231-1/+5
| | | | | | | | | | | | | Also do not use scp command in case copy file from local node. Change-Id: Ie78c77eb0252945867173937391b82001f29c3b0 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1260918 Reviewed-on: http://review.gluster.org/12124 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: Handle Keyboard interruptAravinda VK2015-11-181-43/+53
| | | | | | | | | | | | | | Do not print Python traceback when glusterfind command is interrupted Change-Id: I67383534f965e410fef7ce09798e9d435ef738ae Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1243812 Reviewed-on: http://review.gluster.org/11698 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* build: MKDIR_P is not defined for Makefiles, use mkdir_p insteadNiels de Vos2015-11-101-2/+2
| | | | | | | | | | | Change-Id: Id6d5263eb7b1c53e72a7668e716e9cc4e34b82cd Reported-by: Milind Changire <mchangir@redhat.com> BUG: 1198849 Signed-off-by: Niels de Vos <ndevos@redhat.com> Reviewed-on: http://review.gluster.org/12553 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Milind Changire <mchangir@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* core: use syscall wrappers instead of direct syscalls - miscellaneousKaleb S. KEITHLEY2015-10-281-1/+1
| | | | | | | | | | | | | | | various xlators and other components are invoking system calls directly instead of using the libglusterfs/syscall.[ch] wrappers. If not using the system call wrappers there should be a comment in the source explaining why the wrapper isn't used. Change-Id: I1f47820534c890a00b452fa61f7438eb2b3f667c BUG: 1267967 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/12276 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
generated by cgit (git 2.34.1) at 2024-11-12 22:11:49 +0000