glusterfs.git/tools/glusterfind/src/utils.py, branch v4.1.2

tools/glusterfind: Fix encoding to encode only space,newline and percent chars

2017-07-21T08:41:13+00:00

libgfchangelog was encoding path using spec rfc3986, but encoding only
required for SPACE, NEWLINE and PERCENT chars since the NEWLINE char is
used as record separator and SPACE as field separator in the parsed
changelogs output.

Changed the encoding function to encode only SPACE, NEWLINE and PERCENT chars

BUG: 1451724
Change-Id: Ic1dea824d23493dedcf3db45f353f90572f4e046
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17788
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Milind Changire

tools/glusterfind: add --field-separator option

2017-06-12T04:48:55+00:00

Problem:
Default field separator is a space character.
This gets in the way if the file name itself has embedded spaces.

Solution:
Add --field-separator option to "pre" and "query" commands.
The field separator string will be used to separate strings in the
output lines that get written to the output file.

eg.
old output:
NEW file1.txt
RENAME file2 Copy.txt file3.txt

with --field-separator as "==="
new output:
NEW===file1.txt
RENAME===file2 Copy.txt===file3.txt

Change-Id: I71e878fed58ba1113d97044ac9f6404ee66227c7
BUG: 1453151
Signed-off-by: Milind Changire 
Reviewed-on: https://review.gluster.org/17481
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: kill remote processes and separate run-time directories

2016-10-25T12:00:54+00:00

Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get  changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1382236
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15609
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: add --full option to query command

2016-03-31T04:51:32+00:00

The --full option will use brickfind.py to list all files in the volume.
The output file will contain url-encoded file names prefixed with the
tag string to indicate that all files should be considered as new.
The default tag string for the --full option is "NEW". This can be
changed with the --tag-for-full-find command-line option.

Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792
BUG: 1286279
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/12779
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

tools/glusterfind: Handling Unicode file names

2016-03-30T13:37:18+00:00

Unicode filenames handled cleanly with this patch. Changelog
files and output files are opened with utf-8 encoding using codecs.open.

urllib.quote_plus and unquote_plus will not handle Unicode so, encode
Unicode to 8-bit string version before calling unquote. urllib.quote_plus
requires 8-bit string itself so do not decode to Unicode if we need to use
quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set.

BUG: 1319717
Change-Id: If5561c749ab5529445650d322c831eb4da22b65a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13798
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Kotresh HR

tools/glusterfind: New option --no-encode

2016-02-26T10:11:08+00:00

New option added to skip encoding path in output file. Also handled
Unicode strings.

File paths can have newline characters, to differentiate between each
path patch is encoded according to
RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer
applications have to decode the path before consuming it. With this
option Paths are not encoded, can be directly consumed by
applications.

Unicode encoding is handled automatically

BUG: 1310080
Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13477
Smoke: Gluster Build System 
Reviewed-by: Milind Changire 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System 
Reviewed-by: Venky Shankar

tools/glusterfind: Prepend prefix in case of delete

2015-08-27T06:21:23+00:00

In case of delete operation alone, output prefix adding
was not handled earlier. 
Output prefix is added now.

Change-Id: Ia91444dddbff501b26a864f6185ca4c0aaf4c802
BUG: 1244144
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/11712
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Milind Changire

tools/glusterfind: GFID to Path conversion using Changelog

2015-05-09T04:59:10+00:00

Records fop information collected from Changelogs in sqlite database.
This is only working database, not required after processing.

After post processing, output file is generated by reading these
database files.

This is applicable only in incremental run, When a changelog is
parsed, all the details are saved in Db. GFID to Path is converted
to those files for which information is available in Changelogs.

For all the failed cases, it tries to convert to Path using Pgfid,
if not found GFID to Path is done using find.

BUG: 1201284
Change-Id: I53f168860dae15a0149004835e67f97aebd822be
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10463
Reviewed-by: Kotresh HR 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

tools/glusterfind: Partial Find

2015-05-09T04:19:06+00:00

Enabled by default, if one node fails Glusterfind will not fail
to return list of files from other nodes. This behavior can be changed
using --disable-partial

Now session is maintained in each nodes as well as in initiator node.
Every pre command will pick the status file from respective node and
start collecting list of changes happened after the status time.

--reset-session-time, new option to force reset the session time.
Next incremental run will start from this time.

Change-detector argument is removed since Changelog mode is required
to detect deletes and Renames.

Change-Id: I0b83bc7c0e1b30b13de772b2d21fe968db4ff964
Signed-off-by: Aravinda VK 
BUG: 1201289
Reviewed-on: http://review.gluster.org/10320
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur

tools/glusterfind: Ignore .trashcan dir during Brick Crawl

2015-03-30T12:35:32+00:00

BUG: 1203650
Change-Id: Icf81726ad34bd5cfec6d8e06763b02a159e84c2e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10010
Reviewed-by: Kotresh HR 
Reviewed-by: Prashanth Pai 
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY