| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
Creation of files/directories with non-ascii names fails
to sync to the slave. It crashes with below traceback on
slave.
...
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py", line 118, in worker
res = getattr(self.obj, rmeth)(*in_data[2:])
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py", line 709, in entry_ops
[ESTALE, EINVAL, EBUSY])
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
return call(*arg)
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 83, in lsetxattr
cls.raise_oserr()
File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py", line 38, in raise_oserr
raise OSError(errn, os.strerror(errn))
OSError: [Errno 12] Cannot allocate memory
Cause:
The length calculation arguments passed to blob creation was done before encoding. Hence
was failing in gfid-access layer.
Fix:
It appears that the calculating lenght properly fixes this issue. But it will cause
issues in other places in 'python2' and not in 'python3'. So encoding and decoding
each required string to make geo-rep compatible with both 'python2' and 'python3'
is a nightmare and is not fool proof. Hence kept 'python2' code as is with out
encode/decode and applied encode/decode only to 'python3'
Added non-ascii filename tests to regression
fixes: bz#1650893
Change-Id: I35cfaf848e07b1a0b5cb93c01b98b472f08271a6
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
While syncing metadata, 'os.chmod', 'os.chown',
'os.utime' should be used without de-reference.
But python supports only 'os.chown' without
de-reference. That's mostly because Linux
doesn't support 'chmod' on symlink file itself
but it does support 'chown'.
So while syncing metadata ops, if it's symlink
we should only sync 'chown' and not do 'chmod'
and 'utime'. It will lead to tracebacks with
errors like EROFS, EPERM, ACCESS, ENOENT.
All the three errors (EPERM, ACCESS, ENOENT)
were handled except EROFS. But the way it was
handled was not fool proof. The operation is
tried and failure was handled based on the errors.
All the errors with symlink file for 'chown',
'utime' had to be passed to safe errors list of
'errno_wrap'. This patch handles it better by
avoiding 'chmod' and 'utime' if it's symlink
file.
fixes: bz#1646104
Change-Id: Ic354206455cdc7ab2a87d741d81f4efe1f19d77d
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Geo-rep's automatic error handling does gfid conflict
resolution. But if there are ENOENT errors because the
parent is not synced to slave, it doesn' handle them.
This patch adds the intelligence to create missing
parent directories on slave. It can create the missing
directories upto the depth of 10.
fixes: bz#1643402
Change-Id: Ic97ed1fa5899c087e404d559e04f7963ed7bb54c
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
| |
1. scheduler - Popen
2. syncdutils - corner case on failure
fixes: bz#1643932
Change-Id: I65af97a244a8790e976acedc2728db6ebbf2ae10
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
During gfid-conflict-resolution, geo-rep crashes
with 'ValueError: list.remove(x): x not in list'
Cause and Analysis:
During gfid-conflict-resolution, the entry blob is
passed back to master along with additional
information to verify it's integrity. If everything
looks fine, the entry creation is ignored and is
deleted from the original list. But it is crashing
during removal of entry from the list saying entry
not in list. The reason is that the stat information
in the entry blob was modified and sent back to
master if present.
Fix:
Send back the correct stat information for
gfid-conflict-resolution.
fixes: bz#1642865
Change-Id: I47a6aa60b2a495465aa9314eebcb4085f0b1c4fd
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. ctypes/syscalls
A) arguments is expected to be encoded
B) Raw conversion of return value from bytearray into string
2. struct pack/unpack - Raw converstion of string to bytearray
3. basestring -> str
Updates: #411
Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some of the scripts that have a #!/usr/bin/python3 shebang do not have a
main() like function. These scripts will not get executed but only
imported. They do not need the shebang.
A few others are not installed with 'make install', but do have a main()
like function. These scripts are expected to be used by developers for
different tasks (mostly code generation). Marking these scripts
executable to make it easier to identify them.
Change-Id: I73541471deb7e0830766b804786244e73dfe4221
Updates: #411
Signed-off-by: Niels de Vos <ndevos@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Handle py2 and py3 compatibility for pickling and unpickling.
Geo-rep pickles and unpickles reading sys.stdin and sys.stdout streams.
py2 and py3 compatibility expects the streams to be opened in binary
mode but the sys.stdout objects are different in python2
and python3
python2:
>>> type(sys.stdout)
<type 'file'>
python3:
>>> type(sys.stdout)
<class '_io.TextIOWrapper'>
So in order to access binary stream, using sys.stdin.buffer in python3
Updates: #411
Change-Id: I1a633ccdddff5baf0cf05a8b493add39ddf75bd7
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
| |
1. Fix fdopen used for pid file
2. Fix sha256 checksum calculation
Updates: #411
Change-Id: Ic173d104a73822c29aca260ba6de872cd8d23f86
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
python3 expects byte string for os.write. This works
for both py2 and py3. Fixed the same for geo-rep
mount testing code path.
Updates: #411
Change-Id: I2dfedcb0869457707bcca4d2847ef0d52bff1987
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The file objects for python3 by default is opened
in binary mode where as in python2 it's opened
as text by default.
The geo-rep code parses the output of Popen assuming
it as text, hence used the 'universal_newlines' flag
which provides backward compatibility for the same.
Change-Id: I371a03b6348af9666164cb2e8b93d47475431ad9
Updates: #411
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
'os.pipe' returns pair of file descriptors
which are non-inheritable by child processes.
But geo-rep uses te inheritable nature of
pipe fds to communicate between parent and
child processes. Hence wrote a compatiable
pipe routine which works well both with python2
and python3 with inheritable nature.
Updates: #411
Change-Id: I869d7a52eeecdecf3851d44ed400e69b32a612d9
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
| |
This patch fixes spell mistake in log message.
Change-Id: I84779c64aef6698cbc1a60ae1a82533e8a6a6e3d
updates: bz#1193929
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. '--ignore-mising-args' option for rsync is not
being used even though the rsync version is
greater than 3.1.0. Fixed the same.
2. '--existing' option for rsync is also not being
used. Fixed the same.
3. geo-rep config fails to set rsync-options as the
value contains '--'. Interestingly, python argsparse
treats the value with '--' (e.g., --ignore-missing-args)
as option. But when passed with something like
--value=--ignore-missing-args, it succeeds. Fixed the
same.
Change-Id: Iaeb838acaff1c2920fee9c7f920c99edce13a0a1
Signed-off-by: Kotresh HR <khiremat@redhat.com>
fixes: bz#1629561
|
|
|
|
|
|
| |
Change-Id: Iac241166d7a35dc7cc6cf07850f9f1bce38fe207
Updates: #411
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Commit af0d5a9b5375a5cd87ac10b429e2b9934718ce5b changes
python2 -> python3. This patch fixes a small issue introduced
by it.
Change-Id: Ib23c73683f570e8891f41476b661f37c89635fb5
updates: #411
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
see https://review.gluster.org/#/c/19788/,
https://review.gluster.org/#/c/19871/,
https://review.gluster.org/#/c/19952/,
https://review.gluster.org/#/c/20104/,
https://review.gluster.org/#/c/20162/,
https://review.gluster.org/#/c/20185/,
https://review.gluster.org/#/c/20207/,
https://review.gluster.org/#/c/20227/,
https://review.gluster.org/#/c/20307/,
https://review.gluster.org/#/c/20320/,
https://review.gluster.org/#/c/20332/,
https://review.gluster.org/#/c/20364/,
https://review.gluster.org/#/c/20441/, and
https://review.gluster.org/#/c/20484
shebangs changed from /usr/bin/python2 to /usr/bin/python3.
(Reminder, various distribution packaging guidelines require use
of explicit python version and don't allow '#!/usr/bin/env python',
regardless of how handy that idiom may be.)
glusterfs.spec(.in) package python{2,3}-gluster and python2 or
python3 dependencies as appropriate.
configure(.ac):
+ test for and use python2 or python3 as appropriate. If build
machine has python2 and python3, use python3. Override by
setting PYTHON=/usr/bin/python2 when running configure.
+ PYTHONDEV_CPPFLAGS from python[23]-config --includes is a
better match to the original python sysconfig.get_python_inc().
All those other extraneous flags breaks the build.
+ Only change the shebangs once. Changing them over and over
again, e.g., during a `make glusterrpms` in extras/LinuxRPM
just sends make (is it really make that's looping?) into an
infinite loop. If you figure out why, let me know.
+ Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie).
Everything else has 2.7 or 3.x
+ logic from https://review.gluster.org/c/glusterfs/+/21050, which
needs to be removed/merged after that patch is merged.
Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the
mysterious RHEL > 7.
Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Autmatic gfid conflict resolution needs to be disabled
during failover/failback as it might lead to data loss
in the following scenario.
1. Master went down without syncing directory "dir1" to slave.
2. When slave is failed over to master, if a new file
is written inside "dir1", creating dir1 again if not
present, "dir1" ends up with different gfid on original
slave.
3. When original master is up and failed back, due to
automatic gfid conflict resolution, "dir1" present in
original master is deleted losing all files and only
new file created on original slave is restored.
Hence during failover/failback, automatic gfid conflict
resolution should be disabled. So in these cases, appropriate
decision is taken.
fixes: bz#1622076
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Change-Id: I433616f5d3e13d4b6eb675475bd554ca34928573
|
|
|
|
|
|
|
|
|
|
|
| |
Cleanup the Active/Passive logging code
which is redundant. With new status infra
implmented, this is redundant as every
status switch is logged by status infra.
fixes: bz#1619027
Change-Id: I0a6644cb998f3520e62a5189f21e4d66acc0e7c5
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Analysis:
Monitor process spawns monitor threads (one per brick).
Each monitor thread, forks worker and agent processes.
Each monitor thread, while intializing, updates the
monitor status file. It is synchronized using flock.
The race is that, some thread can fork worker while
other thread opened the status file resulting in
holding the reference of fd in worker process.
Cause:
flock gets unlocked either by specifically unlocking it
or by closing all duplicate fds referring to the file.
The code was relying on fd close, hence a reference
in worker/agent process by fork could cause the deadlock.
Fix:
1. flock is unlocked specifically.
2. Also made sure to update status file in approriate places so that
the reference is not leaked to worker/agent process.
With this fix, both the deadlock and possible fd
leaks is solved.
fixes: bz#1614799
Change-Id: I0d1ce93072dab07d0dbcc7e779287368cd9f093d
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Please review, it's not always just the comments that were fixed.
I've had to revert of course all calls to creat() that were changed
to create() ...
Only compile-tested!
Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. MKDIR/RMDIR is recorded on all bricks. So if
one brick succeeds creating it, other bricks
should ignore it. But this was not happening.
The fix rename of directories in hybrid crawl,
was trying to rename the directory to itself
and in the process crashing with ENOENT if the
directory is removed.
2. If file is created, deleted and a directory is
created with same name, it was failing to sync.
Again the issue is around the fix for rename
of directories in hybrid crawl. Fixed the same.
If the same case was done with hardlink present
for the file, it was failing. This patch fixes
that too.
fixes: bz#1598884
Change-Id: I6f3bca44e194e415a3d4de3b9d03cc8976439284
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
Geo-rep sometimes fails to sync the rename of symlink
if the I/O is as follows
1. touch file1
2. ln -s "./file1" sym_400
3. mv sym_400 renamed_sym_400
4. mkdir sym_400
The file 'renamed_sym_400' failed to sync to slave
Cause:
Assume there are three distribute subvolume (brick1, brick2, brick3).
The changelogs are recorded as follows for above I/O pattern.
Note that the MKDIR is recorded on all bricks.
1. brick1:
-------
CREATE file1
SYMLINK sym_400
RENAME sym_400 renamed_sym_400
MKDIR sym_400
2. brick2:
-------
MKDIR sym_400
3. brick3:
-------
MKDIR sym_400
The operations on 'brick1' should be processed sequentially. But
since MKDIR is recorded on all the bricks, The brick 'brick2/brick3'
processed MKDIR first before 'brick1' causing out of order syncing
and created directory sym_400 first.
Now 'brick1' processed it's changelog.
CREATE file1 -> succeeds
SYMLINK sym_400 -> No longer present in master. Ignored
RENAME sym_400 renamed_sym_400
While processing RENAME, if source('sym_400') doesn't
present, destination('renamed_sym_400') is created. But
geo-rep stats the name 'sym_400' to confirm source file's
presence. In this race, since source name 'sym_400' is
present as directory, it doesn't create destination.
Hence RENAME is ignored.
Fix:
The fix is not rely only on stat of source name during RENAME.
It should stat the name and if the name is present, gfid should
be same. Only then it can conclude the presence of source.
fixes: bz#1600405
Change-Id: I9fbec4f13ca6a182798a7f81b356fe2003aff969
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
| |
This module is no longer being used.
Fixes: bz#1597512
Change-Id: Ie5faf55c5961d9d7b5082c9c257351af712c41d7
Signed-off-by: Nigel Babu <nigelb@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
see https://review.gluster.org/#/c/19788/,
https://review.gluster.org/#/c/19871/,
https://review.gluster.org/#/c/19952/,
https://review.gluster.org/#/c/20104/,
https://review.gluster.org/#/c/20162/,
https://review.gluster.org/#/c/20185/,
https://review.gluster.org/#/c/20207/,
https://review.gluster.org/#/c/20227/,
https://review.gluster.org/#/c/20307/,
https://review.gluster.org/#/c/20320/,
https://review.gluster.org/#/c/20332/, and
https://review.gluster.org/#/c/20364/
Fixes glupy.py python2isms, iteritems -> items, and some overlooked
print() in georep/peer_mountbroker.in
Note: Fedora packaging guidelines and SUSE rpmlint require explicit
shebangs; popular practices like #!/usr/bin/env python and #!/usr/bin/python
are not allowed; they must be #!/usr/bin/python2 or #!/usr/bin/python3
Note: Selected small fixes from 2to3 utility. Specifically apply,
basestring, funcattrs, has_key, idioms, map, numliterals, raise,
set_literal, types, urllib, and zip have already been applied. Also
version agnostic imports for urllib, cpickle, socketserver, _thread,
queue, etc., suggested by Aravinda in https://review.gluster.org/#/c/19767/1
Note: these 2to3 fixes report no changes are necessary: asserts, buffer,
exec, execfile, exitfunc, filter, getcwdu, imports2, input, intern,
itertools, metaclass, methodattrs, ne, next, nonzero, operator, paren,
raw_input, reduce, reload, renames, repr, standarderror, sys_exc, throw,
tuple_params, xreadlines.
Change-Id: Idda031c1ec975417c79323aea33e7b694e752b2a
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Geo-rep mounts are private to worker. It uses
mount namespace using unshare command to achieve
the same. Well, the unshare command has to support
'--propagation' option. So geo-rep breaks on the
systems with older unshare version. The patch
makes it fall back to lazy umount behaviour if
the unshare does not support propagation option.
fixes: bz#1589782
Change-Id: Ia614f068aede288d63ac62fea4461b1865066054
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
| |
1. Import configparser module correctly
2. Import thread module correctly
Updates: #411
Change-Id: I522453d23c256b694fa58d285f413b8c4dd6595c
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
see https://review.gluster.org/#/c/19788/,
https://review.gluster.org/#/c/19871/,
https://review.gluster.org/#/c/19952/, and
https://review.gluster.org/#/c/20104/
https://review.gluster.org/#/c/20162/
This patch changes uses of map() and raise(), and a few cases of print()
that were overlooked in the prior patch that fixed print.
Note: Fedora packaging guidelines require explicit shebangs, so popular
practices like #!/usr/bin/env python and #!/usr/bin/python are not
allowed; they must be #!/usr/bin/python2 or #!/usr/bin/python3
Note: Selected small fixes from 2to3 utility. Specifically apply,
basestring, funcattrs, idioms, numliterals, set_literal, types, urllib,
zip, map, and raise have already been applied. Also version agnostic
imports for urllib, cpickle, socketserver, _thread, queue, etc., suggested
by Aravinda in https://review.gluster.org/#/c/19767/1
Note: these 2to3 fixes report no changes are necessary: asserts, buffer,
exec, execfile, exitfunc, filter, getcwdu, intern, itertools, metaclass,
methodattrs, ne, next, nonzero, operator, paren, raw_input, reduce,
reload, renames, repr, standarderror, sys_exc, throw, tuple_params,
xreadlines.
Change-Id: Id62ea491e4ab5dd390075c5c6d9d889cf6f9da27
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
see https://review.gluster.org/#/c/19788/,
https://review.gluster.org/#/c/19871/, and
https://review.gluster.org/#/c/19952/
This patch adds version agnostic imports for urllib, cpickle,
socketserver, _thread, queue, etc., suggested by Aravinda in
https://review.gluster.org/#/c/19767/1
Note: Fedora packaging guidelines require explicit shebangs, so
popular practices like #!/usr/bin/env python and #!/usr/bin/python
are not allowed; they must be #!/usr/bin/python2 or #!/usr/bin/python3
Note: Selected small fixes from 2to3 utility. Specifically apply,
basestring, funcattrs, idioms, numliterals, set_literal, types, urllib,
and zip have already been applied.
Note: these 2to3 fixes report no changes are necessary: exec, execfile,
exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne,
next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr,
standarderror, sys_exc, throw, tuple_params, xreadlines.
Change-Id: I8d393064a1837874d8b4bc87c8ce05c679664642
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cause and Analysis:
The last synced changelog for entry operations is
marked in current version to avoid re-processing
of already processed entry operations in a batch
during crash/restart of geo-rep. This was not
present in previous versoins.
The marker is maintained in the dictionary with the
key 'last_synced_entry' and dictionary is persisted
into status file. So upgrading to current version in
which the marker is present was failing with KeyError.
Solution:
Load the dictionary with default keys first which
contains all the keys including latest ones and then
load the values from status file instead of doing
otherwise.
fixes: bz#1575490
Change-Id: Ic654e6f9a3c97f616761f1362f890352a2186fb4
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
see https://review.gluster.org/#/c/19788/
use print fn from __future__
Change-Id: If5075d8d9ca9641058fbc71df8a52aa35804cda4
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
If symlink is created on master pointing
to current directory (e.g symlink -> ".") with
non root uid or gid, geo-rep worker crashes
with ENOTSUP.
Cause:
Geo-rep creates the symlink on slave and
fixes the uid and gid using chown cmd.
os.chown dereferences the symlink which is
pointing to ".gfid" which is not supported.
Note that geo-rep operates on aux-gfid-mount
(e.g. "/mnt/.gfid/<gfid-of-symlink-file>").
Solution:
The uid or gid change is acutally on symlink
file. So use os.lchown, i.e, don't deference.
BUG: 1567209
Change-Id: I63575fc589d71f987bef1d350c030987738c78ad
updates: bz#1567209
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines
Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out
The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.
This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.
In a follow-up patch python2 will eventually be changed to python3.
Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.
updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
|
|
|
|
|
|
|
| |
`find_library()` doesn't consider LD_LIBRARY_PATH on Python < 3.6.
Change-Id: Iee26085cb5d14061001f19f032c2664d69a378a8
BUG: 1450593
Signed-off-by: Niklas Hambüchen <mail@nh2.me>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Lazy umounting the master volume by worker causes
issues with rsync's usage of getcwd. Henc removing
the lazy umount and using private mount namespace
for the same. On the slave, the lazy umount is
retained as we can't use private namespace in non
root geo-rep setup.
Change-Id: I403375c02cb3cc7d257a5f72bbdb5118b4c8779a
BUG: 1546129
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
JSON output of `config-get` command now returns in the following
format
{
"name": CONFIG_NAME,
"value": CONFIG_VALUE,
"default_value": DEFAULT_VALUE, # Only if modified == true
"configurable": true|false,
"modified": true|false
}
Change-Id: I6193de48cd33655df7ecef5a0d83d7cb147089cf
Fixes: #361
Signed-off-by: Aravinda VK <avishwan@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Once Geo-replication is started, it runs Gluster commands to get Volume
info from Master and Slave. With this patch, Georep can get Volume info
from Conf file if `--use-gconf-volinfo` argument is specified to monitor
Create a config(Or add to the config if exists) with following fields
[vars]
master-bricks=NODEID:HOSTNAME:PATH,..
slave-bricks=NODEID:HOSTNAME,..
master-volume-id=
slave-volume-id=
master-replica-count=
master-disperse_count=
Note: Exising Geo-replication is not affected since this is activated
only when `--use-gconf-volinfo` is passed while spawning `gsyncd
monitor`
Tiering support is not yet added since Tiering + Glusterd2 is still
under discussion.
Fixes: #396
Change-Id: I281baccbad03686c00f6488a8511dd6db0edc57a
Signed-off-by: Aravinda VK <avishwan@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch re-enables the geo-rep test cases.
Along with it does following optimizations.
1. Use EXPECT_WITHIN instead of sleep
2. Clean up geo-rep ssh key after test
3. Changes to gverify.sh and S56glusterd-geo-rep-create-post.sh
to use the given ssh identity file for geo-rep create
4. Make gluster-command-dir configurable and introduce
slave-gluster-command-dir which points the parent directory
of gluster binaries in master and slave respectively.
Change-Id: Ia7696278d9dd3ba04224dcd7c3564088ca970b04
BUG: 1480491
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
| |
BUG: 1529480
Change-Id: If4775ed9886990c0e1bcf4e44c7dfef95cc4f0c3
Signed-off-by: Aravinda VK <avishwan@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MD5 is not fips compliant. Hence replacing
with SHA256.
NOTE:
The hash is used to form the ctl_path for the ssh connection.
The length of ctl_path for ssh connection should not be > 108.
ssh fails with ctl_path too long if it is so. But when rsync
is piped to ssh, it is not taking > 90. rsync is failing with
error number 12. Hence using first 32 bytes of hash. Hash
collision doesn't matter as only one sock file is created
per directory.
Change-Id: I58aeb32a80b5422f6ac0188cf33fbecccbf08ae7
Updates: #230
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
| |
Fixes: #376
Change-Id: Ib92920c716c7d27e1eeb4bc4ebaf3efb48e0694d
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Fix access-mount and slave-access-mount
configs.
Change-Id: Ib586677755e76a51b9f20093e441b72789b4fecc
Signed-off-by: Kotresh HR <khiremat@redhat.com>
BUG: 1517633
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Slave gsyncd will not use session config files, Slave configs are
stored in Master config file itself and sent as argument to slave
gsyncd.
With this patch, gconf default values are overwritten if argument
name starts with "slave-"
Change-Id: Iebc51f52232c0cd30b29199f03015f97b70ce537
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1517068
|
|
|
|
|
|
|
|
|
| |
For Glusterd2 integration, JSON output of status and config
is very useful from gsyncd
Fixes: #361
Change-Id: I53c61f19033ad4ac601ea49469e4e7c7c8e9af3d
Signed-off-by: Aravinda VK <avishwan@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Fixed Python pep8 issues
- Removed dead code
- Rewritten configuration management
- Rewritten Arguments/subcommands handling
- Added Args upgrade to accommodate all these changes without changing
glusterd code
- use of md5 removed, which was used to hash the brick path for workdir
Both Master and Slave nodes will have subdir for session in the
format "<mastervol>_<primary_slave_host>_<slavevol>
$GLUSTER_LOGDIR/geo-replication/<mastervol>_<primary_slave_host>_<slavevol>
$GLUSTER_LOGDIR/geo-replication-slaves/<mastervol>_<primary_slave_host>_<slavevol>
Log file paths renamed since session info is available with directory
name itself.
$LOG_DIR_MASTER/
- gsyncd.log - Gsyncd, Worker monitor logs
- mnt-<brick-path>.log - Aux mount logs, mounted by each worker
- changes-<brick-path>.log - Changelog related logs(One per brick)
$LOG_DIR_SLAVE/
- gsyncd.log - Slave Gsyncd logs
- mnt-<master-node>-<master-brick-path>.log - Aux mount logs,
mounted for each connection from master-node:master-brick
- mnt-mbr-<master-node>-<master-brick-path>.log - Same as above,
but mountbroker setup
Fixes: #73
Change-Id: I2ec2a21e4e2a92fd92899d026e8543725276f021
Signed-off-by: Aravinda VK <avishwan@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem:
The data is not getting synced if master witnessed
IO as below.
1. echo "test_data" > f1
2. ln f1 f2
3. mv f2 f3
4. unlink f1
On master, 'f3' exists with data "test_data" but on
slave, only f3 exists with zero byte file without
backend gfid link.
Cause:
On master, since 'f2' no longer exists, the hardlink
is skipped during processing. Later, on trying to sync
rename, since source ('f2') doesn't exist, dst ('f3')
is created with same gfid. But in this use case, it
succeeds but backend gfid would not have linked as 'f1'
exists with the same gfid. So, rsync would fail with
ENOENT as backend gfid is not linked with 'f3' and 'f1'
is unlinked.
Fix:
On processing rename, if src doesn't exist on slave,
don't blindly create dst with same gfid. The gfid
needs to be checked, if it exists, hardlink needs
to be created instead of mknod.
Thanks Aravinda for helping in RCA :)
Change-Id: I5af4f99798ed1bcb297598a4bc796b701d1e0130
Signed-off-by: Kotresh HR <khiremat@redhat.com>
BUG: 1512483
Reporter: dimitri.ars@gmail.com
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In hybrid crawl, renames and unlink can't be
synced but directory renames can be detected.
While syncing the directory on slave, if the
gfid already exists, it should be rename.
Hence if directory gfid already exists, rename
it.
Change-Id: Ibf9f99e76a3e02795a3c2befd8cac48a5c365bb6
BUG: 1499566
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Passive brick's stime was not updated to the
status file immediately after updating the brick
root. As a result the last sync time was showing
'0' until it finishes first crawl if passive
worker becomes active after restart. Fix is to
update the status file immediately after upgrading
the brick root.
Change-Id: I248339497303bad20b7f5a1d42ab44a1fe6bca99
BUG: 1500346
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Worker occasionally crashed with EINTR on readlink.
This is not persistent and is transient. Worker restart
invovles re-processing of few entries in changenlogs.
So adding EINTR to retry list to avoid worker restart.
Change-Id: Iefe641437b5d5be583f079fc2a7a8443bcd19f9d
BUG: 1499393
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
During xsync crawl, worker occasionally crashed
with ENODATA on getting gfid from backend. This
is not persistent and is transient. Worker restart
invovles re-processing of few entries in changenlogs.
So adding ENODATA to retry list to avoid worker
restart.
Change-Id: Ib78d1e925c0a83c78746f28f7c79792a327dfd3e
BUG: 1499391
Signed-off-by: Kotresh HR <khiremat@redhat.com>
|