glusterfs.git/geo-replication/syncdaemon/monitor.py, branch v7.1

geo-rep : fix gluster command path for non-root session

2019-11-06T16:59:14+00:00

Problem:
gluster command not found.

Cause:
In Volinfo class we issue command 'gluster vol info' to get information
about volume like getting brick_root to perform various operation.
When geo-rep session is configured for non-root user Volinfo class
fails to issue gluster command due to unavailability of gluster
binary path for non-root user.

Solution:
Use config value 'slave-gluster-command-dir'/'gluster-command-dir' to get path
for gluster command based on caller.

Backport of:
 > Patch: https://review.gluster.org/22920
 > BUG: 1722740
 > Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
 > Signed-off-by: Sunny Kumar 

fixes: bz#1764007
Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
Signed-off-by: Kotresh HR

geo-rep: performance improvement while syncing renames with existing gfid

2019-09-25T04:46:37+00:00

Problem:
The bug[1] addresses issue of data inconsistency when handling RENAME with
existing destination. This fix requires some performance tuning considering
this issue occurs in heavy rename workload.

Solution:
If distribution count for master volume is one do not verify op's on
master and go ahead with rename.

The performance improvement with this patch can only be observed if
master volume has distribution count one.

[1]. https://bugzilla.redhat.com/show_bug.cgi?id=1694820

Backport of:

 > Patch: https://review.gluster.org/23459/
 > BUG: 1753857
 > Change-Id: I8e9bcd575e7e35f40f9f78b7961c92dee642f47b
 > Signed-off-by: Sunny Kumar 
(cherry picked from commit 30d3608c43be119f75da7798d88b94601dedcb29)


Change-Id: I8e9bcd575e7e35f40f9f78b7961c92dee642f47b
Signed-off-by: Sunny Kumar 
fixes: bz#1755212

georep: python2 to python3 compat - syscalls

2018-10-08T16:36:04+00:00

1. ctypes/syscalls
    A) arguments is expected to be encoded
    B) Raw conversion of return value from bytearray into string
2. struct pack/unpack - Raw converstion of string to bytearray
3. basestring -> str

Updates: #411
Change-Id: I80f939adcdec0ed0022c87c0b76d057ad5559e5a
Signed-off-by: Kotresh HR

georep: Fix python3 compatibility (os.pipe)

2018-09-25T17:03:21+00:00

'os.pipe' returns pair of file descriptors
which are non-inheritable by child processes.
But geo-rep uses te inheritable nature of
pipe fds to communicate between parent and
child processes. Hence wrote a compatiable
pipe routine which works well both with python2
and python3 with inheritable nature.

Updates: #411
Change-Id: I869d7a52eeecdecf3851d44ed400e69b32a612d9
Signed-off-by: Kotresh HR

geo-rep: Fix deadlock during worker start

2018-08-13T03:52:07+00:00

Analysis:
Monitor process spawns monitor threads (one per brick).
Each monitor thread, forks worker and agent processes.
Each monitor thread, while intializing, updates the
monitor status file. It is synchronized using flock.
The race is that, some thread can fork worker while
other thread opened the status file resulting in
holding the reference of fd in worker process.

Cause:
flock gets unlocked either by specifically unlocking it
or by closing all duplicate fds referring to the file.
The code was relying on fd close, hence a reference
in worker/agent process by fork could cause the deadlock.

Fix:
1. flock is unlocked specifically.
2. Also made sure to update status file in approriate places so that
the reference is not leaked to worker/agent process.

With this fix, both the deadlock and possible fd
leaks is solved.

fixes: bz#1614799
Change-Id: I0d1ce93072dab07d0dbcc7e779287368cd9f093d
Signed-off-by: Kotresh HR

All: run codespell on the code and fix issues.

2018-07-22T14:40:16+00:00

Please review, it's not always just the comments that were fixed.
I've had to revert of course all calls to creat() that were changed
to create() ...

Only compile-tested!

Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5
updates: bz#1193929
Signed-off-by: Yaniv Kaul

geo-rep: Fix geo-rep for older versions of unshare

2018-06-22T11:37:25+00:00

Geo-rep mounts are private to worker. It uses
mount namespace using unshare command to achieve
the same. Well, the unshare command has to support
'--propagation' option. So geo-rep breaks on the
systems with older unshare version. The patch
makes it fall back to lazy umount behaviour if
the unshare does not support propagation option.

fixes: bz#1589782
Change-Id: Ia614f068aede288d63ac62fea4461b1865066054
Signed-off-by: Kotresh HR

geo-rep: Remove lazy umount and use mount namespaces

2018-02-22T05:40:35+00:00

Lazy umounting the master volume by worker causes
issues with rsync's usage of getcwd. Henc removing
the lazy umount and using private mount namespace
for the same. On the slave, the lazy umount is
retained as we can't use private namespace in non
root geo-rep setup.

Change-Id: I403375c02cb3cc7d257a5f72bbdb5118b4c8779a
BUG: 1546129
Signed-off-by: Kotresh HR

geo-rep: Support for using Volinfo from Conf file

2018-01-23T03:03:01+00:00

Once Geo-replication is started, it runs Gluster commands to get Volume
info from Master and Slave. With this patch, Georep can get Volume info
from Conf file if `--use-gconf-volinfo` argument is specified to monitor

Create a config(Or add to the config if exists) with following fields

    [vars]
    master-bricks=NODEID:HOSTNAME:PATH,..
    slave-bricks=NODEID:HOSTNAME,..
    master-volume-id=
    slave-volume-id=
    master-replica-count=
    master-disperse_count=

Note: Exising Geo-replication is not affected since this is activated
only when `--use-gconf-volinfo` is passed while spawning `gsyncd
monitor`

Tiering support is not yet added since Tiering + Glusterd2 is still
under discussion.

Fixes: #396
Change-Id: I281baccbad03686c00f6488a8511dd6db0edc57a
Signed-off-by: Aravinda VK

geo-rep: Refactoring Config and Arguments parsing

2017-11-15T05:20:08+00:00

- Fixed Python pep8 issues
- Removed dead code
- Rewritten configuration management
- Rewritten Arguments/subcommands handling
- Added Args upgrade to accommodate all these changes without changing
  glusterd code
- use of md5 removed, which was used to hash the brick path for workdir

Both Master and Slave nodes will have subdir for session in the
format "__

  $GLUSTER_LOGDIR/geo-replication/__
  $GLUSTER_LOGDIR/geo-replication-slaves/__

Log file paths renamed since session info is available with directory
name itself.

  $LOG_DIR_MASTER/
      - gsyncd.log - Gsyncd, Worker monitor logs
      - mnt-.log - Aux mount logs, mounted by each worker
      - changes-.log - Changelog related logs(One per brick)

  $LOG_DIR_SLAVE/
      - gsyncd.log - Slave Gsyncd logs
      - mnt--.log - Aux mount logs,
        mounted for each connection from master-node:master-brick
      - mnt-mbr--.log - Same as above,
        but mountbroker setup

Fixes: #73
Change-Id: I2ec2a21e4e2a92fd92899d026e8543725276f021
Signed-off-by: Aravinda VK