glusterfs.git/geo-replication, branch v7.1

geo-rep: Fix py2/py3 compatibility in repce

2019-11-14T13:19:21+00:00

Geo-rep fails to start on python2 only machine like
centos6. It fails with "ImportError no module named _io".
This patch fixes the same.

Backport of:
 > Patch: https://review.gluster.org/23702/
 > BUG: 1771577
 > Change-Id: I8228458a853a230546f9faf29a0e9e0f23b3efec
 > Signed-off-by: Kotresh HR 
(cherry picked from commit 9595ecca3de49fdf37d30b151f5c3e071e0a80d0)

Change-Id: I8228458a853a230546f9faf29a0e9e0f23b3efec
Signed-off-by: Kotresh HR 
fixes: bz#1771840

geo-rep: Fix Permission denied traceback on non root setup

2019-11-06T16:59:15+00:00

Problem:
While syncing rename of directory in hybrid crawl, geo-rep
crashes as below.

Traceback (most recent call last):
  File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/resource.py", line 588, in entry_ops
    src_entry = get_slv_dir_path(slv_host, slv_volume, gfid)
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
    [ENOENT], [ESTALE])
  File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
    return call(*arg)
PermissionError: [Errno 13] Permission denied: '/bricks/brick1/b1/.glusterfs/8e/c0/8ec0fcd4-d50f-4a6e-b473-a7943ab66640'

Cause:
Conversion of gfid to path for a directory uses readlink on backend
.glusterfs gfid path. But this fails for non root user with
permission denied.

Fix:
Use gfid2path interface to get the path from gfid

Backport of:
 > Patch: https://review.gluster.org/23570
 > Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
 > BUG: 1763439
 > Signed-off-by: Kotresh HR 

Change-Id: I9d40c713a1b32cea95144cbc0f384ada82972222
fixes: bz#1764030
Signed-off-by: Kotresh HR

geo-rep: Fix config upgrade on non-participating node

2019-11-06T16:59:14+00:00

After upgrade, if the config files are of old format, it
gets migrated to new format. Monitor process migrates it.
Since monitor doesn't run on nodes where bricks are not
hosted, it doesn't get migrated there. So this patch fixes
the config upgrade on nodes which doesn't host bricks.
This happens during config either on get/set/reset.

Backport of:
Patch: https://review.gluster.org/23555
Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
Signed-off-by: Kotresh HR 
BUG: 1762220

Change-Id: Ibade2f2310b0f3affea21a3baa1ae0eb71162cba
fixes: bz#1764028
Signed-off-by: Kotresh HR

geo-rep: Fix the name of changelog archive file

2019-11-06T16:59:14+00:00

Background:
The processed changelogs are archived each month in a single tar file.
The default format is "archive_YYYYMM.tar" which is specified as "%%Y%%m"
in configuration file.

Problem:
The created changelog archive file didn't have corresponding year
and month. It created as "archive_%Y%m.tar" on python2 only systems.

Cause and Fix:
Geo-rep expects "%Y%m" after the ConfigParser reads it from config file.
Since it was "%%Y%%m" in config file, geo-rep used to get correct value
"%Y%m" in python3 and "%%Y%%m" in python2 which is incorrect.
The fix can be to use "%Y%m" in config file but that fails in python3.
So the fix is to use "RawConfigParser" in geo-rep and use "%Y%m". This
works both in python2 and python3.

Backport of:
 > Patch: https://review.gluster.org/23248
 > Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c
 > BUG: 1741890
 > Signed-off-by: Kotresh HR 

Change-Id: Ie5b7d2bc04d0d53cd1769e064c2d67aaf95d557c
fixes: bz#1764023
Signed-off-by: Kotresh HR

geo-rep: Fix Config Get Race

2019-11-06T16:59:14+00:00

When two threads(sync jobs) in Geo-rep worker calls `gconf.get` and
`gconf.getr`(realtime) at the sametime, `getr` resets the conf object
and other one gets None. Thread Lock is introduced to fix the issue.

```
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py",
  line 368, in twrap
    tf(*aargs)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1987,
  in syncjob
    po = self.sync_engine(pb, self.log_err)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py",
  line 1444, in rsync
    rconf.ssh_ctl_args + \
AttributeError: 'NoneType' object has no attribute 'split'
```

Backport of:
 > Patch: https://review.gluster.org/23158
 > Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5
 > BUG: 1737484
 > Signed-off-by: Aravinda VK 

Change-Id: I9c245e5c36338265354e158f5baa32b119eb2da5
fixes: bz#1764015
Signed-off-by: Kotresh HR

geo-rep: Test case for upgrading config file

2019-11-06T16:59:14+00:00

Added test case for the patch
https://review.gluster.org/#/c/glusterfs/+/22894/4

Also updated if else structure in gsyncdconfig.py to avoid
repeated occurance of values in new configfile.

Backport of:
 > Patch: https://review.gluster.org/22982
 > BUG: 1707731
 > Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
 > Signed-off-by: Shwetha K Acharya 

fixes: bz#1764003
Change-Id: If97e1d37ac52dbd17d47be6cb659fc5a3ccab6d7
Signed-off-by: Kotresh HR

geo-rep : fix gluster command path for non-root session

2019-11-06T16:59:14+00:00

Problem:
gluster command not found.

Cause:
In Volinfo class we issue command 'gluster vol info' to get information
about volume like getting brick_root to perform various operation.
When geo-rep session is configured for non-root user Volinfo class
fails to issue gluster command due to unavailability of gluster
binary path for non-root user.

Solution:
Use config value 'slave-gluster-command-dir'/'gluster-command-dir' to get path
for gluster command based on caller.

Backport of:
 > Patch: https://review.gluster.org/22920
 > BUG: 1722740
 > Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
 > Signed-off-by: Sunny Kumar 

fixes: bz#1764007
Change-Id: I4ec46373da01f5d00ecd160c4e8c6239da8b3859
Signed-off-by: Kotresh HR

geo-rep: Upgrading config file to new version

2019-11-06T16:59:14+00:00

- configuration handling is enhanced with patch
https://review.gluster.org/#/c/glusterfs/+/18257/
- hence, the old configurations are not applied when
Geo-rep session is created in the old version and upgraded.

This patch solves the issue. It,
- checks if the config file is old.
- parses required values from old config file and stores in new
  config file, which ensures that configerations are applied on
  upgrade.
- stores old config file as backup.
- handles changes in options introduced in
  https://review.gluster.org/#/c/glusterfs/+/18257/

Backport of:
 > Patch: https://review.gluster.org/22894
 > BUG: bz#1707731
 > Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0
 > Signed-off-by: Shwetha K Acharya 

updates: bz#1764003
Change-Id: Iad8da6c1e1ae8ecf7c84dfdf8ea3ac6966d8a2a0
Signed-off-by: Kotresh HR

geo-rep : fix sub-command during worker connection

2019-09-25T11:31:54+00:00

Problem:

Geo-rep session for non-root going faulty.

Solution:

During worker start we do not construct slave url and use 'args.resource_remote'
which is basically just slave-hostname.
This works better for root session but fails in non-root session during
ssh command.
Using slave url solves this issue.

Backport of:
 > Patch: https://review.gluster.org/23465
 > BUG: 1753928
 > Change-Id: Ib83552fde77f81c208896494b323514ab37ebf22
 > Signed-off-by: Sunny Kumar 

fixes: bz#1755213
Change-Id: Ib83552fde77f81c208896494b323514ab37ebf22
Signed-off-by: Kotresh HR

geo-rep: performance improvement while syncing renames with existing gfid

2019-09-25T04:46:37+00:00

Problem:
The bug[1] addresses issue of data inconsistency when handling RENAME with
existing destination. This fix requires some performance tuning considering
this issue occurs in heavy rename workload.

Solution:
If distribution count for master volume is one do not verify op's on
master and go ahead with rename.

The performance improvement with this patch can only be observed if
master volume has distribution count one.

[1]. https://bugzilla.redhat.com/show_bug.cgi?id=1694820

Backport of:

 > Patch: https://review.gluster.org/23459/
 > BUG: 1753857
 > Change-Id: I8e9bcd575e7e35f40f9f78b7961c92dee642f47b
 > Signed-off-by: Sunny Kumar 
(cherry picked from commit 30d3608c43be119f75da7798d88b94601dedcb29)


Change-Id: I8e9bcd575e7e35f40f9f78b7961c92dee642f47b
Signed-off-by: Sunny Kumar 
fixes: bz#1755212