glusterfs.git/geo-replication/syncdaemon/gsyncd.py, branch v3.7.16

geo-rep: Use configured log_level for libgfchangelog logs

2016-09-13T11:58:43+00:00

libgfchangelog was not respecting the log_level configured
in Geo-replication. With this patch Libgfchangelog log level
can be configured using `config changelog_log_level TRACE`.
Default Changelog log level is INFO

> Reviewed-on: http://review.gluster.org/15078
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1374606
Change-Id: Ida714931129f6a1331b9d0815da77efcb2b898e3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15449
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Saravanakumar Arumugam 
CentOS-regression: Gluster Build System

geo-rep: Handle Config parser errors

2016-07-28T10:46:24+00:00

Python ConfigParser lib has two methods, readfp and read, it should
be used as follows.

    readfp(open("defaults.conf"))
    read("custom.conf")

ConfigParser.read(path) ignores any file errors, which is intentional
since errors are handled while loading default config.

Geo-rep uses only one config file(Session config in Master side and
Template config on Slave side) so we should use readfp to avoid
skipping OS errors.

config.read is retained in case of `--config-set-rx` where glusterd
creates new template config file.

BUG: 1357759
Change-Id: I15a14d3743facd7b8c7af0edc70fdefaa43efd04
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14777
(cherry picked from commit d94bf608b16b82f2c8f8588a96459cb746773b32)
Reviewed-on: http://review.gluster.org/14946
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
CentOS-regression: Gluster Build System

georep: add reset-sync-time option for session delete

2016-07-21T10:04:23+00:00

Set the stime xattr at all the brick roots to (0,0) if the argument
reset-sync-time has been provided on the command-line.
To avoid testing against directory specific stime, the remote
stime is assumed to be minus_infinity, if the root directory
stime is set to (0,0), before the directory scan begins.
This triggers a full volume resync to slave in the case of a
geo-rep session recreation with the same master-slave volume
pair.

Command synopsis:
gluster volume geo-replication  :: delete \
    [reset-sync-time]

Update gluster cli man page to include new sub-command reset-sync-time.

Change-Id: Ie4ce03b9425ed9bb81eda8681058c0fc6f990948
BUG: 1357772
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/14051
Reviewed-by: Kotresh HR 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK 
(cherry picked from commit 70fd68d94f768c098b3178c151fa92c5079a8cfd)
Reviewed-on: http://review.gluster.org/14952

geo-rep: Fix volume stop with geo-rep session

2016-06-06T06:50:50+00:00

Backport of http://review.gluster.org/14636

The slave volume uuid is appended to slave_url
which is saved in 'gsync_slaves' dictionary with
the commit "http://review.gluster.org/13111".
But the gsyncd expects slave_url with out slave
volume uuid otherwise it fails which in turn
fails volume stop with geo-rep. This patch
fixes the same.

Change-Id: I036f4fce43115ff848daebfe1b2c58765cffab5a
BUG: 1342431
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/14637
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

glusterd/geo-rep: slave volume uuid to identify a geo-rep session

2016-05-19T06:47:09+00:00

Problem:
Currently, it is possible to create multiple geo-rep session from
the Master host to Slave host(s), where Slave host(s) belonging
to the same volume.

For example:
Consider Master Host M1 having volume tv1 and Slave volume tv2,
which spans across two Slave hosts S1 and S2.
Currently, it is possible to create geo-rep session from
M1(tv1) to S1(tv2) as well as from M1(tv1) to S2(tv2).

When the Slave Host is alone modified, it is identified as a new geo-rep
session (as slave host and slave volume together are identifying
Slave side).

Also, it is possible to create both root and non-root geo-rep session between
same Master volume and Slave volume. This should also be avoided.

Solution:
This multiple geo-rep session creation must be avoided and
in order to avoid, use Slave volume uuid to identify a Slave.
This way, we can identify whether a session is already created for
the same Slave volume and avoid creating again (using different host).

When the session creation is forced in the above scenario, rename
the existing geo-rep session directory with new Slave Host mentioned.

Change-Id: I9239759cbc0d15dad63c48b8cf62950bb687c7c8
BUG: 1335728
Signed-off-by: Saravanakumar Arumugam 
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13111
Reviewed-by: Kotresh HR 
Tested-by: Kotresh HR 
Reviewed-by: Atin Mukherjee 
 (cherry picked from commit a9128cda34b1f696b717ba09fa0ac5a929be8969)
Reviewed-on: http://review.gluster.org/14322
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

geo-rep: Fix getting subvol number

2016-01-04T06:06:39+00:00

Fix getting subvol number if the volume
type is tier. If the volume type was tier,
the subvol number was calculated incorrectly
and hence few of workers didn't become ACTIVE
resulting in files not being replicated from
corresponding brick. This patch addresses
the same.

BUG: 1293309
Change-Id: I318de346657d330a2394507514bdff61feb92d27
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12994
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-on: http://review.gluster.org/13059

geo-rep: use cold tier bricks for namespace operations

2015-12-08T06:42:35+00:00

Problem:
symlinks are not getting synced to slave in a Tiering based volume.

Solution:
Now, symlinks are created directly in cold tier bricks( in the backend).

Earlier, cold tier was avoided for namespace operations and only
hot tier was used while processing changelogs.

Now, cold tier is HASH subvolume in a Tiering volume.
So, carry out namespace operation only in cold tier subvolume and
avoid hot tier subvolume to avoid any races.

Earlier, XSYNC was used(and changeloghistory avoided) during initial sync
in order to avoid race while processing historychangelog in Hot tier.
This is no longer required as there is no race from Hot tier.

Also, avoid both live and history changelog ENTRY operations from Hot tier
to avoid any race with cold tier.

Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e
BUG: 1288027
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12844
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
(cherry picked from commit 93f31189ce8f6e2980a39b02568ed17088e0a667)
Reviewed-on: http://review.gluster.org/12891

geo-rep: Avoid cold tier bricks during ENTRY operation

2015-12-07T09:50:47+00:00

This is a series of patch which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier , all fops will be
recorded in HOT tier.
b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot
tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

Now, once the file is demoted to COLD tier, any namespace operation
carried out on the cold tier can be avoided as we directly RECORD
the same in HOT tier.

How?
1. Check whether the brick is cold tier and skip ENTRY operation.
2. Also, if it is cold tier brick, use Xsync(which is used during initial run).
   This will help in getting all cold tier bricks changes using File System crawl
   and helps in avoiding races with hot tier brick(which can happen
   if historychangelog used in cold tier brick).

Dependent patches:
1. http://review.gluster.org/12239
2. http://review.gluster.org/12326

Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12355
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
(cherry picked from commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268)
Reviewed-on: http://review.gluster.org/12429
Reviewed-by: Venky Shankar

geo-rep: Fix Geo-rep logging to log datetime in GMT/UTC

2015-12-02T15:29:13+00:00

Geo-rep is logging in Local time, all other Gluster logs are in
GMT/UTC. It is very difficult to co-relate Geo-rep logs with
other Gluster logs.

BUG: 1284737
Change-Id: Ieae8bda7e4788e587cf4595e21e0e772c210cfbb
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12583
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur 
 (cherry picked from commit 41e900199d7f369862b21b739dd11602d0d7c48d)
Reviewed-on: http://review.gluster.org/12723
Tested-by: NetBSD Build System

geo-rep: New Config option for ssh_port

2015-11-25T07:18:38+00:00

If different port used for SSH instead of 22, Geo-replication
was failing to establish SSH connection.

ssh_port option can be added using config:ssh_command and
config:ssh_command_tar, but user has to remember complete
ssh command used with parameter to add/modify ssh port.

This patch adds new config option for ssh_port,

gluster volume geo-replication   \
        config ssh_port 52022

Change-Id: I7753a09485f0b1f49d2b2a80b962c720817c96f4
Signed-off-by: Aravinda VK 
BUG: 1283060
Reviewed-on: http://review.gluster.org/12444
Tested-by: NetBSD Build System 
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
(cherry picked from commit 7d35eb5926869ed084295600502a85ce13be506f)
Reviewed-on: http://review.gluster.org/12607
Reviewed-by: Kotresh HR