glusterfs.git/geo-replication/syncdaemon/monitor.py, branch v3.7.9

geo-rep: Fix syntax errors in GsyncdError

2016-01-04T06:35:10+00:00

%s was not replaced by actual values in GsyncdError

BUG: 1279644
Change-Id: I3c0a10f07383ca72844a46f930b4aa3d3c29f568
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12566
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur 
 (cherry picked from commit 74699ddd777f7e862991cf3afad91823d30e5b84)
Reviewed-on: http://review.gluster.org/12724
Reviewed-by: Milind Changire 
Reviewed-by: Venky Shankar

geo-rep: Fix getting subvol count

2016-01-04T06:07:05+00:00

Tiering doesn't support disperse volume as hot tier,
hence xml output doesn't give 'hotdisperseCount'.
Remove the usage of 'hotdisperseCount' in geo-rep
and return 0 instead.

BUG: 1293309
Change-Id: I3f50d21cb51db91e31faebf69af4f72360420b73
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/13062
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/13068

geo-rep: Fix getting subvol number

2016-01-04T06:06:39+00:00

Fix getting subvol number if the volume
type is tier. If the volume type was tier,
the subvol number was calculated incorrectly
and hence few of workers didn't become ACTIVE
resulting in files not being replicated from
corresponding brick. This patch addresses
the same.

BUG: 1293309
Change-Id: I318de346657d330a2394507514bdff61feb92d27
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12994
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-on: http://review.gluster.org/13059

geo-rep: use cold tier bricks for namespace operations

2015-12-08T06:42:35+00:00

Problem:
symlinks are not getting synced to slave in a Tiering based volume.

Solution:
Now, symlinks are created directly in cold tier bricks( in the backend).

Earlier, cold tier was avoided for namespace operations and only
hot tier was used while processing changelogs.

Now, cold tier is HASH subvolume in a Tiering volume.
So, carry out namespace operation only in cold tier subvolume and
avoid hot tier subvolume to avoid any races.

Earlier, XSYNC was used(and changeloghistory avoided) during initial sync
in order to avoid race while processing historychangelog in Hot tier.
This is no longer required as there is no race from Hot tier.

Also, avoid both live and history changelog ENTRY operations from Hot tier
to avoid any race with cold tier.

Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e
BUG: 1288027
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12844
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
(cherry picked from commit 93f31189ce8f6e2980a39b02568ed17088e0a667)
Reviewed-on: http://review.gluster.org/12891

geo-rep: Avoid cold tier bricks during ENTRY operation

2015-12-07T09:50:47+00:00

This is a series of patch which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier , all fops will be
recorded in HOT tier.
b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot
tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

Now, once the file is demoted to COLD tier, any namespace operation
carried out on the cold tier can be avoided as we directly RECORD
the same in HOT tier.

How?
1. Check whether the brick is cold tier and skip ENTRY operation.
2. Also, if it is cold tier brick, use Xsync(which is used during initial run).
   This will help in getting all cold tier bricks changes using File System crawl
   and helps in avoiding races with hot tier brick(which can happen
   if historychangelog used in cold tier brick).

Dependent patches:
1. http://review.gluster.org/12239
2. http://review.gluster.org/12326

Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12355
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
(cherry picked from commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268)
Reviewed-on: http://review.gluster.org/12429
Reviewed-by: Venky Shankar

geo-rep: Kill Geo-rep Worker when Agent process dies

2015-11-25T07:17:04+00:00

When Changelog agent process dies, Geo-replication fails to detect
and worker will run without respective Changelog agent. Status shows
Active/Passive without any progress.

With this patch, Worker process gets killed whenever Changelog
agent dies.

Change-Id: I30b4cc77f924f7e8174b8bfe415ac17f0b3851b4
Signed-off-by: Aravinda VK 
BUG: 1279362
Reviewed-on: http://review.gluster.org/12485
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Reviewed-by: Kotresh HR 
(cherry picked from commit 5d1ff7efd6ab3bd29a29922a9ea1e1aaf02544ad)
Reviewed-on: http://review.gluster.org/12550

geo-rep: Status Enhancements

2015-05-06T16:12:35+00:00

Discussion in gluster-devel
http://www.gluster.org/pipermail/gluster-devel/2015-April/044301.html

MASTER NODE - Master Volume Node
MASTER VOL - Master Volume name
MASTER BRICK - Master Volume Brick
SLAVE USER - Slave User to which Geo-rep session is established
SLAVE - :: used in Geo-rep Create command
SLAVE NODE - Slave Node to which Master worker is connected
STATUS - Worker Status(Created, Initializing, Active, Passive, Faulty,
         Paused, Stopped)
CRAWL STATUS - Crawl type(Hybrid Crawl, History Crawl, Changelog Crawl)
LAST_SYNCED - Last Synced Time(Local Time in CLI output and UTC in XML output)
ENTRY - Number of entry Operations pending.(Resets on worker restart)
DATA - Number of Data operations pending(Resets on worker restart)
META - Number of Meta operations pending(Resets on worker restart)
FAILURES - Number of Failures
CHECKPOINT TIME - Checkpoint set Time(Local Time in CLI output and UTC
                  in XML output)
CHECKPOINT COMPLETED - Yes/No or N/A
CHECKPOINT COMPLETION TIME - Checkpoint Completed Time(Local Time in CLI
                             output and UTC in XML output)

XML output:

cliOutput>
  geoRep>
      volume>
        name>
        sessions>
          session>
             session_slave>
             pair>
                master_node>
                master_brick>
                slave_user>
                slave/>
                slave_node>
                status>
                crawl_status>
                entry>
                data>
                meta>
                failures>
                checkpoint_completed>
                master_node_uuid>
                last_synced>
                checkpoint_time>
                checkpoint_completion_time>

BUG: 1218586
Change-Id: I944a6c3c67f1e6d6baf9670b474233bec8f61ea3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10121
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/10574
Tested-by: Gluster Build System

geo-rep: Adhering to the common storage for geo-rep

2015-05-04T09:25:23+00:00

Making geo-rep use the common storage shared by nfs,
snapshot and geo-rep. The meta volume should be named
as gluster_shared_storage, and it should be mounted
at "/var/run/gluster/shared_storage/".

Geo-rep will create a directory called 'geo-rep'
in the meta-volume and all the lock files are created
inside it.

BUG: 1217939
Change-Id: I1d88798376d68340e2b2eff018c7e4f0121a608a
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10196
Reviewed-on: http://review.gluster.org/10503
Tested-by: NetBSD Build System
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

feature/geo-rep: Active Passive Switching logic flock

2015-03-16T04:53:59+00:00

CURRENT DESIGN AND ITS LIMITATIONS:
-----------------------------------
Geo-replication syncs changes across geography using changelogs captured
by changelog translator. Changelog translator sits on server side just
above posix translator. Hence, in distributed replicated setup, both
replica pairs collect changelogs w.r.t their bricks. Geo-replication
syncs the changes using only one brick among the replica pair at a time,
calling it as "ACTIVE" and other non syncing brick as "PASSIVE".

Let's consider below example of distributed replicated setup where
NODE-1 as b1 and its replicated brick b1r is in NODE-2

        NODE-1                         NODE-2
          b1                            b1r

At the beginning, geo-replication chooses to sync changes from NODE-1:b1
and NODE-2:b1r will be "PASSIVE". The logic depends on virtual getxattr
'trusted.glusterfs.node-uuid' which always returns first up subvolume
i.e., NODE-1. When NODE-1 goes down, the above xattr returns NODE-2 and
that is made 'ACTIVE'. But when NODE-1 comes back again, the above xattr
returns NODE-1 and it is made 'ACTIVE' again. So for a brief interval of
time, if NODE-2 had not finished processing the changelog, both NODE-2
and NODE-1 will be ACTIVE causing rename race as mentioned in the bug.

SOLUTION:
---------
1. Have a shared replicated storage, a glusterfs management volume specific
   to geo-replication.

2. Geo-rep creates a file per replica set on management volume.

3. fcntl lock on the above said file is used for synchronization
   between geo-rep workers belonging to same replica set.

4. If management volume is not configured, geo-replication will back
   to previous logic of using first up sub volume.

Each worker tries to lock the file on shared storage, who ever wins will
be ACTIVE. With this, we are able to solve the problem but there is an
issue when the shared replicated storage goes down (when all replicas
goes down). In that case, the lock state is lost. So AFR needs to rebuild the
lock state after brick comes up.

NOTE:
-----
This patch brings in the, pre-requisite step of setting up management volume
for geo-replication during creation.

1. Create mgmt-vol for geo-replicatoin and start it. Management volume should
   be part of master cluster and recommended to be three way replicated
   volume having each brick in different nodes for availability.
2. Create geo-rep session.
3. Configure mgmt-vol created with geo-replication session as follows.
   gluster vol geo-rep  slavenode:: config meta_volume \
   
4. Start geo-rep session.

Backward Compatiability:
-----------------------
If management volume is not configured, it falls back to previous logic of
using node-uuid virtual xattr. But it is not recommended.

Change-Id: I7319d2289516f534b69edd00c9d0db5a3725661a
BUG: 1196632
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/9759
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Fixing the typo errors

2015-02-03T18:03:41+00:00

Change-Id: Iacc67e4ba9ac45e0858f3befe84ffb8fccf7e1c3
BUG: 1075417
Signed-off-by: arao 
Reviewed-on: http://review.gluster.org/9502
Tested-by: Gluster Build System 
Reviewed-by: Humble Devassy Chirammal