glusterfs.git/geo-replication/syncdaemon/syncdutils.py, branch v3.7.9

geo-rep: Do not crash worker on ESTALE

2015-08-06T06:17:25+00:00

Handle ESTALE returned by lstat gracefully
by retrying it. Do not crash the worker.

BUG: 1249547
Change-Id: I57fb9933900153ab41c3d9b73748b1cdaa8d89ca
Reviewed-on: http://review.gluster.org/11772
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Venky Shankar 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/11820
Reviewed-by: Milind Changire

geo-rep: Minimize rm -rf race in Geo-rep

2015-05-06T17:02:27+00:00

While doing RMDIR worker gets ENOTEMPTY because same directory will
have files from other bricks which are not deleted since that worker
is slow processing. So geo-rep does recursive_delete.

Recursive delete was done using shutil.rmtree. once started, it will
not check disk_gfid in between. So it ends up deleting the new files
created by other workers. Also if other worker creates files after one
worker gets list of files to be deleted, then first worker will again
get ENOTEMPTY again.

To fix these races, retry is added when it gets ENOTEMPTY/ESTALE/ENODATA.
And disk_gfid check added for original path for which recursive_delete is
called. This disk gfid check executed before every Unlink/Rmdir. If disk
gfid is not matching with GFID from Changelog, that means other worker
deleted the directory. Even if the subdir/file present, it belongs to
different parent. Exit without performing further deletes.

Retry on ENOENT during create is ignored, since if CREATE/MKNOD/MKDIR
failed with ENOENT will not succeed unless parent directory is created
again.

Rsync errors handling was handling unlinked_gfids_list only for one
Changelog, but when processed in batch it fails to detect unlinked_gfids
and retries again. Finally skips the entire Changelogs in that batch.
Fixed this issue by moving self.unlinked_gfids reset logic before batch
start and after batch end.

Most of the Geo-rep races with rm -rf is eliminated with this patch,
but in some cases stale directories left in some bricks and in mount
point we get ENOTEMPTY.(DHT issue, Error will be logged in Slave log)

BUG: 1218922
Change-Id: I8716b88e4c741545f526095bf789f7c1e28008cb
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10204
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/10599
Tested-by: Gluster Build System

geo-rep: log ENTRY failures from slave on master

2015-05-06T16:07:16+00:00

ENTRY operations failures on slave left no trace for debugging purposes.
This patch captures such failures on slave cluster and forwards them to
the master and logs them. Failures of specific interest are the ones
which return code EEXIST on the failing operations.

Change-Id: Iecab876f16593c746d53f4b7ec2e0783367856bb
Signed-off-by: Milind Changire 
BUG: 1217429
Reviewed-on: http://review.gluster.org/10048
Reviewed-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10517
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System

geo-rep: Adhering to the common storage for geo-rep

2015-05-04T09:25:23+00:00

Making geo-rep use the common storage shared by nfs,
snapshot and geo-rep. The meta volume should be named
as gluster_shared_storage, and it should be mounted
at "/var/run/gluster/shared_storage/".

Geo-rep will create a directory called 'geo-rep'
in the meta-volume and all the lock files are created
inside it.

BUG: 1217939
Change-Id: I1d88798376d68340e2b2eff018c7e4f0121a608a
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10196
Reviewed-on: http://review.gluster.org/10503
Tested-by: NetBSD Build System
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

feature/geo-rep: Active Passive Switching logic flock

2015-03-16T04:53:59+00:00

CURRENT DESIGN AND ITS LIMITATIONS:
-----------------------------------
Geo-replication syncs changes across geography using changelogs captured
by changelog translator. Changelog translator sits on server side just
above posix translator. Hence, in distributed replicated setup, both
replica pairs collect changelogs w.r.t their bricks. Geo-replication
syncs the changes using only one brick among the replica pair at a time,
calling it as "ACTIVE" and other non syncing brick as "PASSIVE".

Let's consider below example of distributed replicated setup where
NODE-1 as b1 and its replicated brick b1r is in NODE-2

        NODE-1                         NODE-2
          b1                            b1r

At the beginning, geo-replication chooses to sync changes from NODE-1:b1
and NODE-2:b1r will be "PASSIVE". The logic depends on virtual getxattr
'trusted.glusterfs.node-uuid' which always returns first up subvolume
i.e., NODE-1. When NODE-1 goes down, the above xattr returns NODE-2 and
that is made 'ACTIVE'. But when NODE-1 comes back again, the above xattr
returns NODE-1 and it is made 'ACTIVE' again. So for a brief interval of
time, if NODE-2 had not finished processing the changelog, both NODE-2
and NODE-1 will be ACTIVE causing rename race as mentioned in the bug.

SOLUTION:
---------
1. Have a shared replicated storage, a glusterfs management volume specific
   to geo-replication.

2. Geo-rep creates a file per replica set on management volume.

3. fcntl lock on the above said file is used for synchronization
   between geo-rep workers belonging to same replica set.

4. If management volume is not configured, geo-replication will back
   to previous logic of using first up sub volume.

Each worker tries to lock the file on shared storage, who ever wins will
be ACTIVE. With this, we are able to solve the problem but there is an
issue when the shared replicated storage goes down (when all replicas
goes down). In that case, the lock state is lost. So AFR needs to rebuild the
lock state after brick comes up.

NOTE:
-----
This patch brings in the, pre-requisite step of setting up management volume
for geo-replication during creation.

1. Create mgmt-vol for geo-replicatoin and start it. Management volume should
   be part of master cluster and recommended to be three way replicated
   volume having each brick in different nodes for availability.
2. Create geo-rep session.
3. Configure mgmt-vol created with geo-replication session as follows.
   gluster vol geo-rep  slavenode:: config meta_volume \
   
4. Start geo-rep session.

Backward Compatiability:
-----------------------
If management volume is not configured, it falls back to previous logic of
using node-uuid virtual xattr. But it is not recommended.

Change-Id: I7319d2289516f534b69edd00c9d0db5a3725661a
BUG: 1196632
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/9759
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Handle ENOENT during cleanup

2015-03-06T02:26:57+00:00

shutil.rmtree was failing to remove file if file was not
exists. Added error handling function to ignore ENOENT if
a file/dir not present.

BUG: 1198101
Change-Id: I1796db2642f81d9e2b5e52c6be34b4ad6f1c9786
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/9792
Tested-by: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Venky Shankar 
Reviewed-by: Kotresh HR 
Reviewed-by: Saravanakumar Arumugam

geo-rep: Fixing the typo errors

2015-02-03T18:03:41+00:00

Change-Id: Iacc67e4ba9ac45e0858f3befe84ffb8fccf7e1c3
BUG: 1075417
Signed-off-by: arao 
Reviewed-on: http://review.gluster.org/9502
Tested-by: Gluster Build System 
Reviewed-by: Humble Devassy Chirammal

geo-rep: Pause and Resume feature for geo-replication

2014-05-09T07:27:40+00:00

Changelog consumption/processing now happens in seperate process
group than monitor. When monitor process group gets SIGSTOP all
worker process, ssh, rsync will be paused except the changelog
processing. When it gets SIGCONT it resumes its operation.

Changelog agent runs as RepceServer, geo-rep worker communicates
with changelog agent using RepceClient.

Change-Id: I35c333e4d8b13d03a7808aed601960eef23cfa04
BUG: 1093602
Signed-off-by: Venky Shankar 
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/7322

geo-rep: Loading libgfchangelog.so only while running geo-rep

2014-05-09T07:03:24+00:00

In source install, libgfchangelog is installed in /usr/local/lib
When glusterd runs /usr/local/libexec/glusterfs/python/gsyncd --version
it fails to find library without LD_LIBRARY_PATH.

This patch avoids loading library when it is run from glusterd
during start.

BUG: 1096026
Change-Id: I59912227ac27ff4877d947a7c8f1fe2e8c5be06e
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/7713
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar

geo-rep: Consume Changelog History API

2014-04-30T14:36:19+00:00

Every time when geo-rep restarts it first does FS crawl using
XCrawl and then switches to Changelog Mode. This is because changelog
only had live API, that is we can get changes only after registering.

Now this(http://review.gluster.org/#/c/6930/) patch introduces History
API for changelogs. If history is available then geo-rep will use it
instead of FS Crawl.

History API returns TS till what time history is available for
given start and end time. If TS < endtime then switch to FS Crawl.
(History => FS Crawl => Live Changelog)

If TS >= endtime, then switch directly to Changelog mode
(History => Live Changelog)

Change-Id: I4922f62b9f899c40643bd35720e0c81c36b2f255
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/6938
Reviewed-by: Venky Shankar 
Reviewed-by: Humble Devassy Chirammal 
Tested-by: Gluster Build System