glusterfs.git/geo-replication/syncdaemon/master.py, branch v3.5.0beta2

gsyncd / geo-rep: geo-replication fixes

2014-01-27T17:43:19+00:00

-> "threaded" hybrid crawl.
-> Enabling metatadata synchronization.
-> Handling EINVAL/ESTALE gracefully while syncing metadata.
-> Improvments to changelog crawl code.
-> Initial crawl changelog generation format.
-> No gsyncd restart when checkpoint updated.
-> Fix symlink handling in hybrid crawl.
-> Slave's xtime key is 'stime'.
-> tar+ssh as data synchronization.
-> Instead of 'raise', just log in warning level for xtime missing cases.
-> Fix for JSON object load failure
-> Get new config value after config value reset.
-> Skip already processed changelogs.
-> Saving status of each individual worker thread.
-> GFID fetch on slave for purges.
-> Add tar ssh keys and config options.
-> Fix nlink count when using backend.
-> Include "data" operation for hardlink.
-> Use changelog time prefix as slave's time.
-> Process changelogs in parallel.

Change-Id: I09fcbb2e2e418149a6d8435abd2ac6b2f015bb06
BUG: 1036539
Signed-off-by: Ajeet Jha 
Reviewed-on: http://review.gluster.org/6404
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati 
Reviewed-on: http://review.gluster.org/6809
Reviewed-by: Vijay Bellur

geo-rep: pack the value of 'readlink()' while crawling symlinks

2013-09-20T18:46:58+00:00

without having 'readlink()' value, symlink fails on slave mounts

Change-Id: Ib84be8db77c033029ba8ba454fd629a496fc3fe1
Signed-off-by: Amar Tumballi 
BUG: 847839
Reviewed-on: http://review.gluster.org/5950
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System

geo-rep: create hardlink as hardlinks in xsync crawl

2013-09-20T18:46:31+00:00

BUG: 847839
Change-Id: Ieaa754f15611392c09afcc7190b9ff9da39d7fce
Signed-off-by: Amar Tumballi 
Reviewed-on: http://review.gluster.org/5934
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System

gsyncd / geo-rep: maximum number of retries per changelog

2013-09-05T03:54:38+00:00

With entry retries MAX_RETRIES is the threashold for the number
of times we retry a changelog. After hitting this limit, the
changelog is marked as processed after warning for incompleteness
in the log.

Change-Id: I777b71b4cdaa63335d9c936f2d64e5c0264a0f26
BUG: 1003803
Signed-off-by: Venky Shankar 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi 
Reviewed-on: http://review.gluster.org/5758
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati

gsyncd / geo-rep: handle ENOENT during readlink()

2013-09-05T03:53:43+00:00

Change-Id: I8611122fda63ea496ad4491473ecabe96bf19cfa
BUG: 1003803
Signed-off-by: Venky Shankar 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi 
Reviewed-on: http://review.gluster.org/5756
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati

gsyncd / geo-rep: overload rename with an unlink

2013-09-05T03:52:02+00:00

This is certainly a "hack" for cases when after a rename
the gfid is unlinked (which is now pointed by a different
pargfid/basename). This was osbserved when a file edited
(uisng vi/vim) on the glusterfs mount point which has a
geo-replication session running.

Change-Id: I0a5d9f0f549aa4403671fa1f11e95510142e5720
BUG: 1003800
Signed-off-by: Avra Sengupta 
Signed-off-by: Venky Shankar 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi 
Reviewed-on: http://review.gluster.org/5755
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

gsyncd / geo-rep: "disjoint" cascading geo-replication sessions

2013-09-05T03:49:12+00:00

Slave's xtime is now stored on the master itself (and that too only on
the root), which implies it cannot be propogated to the cascaded slave.
Thus the intermediate master now makes use of it's own volume information
to propogate volume-mark and xtime.

On starting Geo-Replication "geo-replication.ignore-pid-check" marker
option is enabled, which is an override for the client-pid check in
marker. This options triggers marker update only for geo-replication
auxillary mount (client-pid == -1). Since gsyncd not does setxattr()
directly on the bricks, this option won't trigger a chain of spurious
metadata updates that would need to be processed by gsyncd.

Change-Id: If50c5ef275dfb6b4ff4fd35be2565587e2fdf3e1
BUG: 996371
Original Author: Venky Shankar 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/5592
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati

gsyncd / geo-rep: introduce "recovery" mode

2013-09-05T03:42:31+00:00

This mode introduces and optimized recovery of master,
used during failover-failback cases. Files are only
transferred if they are identified as changed on the slave.

A normal failback mechanism would be just to reverse the
direction of geo-replication syncing. Doing this is the
correct way but could be time consuming. "Recovery" mode
optimizes this by identifying files (as skipping them) duing
the one-shot crawl.

NOTE: Use this mode when you are _absolutely_ sure that the
namespace on the master is intact, ie. there is number of files
in master is equal to the number of files in the slave, but the
slave was written to (after a failover) when master was brought
back up.

- gluster volume geo  :: config special_sync_mode recover
- gluster volume geo  :: start

Change-Id: Idb698949e4e86698b15d04a35d33095562f51714
BUG: 994462
Original Author: Venky Shankar 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/5515
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati

gsyncd / geo-rep: remove deprecated code!

2013-09-05T03:32:15+00:00

This includes "xtime" base crawling and syncing,
"Blind" and "Wrapup" mixins for failover/failback.

Change-Id: I9bc8695dc1c19b1261332f64621305dff2929b32
BUG: 994461
Original Author: Venky Shankar 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/5514
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati

gsyncd / geo-rep: fix regular file creation on rename

2013-09-05T03:28:12+00:00

On creation of a regular file on the slave (this happens
when an unlink()/rmdir() operation is performed on master
after a rename() or a link().

This patch fixes the regular file creation on the slave when
the above mentioned series of operation occurs. entry_ops()
on the slave for a rename() was missing the stat() information.

Change-Id: Idb80c6b3e27c1797dade1113ac8ce920db9c740c
BUG: 994353
Original Author: Venky Shankar 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/5509
Tested-by: Gluster Build System 
Reviewed-by: Avra Sengupta 
Tested-by: Avra Sengupta 
Reviewed-by: Anand Avati