glusterfs.git/geo-replication/syncdaemon/gsyncd.py, branch v3.7.9

geo-rep: Fix getting subvol number

2016-01-04T06:06:39+00:00

Fix getting subvol number if the volume
type is tier. If the volume type was tier,
the subvol number was calculated incorrectly
and hence few of workers didn't become ACTIVE
resulting in files not being replicated from
corresponding brick. This patch addresses
the same.

BUG: 1293309
Change-Id: I318de346657d330a2394507514bdff61feb92d27
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12994
Tested-by: NetBSD Build System 
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-on: http://review.gluster.org/13059

geo-rep: use cold tier bricks for namespace operations

2015-12-08T06:42:35+00:00

Problem:
symlinks are not getting synced to slave in a Tiering based volume.

Solution:
Now, symlinks are created directly in cold tier bricks( in the backend).

Earlier, cold tier was avoided for namespace operations and only
hot tier was used while processing changelogs.

Now, cold tier is HASH subvolume in a Tiering volume.
So, carry out namespace operation only in cold tier subvolume and
avoid hot tier subvolume to avoid any races.

Earlier, XSYNC was used(and changeloghistory avoided) during initial sync
in order to avoid race while processing historychangelog in Hot tier.
This is no longer required as there is no race from Hot tier.

Also, avoid both live and history changelog ENTRY operations from Hot tier
to avoid any race with cold tier.

Change-Id: Ia8fbb7ae037f5b6cb683f36c0df5c3fc2894636e
BUG: 1288027
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12844
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Venky Shankar 
(cherry picked from commit 93f31189ce8f6e2980a39b02568ed17088e0a667)
Reviewed-on: http://review.gluster.org/12891

geo-rep: Avoid cold tier bricks during ENTRY operation

2015-12-07T09:50:47+00:00

This is a series of patch which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier , all fops will be
recorded in HOT tier.
b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot
tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

Now, once the file is demoted to COLD tier, any namespace operation
carried out on the cold tier can be avoided as we directly RECORD
the same in HOT tier.

How?
1. Check whether the brick is cold tier and skip ENTRY operation.
2. Also, if it is cold tier brick, use Xsync(which is used during initial run).
   This will help in getting all cold tier bricks changes using File System crawl
   and helps in avoiding races with hot tier brick(which can happen
   if historychangelog used in cold tier brick).

Dependent patches:
1. http://review.gluster.org/12239
2. http://review.gluster.org/12326

Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde
BUG: 1275173
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12355
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
(cherry picked from commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268)
Reviewed-on: http://review.gluster.org/12429
Reviewed-by: Venky Shankar

geo-rep: Fix Geo-rep logging to log datetime in GMT/UTC

2015-12-02T15:29:13+00:00

Geo-rep is logging in Local time, all other Gluster logs are in
GMT/UTC. It is very difficult to co-relate Geo-rep logs with
other Gluster logs.

BUG: 1284737
Change-Id: Ieae8bda7e4788e587cf4595e21e0e772c210cfbb
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/12583
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur 
 (cherry picked from commit 41e900199d7f369862b21b739dd11602d0d7c48d)
Reviewed-on: http://review.gluster.org/12723
Tested-by: NetBSD Build System

geo-rep: New Config option for ssh_port

2015-11-25T07:18:38+00:00

If different port used for SSH instead of 22, Geo-replication
was failing to establish SSH connection.

ssh_port option can be added using config:ssh_command and
config:ssh_command_tar, but user has to remember complete
ssh command used with parameter to add/modify ssh port.

This patch adds new config option for ssh_port,

gluster volume geo-replication   \
        config ssh_port 52022

Change-Id: I7753a09485f0b1f49d2b2a80b962c720817c96f4
Signed-off-by: Aravinda VK 
BUG: 1283060
Reviewed-on: http://review.gluster.org/12444
Tested-by: NetBSD Build System 
Reviewed-by: Saravanakumar Arumugam 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
(cherry picked from commit 7d35eb5926869ed084295600502a85ce13be506f)
Reviewed-on: http://review.gluster.org/12607
Reviewed-by: Kotresh HR

geo-rep: Update geo-rep status, if monitor process is killed

2015-11-01T14:57:08+00:00

Problem:
When the monitor process itself is getting killed, geo-rep session
still shows as active.

Status command will just pick up the content from the status file
to show the output. Monitor process is the one which updates the Status file.

When the monitor process itself gets killed, there is no way to update
the status file. So, geo-rep session status command ends up showing
last updated Status present in the status file. 

Solution:
While getting the status output, check whether monitor process is running.
If it is NOT running, update the status as STOPPED.

Change-Id: I86a7ac1746dd8f27eef93658e992ef16f6068d9d
BUG: 1276060
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/11873
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Milind Changire 
Reviewed-by: Kotresh HR 
Reviewed-by: Jeff Darcy 
(cherry picked from commit 4d4c7d5dc54850dcf916083b2b1398d9bfe2bfe6)
Reviewed-on: http://review.gluster.org/12448
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

geo-rep: Status Enhancements

2015-05-06T16:12:35+00:00

Discussion in gluster-devel
http://www.gluster.org/pipermail/gluster-devel/2015-April/044301.html

MASTER NODE - Master Volume Node
MASTER VOL - Master Volume name
MASTER BRICK - Master Volume Brick
SLAVE USER - Slave User to which Geo-rep session is established
SLAVE - :: used in Geo-rep Create command
SLAVE NODE - Slave Node to which Master worker is connected
STATUS - Worker Status(Created, Initializing, Active, Passive, Faulty,
         Paused, Stopped)
CRAWL STATUS - Crawl type(Hybrid Crawl, History Crawl, Changelog Crawl)
LAST_SYNCED - Last Synced Time(Local Time in CLI output and UTC in XML output)
ENTRY - Number of entry Operations pending.(Resets on worker restart)
DATA - Number of Data operations pending(Resets on worker restart)
META - Number of Meta operations pending(Resets on worker restart)
FAILURES - Number of Failures
CHECKPOINT TIME - Checkpoint set Time(Local Time in CLI output and UTC
                  in XML output)
CHECKPOINT COMPLETED - Yes/No or N/A
CHECKPOINT COMPLETION TIME - Checkpoint Completed Time(Local Time in CLI
                             output and UTC in XML output)

XML output:

cliOutput>
  geoRep>
      volume>
        name>
        sessions>
          session>
             session_slave>
             pair>
                master_node>
                master_brick>
                slave_user>
                slave/>
                slave_node>
                status>
                crawl_status>
                entry>
                data>
                meta>
                failures>
                checkpoint_completed>
                master_node_uuid>
                last_synced>
                checkpoint_time>
                checkpoint_completion_time>

BUG: 1218586
Change-Id: I944a6c3c67f1e6d6baf9670b474233bec8f61ea3
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/10121
Reviewed-by: Kotresh HR 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/10574
Tested-by: Gluster Build System

geo-rep: Adhering to the common storage for geo-rep

2015-05-04T09:25:23+00:00

Making geo-rep use the common storage shared by nfs,
snapshot and geo-rep. The meta volume should be named
as gluster_shared_storage, and it should be mounted
at "/var/run/gluster/shared_storage/".

Geo-rep will create a directory called 'geo-rep'
in the meta-volume and all the lock files are created
inside it.

BUG: 1217939
Change-Id: I1d88798376d68340e2b2eff018c7e4f0121a608a
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10196
Reviewed-on: http://review.gluster.org/10503
Tested-by: NetBSD Build System
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

geo-rep: Log Rsync performance

2015-04-02T13:47:16+00:00

Introducing configurable option to log the rsync performance.

gluster volume geo-replication  :: \
        config log-rsync-performance true

Default value is False.

Example log:
[2015-03-31 16:48:34.572022] I [resource(/bricks/b1):857:rsync] SSH: rsync
performance: Number of files: 2 (reg: 1, dir: 1), Number of regular files
transferred: 1, Total file size: 178 bytes, Total transferred file
size: 178 bytes, Literal data: 178 bytes, Matched data: 0 bytes,
Total bytes sent: 294, Total bytes received: 32, sent 294 bytes
received 32 bytes  652.00 bytes/sec

Change-Id: If11467e29e6ac502fa114bd5742a8434b7084f98
Signed-off-by: Aravinda VK 
BUG: 764827
Reviewed-on: http://review.gluster.org/10070
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

geo-rep: Add support for acls

2015-03-30T13:57:16+00:00

This patch adds support for ACLS. When it sees SETXATTR
in Changelog, it adds the file to data queue. rsync/tar+ssh
will take care of syncing ACLS. User set ACLS will be
synced to Slave.

This requires "system.posix_acl_access" to go through when
client-pid is equal GF_CLIENT_PID_GSYNCD in fuse layer.

New config interface is introduced, sync-acls
Which can be set using geo-rep config(Default is True)

gluster volume geo-replication  :: \
               config sync-acls false

Change-Id: I7eb3523fa72b8fed830efc98138891244e830d65
BUG: 1187021
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10001
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar