glusterfs.git/geo-replication/syncdaemon/gsyncdstatus.py, branch v4.1.8

geo-rep: Fix deadlock during worker start

2018-09-21T13:25:43+00:00

Analysis:
Monitor process spawns monitor threads (one per brick).
Each monitor thread, forks worker and agent processes.
Each monitor thread, while intializing, updates the
monitor status file. It is synchronized using flock.
The race is that, some thread can fork worker while
other thread opened the status file resulting in
holding the reference of fd in worker process.

Cause:
flock gets unlocked either by specifically unlocking it
or by closing all duplicate fds referring to the file.
The code was relying on fd close, hence a reference
in worker/agent process by fork could cause the deadlock.

Fix:
1. flock is unlocked specifically.
2. Also made sure to update status file in approriate places so that
the reference is not leaked to worker/agent process.

With this fix, both the deadlock and possible fd
leaks is solved.

Backport of:
 > Patch: https://review.gluster.org/20704
 > BUG: bz#1614799
 > Change-Id: I0d1ce93072dab07d0dbcc7e779287368cd9f093d
 > Signed-off-by: Kotresh HR 

fixes: bz#1630145
Change-Id: I0d1ce93072dab07d0dbcc7e779287368cd9f093d
Signed-off-by: Kotresh HR

geo-rep: Fix upgrade issue

2018-08-15T18:40:31+00:00

Cause and Analysis:
The last synced changelog for entry operations is
marked in current version to avoid re-processing
of already processed entry operations in a batch
during crash/restart of geo-rep. This was not
present in previous versoins.

The marker is maintained in the dictionary with the
key 'last_synced_entry' and dictionary is persisted
into status file. So upgrading to current version in
which the marker is present was failing with KeyError.

Solution:
Load the dictionary with default keys first which
contains all the keys including latest ones and then
load the values from status file instead of doing
otherwise.

Backport of:
 > BUG: 1575490
 > Change-Id: Ic654e6f9a3c97f616761f1362f890352a2186fb4
 > Signed-off-by: Kotresh HR 
 (cherry picked from commit 23c1385b5f6f6103e820d15ecfe1df31940fdb45)

fixes: bz#1611104
Change-Id: Ic654e6f9a3c97f616761f1362f890352a2186fb4
Signed-off-by: Kotresh HR 
(cherry picked from commit 23c1385b5f6f6103e820d15ecfe1df31940fdb45)

core/various: python3 compat, prepare for python2 -> python3

2018-05-02T11:28:46+00:00

see https://review.gluster.org/#/c/19788/

use print fn from __future__

Change-Id: If5075d8d9ca9641058fbc71df8a52aa35804cda4
updates: #411
Signed-off-by: Kaleb S. KEITHLEY

core/build/various: python3 compat, prepare for python2 -> python3

2018-04-12T11:04:27+00:00

Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines

Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.

This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.

In a follow-up patch python2 will eventually be changed to python3.

Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.

updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY

geo-rep: JSON output for status and config

2017-11-24T13:59:24+00:00

For Glusterd2 integration, JSON output of status and config
is very useful from gsyncd

Fixes: #361
Change-Id: I53c61f19033ad4ac601ea49469e4e7c7c8e9af3d
Signed-off-by: Aravinda VK

geo-rep: Structured log support

2017-06-20T06:00:47+00:00

Changed all log messages to structured log format

Change-Id: Idae25f8b4ad0bbae38f4362cbda7bbf51ce7607b
Updates: #240
Signed-off-by: Aravinda VK 
Reviewed-on: https://review.gluster.org/17551
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep/eventsapi: Add Master node information in Geo-rep Events

2016-12-02T10:16:32+00:00

Added Master node information to GEOREP_ACTIVE, GEOREP_PASSIVE, GEOREP_FAULTY
and GEOREP_CHECKPOINT_COMPLETED events.

EVENT_GEOREP_ACTIVE(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_ACTIVE",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH
    }
}

EVENT_GEOREP_PASSIVE(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_PASSIVE",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH
    }
}

EVENT_GEOREP_FAULTY(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_FAULTY",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "current_slave_host": CURRENT_SLAVE_HOST,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH
    }
}

EVENT_GEOREP_CHECKPOINT_COMPLETED(master_node and master_node_id are new fields)
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": "GEOREP_CHECKPOINT_COMPLETED",
    "message": {
        "master_volume": MASTER_VOLUME_NAME,
        "master_node": MASTER_NODE,
        "master_node_id": MASTER_NODE_ID,
        "slave_host": SLAVE_HOST,
        "slave_volume": SLAVE_VOLUME,
        "brick_path": BRICK_PATH,
        "checkpoint_time": CHECKPOINT_TIME,
        "checkpoint_completion_time": CHECKPOINT_COMPLETION_TIME
    }
}

BUG: 1395660
Change-Id: Ic91af52fa248c8e982e93a06be861dfd69689f34
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15858
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Logging improvements

2016-10-24T07:01:48+00:00

- Redundant log messages removed.
- Worker and connected slave node details added in "starting worker" log
- Added log for Monitor state change
- Added log for Worker status change(Initializing/Active/Passive/Faulty)
- Added log for Crawl status Change
- Added log for config set and reset
- Added log for checkpoint set, reset and completion

BUG: 1359612
Change-Id: Icc7173ff3c93de4b862bdb1a61760db7eaf14271
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15684
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep/eventsapi: Additional Events

2016-10-18T10:56:03+00:00

Added following events
EVENT_GEOREP_ACTIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }

EVENT_GEOREP_PASSIVE
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_PASSIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH
        }
    }
EVENT_GEOREP_CHECKPOINT_COMPLETED
    {
        "nodeid": NODEID,
        "ts": TIMESTAMP,
        "event": "GEOREP_ACTIVE",
        "message": {
            "master_volume": MASTER_VOLUME_NAME,
            "slave_host": SLAVE_HOST,
            "slave_volume": SLAVE_VOLUME,
            "brick_path": BRICK_PATH,
            "checkpoint_time": CHECKPOINT_TIME,
            "checkpoint_completion_time": CHECKPOINT_COMPLETION_TIME
        }
    }

BUG: 1379330
Change-Id: I90716175868c59dd65c8d202e73e0ede90347b6a
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15630
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
Tested-by: Kotresh HR

geo-rep: Fix Geo-rep status if monitor.pid file not exists

2016-09-08T16:15:15+00:00

If monitor.pid file not exists, gsyncd fails with following traceback

Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py",
  line 201, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py",
  line 681, in main_i
    brick_status.print_status(checkpoint_time=checkpoint_time)
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py",
  line 343, in print_status
    for key, value in self.get_status(checkpoint_time).items():
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py",
  line 262, in get_status
    with open(self.monitor_pid_file, "r+") as f:
IOError: [Errno 2] No such file or directory: '/var/lib/glusterd/
  geo-replication/master_node_slave/monitor.pid'

If Georep status command this worker's status will not be displayed
since not returning expected status output.

BUG: 1373741
Change-Id: I600a2f5d9617f993d635b9bc6e393108500db5f9
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15416
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR