glusterfs.git, branch v3.9.0rc2

tools/glusterfind: kill remote processes and separate run-time directories

2016-10-26T09:09:34+00:00

Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get  changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

> Reviewed-on: http://review.gluster.org/15609
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Aravinda VK 
(cherry picked from commit feea851fad4f89b48bfe89fe3b75250cc7bd6501)


Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1388731
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15729
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

CLI/TIER: throw warning regarding the removal of the older commands.

2016-10-26T07:04:29+00:00

The older tier commands for attach tier and detach tier have to be
removed from code. This patch sends a warning asking to use new
command as older ones are depricated and will be removed.

>Change-Id: Ie1c62947bad6ff106f40331ff6134838a6c72a7a
>BUG: 1388062
>Signed-off-by: hari 
>Reviewed-on: http://review.gluster.org/15713
>Tested-by: hari gowtham 
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Dan Lambright 

Change-Id: I28bcb0b0d2f362fbdd28aab9bea95e3a2947caa6
BUG: 1388470
Signed-off-by: hari 
Reviewed-on: http://review.gluster.org/15723
Tested-by: hari gowtham 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee

events: add TIER_START and TIER_START_FORCE events

2016-10-26T05:11:54+00:00

Add TIER_START and TIER_START_FORCE events
Conditionally generate DETACH events as per user confirmation.

> Reviewed-on: http://review.gluster.org/15675
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Dan Lambright 
(cherry picked from commit 8f8e23b0142a21779648802664db5b3d5253e4e0)

Change-Id: I205dc14884d707087edce42e8cf4208bd89d31dc
BUG: 1387981
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15708
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

cluster/dht: Incorrect volname in rebalance events

2016-10-26T02:57:10+00:00

The rebalance event code was using strtok to parse the
volume name which is incorrect.
Reworked the code to get the correct volume name using
strstr.

> Change-Id: Ib5f3305a34e6bf1ecfef677d87c5aff96bdeb0e6
> BUG: 1388010
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/15712
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Raghavendra G 
> CentOS-regression: Gluster Build System 
(cherry picked from commit f4efbf0e3092bd2181f62be1e1f30f202678c866)

Change-Id: Ia22d6a8ce23e9dcab8281b4995277cbb3a196fda
BUG: 1388563
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/15725
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

rpc/socket: Close pipe on disconnection

2016-10-25T11:09:08+00:00

Encrypted connections create a pipe, which isn't closed when the
connection disconnects. This leaks fds, and gluster eventually ends up
in a situation with fd starvation which leads to operation failures.

> Change-Id: I144e1f767cec8c6fc1aa46b00cd234129d2a4adc
> BUG: 1336371
> Signed-off-by: Kaushal M 
> Reviewed-on: http://review.gluster.org/14356
> Tested-by: MOHIT AGRAWAL 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Raghavendra G 
> Signed-off-by: Kaushal M 

Change-Id: I144e1f767cec8c6fc1aa46b00cd234129d2a4adc
BUG: 1387960
Signed-off-by: Kaushal M 
Reviewed-on: http://review.gluster.org/15702
Tested-by: Atin Mukherjee 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Raghavendra G

rpc: Fix the race between notification and reconnection

2016-10-25T11:00:46+00:00

Problem:
There was a hang because unlock on an entry failed with
ENOTCONN.
Client thinks the connection is down where as server thinks
the connection is up.

This is the race we are seeing:
1) Connection from client to the brick disconnects.
2) Saved frames unwind is called which unwinds all
   frames that were wound before disconnect.
3) connection from client to the brick happens and
   setvolume.
4) Disconnect notification for the connection in 1)
   comes now and calls client_rpc_notify() which
   marks the connection to be offline even when the
   connection is up.

This is happening because I/O can retrigger connection
before disconnect notification is sent to the higher
layers in rpc.

Fix:
Notify the higher layers that a disconnect happened and then
go ahead with reconnect logic.

For the logs which point to the information above check:
https://bugzilla.redhat.com/show_bug.cgi?id=1386626#c1

Thanks to Raghavendra G for suggesting the correct fix.

 >BUG: 1386626
 >Change-Id: I3c84ba1f17010bd69049fa88ec5f0ae431f8cda9
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15681
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Niels de Vos 
 >CentOS-regression: Gluster Build System 
 >Smoke: Gluster Build System 
 >Reviewed-by: Raghavendra G 
 >(cherry picked from commit a6b63e11b7758cf1bfcb67985e25ec02845f0995)

Change-Id: Ifa721193c26b70e26b47b7698c077da0ad5f2e1d
BUG: 1388323
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15717
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

storage/posix: Fix race in posix_pstat

2016-10-25T10:19:16+00:00

Problem:
When one thread is in the process of creating a file/directory
and the other thread is doing readdirp, there is a chance that
posix_pstat, creation fops race in the following manner which
will lead to wrong stat values to be read by parent xlators
like posix-acl.

Creation fops                           posix_pstat() as part of readdirp
1) file is created with uid/gid 0/0     1) does stat of the path that
                                           is created just now.
2) Does chown to set the correct
   uid/gid
3) Sets the acl/user/internal xattrs
4) Sets the gfid on the entry and
   completes the creation of the file/dir
                                        2) fills the gfid in the iatt

If unwind of readdirp hits server xlator before creation fop, then
posix-acl remembers uid/gid of the file to be root/root and fails
fops like open etc on it.

Fix:
Reverse the order of filling gfid and filling lstat() values in
posix_pstat() so that if there is gfid in iatt buffer uid/gid
are valid.

 >Change-Id: I46caa7f6da7abfa40a0b1d70e35b88de9c64959c
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15564
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Smoke: Gluster Build System 
 >Reviewed-by: Shyamsundar Ranganathan 

BUG: 1386072
Change-Id: Ida2bd7297de8e7f51bcb7deda8936ce64ea8ec2b
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15664
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

bitrot/cli: Add ondemand scrub event

2016-10-25T09:10:58+00:00

Following Bitrot Events are added

BITROT_SCRUB_ONDEMAND
{
     "nodeid": NODEID,
     "ts": TIMESTAMP,
     "event": EVENT_TYPE,
     "message": {
        "name": VOLUME_NAME,
     }
}

> Change-Id: I85e668e254e6f29c447ddb4ad2ce2fc04f98bf3c
> BUG: 1387864
> Signed-off-by: Kotresh HR 
> Reviewed-on: http://review.gluster.org/15700
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 

Change-Id: I85e668e254e6f29c447ddb4ad2ce2fc04f98bf3c
BUG: 1387964
Signed-off-by: Kotresh HR 
(cherry picked from commit 255cc64375abe2925c7da1e13e45018dad4462df)
Reviewed-on: http://review.gluster.org/15705
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Aravinda VK

geo-rep: Upgrade conf file only if it is session config

2016-10-25T07:22:18+00:00

Ignore config upgrade if it is template config file present in
/var/lib/glusterd/geo-replication/gsyncd_template.conf

> Reviewed-on: http://review.gluster.org/15669
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 
> NetBSD-regression: NetBSD Build System 

BUG: 1388150
Change-Id: I2cbba3103b6801c16ff57f778a90b9a0bb2467cf
Signed-off-by: Aravinda VK 
(cherry picked from commit 1506c7a98d8d3b31e68d0f214ab331f28ffa9fb5)
Reviewed-on: http://review.gluster.org/15715
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

geo-rep: Logging improvements

2016-10-25T07:22:01+00:00

- Redundant log messages removed.
- Worker and connected slave node details added in "starting worker" log
- Added log for Monitor state change
- Added log for Worker status change(Initializing/Active/Passive/Faulty)
- Added log for Crawl status Change
- Added log for config set and reset
- Added log for checkpoint set, reset and completion

> Reviewed-on: http://review.gluster.org/15684
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 

BUG: 1387990
Change-Id: Icc7173ff3c93de4b862bdb1a61760db7eaf14271
Signed-off-by: Aravinda VK 
(cherry picked from commit cdc30ed8eacb6772e0dabb863ef51cef794d60dd)
Reviewed-on: http://review.gluster.org/15710
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR