glusterfs.git/xlators/cluster, branch v3.9.0

features/shard: Fill loc.pargfid too for named lookups on individual shards

2016-11-09T07:34:31+00:00

        Backport of: http://review.gluster.org/#/c/15788/

On a sharded volume when a brick is replaced while IO is going on, named
lookup on individual shards as part of read/write was failing with
ENOENT on the replaced brick, and as a result AFR initiated name heal in
lookup callback. But since pargfid was empty (which is what this patch
attempts to fix), the resolution of the shards by protocol/server used
to fail and the following pattern of logs was seen:

Brick-logs:

[2016-11-08 07:41:49.387127] W [MSGID: 115009]
[server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
for (null) (LOOKUP)
[2016-11-08 07:41:49.387157] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
(00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
==> (Invalid argument) [Invalid argument]

Client-logs:
[2016-11-08 07:41:27.497687] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.497755] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.498500] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.499680] E [MSGID: 133010]

Also, this patch makes AFR by itself choose a non-NULL pargfid even if
its ancestors fail to initialize all pargfid placeholders.

Change-Id: I1ad027b7b2ccd6b5c0162726b63b06d261b108e8
BUG: 1392844
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15795
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Ravishankar N 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

afr,ec: Heal device files with correct major, minor numbers

2016-10-27T06:23:26+00:00

Thanks a lot to xiaoping.wu@nokia.com from Nokia for the bug and the
fix.

 >BUG: 1384297
 >Change-Id: Ie443237e85d34633b5dd30f85eaa2ac34e45754c
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15728
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Xavier Hernandez 
 >CentOS-regression: Gluster Build System 

Change-Id: Ie7977caf7c98c91fca64752c56731c37ad27df4d
BUG: 1388912
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15734
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Xavier Hernandez

cluster/dht: Incorrect volname in rebalance events

2016-10-26T02:57:10+00:00

The rebalance event code was using strtok to parse the
volume name which is incorrect.
Reworked the code to get the correct volume name using
strstr.

> Change-Id: Ib5f3305a34e6bf1ecfef677d87c5aff96bdeb0e6
> BUG: 1388010
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/15712
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Raghavendra G 
> CentOS-regression: Gluster Build System 
(cherry picked from commit f4efbf0e3092bd2181f62be1e1f30f202678c866)

Change-Id: Ia22d6a8ce23e9dcab8281b4995277cbb3a196fda
BUG: 1388563
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/15725
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

cluster/afr: Prevent dict_set() on NULL dict

2016-10-16T02:38:24+00:00

In afr lookup when NULL dict is received in lookup, afr
is supposed to set all the xattrs it requires in a new dict
it creates, but for 'link-count' it is trying to set to the
dict that is passed in lookup which can be NULL sometimes.
This is leading to error logs. Fixed the same in this patch.

 >BUG: 1385104
 >Change-Id: I679af89cfc410cbc35557ae0691763a05eb5ed0e
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15646
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Ravishankar N 

BUG: 1385236
Change-Id: I802e74e7ad24e183b6653101ad7bf5ab0bf6e55b
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15650
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Ravishankar N

afr: Take full locks in arbiter only for data transactions

2016-10-15T17:02:31+00:00

Problem:
Sharding exposed a bug in arbiter config. where `dd` throughput was
extremely slow. Shard xlator was sending a fxattrop to update the file
size immediately after a writev. Arbiter was incorrectly over-riding the
LLONGMAX-1 start offset (for metadata domain locks) for this fxattrop,
causing the inodelk to be taken on the data domain. And since the
preceeding writev hadn't released the lock (afr does a 'lazy'
unlock if write succeeds on all bricks), this degraded to a blocking
lock causing extra lock/unlock calls and delays.

Fix:
Modify flock.l_len and flock.l_start to take full locks only for data
transactions.

> Reviewed-on: http://review.gluster.org/15641
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Pranith Kumar Karampuri 
(cherry picked from commit 3a97486d7f9d0db51abcb13dcd3bc9db935e3a60)

Change-Id: I906895da2f2d16813607e6c906cb4defb21d7c3b
BUG: 1385224
Signed-off-by: Ravishankar N 
Reported-by: Max Raba 
Reviewed-on: http://review.gluster.org/15648
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

cluster/ec: set/unset dirty flag for data/metadata update

2016-10-13T20:04:11+00:00

Currently, for all the update operations, metadata or data,
we set the dirty flag at the end of the operation only if
a brick is down. This leads to delay in healing and in some
cases not at all.
In this patch we set (+1) the dirty flag
at the start of the metadata or data update operations and
after successfull completion of the fop, we unset (-1) it again.

>Change-Id: Ide5668bdec7b937a61c5c840cdc79a967598e1e9
>BUG: 1316873
>Signed-off-by: Ashish Pandey 
>Reviewed-on: http://review.gluster.org/13733
>Tested-by: Pranith Kumar Karampuri 
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Xavier Hernandez 

Change-Id: Ide5668bdec7b937a61c5c840cdc79a967598e1e9
BUG: 1377570
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15534
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/ec: Implement heal info with lock

2016-10-13T14:35:46+00:00

Problem: Currently heal info command prints all
the files/directories if the index for the
file/directory is present in .glusterfs/indices folder.
After implementing patch http://review.gluster.org/#/c/13733/
indices of the file which is going through update fop
will also be present in .glusterfs/indices even
if the fop is successful on all the brick. At this time
if heal info command is being used, it will also display this
file which is actually healthy and does not require any heal.

Solution: Take lock on a file corresponding to the indices
and inspect xattrs to decide if the file needs heal or not.

>Reviewed-on: http://review.gluster.org/15543
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Pranith Kumar Karampuri 
>Reviewed-by: Xavier Hernandez 
>CentOS-regression: Gluster Build System 
>Smoke: Gluster Build System 
>Signed-off-by: Ashish Pandey 

Change-Id: I6361e2813ece369be12d02e74816df4eddb81cfa
BUG: 1383913
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15627
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Xavier Hernandez

afr: Ignore gluster internal (virtual) xattrs in metadata heal check

2016-09-29T07:08:35+00:00

Problem:
In arbiter configuration, posix-xlator in the arbiter brick always sets
the GF_CONTENT_KEY in the response dict with a value 0. If the file size on
the data bricks is more than quick-read's max-file-size (64kb default),
those bricks don't set the key. Because of this difference in the no. of dict
elements, afr triggers metadata heal in lookup code path, in turn
leading to extra lookups+inodelks.

Fix:
Changed afr dict comparison logic to ignore all virtual xattrs and the
on-disk ones that we should not be healing.

Also removed is_virtual_xattr() function. The original callers to this
function (upcall) don't seem to need it anymore.


> Reviewed-on: http://review.gluster.org/15548
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit 5afc6aba906a21aee19c2f1baaa7d9eb762ae0ac)

Change-Id: I05730bdd39d8fb0b9a49a5fc9c0bb01f0d3bb308
BUG: 1379528
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/15577
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

afr: Modifications to afr events

2016-09-25T01:28:05+00:00

Modified afr event message to add a 'type' key as detailed in the BZ.
Also added events for data and metadata split-brain.


> Reviewed-on: http://review.gluster.org/15550
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit f6a3c541941df6fd19ef57185aca5c4bcec2dec3)
Change-Id: I8156674b4b6a501499fc10fd68e05115fdaef3e4
BUG: 1379028
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/15565
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

dht/events: Added rebalance events

2016-09-17T10:51:48+00:00

The rebalance process will now send an event when it is
complete.
Also fixed a problem where the run-time was not always
set causing spurious rebalance failure events to be sent.

> Change-Id: Ib445171c78c9560940022bca20c887d31a9bb1ca
> BUG: 1371874
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/15501
> Reviewed-by: Raghavendra G 
> Reviewed-by: Shyamsundar Ranganathan 
(cherry picked from commit e872917dbf4563703b8f396d7d30f8168ed646cd)

Change-Id: Ia24038c04548a98aac899df10414585b988caa76
BUG: 1376477
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/15520
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee