glusterfs.git/xlators/cluster/dht/src/dht-common.c, branch v3.10.4

cluster/dht: Include dirs in rebalance estimates

2017-06-20T04:57:14+00:00

Empty directories were not being considered while
calculating rebalance estimates leading to negative
time-left values being displayed as part of the
rebalance status.

> BUG: 1457985
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17448
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Amar Tumballi 
> Reviewed-by: Raghavendra G 

Change-Id: I48d41d702e72db30af10e6b87b628baa605afa98
BUG: 1460914
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17530
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Talur

cluster/dht: Rebalance on all nodes should migrate files

2017-05-18T08:06:59+00:00

Problem:
Rebalance compares the node-uuid of a file against its own
to and migrates a file only if they match. However, the
current behaviour in both AFR and EC is to return
the node-uuid of the first brick in a replica set for all
files. This means a single node ends up migrating all
the files if the first brick of every replica set is on the
same node.

Fix:
AFR and EC will return all node-uuids for the replica set.
The rebalance process will divide the files to be migrated
among all the nodes by hashing the gfid of the file and
using that value to select a node to perform the migration.
This patch makes the required DHT and tiering changes.

Some tests in rebal-all-nodes-migrate.t will need to be
uncommented once the AFR and EC changes are merged.

> BUG: 1366817
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17239
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Amar Tumballi 
> Reviewed-by: Jeff Darcy 
> Reviewed-by: Shyamsundar Ranganathan 
(cherry picked from commit b23bd3dbc2c153171d0bb1205e6804afe022a55f)

Change-Id: I5ce41600f5ba0e244ddfd986e2ba8fa23329ff0c
BUG: 1451561
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17311
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Fix crash in dht rmdir

2017-05-18T08:06:43+00:00

Using local->call_cnt to check STACK_WINDs can
cause dht_rmdir_do to be called erroneously if
dht_rmdir_readdirp_cbk unwinds before we check if
local->call_cnt is zero in dht_rmdir_opendir_cbk.
This can cause frame corruptions and crashes.

Thanks to Shyam (srangana@redhat.com) for the
analysis.

> BUG: 1451083
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17305
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
(cherry picked from commit 6f7d55c9d58797beaf8d5393c03a5a545bed8bec)
Change-Id: I5362cf78f97f21b3fade0b9e94d492002a8d4a11
BUG: 1451371
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17309
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

dht: Add missing braces in dht_opendir

2017-05-14T08:35:03+00:00

>Change-Id: I6adce98f52e17953f501bc590ff7189cceac3c31
>BUG: 1431908
>Signed-off-by: Poornima G 
>Reviewed-on: https://review.gluster.org/17057
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Vijay Bellur 
(cherry picked from commit af218797fa98f2f75594fc9ae595f184682f1a0d)

Change-Id: I6adce98f52e17953f501bc590ff7189cceac3c31
BUG: 1435942
Reviewed-on: https://review.gluster.org/17285
Tested-by: Raghavendra Talur 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Talur

cluster/dht: Pass the correct xdata in fremovexattr fop

2017-05-01T10:02:52+00:00

Backport of:
> Change-Id: Id84bc87e48f435573eba3b24d3fb3c411fd2445d
> BUG: 1440051
> Reviewed-on: https://review.gluster.org/17126
> (cherry-picked from ab88f655e6423f51e2f2fac9265ff4d4f5c3e579)

Change-Id: Id84bc87e48f435573eba3b24d3fb3c411fd2445d
BUG: 1426508
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17134
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan 
Smoke: Gluster Build System

cluster/dht Remove redundant logs in dht rmdir

2017-04-27T22:07:53+00:00

Removing redundant logs were introduced in
https://review.gluster.org/#/c/17065/

> BUG: 1445590
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17118
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Susant Palai 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 

(cherry picked from commit 25f0a7b153b30b2c0e8278b0ce11d1199c3fb006)
Change-Id: I0d6055488b51a13c91d2121e87f653cdb94888b0
BUG: 1446227
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17130
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Pass the req dict instead of NULL in dht_attr2()

2017-04-27T10:51:32+00:00

    Backport of:
    > Change-Id: Id7823fd932b4e5a9b8779ebb2b612a399c0ef5f0
    > BUG: 1440051
    > Reviewed on: https://review.gluster.org/17085
    > (cherry-picked from commit d60ca8e96bbc16b13f8f3456f30ebeb16d0d1e47)

This bug was causing VMs to pause during rebalance. When qemu winds
down a STAT, shard fills the trusted.glusterfs.shard.file-size attribute
in the req dict which DHT doesn't wind its STAT fop with upon detecting
the file has undergone migration. As a result shard doesn't find the
value to this key in the unwind path, causing it to fail the STAT
with EINVAL.

Also, the same bug exists in other fops too, which is also fixed in
this patch.

Change-Id: Id7823fd932b4e5a9b8779ebb2b612a399c0ef5f0
BUG: 1426508
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17119
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System

cluster/dht: rm -rf fails if dir has stale linkto files

2017-04-27T10:49:57+00:00

rm -rf  fails with ENOENT if dir contains a lot of
stale linkto files. This is because a single
readdirp is sent as part of the rmdir which would return
and delete only as many linkto files on the bricks as would fit
in one readdirp buffer. Running rm -rf  multiple times
will eventually delete all the files. The fix sends readdirp
on each subvol until no more entries are returned.

> BUG: 1442724
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17065
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit e5f9ba138571bd18226462c49ff6a55f5c3ed3a4)

Change-Id: I447f2d193de4bd8ac16e4541c6b919d22250e39e
BUG: 1444540
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17102
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

dht: The xattrs sent in readdirp should be sent in opendir aswell

2017-04-27T10:49:47+00:00

As readdir-ahead can be loaded as a child of dht, dht has to specify
the xattrs it is intrested in, as part of opendir call itself.

>Reviewed-on: https://review.gluster.org/16902
>Smoke: Gluster Build System 
>Reviewed-by: Raghavendra G 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>(cherry picked from commit 0f71338e1d7c0b70f4fe3b19c68612fe730d9de2)

Change-Id: I012ef96cc143b0cef942df78aa7150d85ec38606
BUG: 1435942
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16947
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Raghavendra G

cluster/dht: Modify local->loc.gfid in thread safe manner

2017-04-13T15:31:49+00:00

	Backport of https://review.gluster.org/16986

Problem:
local->loc.gfid in dht_lookup_directory() will be null-gfid for a fresh lookup.
dht_lookup_dir_cbk() updates local->loc.gfid while in other thread dht_lookup_directory()
is still winding lookup calls to subvolumes so there is a chance of partial gfid being
seen by EC.

We saw in 12x(4+2) volume, ec is receiving an loc where the gfid has last 10 bytes matching
with the gfid of the directory and the first 4 bytes are all-zeros. This is leading to EC
erroring out the lookup with EINVAL which leads to NFS failing lookup with EIO.

snip from gdb:
$37 = (dht_local_t *) 0x7fde5de5b3cc
(gdb) p /x $37->loc.gfid
$39 = {0x3b, 0x82, 0x10, 0x5e, 0x40, 0x65, 0x43, 0x14, 0xa0, 0xc6, 0x8, 0xf5,
0x6c, 0x2c, 0xb8, 0x56}
(gdb) fr 7
state=) at ec-generic.c:837
837	                ec_lookup_rebuild(fop->xl->private, fop, cbk);
(gdb) p /x fop->loc[0].gfid
$40 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x43, 0x14, 0xa0, 0xc6, 0x8, 0xf5, 0x6c,
0x2c, 0xb8, 0x56}

snip from log:
[2017-01-29 03:22:30.132328] W [MSGID: 122019]
[ec-helpers.c:354:ec_loc_gfid_check] 0-butcher-disperse-4: Mismatching GFID's
in loc [2017-01-29 03:22:30.132709] W [MSGID: 112199]
[nfs3-helpers.c:3515:nfs3_log_newfh_res] 0-nfs-nfsv3:
/linux-4.9.5/Documentation => (XID: b27b9474, MKDIR: NFS: 5(I/O error), POSIX:
5(Input/output error)), FH: exportid 00000000-0000-0000-0000-000000000000, gfid
00000000-0000-0000-0000-000000000000, mountid
00000000-0000-0000-0000-000000000000 [Invalid argument]

Fix:
update local->loc.gfid in last-call to make sure there are no races.

 >BUG: 1438411
 >Change-Id: Ifcb7e911568c1f1f83123da6ff0cf742b91800a0
 >Signed-off-by: Pranith Kumar K 

BUG: 1438423
Change-Id: I804822a1d50215301881ac18318282c1a6951cfb
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/16987
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G