glusterfs.git/xlators/cluster/dht/src, branch v3.11.0

features/dht: Initialize local hashed_subvol

2017-05-25T13:07:01+00:00

Self heal directory code path doesn't always
have local->hashed_subvol populated. Populating
the same which otherwise would fail the self
heal.

> Change-Id: I03b64709fd7a68e28f9e7438243e817c53c6ef5d
> BUG: 1455104
> Signed-off-by: Kotresh HR 
> Reviewed-on: https://review.gluster.org/17381
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Amar Tumballi 
> Reviewed-by: N Balachandran 
> Reviewed-by: Raghavendra G 
(cherry picked from commit 90df37558d488f9a794f62ed74ec6d72879ed895)

Change-Id: I03b64709fd7a68e28f9e7438243e817c53c6ef5d
BUG: 1455423
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17388
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/dht: Fix ret check

2017-05-23T23:48:54+00:00

Fixed an incorrect return code check in the rebalance
code.

> BUG: 1448640
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17197
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit 67598f538efb24a9e5ac561b294a05e707e15761)
Change-Id: I60804ff121cec7a2f0419e2ee70dd22ea7533c0c
BUG: 1454853
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17373
Smoke: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/dht: Fix crash in dht_selfheal_dir_setattr

2017-05-22T14:17:58+00:00

Use a local variable to store the call cnt used in the
for loop for the STACK_WIND so as not to access local
which may be freed by STACK_UNWIND after all fops return.

> BUG: 1452102
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17343
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> CentOS-regression: Gluster Build System 
(cherry picked from commit 17784aaa311494e4538c616f02bf95477ae781bc)
Change-Id: I24f49b6dbd29a2b706e388e2f6d5196c0f80afc5
BUG: 1453050
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17348
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Rebalance on all nodes should migrate files

2017-05-17T13:52:46+00:00

Problem:
Rebalance compares the node-uuid of a file against its own
to and migrates a file only if they match. However, the
current behaviour in both AFR and EC is to return
the node-uuid of the first brick in a replica set for all
files. This means a single node ends up migrating all
the files if the first brick of every replica set is on the
same node.

Fix:
AFR and EC will return all node-uuids for the replica set.
The rebalance process will divide the files to be migrated
among all the nodes by hashing the gfid of the file and
using that value to select a node to perform the migration.
This patch makes the required DHT and tiering changes.

Some tests in rebal-all-nodes-migrate.t will need to be
uncommented once the AFR and EC changes are merged.

> BUG: 1366817
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17239
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Amar Tumballi 
> Reviewed-by: Jeff Darcy 
> Reviewed-by: Shyamsundar Ranganathan 

(cherry picked from commit b23bd3dbc2c153171d0bb1205e6804afe022a55f)
Change-Id: I5ce41600f5ba0e244ddfd986e2ba8fa23329ff0c
BUG: 1451573
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17312
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Fix crash in dht rmdir

2017-05-17T13:49:43+00:00

Using local->call_cnt to check STACK_WINDs can
cause dht_rmdir_do to be called erroneously if
dht_rmdir_readdirp_cbk unwinds before we check if
local->call_cnt is zero in dht_rmdir_opendir_cbk.
This can cause frame corruptions and crashes.

Thanks to Shyam (srangana@redhat.com) for the
analysis.

> BUG: 1451083
> Signed-off-by: N Balachandran 
> Reviewed-on: https://review.gluster.org/17305
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
(cherry picked from commit 6f7d55c9d58797beaf8d5393c03a5a545bed8bec)
Change-Id: I5362cf78f97f21b3fade0b9e94d492002a8d4a11
BUG: 1451586
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17314
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Make rebalance throttle option tuned by number

2017-04-29T14:29:34+00:00

Current rebalance throttle options: lazy/normal/aggressive may not always be
sufficient for the purpose of throttling.  In our recent test, we observed for
certain setups, normal and aggressive modes behaved similarly consuming full
disk bandwidth. So in cases like this admin should be able to  tune it
down(or vice versa) depending on the need.

Along with old throttle configurations, thread counts are tuned based on number.
e.g. gluster v set vol-name cluster-rebal.throttle  5.

Admin can tune up/down between 0 and the number of cores available.

Note: For heterogenous servers, validation will fail on the old server if "number"
is given for throttle configuration.
The message looks something like this:
"volume set: failed: Staging failed on vm2. Error: cluster.rebal-throttle should be {lazy|normal|aggressive}"

Test: Manual test by logging active thread number after reconfiguring throttle option.
testcase: tests/basic/distribute/throttle-rebal.t

Change-Id: I46e3cde546900307831028b344ecf601fd9b02c3
BUG: 1438370
Signed-off-by: Susant Palai 
Reviewed-on: https://review.gluster.org/16980
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Raghavendra G

dht: send lookup on old name inside rename with bname and pargfid

2017-04-29T14:28:38+00:00

Inside rename, a lookup is done on the source name to make sure that
the file is there. But we used to do a gfid based lookup and hence,
even if the source name was renamed to a new name from some other client,
lookup will be successful as server3_3_lookup will fetch the new path
based on the gfid.

So even if the source file does not exist any more rename will carry on,
and as server3_3_link(destination is hashed to a different brick other
than source cached scenario) also does gfid based resolve, it wont
detect that the source name does not exist and hardlink creation will be
successful (since gfid based resolve will get the new dentry).

To solve this problem, do a name based lookup inside rename. So that
rename will fail right away if the source does not exist.

Change-Id: Ieba8bdd6675088dbf18de90ed4622df043d163bd
BUG: 1412135
Signed-off-by: Susant Palai 
Reviewed-on: https://review.gluster.org/16375
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/dht: rebalance perf enhancement

2017-04-29T14:28:19+00:00

Problem: Throttle settings "normal" and "aggressive" for rebalance
did not have performance difference.

normal mode spawns $(no. of cores - 4)/2 threads and aggressive
spawns $(no. of cores - 4) threads. Though aggressive mode has twice
the number of threads compared to that of normal mode, there was no
performance gain when switched to aggressive mode from normal mode.

RCA:
During the course of debugging the above problem, we tried assigning
migration job to migration threads spawned by rebalance, rather than
synctasks(as there is more overhead associated to manage the task
queue and threads). This gave us a significant improvement over rebalance
under synctasks. This patch does not really gurantee that there will be a
clear performance difference between normal and aggressive mode, but this
patch certainly maximized the disk utilization for 1GBfiles run.

Results:

Test enviroment:
Gluster Config:
Number of Bricks: 2 (one brick per disk(RAID-6 12 disk))
Bricks:
Brick1: server1:/brick/test1/1
Brick2: server2:/brick/test1/1
Options Reconfigured:
performance.readdir-ahead: on
server.event-threads: 4
client.event-threads: 4

1000 files with 1GB each were created/renamed such that all files will have
server1 as cached and server2 as hashed, so that all files will be migrated.

Test machines had 24 cores each.

Results  with/without synctask based migration:
-----------------------------------------------

mode                    normal(10threads)          aggressive(20threads)

timetaken               0:55:30 (h:m:s)            0:56:3 (h:m:s)
withsynctask

timetaken
with migrator           0:38:3 (h:m:s)             0:23:41 (h:m:s)
threads

From above table it can be seen that, there is a clear 2x perf gain between
rebalance with synctask vs rebalance with migrator threads.

Additionally this patch modifies the code so that caller will have the exact error
number returned by dht_migrate_file(earlier the errno meaning was overloaded). This
will help avoiding scenarios where migration failure due to ENOENT, can result in
rebalance abort/failure.

Change-Id: I8904e2fb147419d4a51c1267be11a08ffd52168e
BUG: 1420166
Signed-off-by: Susant Palai 
Reviewed-on: https://review.gluster.org/16427
Smoke: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/dht: Pass the correct xdata in fremovexattr fop

2017-04-28T04:27:14+00:00

Change-Id: Id84bc87e48f435573eba3b24d3fb3c411fd2445d
BUG: 1440051
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17126
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

cluster/dht Remove redundant logs in dht rmdir

2017-04-26T09:44:06+00:00

Removing redundant logs were introduced in
https://review.gluster.org/#/c/17065/

Change-Id: I0d6055488b51a13c91d2121e87f653cdb94888b0
BUG: 1445590
Signed-off-by: N Balachandran 
Reviewed-on: https://review.gluster.org/17118
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Susant Palai 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G