glusterfs.git/xlators/cluster/dht/src, branch v3.8.8

dht/rebalance: remove errno check for failure detection

2017-01-08T11:03:50+00:00

> BUG: 1410355
> Change-Id: I867419ca36a81ef7209e6911a46c1c2c898b8eab
> Signed-off-by: Susant Palai 
> Reviewed-on: http://review.gluster.org/16328
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
>Reviewed-by: Raghavendra G 
(cherry picked from commit 451ca272d12f2d49522c845d53585520f71525f8)

Change-Id: If05094346fc1fc48f4982db6a195740171ae2fad
BUG: 1410764
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/16348
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht: Fix dict_leak in migration check tasks

2017-01-08T10:53:49+00:00

Fixed a memleak where dict was not being unrefed
in the dht_migration_complete_check_task and
dht_rebalance_inprogress_task functions.

BUG: 1410369
> Change-Id: I3d42e9a2e5c8596c985bf6431a68fd3905227383
> BUG: 1409186
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/16308
> Smoke: Gluster Build System 
> Reviewed-by: MOHIT AGRAWAL 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> (cherry picked from commit 11b6a2c9fc5232b58774cab29873406c0fbfef19)

Change-Id: I4e92e4c87b52f6f5ffb859005aa3d525225b4bda
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/16331
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht: Fix memory corruption while accessing regex stored in

2017-01-04T03:41:24+00:00

private

If reconfigure is executed parallely (or concurrently with dht_init),
there are races that can corrupt memory. One such race is modification
of regexes stored in conf (conf->rsync_regex_valid and
conf->extra_regex_valid) through dht_init_regex. With change [1],
reconfigure codepath can get executed parallely (with itself or with
dht_init) and this fix is needed.

Also, a reconfigure can race with any thread doing dht_layout_search,
resulting in dht_layout_search accessing regex freed up by reconfigure
(like in bz 1399134).

[1] http://review.gluster.org/15046

>Change-Id: I039422a65374cf0ccbe0073441f0e8c442ebf830
>BUG: 1399134
>Signed-off-by: Raghavendra G 
>Reviewed-on: http://review.gluster.org/15945
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: N Balachandran 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Shyamsundar Ranganathan 

Change-Id: I039422a65374cf0ccbe0073441f0e8c442ebf830
BUG: 1399423
Signed-off-by: Raghavendra G 
(cherry picked from commit 64451d0f25e7cc7aafc1b6589122648281e4310a)
Reviewed-on: http://review.gluster.org/15793
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

dht/rename : Incase of failure remove linkto file properly

2017-01-02T13:07:25+00:00

Generally linkto file is created using root user. Consider following
case, a user is trying to rename a file which he is not permitted.
So the rename fails with EACESS and when rename tries to cleanup the
linkto file, it fails.

The above issue happens when rename/00.t test executed on nfs-ganesha
clients :
Steps executed in script
* create a file "abc" using root
* rename the file "abc" to "xyz" using a non root user, it fails with EACESS
* delete "abc"
* create directory "abc" using root
* again try ot rename "abc" to "xyz" using non root user, test hungs here
which slowly leds to OOM kill of ganesha process

RCA put forwarded by Du for OOM kill of ganesha
Note that when we hit this bug, we've a scenario of a dentry being
present as:
    * a linkto file on one subvol
    * a directory on rest of subvols

When a lookup happens on the dentry in such a scenario, the control flow
goes into an infinite loop of:

    dht_lookup_everywhere
    dht_lookup_everywhere_cbk
    dht_lookup_unlink_cbk
    dht_lookup_everywhere_done
    dht_lookup_directory (as local->dir_count > 0)
    dht_lookup_dir_cbk (sets to local->need_selfheal = 1 as the entry is a linkto file on one of the subvol)
    dht_lookup_everywhere (as need_selfheal = 1).

This infinite loop can cause increased consumption of memory due to:
1) dht_lookup_directory assigns a new layout to local->layout unconditionally
2)  Most of the functions in this loop do a stack_wind of various fops.

This results in growing of call stack (note that call-stack is destroyed only after lookup response is
received by fuse - which never happens in this case)

Thanks Du for root causing the oom kill and Sushant for suggesting the fix

Upstream reference :
>Change-Id: I1e16bc14aa685542afbd21188426ecb61fd2689d
>BUG: 1397052
>Signed-off-by: Jiffin Tony Thottan 
>Reviewed-on: http://review.gluster.org/15894
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Smoke: Gluster Build System 
>Reviewed-by: Raghavendra G 
>(cherry picked from commit 57d59f4be205ae0c7888758366dc0049bdcfe449)

Change-Id: I1e16bc14aa685542afbd21188426ecb61fd2689d
BUG: 1401029
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: http://review.gluster.org/16015
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/dht: Check for null inode

2017-01-02T13:04:39+00:00

Check for NULL inode before attempting to
set dht inode ctx.

> Change-Id: I7693c18445f138221d8417df5e95b118cedb818a
> BUG: 1395261
> Signed-off-by: N Balachandran 
> Reviewed-on: http://review.gluster.org/15847
> Smoke: Gluster Build System 
> Reviewed-by: Shyamsundar Ranganathan 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
(cherry picked from commit 8313d53accaa22feb14d284fb91245be0a32e16e)

Change-Id: I7607d32d38d707dd5d71b98efffd1a458ffe90d7
BUG: 1395510
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/15850
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Niels de Vos

dht/rebalance: reverify lookup failures

2016-12-28T05:54:41+00:00

race: readdirp has read one entry, and doing a lookup on
that entry, but user might have renamed/removed that entry just
after readdirp but before lookup.

Since remove-brick is a costly opertaion,will ingore any
ENOENT/ESTALE failures and move on.

> Change-Id: I62c7fa93c0b9b7e764065ad1574b97acf51b5996
> BUG: 1408115
> Signed-off-by: Susant Palai 
> Reviewed-on: http://review.gluster.org/15846
> Reviewed-by: Raghavendra G 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 

Change-Id: I62c7fa93c0b9b7e764065ad1574b97acf51b5996
BUG: 1408414
Reviewed-on: http://review.gluster.org/15846
Reviewed-by: Raghavendra G 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
(cherry picked from commit c20febcb1ffaef3fa29563987e7a3b554aea27b3)
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/16278

cluster/dht: A hard link is lost during rebalance + lookup

2016-12-15T05:32:04+00:00

Problem: A hard link is lost during rebalance + lookup.Rebalance skip
         files if file has hardlink.In dht_migrate_file
         __is_file_migratable () function checks if a file has hardlink,
         if yes file is not migrated but if link is created after call
         this function then link will lost.

Solution: Call __check_file_has_hardlink to check hardlink existence
          after (S+T) bits in migration process ,if file has hardlink
          then skip the file for migrate rebalance process.

> BUG: 1396048
> Change-Id: Ia53c07ef42f1128c2eedf959a757e8df517b9d12
> Signed-off-by: Mohit Agrawal 
> Reviewed-on: http://review.gluster.org/15866
> Reviewed-by: Susant Palai 
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: N Balachandran 
> Reviewed-by: Raghavendra G 
> (cherry picked from commit 71dd2e914d4a537bf74e1ec3a24512fc83bacb1d)

BUG: 1399432
Change-Id: I30e21efd5a054d8a3e640ab3ed8aa7955d083926
Signed-off-by: Mohit Agrawal 
Reviewed-on: http://review.gluster.org/15954
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Raghavendra G

dht: Proper log message if data migration is skipped

2016-11-03T05:33:02+00:00

There was a misleading message from logs about 
available disk space while rebalancing of bricks
while calculating free space.

Bug: 1390870

Backprt of http://review.gluster.org/#/c/15345/

Change-Id: Ie9df0b2cbf00faaf13a0a3f0dbd657377a082755
Signed-off-by: ankitraj 
Reviewed-on: http://review.gluster.org/15765
Tested-by: ankitraj
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

dht: udpate stbuf from servers those have layout

2016-09-29T09:37:15+00:00

Problem: For healing of uid/gid we check if local->stbuf.ia_ctime is
lesser than stbuf->ia_ctime (received from brick). If yes then uid/gid
is updated to local->prebuf(source of healing).

But we merge local->stbuf also form the newly added brick. So if we
receive response from the newly added brick first and update the
local->stbuf, then local->prebuf will remain empty since the newly added
brick will have the latest ctime among all servers. And this can result
in healing wrong uid/gids to the rest of servers.

Hence, we should update local->stbuf from servers with a layout which
will ignore merging stbufs from newly added bricks.

> Reviewed-on: http://review.gluster.org/15126
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Raghavendra G 
(cherry picked from commit 36af81ac7cb2d459f9bfc0c436f0038a68f85235)

Change-Id: If4b64f75a0ea669abdbe9f5a3d1d18ff19374c2f
BUG: 1375096
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/15464
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

dht: "replica.split-brain-status" attribute value is not correct

2016-09-27T06:49:45+00:00

Problem: In a distributed-replicate volume attribute
         "replica.split-brain-status" value does not display split-brain
           condition though directory is in split-brain.
         If directory is in split brain on mutiple replica-pairs
         it does not show full list of replica pairs.

Solution: Update the dht_aggregate code to aggregate the xattr
          value in this specific condition.

Fix:      1) function getChoices returns the choices from split-brain
             status string.
          2) function add_opt adding the choices to local buffer to
             store in dictionary
          3) For the key "replica.split-brain-status" function dht_aggregate
             call dht_aggregate_split_brain_xattr to prepare the list.

Test:     To verify the patch followed below steps
          1) Create a distributed replica volume and create mount point
          2) Stop heal daemon
          3) Touch file and directories on mount point
             mkdir test{1..5};touch tmp{1..5}
          4) Down brick process on one of the replica set
             pkill -9 glusterfsd
          5) Change permission of dir on mount point
             chmod 755 test{1..5}
          6) Restart brick process on node with force option
          7) kill brick process on other node in same replica set
          8) Change permission of dir again on mount point
             chmod 766 test{1..5}
          9) Reexecute same step from 4-9 on other replica set also
          10) After check heal status on server it will show dir's are
              in split brain on all replica sets
          11) After check the replica.split-brain-status attr on mount
              point it will show wrong status of split brain.
          12) After apply the patch the attribute shows correct value.

> Change-Id: Icdfd72005a4aa82337c342762775a3d1761bbe4a
> Signed-off-by: Mohit Agrawal 
> Reviewed-on: http://review.gluster.org/15201
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 
> (cherry picked from commit c4e9ec653c946002ab6d4c71ee8e6df056438a04)

Change-Id: I85a5ae60189066d9e80799f00f1352c2f33ef4f8
Backport of commit c4e9ec653c946002ab6d4c71ee8e6df056438a04
BUG: 1375098
Signed-off-by: Mohit Agrawal 
Reviewed-on: http://review.gluster.org/15467
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G