glusterfs.git/xlators/cluster/dht/src/dht-common.h, branch v3.8.3

cluster/tier: dont promote if estimated block consumption > hi watermark

2016-08-05T08:12:11+00:00

Add test to fail promotion if estimated block consumption grows
beyond hi watermark.

Skip file migrations until next cycle if tier_get_fs_stat() fails
in tier_migrate_using_query_file()

> Reviewed-on: http://review.gluster.org/14780
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: mohammed rafi  kc 
> Reviewed-by: Dan Lambright 

(cherry picked from commit 1f4e41e8c2f5f4af4564caba0a08996853f089f4)

Change-Id: Ice04572fa739c09109c4433e65965197482a7beb
BUG: 1362198
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15065
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

dht:remember locked subvol and send unlock to the same

2016-06-20T11:56:16+00:00

During locking we send lock request to cached subvol,
and normally we unlock to the cached subvol
But with parallel fresh lookup on a directory, there
is a race window where the cached subvol can change
and the unlock can go into a different subvol from
which we took lock.

This will result in a stale lock held on one of the
subvol.

So we will store the details of subvol which we took the lock
and will unlock from the same subvol

Back port of>
>Change-Id: I47df99491671b10624eb37d1d17e40bacf0b15eb
>BUG: 1311002
>Signed-off-by: Mohammed Rafi KC 
>Reviewed-on: http://review.gluster.org/13492
>Reviewed-by: N Balachandran 
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Raghavendra G 
>CentOS-regression: Gluster Build System 

(cherry picked from commit ef0db52bc55a51fe5e3856235aed0230b6a188fe)

Change-Id: Ib821e7355b4937b86d2f9f11e2c8311b7301b6c7
BUG: 1347524
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/14750
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/dht: handle EREMOTE in dht lk/flush

2016-05-02T01:05:08+00:00

With lock-migration, we need to send requests to destination
brick post migration. Once, the source brick marks the lock
structure to be already migrated, the requests will be redirected
to destination brick by dht_lk2/flush2.

Change-Id: I50b14011c5ab68c34826fb7ba7f8c8d42a68ad97
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/13493
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd: volume set changes for lock migration

2016-05-02T01:05:03+00:00

Change-Id: I48c6f9cdda47503615ba65882acd5eedf0a70c89
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14024
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tier/migrator: Fetch the next query file for the next cycle

2016-04-30T14:30:49+00:00

Problem: When we spawn promote and demote thread, query files
are build. And only query file with index 0 is picked for migration
as the first query file. This may not be suitable for scenarios,
where the file in the query are too big to move in the first cycle,
as a result file in the other query files always get missed. We need to
shuffle so that other query files also get a chance.

Fix: Remember the previous first query file and shift it by one index,
before the migration starts.

Change-Id: I704947bcf4bab6b20b1179a6d9ae4a15a3d51bd9
BUG: 1330353
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/14068
Tested-by: Joseph Fernandes
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright 
NetBSD-regression: NetBSD Build System

tier/dht: check for rebalance completion for EIO error

2016-04-25T14:55:26+00:00

When an ongoing rebalance completion check task been
triggered by dht, there is a possibility of a race
between afr setting subvol as non-readable and dht updates
the cached subvol. In this window a write can fail with EIO.

Change-Id: I42638e6d4104c0dbe893d1bc73e1366188458c5d
BUG: 1329503
Signed-off-by: Mohammed Rafi KC 
Reviewed-on: http://review.gluster.org/14049
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Jeff Darcy

dht: Add lease() fop

2016-04-25T14:30:16+00:00

Change-Id: I0bbc2c2ef115c78393f6570815a5b80316e7e4be
BUG: 1319992
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/11720
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Talur 
Reviewed-by: Rajesh Joseph 
Reviewed-by: Jeff Darcy

cluster/distribute: detect stale layouts in entry fops

2016-04-22T17:28:54+00:00

dht_mkdir ()
{
      first-hashed-subvol = hashed-subvol for "bname" in in-memory
                            layout of "parent";
      inodelk (SETLKW, parent, "LAYOUT_HEAL_DOMAIN", "can be any
               subvol, but we choose first-hashed-subvol randomly");
      {
begin:
            hashed-subvol = hashed-subvol for "bname" in in-memory
                            layout of "parent";
            hash-range = extract hashe-range from layout of "parent";

            ret = mkdir (parent/bname, hashed-subvol, hash-range);
            if (ret == "hash-value doesn't fall into layout stored on
                       the brick (this error is returned by posix-mkdir)")
            {
                refresh_parent_layout ();
                goto begin;
            }

      }
      inodelk (UNLCK, parent, "LAYOUT_HEAL_DOMAIN",
               "first-hashed-subvol");

      proceed with other parts of dht_mkdir;
}

posix_mkdir (parent/bname, client-hash-range)
{

       disk-hash-range = getxattr (parent, "dht-layout-key");
       if (disk-hash-range != client-hash-range) {
              fail-with-error ("hash-value doesn't fall into layout
                                stored on the brick");
              return 0;
       }

       continue-with-posix-mkdir;
}

Similar changes need to be done for dentry operations like create,
symlink, link, unlink, rmdir, rename. These will be addressed in
subsequent patches. This patch addresses only mkdir codepath.

This change breaks stripe tests, as on some striped subvols dht layout
xattrs are not set for some reason. This results in failure of
mkdir. Since striped volumes are always created with dht, some tests
associated with stripe also fail. So, I am making following tests
changes (since stripe is out of maintainance):
* modify ./tests/basic/rpc-coverage.t to not to use striped volumes
* mark all (2) tests in tests/bugs/stripe/ as bad tests

Change-Id: Idd1ae879f24a48303dc743c1bb4d91f89a629e25
BUG: 1323040
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/13885
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: N Balachandran

dht: lock on subvols to prevent lookup vs rmdir race

2016-04-06T05:06:02+00:00

There is a possibility that while an rmdir is completed on
some non-hashed subvol and proceeding to others, a lookup
selfheal can recreate the same directory on those subvols
for which the rmdir had succeeded. Now the deletion of the
parent directory will fail with an ENOTEMPTY.

To fix this take blocking inodelk on the subvols before
starting rmdir. Selfheal must also take blocking inodelk
before creating the entry.

Change-Id: I168a195c35ac1230ba7124d3b0ca157755b3df96
BUG: 1245065
Signed-off-by: Sakshi 
Reviewed-on: http://review.gluster.org/13528
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

tier/dht : Attach tier fix layout to run in background

2016-03-28T16:17:59+00:00

1. Spawn a thread for background fix-layout for tier process.

2. Once the fix-layout is completed a marker xttr is set on the root of
volume to mark the completion of the background fixlayout, so that
even if the tier process is spawned again, fixlayout will not be
issued, if it was completed last time.

3. Please note that promotion of legacy files will happen eventually as
the ctr lookup heal in the fixlayout slowly heals the ctr db for legacy
files OR the ctr lookup heal happend due to a name lookup.

4. When a detach tier is successful in evacuation data from hot tier, we remove
the marker xattr is removed. So that next attach tier runs the background
tier fixlayout.

what is remaining ?
1. Instead of clearing the marker xattr of tiering fix layout at the end of detach start
clear it during detach commit. But the issue is detach commit is a glusterd operation
and the volume is not mounted in glusterd.
The reason we want to do it in detach commit is that if the admin wants to attach the
same tier again, then a background fixlayout will be triggered, which would not be needed.
2. Clearing the CTR DB of the cold bricks when there is a detach commit, as it will be having
entries which will be stale when the volume is used, with ctr off (ctr is switched off only when
we have detach commit.)

Change-Id: Ibe343572e95865325cd0eef4d0b976b626a3c0c5
BUG: 1313228
Signed-off-by: Joseph Fernandes
Reviewed-on: http://review.gluster.org/13491
Smoke: Gluster Build System
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System
CentOS-regression: Gluster Build System
Reviewed-by: Dan Lambright