tier/dht : Attach tier fix layout to run in background

1. Spawn a thread for background fix-layout for tier process.

2. Once the fix-layout is completed a marker xttr is set on the root of
   volume to mark the completion of the background fixlayout, so that
   even if the tier process is spawned again, fixlayout will not be
   issued, if it was completed last time.

3. Please note that promotion of legacy files will happen eventually as
   the ctr lookup heal in the fixlayout slowly heals the ctr db for legacy
   files OR the ctr lookup heal happend due to a name lookup.

4. When a detach tier is successful in evacuation data from hot tier, we remove
   the marker xattr is removed. So that next attach tier runs the background
   tier fixlayout.

what is remaining ?
1. Instead of clearing the marker xattr of tiering fix layout at the end of detach start
   clear it during detach commit. But the issue is detach commit is a glusterd operation
   and the volume is not mounted in glusterd.
   The reason we want to do it in detach commit is that if the admin wants to attach the
   same tier again, then a background fixlayout will be triggered, which would not be needed.
2. Clearing the CTR DB of the cold bricks when there is a detach commit, as it will be having
   entries which will be stale when the volume is used, with ctr off (ctr is switched off only when
   we have detach commit.)

Change-Id: Ibe343572e95865325cd0eef4d0b976b626a3c0c5
BUG: 1313228
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/13491
Smoke: Gluster Build System <jenkins@build.gluster.com>
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>

3 files changed, 19 insertions, 5 deletions

diff --git a/tests/basic/tier/legacy-many.t b/tests/basic/tier/legacy-many.t
index 851f0a1b0cc..a9fcba85209 100644
--- a/tests/basic/tier/legacy-many.t
+++ b/tests/basic/tier/legacy-many.t
@@ -58,6 +58,11 @@ TEST $CLI volume set $V0 cluster.write-freq-threshold 0
 # wait a little for lookup heal to finish
 sleep 10
 
+# make sure fix layout completed
+CPATH=$B0/${V0}0
+echo $CPATH > /tmp/out
+TEST getfattr -n "trusted.tier.fix.layout.complete" $CPATH
+
 # Read "legacy" files
 drop_cache $M0
 
@@ -69,6 +74,14 @@ TEST read_all
 sleep $PROMOTE_TIMEOUT
 EXPECT_WITHIN $PROMOTE_TIMEOUT "0" check_counters $NUM_FILES 0
 
+EXPECT_WITHIN $PROCESS_UP_TIMEOUT "0" detach_start $V0
+EXPECT_WITHIN $REBALANCE_TIMEOUT "completed" remove_brick_status_completed_field "$V0 $H0:$B0/${V0}${CACHE_BRICK_FIRST}"
+
+TEST $CLI volume tier $V0 detach commit
+
+# fix layout flag should be cleared
+TEST ! getfattr -n "trusted.tier.fix.layout.complete" $CPATH
+
 cd;
 cleanup
 #G_TESTDEF_TEST_STATUS_NETBSD7=KNOWN_ISSUE,BUG=000000
diff --git a/tests/basic/tier/tier.t b/tests/basic/tier/tier.t
index deff2e299ef..7bac2dd03bb 100755
--- a/tests/basic/tier/tier.t
+++ b/tests/basic/tier/tier.t
@@ -13,11 +13,6 @@ MIGRATION_TIMEOUT=10
 DEMOTE_FREQ=4
 PROMOTE_FREQ=12
 
-function detach_start {
-        $CLI volume tier $1 detach start
-        echo $?;
-}
-
 function file_on_slow_tier {
     found=0
 
diff --git a/tests/tier.rc b/tests/tier.rc
index dd220fe09d2..ee37e0704fa 100644
--- a/tests/tier.rc
+++ b/tests/tier.rc
@@ -60,6 +60,12 @@ function check_counters {
 }
 
 
+function detach_start {
+        $CLI volume tier $1 detach start
+        echo $?;
+}
+
+
 # Grab md5sum without file path (failed attempt notifications are discarded)
 function fingerprint {
     md5sum $1 2> /dev/null | grep --only-matching -m 1 '^[0-9a-f]*'