Add option to toggle x-halo fail-over

Summary:
- Adds "halo-failover-enabled" option to enable/disable failing over to a brick outside of the defined halo to satisfy min-replicas
- There are some use-cases where failing over to a brick which is out of region will be undesirable.  I such cases we will more than likely opt to have more replicas within the region to tolerate the loss of a single replica in that region without losing quorum.
- Fixed quorum accounting problem as well, now correctly goes RO in case where we lose a brick and aren't able to swap one in for some reason (fail-over not enabled or otherwise)

Test Plan:
- run prove -v tests/basic/halo.t
- run prove -v tests/basic/halo-disable.t
- run prove -v tests/basic/halo-failover-enabled.t
- run prove -v tests/basic/halo-failover-disabled.t

Reviewers: dph, cjh, jackl, mmckeen

Reviewed By: mmckeen

Conflicts:
	xlators/cluster/afr/src/afr.h
	xlators/mount/fuse/utils/mount.glusterfs.in

Change-Id: Ia3ebf83f34b53118ca4491a3c4b66a178cc9795e
Signed-off-by: Kevin Vigor <kvigor@fb.com>
Reviewed-on: http://review.gluster.org/16275
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Shreyas Siravara <sshreyas@fb.com>
Smoke: Gluster Build System <jenkins@build.gluster.org>

1 files changed, 7 insertions, 0 deletions

diff --git a/xlators/mount/fuse/utils/mount.glusterfs.in b/xlators/mount/fuse/utils/mount.glusterfs.in
index b5c90ba1dff..598f62fee7a 100755
--- a/xlators/mount/fuse/utils/mount.glusterfs.in
+++ b/xlators/mount/fuse/utils/mount.glusterfs.in
@@ -186,6 +186,10 @@ start_glusterfs ()
     fi
 
 #options with values start here
+    if [ -n "$halo_failover_enabled" ]; then
+      cmd_line=$(echo "$cmd_line --xlator-option \
+ *replicate*.halo-failover-enabled=$halo_failover_enabled");
+    fi
     if [ -n "$halo_max_latency" ]; then
       cmd_line=$(echo "$cmd_line --xlator-option \
  *replicate*.halo-max-latency=$halo_max_latency");
@@ -503,6 +507,9 @@ with_options()
         "halo-min-replicas")
           halo_min_replicas=$value
           ;;
+        "halo-failover-enabled")
+          halo_failover_enabled=$value
+          ;;
         x-*)
             # comments or userspace application-specific options, drop them
             ;;