performance/write-behind: retry "failed syncs to backend"

1. When sync fails, the cached-write is still preserved unless there
   is a flush/fsync waiting on it.

2. When a sync fails and there is a flush/fsync waiting on the
   cached-write, the cache is thrown away and no further retries will
   be made. In other words flush/fsync act as barriers for all the
   previous writes. The behaviour of fsync acting as a barrier is
   controlled by an option (see below for details). All previous
   writes are either successfully synced to backend or forgotten in
   case of an error. Without such barrier fop (especially flush which
   is issued prior to a close), we end up retrying for ever even after
   fd is closed.

3. If a fop is waiting on cached-write and syncing to backend fails,
   the waiting fop is failed.

4. sync failures when no fop is waiting are ignored and are not
   propagated to application. For eg.,
   a. first attempt of sync of a cached-write w1 fails
   b. second attempt of sync of w1 succeeds

   If there are no fops dependent on w1 are issued b/w a and b,
   application won't know about failure encountered in a.

5. The effect of repeated sync failures is that, there will be no
   cache for future writes and they cannot be written behind.

fsync as a barrier and resync of cached writes post fsync failure:
==================================================================
Whether to keep retrying failed syncs post fsync is controlled by an
option "resync-failed-syncs-after-fsync". By default, this option is
set to "off".

If sync of "cached-writes issued before fsync" (to backend) fails,
this option configures whether to retry syncing them after fsync or
forget them. If set to on, cached-writes are retried till a "flush"
fop (or a successful sync) on sync failures. fsync itself is failed
irrespective of the value of this option, when there is a sync failure
of any cached-writes issued before fsync.

Change-Id: I6097c0257bfb9ee5b1f616fbe6a0576ae9af369a
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
BUG: 1293534
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/13057
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>

1 files changed, 14 insertions, 0 deletions

diff --git a/xlators/mgmt/glusterd/src/glusterd-volume-set.c b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
index 7a81fad13ae..39db2297592 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volume-set.c
+++ b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
@@ -1282,6 +1282,20 @@ struct volopt_map_entry glusterd_volopt_map[] = {
           .op_version = 1,
           .flags      = OPT_FLAG_CLIENT_OPT
         },
+        { .key        = "performance.resync-failed-syncs-after-fsync",
+          .voltype    = "performance/write-behind",
+          .option     = "resync-failed-syncs-after-fsync",
+          .op_version = GD_OP_VERSION_3_7_7,
+          .flags      = OPT_FLAG_CLIENT_OPT,
+          .description = "If sync of \"cached-writes issued before fsync\" "
+                         "(to backend) fails, this option configures whether "
+                         "to retry syncing them after fsync or forget them. "
+                         "If set to on, cached-writes are retried "
+                         "till a \"flush\" fop (or a successful sync) on sync "
+                         "failures. "
+                         "fsync itself is failed irrespective of the value of "
+                         "this option. ",
+        },
         { .key        = "performance.nfs.write-behind-window-size",
           .voltype    = "performance/write-behind",
           .option     = "cache-size",