geo-rep / gsyncd: recognize ECONNABORTED as termination of aux glusterfs

Don't dump stack, rather log the "glusterfs session went down" message.

If the aux glusterfs is already dead when we try to do some file
operation, we get a failure with ENOTCONN, which is already handled
as above. However, it's also possible that glusterfs dies while we
are in a syscall into it -- in that case we get ECONNABORTED, and
so far then we end up with an ugly stack strace. From now on we
take ECONNABORTAD as well into consideration.

Nb. wrt. testing: it's not easy to synthetically force the aux glusterfs
to end this way; for that we have to provoke gsyncd into intensive
synchronization. I succeeded in that with the following ruby oneliner:

ruby -rcgi -e '
  Dir.chdir($*[0])
  a=[]
  Thread.new { loop { while a.size >= 100; File.delete a.shift; end; sleep 1 }}
  loop { a<<CGI.escape(STDIN.read 10); open(a[-1], "w") {}}' MTPT < /dev/urandom

where the geo-rep master is mounted at MTPT. With this going on, deliver a
SIGKILL to the geo-rep session's aux glusterfs. (It is giving ECONNABORTED
non-deterministically, actually in the minority of cases.)

Change-Id: I24fd8d0295cdba91d8b994057a1255ca8e2d1a67
BUG: 764510
Signed-off-by: Csaba Henk <csaba@redhat.com>
Reviewed-on: http://review.gluster.com/3078
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Vijay Bellur <vijay@gluster.com>

1 files changed, 3 insertions, 3 deletions

diff --git a/xlators/features/marker/utils/syncdaemon/syncdutils.py b/xlators/features/marker/utils/syncdaemon/syncdutils.py
index 11c2063b750..1a7dbd4dbbe 100644
--- a/xlators/features/marker/utils/syncdaemon/syncdutils.py
+++ b/xlators/features/marker/utils/syncdaemon/syncdutils.py
@@ -6,7 +6,7 @@ import fcntl
 import shutil
 import logging
 from threading import Lock, Thread as baseThread
-from errno import EACCES, EAGAIN, EPIPE, ENOTCONN, EINTR
+from errno import EACCES, EAGAIN, EPIPE, ENOTCONN, ECONNABORTED, EINTR, errorcode
 from signal import SIGTERM, SIGKILL
 from time import sleep
 import select as oselect
@@ -174,8 +174,8 @@ def log_raise_exception(excont):
             if hasattr(gconf, 'transport'):
                 gconf.transport.wait()
                 gconf.transport.terminate_geterr()
-        elif isinstance(exc, OSError) and exc.errno == ENOTCONN:
-            logging.error('glusterfs session went down')
+        elif isinstance(exc, OSError) and exc.errno in (ENOTCONN, ECONNABORTED):
+            logging.error('glusterfs session went down [%s]', errorcode[exc.errno])
         else:
             logtag = "FAIL"
         if not logtag and logging.getLogger().isEnabledFor(logging.DEBUG):