gluster: Sometimes Brick process is crashed at the time of stopping brick

Problem: Sometimes brick process is getting crashed at the time
         of stop brick while brick mux is enabled.

Solution: Brick process was getting crashed because of rpc connection
          was not cleaning properly while brick mux is enabled.In this patch
          after sending GF_EVENT_CLEANUP notification to xlator(server)
          waits for all rpc client connection destroy for specific xlator.Once rpc
          connections are destroyed in server_rpc_notify for all associated client
          for that brick then call xlator_mem_cleanup for for brick xlator as well as
          all child xlators.To avoid races at the time of cleanup introduce
          two new flags at each xlator cleanup_starting, call_cleanup.

BUG: 1544090
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>

Note: Run all test-cases in separate build (https://review.gluster.org/#/c/19700/)
      with same patch after enable brick mux forcefully, all test cases are
      passed.

Change-Id: Ic4ab9c128df282d146cf1135640281fcb31997bf
updates: bz#1544090

2 files changed, 14 insertions, 0 deletions

diff --git a/xlators/features/changelog/src/changelog-rpc.c b/xlators/features/changelog/src/changelog-rpc.c
index 4b2b38cad51..852c0694f9a 100644
--- a/xlators/features/changelog/src/changelog-rpc.c
+++ b/xlators/features/changelog/src/changelog-rpc.c
@@ -258,6 +258,15 @@ changelog_handle_probe (rpcsvc_request_t *req)
         changelog_probe_req   rpc_req = {0,};
         changelog_probe_rsp   rpc_rsp = {0,};
 
+
+        this = req->trans->xl;
+        if (this->cleanup_starting) {
+                gf_msg (this->name, GF_LOG_DEBUG, 0,
+                        CHANGELOG_MSG_HANDLE_PROBE_ERROR,
+                        "cleanup_starting flag is already set for xl");
+                return 0;
+        }
+
         ret = xdr_to_generic (req->msg[0],
                               &rpc_req, (xdrproc_t)xdr_changelog_probe_req);
         if (ret < 0) {
diff --git a/xlators/features/changelog/src/changelog.c b/xlators/features/changelog/src/changelog.c
index 19d66b605bb..12997181da4 100644
--- a/xlators/features/changelog/src/changelog.c
+++ b/xlators/features/changelog/src/changelog.c
@@ -2894,6 +2894,7 @@ void
 fini (xlator_t *this)
 {
         changelog_priv_t *priv = NULL;
+        struct list_head  queue = {0, };
 
         priv = this->private;
 
@@ -2901,6 +2902,10 @@ fini (xlator_t *this)
                 /* terminate RPC server/threads */
                 changelog_cleanup_rpc (this, priv);
 
+                /* call barrier_disable to cancel timer */
+                if (priv->barrier_enabled)
+                        __chlog_barrier_disable (this, &queue);
+
                 /* cleanup barrier related objects */
                 changelog_barrier_pthread_destroy (priv);