rpc,glusterd: Use rpc_clnt notifyfn to cleanup mydata

rpc:
- On a RPC_TRANSPORT_CLEANUP event, rpc_clnt_notify calls the registered
  notifyfn with a RPC_CLNT_DESTROY event. The notifyfn should properly
  cleanup the saved mydata on this event.
- Break the reconnect chain when an rpc client is disabled. This will
  prevent new disconnect events which can lead to crashes.

glusterd:
- Added support for RPC_CLNT_DESTROY in glusterd_brick_rpc_notify
- Use a common glusterd_rpc_clnt_unref() function throught glusterd in
  place of rpc_clnt_unref(). This function correctly gives up the
  big-lock before performing the unref.

Change-Id: I93230441c5089039643fc9f5632477ef1b695348
BUG: 962619
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/5512
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>

2 files changed, 10 insertions, 2 deletions

diff --git a/rpc/rpc-lib/src/rpc-clnt.c b/rpc/rpc-lib/src/rpc-clnt.c
index ac98a5c9110..1e9f307be40 100644
--- a/rpc/rpc-lib/src/rpc-clnt.c
+++ b/rpc/rpc-lib/src/rpc-clnt.c
@@ -416,7 +416,7 @@ rpc_clnt_reconnect (void *trans_ptr)
                                               conn->reconnect);
                 conn->reconnect = 0;
 
-                if (conn->connected == 0) {
+                if ((conn->connected == 0) && !clnt->disabled) {
                         ts.tv_sec = 3;
                         ts.tv_nsec = 0;
 
@@ -834,6 +834,7 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
         rpc_request_info_t     *req_info    = NULL;
         rpc_transport_pollin_t *pollin      = NULL;
         struct timespec         ts          = {0, };
+        void                   *clnt_mydata = NULL;
 
         conn = mydata;
         if (conn == NULL) {
@@ -870,6 +871,12 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
         }
 
         case RPC_TRANSPORT_CLEANUP:
+                if (clnt->notifyfn) {
+                        clnt_mydata = clnt->mydata;
+                        clnt->mydata = NULL;
+                        ret = clnt->notifyfn (clnt, clnt_mydata,
+                                              RPC_CLNT_DESTROY, NULL);
+                }
                 rpc_clnt_destroy (clnt);
                 ret = 0;
                 break;
diff --git a/rpc/rpc-lib/src/rpc-clnt.h b/rpc/rpc-lib/src/rpc-clnt.h
index 584963ad036..2596c3508c2 100644
--- a/rpc/rpc-lib/src/rpc-clnt.h
+++ b/rpc/rpc-lib/src/rpc-clnt.h
@@ -19,7 +19,8 @@
 typedef enum {
         RPC_CLNT_CONNECT,
         RPC_CLNT_DISCONNECT,
-        RPC_CLNT_MSG
+        RPC_CLNT_MSG,
+        RPC_CLNT_DESTROY
 } rpc_clnt_event_t;