protocol/client: Fix double free of client fdctx destroy

This patch fixes the race between fd re-open code and fd release code,
both of which free the fd context due to a race in certain variable
checks as explained below:

1. client process (shd in the case of this BZ) sends an opendir to its
children (client xlators) which send the fop to the bricks to get a valid fd.

2. Client xlator loses connection to the brick. fdctx->remotefd is -1

3. Client re-establishes connection. After handshake, it reopens the dir
and sets fdctx->remotefd to a valid fd in client3_3_reopendir_cbk().

4. Meanwhile, shd sends a fd unref after it is done with the opendir.
This triggers a releasedir (since fd->refcount becomes 0).

5. client3_3_releasedir() sees that fdctx-->remotefd is a valid number
(i.e not -1), sets fdctx->released=1 and calls  client_fdctx_destroy()

6. As a continuation of step3, client_reopen_done() is called by
client3_3_reopendir_cbk(), which sees that fdctx->released==1 and
again calls client_fdctx_destroy().

Depending on when step-5 does GF_FREE(fdctx), we may crash at any place in
step-6 in client3_3_reopendir_cbk() when it tries to access
fdctx->{whatever}.

> Reviewed-on: https://review.gluster.org/16521
> CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
> Smoke: Gluster Build System <jenkins@build.gluster.org>
> Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
(cherry picked from commit 25fc74f9d1f2b1e7bab76485a99f27abadd10b7b)

Change-Id: Ia50873d11763e084e41d2a1f4d53715438e5e947
BUG: 1422352
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-on: https://review.gluster.org/16621
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>

1 files changed, 3 insertions, 2 deletions

diff --git a/xlators/protocol/client/src/client.h b/xlators/protocol/client/src/client.h
index 46a7875682e..4e278ae95be 100644
--- a/xlators/protocol/client/src/client.h
+++ b/xlators/protocol/client/src/client.h
@@ -195,7 +195,7 @@ typedef struct _client_fd_ctx {
         pthread_mutex_t   mutex;
         lk_heal_state_t   lk_heal_state;
         uuid_t            gfid;
-        void (*reopen_done) (struct _client_fd_ctx*, xlator_t *);
+        void (*reopen_done)(struct _client_fd_ctx*, int64_t rfd, xlator_t *);
         struct list_head  lock_list;     /* List of all granted locks on this fd */
         int32_t           reopen_attempts;
 } clnt_fd_ctx_t;
@@ -325,7 +325,8 @@ int client_mark_fd_bad (xlator_t *this);
 int client_set_lk_version (xlator_t *this);
 
 int client_fd_lk_list_empty (fd_lk_ctx_t *lk_ctx, gf_boolean_t use_try_lock);
-void client_default_reopen_done (clnt_fd_ctx_t *fdctx, xlator_t *this);
+void client_default_reopen_done (clnt_fd_ctx_t *fdctx, int64_t rfd,
+                                 xlator_t *this);
 void client_attempt_reopen (fd_t *fd, xlator_t *this);
 int client_get_remote_fd (xlator_t *this, fd_t *fd, int flags,
                           int64_t *remote_fd);