protocol/server: Remove connection from conf->conns w.o. race

1) Adding the connection to conf->conns used to
happen in conf->mutex, but removing happened under conn->lock.
Fixed that as below.
When the connection object is created conn's ref, bind_ref count
is set to '1'. For bind_ref ref/unref happens under conf->mutex
whenever server_connection_get, put is called.
When bind_ref goes to '0' connection object is removed from
conf->conns under conf->mutex.  After it is removed from the list,
conn_unref is called outside the conf->mutex.
conn_ref/unref still happens under conn->lock.

2) Fixed races in server_connection_cleaup in grace_timer_handler
and server_setvolume.

Change-Id: Ie7b63b10f658af909a11c3327066667f5b7bd114
BUG: 801675
Signed-off-by: Pranith Kumar K <pranithk@gluster.com>
Reviewed-on: http://review.gluster.com/2911
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Vijay Bellur <vijay@gluster.com>

1 files changed, 7 insertions, 2 deletions

diff --git a/xlators/protocol/server/src/server.c b/xlators/protocol/server/src/server.c
index 7ac8590a42e..b75ad172e6f 100644
--- a/xlators/protocol/server/src/server.c
+++ b/xlators/protocol/server/src/server.c
@@ -42,6 +42,7 @@ grace_time_handler (void *data)
         server_connection_t     *conn = NULL;
         xlator_t                *this = NULL;
         gf_boolean_t            cancelled = _gf_false;
+        gf_boolean_t            detached = _gf_false;
 
         conn = data;
         this = conn->this;
@@ -49,11 +50,15 @@ grace_time_handler (void *data)
         GF_VALIDATE_OR_GOTO (THIS->name, conn, out);
         GF_VALIDATE_OR_GOTO (THIS->name, this, out);
 
-        gf_log (this->name, GF_LOG_INFO, "grace timer expired");
+        gf_log (this->name, GF_LOG_INFO, "grace timer expired for %s", conn->id);
 
         cancelled = server_cancel_conn_timer (this, conn);
         if (cancelled) {
-                server_connection_cleanup (this, conn);
+                //conn should not be destroyed in conn_put, so take a ref.
+                server_conn_ref (conn);
+                server_connection_put (this, conn, &detached);
+                if (detached)//reconnection did not happen :-(
+                        server_connection_cleanup (this, conn);
                 server_conn_unref (conn);
         }
 out: