glusterd: On gaining quorum spawn_daemons in new thread

During startup, if a glusterd has peers, it waits till quorum is
obtained to spawn bricks and other services. If peers are not present,
the daemons are started during glusterd' startup itself.

The spawning of daemons as a quorum action was done without using a
seperate thread, unlike the spawn on startup. Since, quotad was launched
using the blocking runner_run api, this leads to the thread being
blocked. The calling thread is almost always the epoll thread and this
leads to a deadlock. The runner_run call blocks the epoll thread waiting
for quotad to start, as a result glusterd cannot serve any requests. But
the startup of quotad is blocked as it cannot fetch the volfile from
glusterd.

The fix for this is to launch the spawn daemons task in a seperate
thread. This will free up the epoll thread and prevents the above
deadlock from happening.

Change-Id: Ife47b3591223cdfdfb2b4ea8dcd73e63f18e8749
BUG: 1095585
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/7703
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: Krishnan Parthasarathi <kparthas@redhat.com>

1 files changed, 10 insertions, 1 deletions

diff --git a/xlators/mgmt/glusterd/src/glusterd-sm.c b/xlators/mgmt/glusterd/src/glusterd-sm.c
index 6dc35109d85..b14b7d729d0 100644
--- a/xlators/mgmt/glusterd/src/glusterd-sm.c
+++ b/xlators/mgmt/glusterd/src/glusterd-sm.c
@@ -1039,6 +1039,13 @@ glusterd_friend_sm ()
         gf_boolean_t                     is_await_conn = _gf_false;
         gf_boolean_t                     quorum_action = _gf_false;
         glusterd_friend_sm_state_t       old_state = GD_FRIEND_STATE_DEFAULT;
+        xlator_t                        *this = NULL;
+        glusterd_conf_t                 *priv = NULL;
+
+        this = THIS;
+        GF_ASSERT (this);
+        priv = this->private;
+        GF_ASSERT (priv);
 
         while (!list_empty (&gd_friend_sm_queue)) {
                 list_for_each_entry_safe (event, tmp, &gd_friend_sm_queue, list) {
@@ -1137,7 +1144,9 @@ out:
              * the functions spawn process(es) only if they are not started yet.
              *
              * */
-                glusterd_spawn_daemons (NULL);
+                synclock_unlock (&priv->big_lock);
+                glusterd_launch_synctask (glusterd_spawn_daemons, NULL);
+                synclock_lock (&priv->big_lock);
                 glusterd_do_quorum_action ();
         }
         return ret;