io-threads: Change mutexes/condvars to spinlocks/semaphores

It seems that use of mutexes is resulting in pretty high thread
sleep and wake-up cost. What is worse, if a worker thread has
acquired a  lock, there is a possibility of the main glusterfs thread
being put to sleep. We change the use of mutexes into spinlock.
At the same time, we cannot anymore use condvars for notification since
the condvar interface depends on mutexes itself. Semaphores come to
out rescue. Luckily, even the pthread semaphores have a timedwait
interface to allow our idle worker threads to make an exit decision.

Further, it is possible that spinlocks are not available on all systems
so all this is curtained behind #defines so we can fall back to
mutexes and condvars implementation.

Signed-off-by: Anand V. Avati <avati@dev.gluster.com>

1 files changed, 8 insertions, 2 deletions

diff --git a/xlators/performance/io-threads/src/io-threads.h b/xlators/performance/io-threads/src/io-threads.h
index c5ca760000a..ec17356942b 100644
--- a/xlators/performance/io-threads/src/io-threads.h
+++ b/xlators/performance/io-threads/src/io-threads.h
@@ -34,6 +34,8 @@
 #include "common-utils.h"
 #include "list.h"
 #include <stdlib.h>
+#include "locking.h"
+#include <semaphore.h>
 
 #define min(a,b) ((a)<(b)?(a):(b))
 #define max(a,b) ((a)>(b)?(a):(b))
@@ -88,9 +90,13 @@ typedef enum {
 struct iot_worker {
         struct list_head rqlist;      /* List of requests assigned to me. */
         struct iot_conf  *conf;
+#ifndef HAVE_SPINLOCK
+        pthread_cond_t   notifier;
+#else
+        sem_t            notifier;
+#endif
         int64_t          q,dq;
-        pthread_cond_t   dq_cond;
-        pthread_mutex_t  qlock;
+        gf_lock_t        qlock;
         int32_t          queue_size;
         pthread_t        thread;
         iot_state_t      state;            /* What state is the thread in. */