epoll: Fix broken RPC throttling due to MT epoll

The RPC throttle which kicks in by setting the poll-in event on a
socket to false, is broken with the MT epoll commit. This is due
to the event handler of poll-in attempting to read as much out of
the socket till it receives an EAGAIN. Which may never happen and
hence we would be processing far more RPCs that we want to.

This is being fixed by changing the epoll from ET to LT, and
reading request by request, so that we honor the throttle.

The downside is that we do not drain the socket, but go back to
epoll_wait before reading the next request, but when kicking in
throttle, we need to anyway and so a busy connection would degrade
to LT anyway to maintain the throttle. As a result this change
should not cause deviation in the performance much for busy
connections.

Change-Id: I522d284d2d0f40e1812ab4c1a453c8aec666464c
BUG: 1192114
Signed-off-by: Shyam <srangana@redhat.com>
Reviewed-on: http://review.gluster.org/9726
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Tested-by: Raghavendra G <rgowdapp@redhat.com>

1 files changed, 3 insertions, 11 deletions

diff --git a/rpc/rpc-transport/socket/src/socket.c b/rpc/rpc-transport/socket/src/socket.c
index a7e2bb0cfdf..2b61eb417d2 100644
--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -2263,16 +2263,9 @@ socket_event_poll_in (rpc_transport_t *this)
         rpc_transport_pollin_t *pollin = NULL;
         socket_private_t       *priv = this->private;
 
-	do {
-		/* consume all we can, this is our only chance
-		   (Edge Triggered polling in epoll)
-		*/
-		pollin = NULL;
-		ret = socket_proto_state_machine (this, &pollin);
-
-		if (!pollin)
-			break;
+	ret = socket_proto_state_machine (this, &pollin);
 
+	if (pollin) {
                 priv->ot_state = OT_CALLBACK;
                 ret = rpc_transport_notify (this, RPC_TRANSPORT_MSG_RECEIVED,
                                             pollin);
@@ -2280,8 +2273,7 @@ socket_event_poll_in (rpc_transport_t *this)
                         priv->ot_state = OT_RUNNING;
                 }
                 rpc_transport_pollin_destroy (pollin);
-
-        } while (pollin);
+        }
 
         return ret;
 }