glusterfs.git/xlators/protocol/server/src, branch v3.8.13

rpcsvc: Add rpchdr and proghdr to iobref before submitting to transport

2017-04-07T12:05:09+00:00

Backport of https://review.gluster.org/16613

Issue:
When fio is run on multiple clients (each client writes to its own files),
and meanwhile the clients does a readdirp, thus the client which did
a readdirp will now recieve the upcalls. In this scenario the client
disconnects with rpc decode failed error.

RCA:
Upcall calls rpcsvc_request_submit to submit the request to socket:
rpcsvc_request_submit currently:
rpcsvc_request_submit () {
   iobuf = iobuf_new
   iov = iobuf->ptr
   fill iobuf to contain xdrised upcall content - proghdr
   rpcsvc_callback_submit (..iov..)
   ...
   if (iobuf)
       iobuf_unref (iobuf)
}

rpcsvc_callback_submit (... iov...) {
   ...
   iobuf = iobuf_new
   iov1 = iobuf->ptr
   fill iobuf to contain xdrised rpc header - rpchdr
   msg.rpchdr = iov1
   msg.proghdr = iov
   ...
   rpc_transport_submit_request (msg)
   ...
   if (iobuf)
       iobuf_unref (iobuf)
}

rpcsvc_callback_submit assumes that once rpc_transport_submit_request()
returns the msg is written on to socket and thus the buffers(rpchdr, proghdr)
can be freed, which is not the case. In especially high workload,
rpc_transport_submit_request() may not be able to write to socket immediately
and hence adds it to its own queue and returns as successful. Thus, we have
use after free, for rpchdr and proghdr. Hence the clients gets garbage rpchdr
and proghdr and thus fails to decode the rpc, resulting in disconnect.

To prevent this, we need to add the rpchdr and proghdr to a iobref and send
it in msg:
   iobref_add (iobref, iobufs)
   msg.iobref = iobref;
The socket layer takes a ref on msg.iobref, if it cannot write to socket and
is adding to the queue. Thus we do not have use after free.

Thank You for discussing, debugging and fixing along:
Prashanth Pai 
Raghavendra G 
Rajesh Joseph 
Kotresh HR 
Mohammed Rafi KC 
Soumya Koduri 

> Reviewed-on: https://review.gluster.org/16613
> Reviewed-by: Prashanth Pai 
> Smoke: Gluster Build System 
> Reviewed-by: soumya k 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 

Change-Id: Ifa6bf6f4879141f42b46830a37c1574b21b37275
BUG: 1422788
Signed-off-by: Poornima G 
Reviewed-on: https://review.gluster.org/16638
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Prashanth Pai 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

protocol/server: capture offset in seek

2016-11-29T08:46:45+00:00

Problem:
http://review.gluster.org/11482 implemented seek FOP but
http://review.gluster.org/#/c/14137/ 'undid' the change where we pack
the offset returned by seek in server xlator before sending it to the client.
As a result, seek always returns zero to the client for SEEK_HOLE/
SEEK_DATA.

Fix:
I think 14137 removed it unintentionally, hence adding it back again.


> Reviewed-on: http://review.gluster.org/15920
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit cc37e5929d1e3ea4eaf4c4576a82066bf131ad05)
Signed-off-by: Ravishankar N 
Change-Id: I67a1f7b53214b043c5291f5704be4a50b698f91c
BUG: 1399130
Reviewed-on: http://review.gluster.org/15943
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

protocol/server: readlink rsp xdr will fail while readlink got an error

2016-09-01T15:20:41+00:00

set gfs3_readlink_rsp.path with an empty string while error happen, to
make xdr_gfs3_readlink_rsp happy. otherwise the original errno will be
lost, and return an rpc internal errno instead.

Change-Id: I36655b66df8b9f164e5bd21eb17244722c2f5a52
BUG: 1370172
Signed-off-by: Ryan Ding 
Reviewed-on: http://review.gluster.org/15312
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

libglusterfs: Negate all but O_DIRECT flag if present on anon fds

2016-06-28T05:57:11+00:00

        Backport of: http://review.gluster.org/14665

This is to prevent any unforeseen problems that might arise due to
writevs and readvs being wound with @flag parameter containing
O_TRUNC or O_APPEND especially wrt translators like sharding and ec
where O_TRUNC write or O_APPEND write on individual shards/fragments
is not the same as O_TRUNC write or O_APPEND write as expected by the
application.

 >Change-Id: Ib0110731d6099bc888b7ada552a2abbea8e76051
 >BUG: 1342903
 >Signed-off-by: Krutika Dhananjay 
 >Reviewed-on: http://review.gluster.org/14735
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Pranith Kumar Karampuri 
 >CentOS-regression: Gluster Build System 

Change-Id: I7ffa4fa366f727f7e345ab0bf4c8eb009710074b
BUG: 1347553
Signed-off-by: Krutika Dhananjay 
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14755
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

core, shard: Make shards inherit main file's O_DIRECT flag if present

2016-06-27T15:17:55+00:00

        Backport of: http://review.gluster.org/14191

If the application opens a file with O_DIRECT, the shards'
anon fds would also need to inherit the flag. Towards this,
shard xl would be passing the odirect flag in the @flags parameter
to the WRITEV fop. This will be used in anon fd resolution
and subsequent opening by posix xl.

 >Change-Id: I3a0593fa46cc25e390a5762a0354b469c2a1532d
 >BUG: 1342903
 >Signed-off-by: Krutika Dhananjay 
 >Reviewed-on: http://review.gluster.org/14663
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Jeff Darcy 

Change-Id: Ibfc164aa7f9eecd6993255f1c03557f2ec35ac8c
BUG: 1347553
Signed-off-by: Krutika Dhananjay 
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14754
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

protocol: Add framework to send transaction id with recall

2016-06-10T11:03:54+00:00

Backport of: http://review.gluster.org/#/c/14647/

Issue:
The upcall(cache invalidation/recall) event is sent from the bricks
to clients. In AFR/EC setup, it can so happen that all the bricks
will send the upcall for the same event, and if AFR/EC doesn't filter
out these duplicate notifications, the logic above cluster xlators
can fail.

Solution:
Use transaction id to filter out duplicate notifications.

This patch adds framework for duplicate notifications.
AFR/EC can build up on this patch for deduping the notifications

Change-Id: I66b08e63b8799bc5932f2b2545376138a5701168
BUG: 1337638
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/14648
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

cluster/ec: Use correct log levels

2016-05-30T14:38:00+00:00

Problem :
Misleading messages are getting logged in mount logs
and bricks log.
"Mismatching xdata" and "Heal failed" are getting logged

Solution :
Reduce the level of logs from INFO, WARNING and NOTICE
to DEBUG level wherever applicable OR use fop_log_level
to get proper log level.

Backport of commit 02b2750ecc35f88c3262015b401dda962381f9da:
> Change-Id: Ia824c71e75ab683d3cb8949e1966ea09c9ccce72
> BUG: 1231224
> Signed-off-by: Ashish Pandey 
> Reviewed-on: http://review.gluster.org/13266
> Smoke: Gluster Build System 
> Reviewed-by: Xavier Hernandez 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Pranith Kumar Karampuri 

Change-Id: Ia824c71e75ab683d3cb8949e1966ea09c9ccce72
BUG: 1254934
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/14520
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Niels de Vos

core: Honour mandatory lock flags during lock migration

2016-05-27T14:50:37+00:00

lk_flags from posix_lock_t structure is the primary key used to
differentiate locks as either advisory and mandatory type. During
lock migration this field is not read in getactivelk() call path.
So in order to copy the exact lock state from source to destination
it is necessary to include lk_flags within lock_migration_info_t
structure to maintain accurate state. This change also includes
minor modifications to setactivelk() call to consider lk_flags
during lock migration.

> Reviewed-on: http://review.gluster.org/14189
> Smoke: Gluster Build System 
> Reviewed-by: Susant Palai 
> Reviewed-by: Poornima G 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit deaf8439fc42435988aae6a7b9ab681cc0d36b09)

Change-Id: I20a7b6b6a0f3bdac5734cce8a2cd2349eceff195
BUG: 1337805
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/14457
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

glusterd: add defence mechanism to avoid brick port clashes

2016-05-10T07:11:33+00:00

Intro:
Currently glusterd maintain the portmap registry which contains ports that
are free to use between 49152 - 65535, this registry is initialized
once, and updated accordingly as an then when glusterd sees they are been
used.

Glusterd first checks for a port within the portmap registry and gets a FREE
port marked in it, then checks if that port is currently free using a connect()
function then passes it to brick process which have to bind on it.

Problem:
We see that there is a time gap between glusterd checking the port with
connect() and brick process actually binding on it. In this time gap it could
be so possible that any process would have occupied this port because of which
brick will fail to bind and exit.

Case 1:
To avoid the gluster client process occupying the port supplied by glusterd :

we have separated the client port map range with brick port map range more @
http://review.gluster.org/#/c/13998/

Case 2: (Handled by this patch)
To avoid the other foreign process occupying the port supplied by glusterd :

To handle above situation this patch implements a mechanism to return EADDRINUSE
error code to glusterd, upon which a new port is allocated and try to restart
the brick process with the newly allocated port.

Note: Incase of glusterd restarts i.e. runner_run_nowait() there is no way to
handle Case 2, becuase runner_run_nowait() will not wait to get the return/exit
code of the executed command (brick process). Hence as of now in such case,
we cannot know with what error the brick has failed to connect.

This patch also fix the runner_end() to perform some cleanup w.r.t
return values.

Backport of:
> Change-Id: Iec52e7f5d87ce938d173f8ef16aa77fd573f2c5e
> BUG: 1322805
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/14043
> Tested-by: Prasanna Kumar Kalever 
> Reviewed-by: Atin Mukherjee 
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 

Change-Id: Id7d8351a0082b44310177e714edc0571ad0f7195
BUG: 1333711
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/14235
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

protocol/server: address double free's

2016-05-03T12:16:55+00:00

Backport of:
> Change-Id: Ic8a8fe85cf91c5c7aa93dce872cedbc67464e4ea
> BUG: 1227667
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/14150
> Reviewed-by: Jeff Darcy 
> Tested-by: Prasanna Kumar Kalever 
> Smoke: Gluster Build System 
> Reviewed-by: Niels de Vos 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 

Change-Id: I895457636c99cc77d643985b64180d659ac356ba
BUG: 1332414
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/14179
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
Reviewed-by: Susant Palai 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System