glusterfs.git/libglusterfs, branch v4.0.1

rpcsvc: scale rpcsvc_request_handler threads

2018-03-20T12:17:40+00:00

Scale rpcsvc_request_handler threads to match the scaling of event
handler threads.

Please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c51
for a discussion about why we need multi-threaded rpcsvc request
handlers.

mainline:
> Reviewed-on: https://review.gluster.org/19337
> Reviewed-by: Raghavendra G 
> Signed-off-by: Milind Changire 
(cherry picked from commit 7d641313f46789ec0a7ba0cc04f504724c780855)

Change-Id: Ib6838fb8b928e15602a3d36fd66b7ba08999430b
BUG: 1550946
Signed-off-by: Milind Changire

libglusterfs: move compat RPC/XDR #defines to eliminate warnings

2018-03-16T13:41:47+00:00

Building with libtirpc (versus legacy glibc rpc) results in many
warnings about xdr macros that are redefined in libtirpc headers
because of the way compat.h and glusterfs.h are usually #included.

And these xdr macros in libglusterfs/src/compat.h - which were copied
from legacy glibc's rpc headers - are different than the same-name macros
in libtirpc. I haven't checked to see that any of the macros are
expanded (incorrectly) between the definition in compat.h and the
redefinition in tirpc/rpc/xdr.h; the risk seems pretty minimal. Regardless
it seems better, from a truth-and-beauty perspective to not have the
old, incorrect definitions in the first place.

Not to mention that any file that #includes compat.h and not glusterfs.h
does not need these xdr macro definitions at all. They're really only
needed when using really old glibc rpc, which would only be evident if
including glusterfs.h and/or glusterfs-fops.h. (Which by the way, nothing
currently #includes glusterfs-fops.h by itself. And maybe nothing ever
should?)

Change-Id: Ic11e4407d6ab7c498a8745a99379cbf4788a24e8
BUG: 1555309
Signed-off-by: Kaleb S. KEITHLEY

protocol: Fix 4.0 client, parsing older iatt in dict

2018-03-11T10:22:22+00:00

In a mixed mode cluster involving 4.0 and older 3.x bricks, if
clients are newer, then the iatt encoded in the dictionary can be
of the older iatt format, which a newer client will map incorrectly
to the newer structure.

This causes failures in FOPs that depend on this iatt for some
functionality (seen in mkdir operations failing as EIO, when DHT
hits its internal setxattr call).

The fix provided is to convert the iatt in the dict, based on which
RPC version is used to communicate with the server.

IOW, this is the reverse of change in commit "b966c7790e"

Tested using a mixed mode cluster (i.e bricks in 3.12 and 4.0 versions)
and a mixed set of clients, 3.12 and 4.0 clients.

There is no regression test provided, as this needs a mixed mode cluster
to test and validate.

>Change-Id: I454e54651ca836b9f7c28f45f51d5956106aefa9
>BUG: 1554053
>Signed-off-by: ShyamsundarR 

Change-Id: I454e54651ca836b9f7c28f45f51d5956106aefa9
BUG: 1554077
Signed-off-by: ShyamsundarR 
Signed-off-by: Raghavendra G

protocol: Added iatt conversion to older format

2018-03-11T10:22:22+00:00

Added iatt conversion to an older format, when dealing with
older RPC versions. This enables iatt structure conformance
when dealing with older clients.

This helps fix rolling upgrade from 3.x versions to 4.0 version
of gluster by sending the right iatt in the dictionary when DHT
requests the same.

(cherry picked from commit b966c7790e35de353ae09ee48d4e2f55e0117f7e)

Change-Id: Ieaf925f81f8c7798a8fba1e90a59fa9dec82856c
BUG: 1551112
Signed-off-by: ShyamsundarR

posix/afr: handle backward compatibility for rchecksum fop

2018-02-20T03:12:02+00:00

Added a volume option 'fips-mode-rchecksum' tied to op version 4.
If not set, rchecksum fop will use MD5 instead of SHA256.

updates: #230
Change-Id: Id8ea1303777e6450852c0bc25503cda341a6aec2
Signed-off-by: Ravishankar N 
(cherry picked from commit 6daa6535692b2c68b493636a9bbfdcbc475b3d80)

core: move logs which are only developer relevant to DEBUG level

2018-02-09T15:35:35+00:00

Change-Id: I8b38e231b6160db8075f73773d4e7dc115a90d95
BUG: 1542829
Signed-off-by: Amar Tumballi

cluster/dht: avoid overwriting client writes during migration

2018-02-06T14:33:30+00:00

For more details on this issue see
https://github.com/gluster/glusterfs/issues/308

Solution:
This is a restrictive solution where a file will not be migrated
if a client writes to it during the migration. This does not
check if the writes from the rebalance and the client actually
do overlap.

If dht_writev_cbk finds that the file is being migrated (PHASE1)
it will set an xattr on the destination file indicating the file
was updated by a non-rebalance client.
Rebalance checks if any other client has written to the dst file
and aborts the file migration if it finds the xattr.

updates gluster/glusterfs#308

Change-Id: I73aec28bc9dbb8da57c7425ec88c6b6af0fbc9dd
Signed-off-by: Susant Palai 
Signed-off-by: Raghavendra G 
Signed-off-by: N Balachandran 
(cherry picked from commit 545a7ce6762a1b3a7b989b43a9d18b5b1b299df0)

gfapi : New APIs have been added to use lease feature in gluster

2018-02-01T11:39:52+00:00

Following APIs glfs_h_lease(), glfs_lease() added, so that gfapi applications
can set and get lease which enables more efficient client side caching.

Updates: #350
Change-Id: Iede85be9af1d4df969b890d0937ed0afa4ca6596
Signed-off-by: Poornima G 
Signed-off-by: Soumya Koduri 
Signed-off-by: Jiffin Tony Thottan

build: use libtirpc by default, even if ipv6 is not the default

2018-01-30T18:36:28+00:00

Another error snuck in with Change-Id I86f847dfd, or more
accurately I think, with Change-Id: Ic47065e9c2...

All libs, not just libgfrpc, need to be linked with libtirpc,
especially on systems that still have xdr functions in (g)libc
where you will get a mixture of calls to libtirpc functions
and glibc functions, with catas

Change-Id: I97dc39c7844f44c36fe210aa813480c219e1e415
BUG: 1538723
Signed-off-by: Kaleb S. KEITHLEY

dentry fop serializer: added new server side xlator for dentry fop serialization

2018-01-30T18:36:01+00:00

Problems addressed by this xlator :

[1]. To prevent race between parallel mkdir,mkdir and lookup etc.

Fops like mkdir/create, lookup, rename, unlink, link that happen on a
particular dentry must be serialized to ensure atomicity.

Another possible case can be a fresh lookup to find existance of a path
whose gfid is not set yet. Further, storage/posix employs a ctime based
heuristic 'is_fresh_file' (interval time is less than 1 second of current
time) to check fresh-ness of file. With serialization of these two fops
(lookup & mkdir), we eliminate the race altogether.

[2]. Staleness of dentries

This causes exponential increase in traversal time for any inode in the
subtree of the directory pointed by stale dentry.

Cause :  Stale dentry is created because of following two operations:

      a. dentry creation due to inode_link, done during operations like
         lookup, mkdir, create, mknod, symlink, create and
      b. dentry unlinking due to various operations like rmdir, rename,
         unlink.

       The reason is __inode_link uses __is_dentry_cyclic, which explores
       all possible path to avoid cyclic link formation during inode
       linkage. __is_dentry_cyclic explores stale-dentry(ies) and its
       all ancestors which is increases traversing time exponentially.

Implementation : To acheive this all fops on dentry must take entry locks
before they proceed, once they have acquired locks, they perform the fop
and then release the lock.

Some documentation from email conversation:
[1] http://www.gluster.org/pipermail/gluster-devel/2015-December/047314.html

[2] http://www.gluster.org/pipermail/gluster-devel/2015-August/046428.html

With this patch, the feature is optional, enable it by running:

 `gluster volume set $volname features.sdfs enable`

Also the feature is tested for a month without issues in the
experiemental branch for all the regression.

Change-Id: I6e80ba3cabfa6facd5dda63bd482b9bf18b6b79b
Fixes: #397
Signed-off-by: Sakshi Bansal 
Signed-off-by: Amar Tumballi 
Signed-off-by: Sunny Kumar