glusterfs.git/xlators, branch v3.8.6

glusterd: clean up old port and allocate new one on every restart

2016-11-23T10:37:16+00:00

    Backport of http://review.gluster.org/#/c/15005/9.

GlusterD as of now was blindly assuming that the brick port which was
already allocated would be available to be reused and that assumption
is absolutely wrong.

Solution : On first attempt, we thought GlusterD should check if the
already allocated brick ports are free, if not allocate new port and
pass it to the daemon. But with that approach there is a possibility
that if PMAP_SIGNOUT is missed out, the stale port will be given back
to the clients where connection will keep on failing. Now given the
port allocation always start from base_port, if everytime a new port
has to be allocated for the daemons, the port range will still be
under control. So this fix tries to clean up old port using
pmap_registry_remove () if any and then goes for pmap_registry_alloc ()

This patch is being ported to 3.8 branch because, the brick process
blindly re-using old port, without registering with the pmap server,
causes snapd daemon to not start properly, even though snapd registers
with the pmap server. With this patch, all the brick processes and
snapd will register with the pmap server to either get the same port,
or a new port, and avoid port collision.

> Reviewed-on: http://review.gluster.org/15005
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Avra Sengupta 

(cherry picked from commit c3dee6d35326c6495591eb5bbf7f52f64031e2c4)

Change-Id: If54a055d01ab0cbc06589dc1191d8fc52eb2c84f
BUG: 1369766
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/15308
Tested-by: Avra Sengupta 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos

xlators/trash : Remove upper limit for trash max file size

2016-11-23T10:19:03+00:00

Currently file which size exceeds more than 1GB never moved to
trash directory. This is due to the hard coded check using
GF_ALLOWED_MAX_FILE_SIZE.

Upstream reference :
>Change-Id: I2ed707bfe1c3114818896bb27a9856b9a164be92
>BUG: 1386766
>Signed-off-by: Jiffin Tony Thottan 
>Reviewed-on: http://review.gluster.org/15689
>Smoke: Gluster Build System 
>Reviewed-by: Anoop C S 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Vijay Bellur 
>(cherry picked from commit cd9be49f6fe05d424989c0686a7e55a3f3ead27e)

Change-Id: I2ed707bfe1c3114818896bb27a9856b9a164be92
BUG: 1392364
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: http://review.gluster.org/15785
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Anoop C S 
Reviewed-by: Niels de Vos

marker: Fix inode value in loc, in setxattr fop

2016-11-22T05:43:38+00:00

Backport of http://review.gluster.org/15826

On recieving a rename fop, marker_rename() stores the,
oldloc and newloc in its 'local' struct, once the rename
is done, the xtime marker(last updated time) is set on
the file, but sending a setxattr fop. When upcall
receives the setxattr fop, the loc->inode is NULL and
it crashes. The loc->inode can be NULL only in one valid
case, i.e. in rename case where the inode of new loc
can be NULL. Hence, marker should have filled the inode
of the new_loc before issuing a setxattr.

> Reviewed-on: http://review.gluster.org/15826
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kotresh HR 
> Smoke: Gluster Build System 
> Reviewed-by: Rajesh Joseph 
(cherry picked from commit 46e5466850311ee69e6ae9a11c2bba2aabadd5de)

Change-Id: Id638f678c3daaf4a5c29b970b58929d377ae8977
BUG: 1396418
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15878
Reviewed-by: Rajesh Joseph 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

upcall: Fix a log level

2016-11-18T11:20:42+00:00

In upcall_cache_invalidation(), the gfid can be NULL in certain
valid test cases(eg: entry for ".." in readdirp), hence change
the log level from WARNING to DEBUG.

Backport of http://review.gluster.org/15777

> Reviewed-on: http://review.gluster.org/15777
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Rajesh Joseph 
> Reviewed-by: Jeff Darcy 
(cherry picked from commit 68d1480e6056d1be91cde5129a6809642eeee857)

Change-Id: Ic90167a0e2076694e9131913114460df7b939b30
BUG: 1394187
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15828
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Rajesh Joseph 
Smoke: Gluster Build System

cluster/afr: When failing fop due to lack of quorum, also log error string

2016-11-11T12:23:54+00:00

        Backport of: http://review.gluster.org/#/c/15800/

Change-Id: I2dd7ed69a456e8b9e54a4093f14dc16950bef081
BUG: 1393630
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15813
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri

performance/open-behind: Avoid deadlock in statedump

2016-11-11T07:22:30+00:00

Problem:
open-behind is taking fd->lock then inode->lock where as statedump is taking
inode->lock then fd->lock, so it is leading to deadlock

In open-behind, following code exists:
void
ob_fd_free (ob_fd_t *ob_fd)
{
        loc_wipe (&ob_fd->loc); <<--- this takes (inode->lock)
.......
}

int
ob_wake_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
             int op_ret, int op_errno, fd_t *fd_ret, dict_t *xdata)
{
	.......
        LOCK (&fd->lock); <<---- fd->lock
        {
	.......
                __fd_ctx_del (fd, this, NULL);
                ob_fd_free (ob_fd); <<<---------------
        }
        UNLOCK (&fd->lock);
.......
}
=================================================================
In statedump this code exists:
inode_dump (inode_t *inode, char *prefix)
{
.......
	ret = TRY_LOCK(&inode->lock); <<---- inode->lock
.......
	fd_ctx_dump (fd, prefix); <<<-----
.......
}
fd_ctx_dump (fd_t *fd, char *prefix)
{
.......
        LOCK (&fd->lock); <<<------------------ this takes fd-lock
        {
.......
}

Fix:
Make sure open-behind doesn't call ob_fd_free() inside fd->lock

 >BUG: 1393259
 >Change-Id: I4abdcfc5216270fa1e2b43f7b73445f49e6d6e6e
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15808
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Poornima G 
 >Reviewed-by: Raghavendra G 

BUG: 1393682
Change-Id: I45a0fbed683ef6acb7900df87534927f332fdaaa
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15818
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

posix-acl: check dictionary before using it

2016-11-10T06:26:40+00:00

If extended attributes are not present in md-cache it returns NULL as xattr.
posix acl xlator should check for NULL before using xattr.

If normal and default ACLs are not set on file then md-cache will not contain
system.posix_acl_access and system.posix_acl_default extended attributes in
its cache.

Therefore posix_acl_lookup_cbk should check xattr before using it, otherwise
the logs will get filled with dictionary errors.

> Reviewed-on: http://review.gluster.org/15769
> Reviewed-by: Raghavendra Talur 
> Smoke: Gluster Build System 
> Reviewed-by: jiffin tony Thottan 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Vijay Bellur 
> CentOS-regression: Gluster Build System 

(cherry picked from commit de7fe24663713fff364cfc2b52b675e3e979ee68)

Change-Id: Icebf73cf0b313bd3e82ca8cbda63786dd0fa47da
BUG: 1392868
Signed-off-by: Rajesh Joseph 
Reviewed-on: http://review.gluster.org/15799
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra Talur

features/shard: Fill loc.pargfid too for named lookups on individual shards

2016-11-09T07:28:19+00:00

        Backport of: http://review.gluster.org/#/c/15788/

On a sharded volume when a brick is replaced while IO is going on, named
lookup on individual shards as part of read/write was failing with
ENOENT on the replaced brick, and as a result AFR initiated name heal in
lookup callback. But since pargfid was empty (which is what this patch
attempts to fix), the resolution of the shards by protocol/server used
to fail and the following pattern of logs was seen:

Brick-logs:

[2016-11-08 07:41:49.387127] W [MSGID: 115009]
[server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
for (null) (LOOKUP)
[2016-11-08 07:41:49.387157] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
(00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
==> (Invalid argument) [Invalid argument]

Client-logs:
[2016-11-08 07:41:27.497687] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.497755] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.498500] W [MSGID: 114031]
[client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[Invalid argument]
[2016-11-08 07:41:27.499680] E [MSGID: 133010]

Also, this patch makes AFR by itself choose a non-NULL pargfid even if
its ancestors fail to initialize all pargfid placeholders.

Change-Id: Ica9e1b5b196ac37aafe6128e7aa0694a07245fdb
BUG: 1392846
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15796
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Ravishankar N 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd/quota: upgrade quota.conf file during an upgrade

2016-11-08T16:46:41+00:00

Problem
=======
When quota is enabled on 3.6, it will have quota conf version in quota.conf
as v1.1. This node gets upgraded to 3.7 but it will still have quota conf
version as v1.1 until a quota enable/disable/set limit is initiated. When
this is not initiated and when this node tries to peer probe a node which
is a fresh install of 3.7 (which will have quota conf version as v1.2), then this
will result in "Peer rejected" state. This patch fixes the issue.

Solution
========
When an upgrade happens from 3.6 to 3.7, quota.conf file needs
to be modified as well. With 3.6, in quota.conf the version will be
v1.1 and it needs to be changed to v1.2 from 3.7. This is because in
3.7, inode quota feature is introduced. So when an op-version bumpup
happens quota.conf needs to be upgraded with quota conf version v1.2
and all the 16 byte uuid needs to be changed to 17 bytes uuid as well.

Previously, when the cluster version is upgraded to 3.7, the quota.conf
got upgraded as well. But, the upgradation was done only when quota
enable/disable/set limit is done. With this patch, the upgradation is done
during a cluster op version bump up as well.

> Reviewed-on: http://review.gluster.org/15352
> Tested-by: Atin Mukherjee 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
(cherry picked from commit 4b2cff614462508eef529c5d128e0974720e3f50)

Change-Id: Idb5ba29d3e1ea0e45c85d87c952c75da9e0f99f0
BUG: 1392716
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/15791
Smoke: Gluster Build System 
Reviewed-by: Manikandan Selvaganesh 
Tested-by: Manikandan Selvaganesh 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd/shared storage: Check for hook-script at staging

2016-11-07T05:18:28+00:00

Check if S32gluster_enable_shared_storage.sh is present
at /var/lib/glusterd/hooks/1/set/post/ at staging
before proceeding with the command. Fail the command
with the appropriate error message in case it is not
present.

> Reviewed-on: http://review.gluster.org/15718
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 

(cherry picked from commit 29587a91716e1e55bd172d63340c40249fb343c9)

Change-Id: I84e3912f1cdffb927f8a40d74d52be43ee69388b
BUG: 1377448
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/15741
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee