glusterfs.git/tests, branch v3.9.0rc2

compound fops: Fix file corruption issue

2016-10-25T04:14:54+00:00

        Backport of: http://review.gluster.org/#/c/15654/

1. Address of a local variable @args is copied into state->req
in server3_3_compound (). But even after the function has gone out of
scope, in server_compound_resume () this pointer is accessed and
dereferenced. This patch fixes that.

2. Compound fops, by virtue of NOT having a vector sizer (like the one
writev has), ends up having both the header and the data (in case one of
its member fops is WRITEV) in the same hdr_iobuf. This buffer was not
being preserved through the lifetime of the compound fop, causing it to
be overwritten by a parallel write fop, even when the writev associated
with the currently executing compound fop is yet to hit the desk, thereby
corrupting the file's data. This is fixed by associating the hdr_iobuf with
the iobref so its memory remains valid through the lifetime of the fop.

3. Also fixed a use-after-free bug in protocol/client in compound fops cbk,
missed by Linux but caught by NetBSD.

Finally, big thanks to Pranith Kumar K and Raghavendra Gowdappa for their
help in debugging this file corruption issue.

Change-Id: I58da39ae544ad81192849926399a971c4c01c986
BUG: 1387984
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15709
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

cluster/ec: set/unset dirty flag for data/metadata update

2016-10-13T20:04:11+00:00

Currently, for all the update operations, metadata or data,
we set the dirty flag at the end of the operation only if
a brick is down. This leads to delay in healing and in some
cases not at all.
In this patch we set (+1) the dirty flag
at the start of the metadata or data update operations and
after successfull completion of the fop, we unset (-1) it again.

>Change-Id: Ide5668bdec7b937a61c5c840cdc79a967598e1e9
>BUG: 1316873
>Signed-off-by: Ashish Pandey 
>Reviewed-on: http://review.gluster.org/13733
>Tested-by: Pranith Kumar Karampuri 
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Xavier Hernandez 

Change-Id: Ide5668bdec7b937a61c5c840cdc79a967598e1e9
BUG: 1377570
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15534
Reviewed-by: Pranith Kumar Karampuri 
Tested-by: Pranith Kumar Karampuri 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/ec: Implement heal info with lock

2016-10-13T14:35:46+00:00

Problem: Currently heal info command prints all
the files/directories if the index for the
file/directory is present in .glusterfs/indices folder.
After implementing patch http://review.gluster.org/#/c/13733/
indices of the file which is going through update fop
will also be present in .glusterfs/indices even
if the fop is successful on all the brick. At this time
if heal info command is being used, it will also display this
file which is actually healthy and does not require any heal.

Solution: Take lock on a file corresponding to the indices
and inspect xattrs to decide if the file needs heal or not.

>Reviewed-on: http://review.gluster.org/15543
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Pranith Kumar Karampuri 
>Reviewed-by: Xavier Hernandez 
>CentOS-regression: Gluster Build System 
>Smoke: Gluster Build System 
>Signed-off-by: Ashish Pandey 

Change-Id: I6361e2813ece369be12d02e74816df4eddb81cfa
BUG: 1383913
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/15627
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Xavier Hernandez

gfapi: redesign the public interface for upcall consumers

2016-10-11T12:18:56+00:00

The glfs_callback_arg and glfs_callback_inode_arg were allocated by
gfapi, and expected to be free()'d by the application. However it is not
reasonable to expect that applications use the same memory allocator to
as the compiled libgfapi.so. For instance, it is possible that gfapi
uses glibc malloc/free, and an application like NFS-Ganesha the versions
from jemalloc. Mismatching of the malloc() and free() functions causes
segmentation faults at best.

In order to prevent problems like this in the future, the API for
applications that consume upcalls has been remodeled. Any of the
structures that gfapi allocates, should be free'd with glfs_free(). The
members of the structures can not be accessed directly anymore, each
has its own function to access now.

Correcting the naming of the functions, structures and constants is a
continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These
new improvements not only have correct prefixes for the functions and
structures, the naming also reflects more to the upcall framework and
does not use "callback" anymore.

Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9:
> Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
> BUG: 1344714
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/14701
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 
> Reviewed-by: soumya k 
> Reviewed-by: jiffin tony Thottan 

Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
BUG: 1378948
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15597
NetBSD-regression: NetBSD Build System 
Reviewed-by: jiffin tony Thottan 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

socket: pollerr event shouldn't trigger socket_connnect_finish

2016-09-21T15:21:18+00:00

If connect fails with any other error than EINPROGRESS we cannot get
the error status using getsockopt (... SO_ERROR ... ). Hence we need
to remember the state of connect and take appropriate action in the
event_handler for the same.

As an added note, a event can come where poll_err is HUP and we have
poll_in as well (i.e some status was written to the socket), so for
such cases we need to finish the connect, process the data and then
the poll_err as is the case in the current code.

Special thanks to Kaushal M & Raghavendra G for figuring out the issue.

>Signed-off-by: Shyam 
>Reviewed-on: http://review.gluster.org/15440
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Raghavendra G 

Change-Id: Ic45ad59ff8ab1d0a9d2cab2c924ad940b9d38528
BUG: 1377386
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/15533
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

tests: fix bug-963541.t spurious failure

2016-09-19T05:40:34+00:00

wait for remove brick to complete before attempt for a commit.

>Reviewed-on: http://review.gluster.org/15457
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Vijay Bellur 

Change-Id: I66ea6c48b6a69fe33d79f9d9080b6f2c1462578e
BUG: 1375042
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/15458
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

feature/bitrot: Fix recovery of corrupted hardlink

2016-09-19T05:37:55+00:00

Problem:
When a file with hardlink is corrupted in ec volume,
the recovery steps mentioned was not working.
Only name and metadata was healing but not the data.

Cause:
The bad file marker in the inode context is not removed.
Hence when self heal tries to open the file for data
healing, it fails with EIO.

Background:
The bitrot deletes inode context during forget.

Briefly, the recovery steps involves following steps.
   1. Delete the entry marked with bad file xattr
      from backend. Delete all the hardlinks including
      .glusters hardlink as well.
   2. Access the each hardlink of the file including
      original from the mount.

The step 2 will send lookup to the brick where the files
are deleted from backend and returns with ENOENT. On
ENOENT, server xlator forgets the inode if there are
no dentries associated with it. But in case hardlinks,
the forget won't be called as dentries (other hardlink
files) are associated with the inode. Hence bitrot stube
won't delete it's context failing the data self heal.

Fix:
Bitrot-stub should delete the inode context on getting
ENOENT during lookup.

>Change-Id: Ice6adc18625799e7afd842ab33b3517c2be264c1
>BUG: 1373520
>Signed-off-by: Kotresh HR 
>Reviewed-on: http://review.gluster.org/15408
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Raghavendra Bhat 
(cherry picked from commit b86a7de9b5ea9dcd0a630dbe09fce6d9ad0d8944)

Change-Id: Ice6adc18625799e7afd842ab33b3517c2be264c1
BUG: 1374567
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/15434
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Aravinda VK

cluster/ec: Add support for hardware acceleration

2016-09-10T02:19:13+00:00

This patch implements functionalities for fast encoding/decoding
using hardware support. Currently optimized x86_64, SSE and AVX is
added.

Additionally this patch implements a caching mecanism for inverse
matrices to reduce computation time, as well as a new method for
computing the inverse that takes quadratic time instead of cubic.

Finally some unnecessary memory copies have been eliminated to
further increase performance.

 >Change-Id: I26c75f26fb4201bd22b51335448ea4357235065a
 >BUG: 1289922
 >Signed-off-by: Xavier Hernandez 
 >Reviewed-on: http://review.gluster.org/12837
 >Tested-by: Pranith Kumar Karampuri 
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

BUG: 1374841
Change-Id: I83731663922ed11ca84536deab5737463416e1e0
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15455
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

tests: Fix one of the upcall tests

2016-09-03T07:05:43+00:00

Backport of http://review.gluster.org/#/c/15385/

Currently md-cache invalidation feature is enabled by setting
"performance.cache-invalidation", but this case was sent when
"features.cache-invalidation" was enabling md-cache invalidation.
Hence, fix the same.

Change-Id: If044f6208179748a120fbe1d63b676367e707f73
BUG: 1372586
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15386
Smoke: Gluster Build System 
Reviewed-by: Raghavendra Talur 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

io-stats: Add stats for upcall notifications

2016-09-01T03:18:42+00:00

With this patch, there will be additional entries seen in
the profile info:

UPCALL : Total number of upcall events that were sent from
         the brick(in brick profile), and number of upcall
         notifications recieved by client(in client profile)

Cache invalidation events:
-------------------------
CI_IATT : Number of upcalls that were cache invalidation and
         had one of the IATT_UPDATE_FLAGS set. This indicates
         that one of the iatt value was changed.

CI_XATTR : Number of upcalls that were cache invalidation, and
         had one of the UP_XATTR or UP_XATTR_RM set. This indicates
         that an xattr was updated or deleted.

CI_RENAME : Number of upcalls that were cache invalidation,
         resulted by the renaming of a file or directory

CI_UNLINK : Number of upcalls that were cache invalidation,
         resulted by the unlink of a file.

CI_FORGET : Number of upcalls that were cache invalidation,
         resulted by the forget of inode on the server side.

Lease events:
------------
LEASE_RECALL : Number of lease recalls sent by the brick (in
         brick profile), and number of lease recalls recieved
         by client(in client profile)

Note that the sum of CI_IATT, CI_XATTR, CI_RENAME, CI_UNLINK,
CI_FORGET, LEASE_RECALL may not be equal to UPCALL. This is
because, each cache invalidation can carry multiple flags.
Eg:
- Every CI_XATTR will have CI_IATT
- Every CI_UNLINK will also increment CI_IATT as link count is an
iatt attribute.

Also UP_PARENT_DENTRY_FLAGS is currently not accounted for,
as CI_RENAME and CI_UNLINK will always have the flag
UP_PARENT_DENTRY_FLAGS

Change-Id: Ieb8cd21dde2c4c7618f12d025a5e5156f9cc0fe9
BUG: 1371543
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15193
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Rajesh Joseph 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri