glusterfs.git/xlators/features/bit-rot/src/stub, branch v3.12dev

libglusterfs/stack.h: reduce duplication of code

2017-04-27T14:54:40+00:00

* Use STACK_UNWIND_STRICT everywhere.
* Provide STACK_WIND_COMMON as both STACK_WIND_COOKIE
  and STACK_WIND differ by just 1 line and 1 option.

Updates gluster/glusterfs#137

Change-Id: Ifbb6b9c4702b02f4a02834824f509fd10c78f0ce
Signed-off-by: Amar Tumballi 
Reviewed-on: https://review.gluster.org/16915
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

features/bit-rot-stub: bring in optional versioning

2017-04-19T03:29:46+00:00

* As of now bit-rot-stub does versioning always. This leads
  lots of getxattr calls being made in lookups. So make
  object versioning optional.

Change-Id: I83713e45ae59fb28004bb3cfa008f2d69edebbfa
BUG: 1359599
Signed-off-by: Raghavendra Bhat 
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/14442
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

features/bit-rot-stub: use the correct spelling of quarantine for bad objects

2017-01-30T20:17:51+00:00

                       container

The directory for containing the list of bad objects was named "quanrantine"
instead of "quarantine"

Change-Id: I8c20381ac637201d9d1a224f5223e8dfbed53f1e
BUG: 1401571
Signed-off-by: Raghavendra Bhat 
Reviewed-on: https://review.gluster.org/16027
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

feature/bitrot: Fix recovery of corrupted hardlink

2016-09-08T17:09:33+00:00

Problem:
When a file with hardlink is corrupted in ec volume,
the recovery steps mentioned was not working.
Only name and metadata was healing but not the data.

Cause:
The bad file marker in the inode context is not removed.
Hence when self heal tries to open the file for data
healing, it fails with EIO.

Background:
The bitrot deletes inode context during forget.

Briefly, the recovery steps involves following steps.
   1. Delete the entry marked with bad file xattr
      from backend. Delete all the hardlinks including
      .glusters hardlink as well.
   2. Access the each hardlink of the file including
      original from the mount.

The step 2 will send lookup to the brick where the files
are deleted from backend and returns with ENOENT. On
ENOENT, server xlator forgets the inode if there are
no dentries associated with it. But in case hardlinks,
the forget won't be called as dentries (other hardlink
files) are associated with the inode. Hence bitrot stube
won't delete it's context failing the data self heal.

Fix:
Bitrot-stub should delete the inode context on getting
ENOENT during lookup.

Change-Id: Ice6adc18625799e7afd842ab33b3517c2be264c1
BUG: 1373520
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/15408
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Bhat

bit-rot: fix unused variable warnings/errors

2016-08-29T16:21:41+00:00

http://review.gluster.org/14085 fixes a/the "leak" - via the
generated rpc/xdr headers - of pragmas that mask these warnings.

However 14085 won't pass the smoke test until all the warnings are
fixed.

Change-Id: I566fcbcae5aec575bfca40975b941c53546d4d97
BUG: 1369124
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/15245
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR

features/bitrot: Move throttling code to libglusterfs

2016-07-18T12:03:17+00:00

Since throttling is a separate feature by itself,
move throttling code to libglusterfs.

Change-Id: If9b99885ceb46e5b1865a4af18b2a2caecf59972
BUG: 1352019
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/14846
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Ravishankar N 
Reviewed-by: Jeff Darcy

core: use readdir(3) with glibc, and associated cleanup

2016-07-18T11:59:42+00:00

Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3)
is marked as deprecated. Specifically the function decl in 
has the deprecated attribute, and now warnings are thrown during the
compile on Fedora 25 builds.

The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say
that glibc's readdir(3) is, and always has been, MT-SAFE as long as
only one thread is accessing the directory object returned by opendir().
World+Dog also says there is a potential buffer overflow in readdir_r().
World+Dog suggests that it is preferable to simply use readdir(). There's
an implication that eventually readdir_r(3) will be removed from glibc.
POSIX has, apparently deprecated it in the standard, or even removed it
entirely.

Over and above that, our source near the various uses of readdir(_r)(3)
has a few unsafe uses of strcpy()+strcat().

(AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see
if the same is true on those platforms, and we can't be sure of MacOS
even though we know it's based on *BSD.)

Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568
BUG: 1356998
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/14838
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Jeff Darcy

features/bit-rot-stub: get frame->local before unwinding

2016-03-09T15:14:10+00:00

In bit-rot-stub, if unlink fails, then it was unwinding
directly. Then it was trying to cleanup local. But local
would be NULL, since it was unwinding directly without getting
the value of frame->local. The NULL cleanup of local was
causing the brick process to crash.

Change-Id: I8544ba73b2e8dc0c50b1a53ff8027d85588d087b
BUG: 1315465
Signed-off-by: Raghavendra Bhat 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/13628
Smoke: Gluster Build System 
Reviewed-by: Kotresh HR 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

features/bitrot: do not remove the quarantine handle in forget

2016-03-01T03:18:55+00:00

If an object is marked as bad, then an entry is corresponding to the
bad object is created in the .glusterfs/quarantine directory to help
scrub status. The entry name is the gfid of the corrupted object.
The quarantine handle is removed in below 2 cases.

1) When protocol/server revceives the -ve lookup on an entry whose inode
   is there in the inode table (it can happen when the corrupted object
   is deleted directly from the backend for recovery purpose) it sends a
   forget on the inode and bit-rot-stub removes the quarantine handle in
   upon getting the forget.
   refer to the below commit
   f853ed9c61bf65cb39f859470a8ffe8973818868:
   http://review.gluster.org/12743)

2) When bit-rot-stub itself realizes that lookup on a corrupted object
   has failed with ENOENT.

But with step1, there is a problem when the bit-rot-stub receives forget
due to lru limit exceeding in the inode table. In such cases, though the
corrupted object is not deleted (either from the mount point or from the
backend), the handle in the quarantine directory is removed and that object
is not shown in the bad objects list in the scrub status command.

So it is better to follow only 2nd step (i.e. bit-rot-stub removing the handle
from the quarantine directory in -ve lookups). Also the handle has to be removed
when a corrupted object is unlinked from the mount point itself.

Change-Id: Ibc3bbaf4bc8a5f8986085e87b729ab912cbf8cf9
BUG: 1308961
Original author: Raghavendra Bhat 
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/13472
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Venky Shankar