glusterfs.git/xlators/features/index, branch v3.11dev

features/index: Delete granular entry indices of already healed directories during crawl

2016-11-25T03:03:14+00:00

If granular name indices are already in existence for a volume, and
before they are healed, granular entry heal be disabled, a crawl on
indices/xattrop will clear the changelogs on these directories. When
their corresponding entry-changes indices are crawled subsequently,
if it is found that the directories don't need heal anymore, the
granular indices are not cleaned up.
This patch fixes that problem by ensuring that the zero-xattrop
also deletes the stale indices at the level of index translator.

Change-Id: Ifbaa6bec2a14e3041addfee4054131babbf4d35e
BUG: 1370410
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15880
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

index: fix unused variable warnings/errors

2016-09-13T04:01:25+00:00

http://review.gluster.org/14085 fixes a/the "leak" - via the
generated rpc/xdr headers - of pragmas that mask these warnings.

However 14085 won't pass the smoke test until all the warnings are
fixed.

Change-Id: Ibbc34782fdbffb7d9deaf454ebe946d04696035e
BUG: 1369124
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/15247
Reviewed-by: Krutika Dhananjay 
Smoke: Gluster Build System 
Reviewed-by: Ravishankar N 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

core: use readdir(3) with glibc, and associated cleanup

2016-07-18T11:59:42+00:00

Starting with glibc-2.23 (i.e. what's in Fedora 25), readdir_r(3)
is marked as deprecated. Specifically the function decl in 
has the deprecated attribute, and now warnings are thrown during the
compile on Fedora 25 builds.

The readdir(_r)(3) man page (on Fedora 25 at least) and World+Dog say
that glibc's readdir(3) is, and always has been, MT-SAFE as long as
only one thread is accessing the directory object returned by opendir().
World+Dog also says there is a potential buffer overflow in readdir_r().
World+Dog suggests that it is preferable to simply use readdir(). There's
an implication that eventually readdir_r(3) will be removed from glibc.
POSIX has, apparently deprecated it in the standard, or even removed it
entirely.

Over and above that, our source near the various uses of readdir(_r)(3)
has a few unsafe uses of strcpy()+strcat().

(AFAIK nobody has looked at the readdir(3) implemenation in *BSD to see
if the same is true on those platforms, and we can't be sure of MacOS
even though we know it's based on *BSD.)

Change-Id: I5481f18ba1eebe7ee177895eecc9a80a71b60568
BUG: 1356998
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/14838
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Kotresh HR 
Reviewed-by: Jeff Darcy

features/index: Move gf_log -> gf_msg

2016-07-18T02:26:28+00:00

Change-Id: I88e5d69a24309bf9d3cb855950c9a5349bdd3d66
BUG: 1356504
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14924
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
Tested-by: Krutika Dhananjay 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Zhou Zhengping 
Tested-by: Zhou Zhengping

features/index: Ignore regular files when examining state of the inode wrt ENTRY_CHANGES

2016-07-12T07:22:44+00:00

This prevents unnecessary sys_lstat() syscall when xattrop
is operating on regular files.

Also, to avoid (even) the one-time execution of sys_lstat()
syscalls on disperse volumes or replicate volumes when granular
entry self-heal is disabled, we pass a key in afr_selfheal_undo_pending()
indicating to index that this xattrop is on a granular directory index.
Index will accordingly decide whether or not to attempt the index state
initialization.

Change-Id: I616d9a4b3704066f4a2770d3ca307f7dca9122e0
BUG: 1331323
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14870
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Anuradha Talur 
Reviewed-by: Pranith Kumar Karampuri

afr, index: Clean up stale directory and file indices in granular entry sh

2016-07-12T06:13:53+00:00

Specifically when a directory tree is removed (rm -rf)
while a brick is down, both the directory index and the
name indices of the files and subdirs under it will remain.
Self-heal will need to pick up these and remove them.

Towards this, afr sh will now also crawl indices/entry-changes
and call an rmdir on the dir if the directory index is stale.

On the brick side, rmdir fop has been implemented for index xl,
which would delete the directory index and its contents if present
in a synctask.

Change-Id: I8b527331c2547e6c141db6c57c14055ad1198a7e
BUG: 1331323
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14832
Reviewed-by: Ravishankar N 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri 
CentOS-regression: Gluster Build System

features/index: Delete parent dir indices when heal on it is complete

2016-06-28T08:13:57+00:00

In this patch, the state information about whether a directory
gfid index is present or not is stored in the inode ctx with
values IN and NOTIN. This saves index xl the need to perform
stat() everytime an index_entry_create() is called.

When a brick is restarted these in-memory inode ctx records will
be gone. So when granular entry heal happens after a brick is restarted,
and a post-op is done on the parent, if the state gotten from inode ctx
is UNKNOWN, then index xl does a stat to initialize the state as IN or
NOTIN. Note that this is a one-time operation for the lifetime of the
brick. Such a change also helps avoid calling index_del() in
xattrop_index_action() periodically even when granular self-heal is
disabled or when the volume type is disperse.

Change-Id: Ib92c17350e6531b91ab81477410fe1e7a5856207
BUG: 1331323
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14781
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri

features/index: Do deletion of name-index in unwind path

2016-06-24T04:14:08+00:00

Previously the deletion of name index was being done before
winding the xattrop to posix. This order needs to be changed
to ensure we don't lose the index in the event the xattrop fails
in posix xl.

Change-Id: I4aa892b102cbf5269cede041b871064563cb7348
BUG: 1331323
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/14742
Smoke: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Ravishankar N 
CentOS-regression: Gluster Build System

features/index: Exclude gfid-type for '.', '..'

2016-06-01T07:34:38+00:00

index_get_gfid_type is assuming all names are gfids where as some of these
entry->d_names can be '.' or '..'

Thanks a lot to Nithya for RC

BUG: 1336630
Change-Id: I06ad688a5865ab25b4f6c8a91af8c7fb2ed62186
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/14589
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Ravishankar N 
Reviewed-by: Krutika Dhananjay