glusterfs.git/xlators/performance/md-cache/src, branch exp

dht/md-cache: Filter invalidate if the file is made a linkto file

2016-12-02T10:46:58+00:00

Upcall as a part of setattr, sends an invalidation and the
invalidation carries the resulting stat value. When a file
is converted to linkto files, even then an invalidation
is set and as a result the mountpoint shows the sticky
bit in the stat of the file.
eg: ---------T. 945 root root 0 Nov  8 10:14 hardlink.999

Fix:
When dht recieves a notification of sticky bit change, it updates
the flag, to indicate md-cache to send the subsequent lookup.

Change-Id: Ic2fd7a5b196db0754f9b97072e644e6bf69da606
BUG: 1392713
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15789
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
Reviewed-by: Susant Palai 
Reviewed-by: Rajesh Joseph

afr,dht,ec: Replace GF_EVENT_CHILD_MODIFIED with event SOME_DESCENDENT_DOWN/UP

2016-11-21T09:32:05+00:00

Currently these are few events related to child_up/down:
GF_EVENT_CHILD_UP :  Issued when any of the protocol client
connects.
GF_EVENT_CHILD_MODIFIED : Issued by afr/dht/ec
GF_EVENT_CHILD_DOWN : Issued when any of the protocol client
disconnects.
These events get modified at the dht/afr/ec layers. Here is a
brief on the same.

DHT:
- All the subvolumes reported once, and atleast one child came
  up, then GF_EVENT_CHILD_UP is issued
- connect GF_EVENT_CHILD_UP is issued
- disconnect GF_EVENT_CHILD_MODIFIED is issued
- All the subvolumes disconnected, GF_EVENT_CHILD_DOWN is issued

AFR:
- First subvolume came up, then GF_EVENT_CHILD_UP is issued
- Subsequent subvolumes coming up, results in GF_EVENT_CHILD_MODIFIED
- Any of the subvolumes go down, then GF_EVENT_SOME_CHILD_DOWN is issued
- Last up subvolume goes down, then GF_EVENT_CHILD_DOWN is issued

Until the patch [1] introduced GF_EVENT_SOME_CHILD_UP,
GF_EVENT_CHILD_MODIFIED was issued by afr/dht when any of the subvolumes
go up or down.

Now with md-cache changes, there is a necessity to differentiate between
child up and down. Hence, introducing GF_EVENT_SOME_DESCENDENT_DOWN/UP and
getting rid of GF_EVENT_CHILD_MODIFIED.

[1] http://review.gluster.org/12573

Change-Id: I704140b6598f7ec705493251d2dbc4191c965a58
BUG: 1396038
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15764
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Rajesh Joseph

md-cache: Invalidate cache entry for open() with O_TRUNC

2016-10-27T05:46:32+00:00

When a file is opened with O_TRUNC flag set, its size gets
set to '0'. This case needs to be handled in md-cache to
avoid sending incorrect cached stat.

Change-Id: I95d1f8a6634734898883ede010c3e7b0b7eb97d9
BUG: 1382266
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/15618
Smoke: Gluster Build System 
Reviewed-by: jiffin tony Thottan 
Tested-by: jiffin tony Thottan 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

xlators/md-cache: fix unused variable warnings/errors

2016-09-14T06:32:39+00:00

http://review.gluster.org/14085 fixes a "pragma leak" where the
generated rpc/xdr headers have a pair of pragmas that disable these
warnings. With the warnings disabled, many unused variables have
crept into the code base.

And 14085 won't pass its own smoke test until all these warnings are
fixed.

BUG: 1369124
Change-Id: I5904956b2033993abee0a29ff615e058a52c9ac0
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/15481
Reviewed-by: Prashanth Pai 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

md-cache: Process all the cache invalidation flags

2016-08-31T06:40:52+00:00

Currently, md-cache only processes IATT_UPDATE_FLAGS, UP_XATTR and
UP_XATTR_RM. We also need to process UP_RENAME_FLAGS, UP_FORGET,
UP_PARENT_DENTRY_FLAGS and UP_NLINK_FLAGS. Otherwise the files
unlinked or renamed will not be reflected on other mounts.

Change-Id: Icb8b03da51482c3fc2e2a7292d16d56e11a341d9
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15324
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

dht, md-cache, upcall: Add invalidation of IATT when the layout changes

2016-08-31T06:08:54+00:00

Issue:
dht_layout is built as a part of lookup only. The layout can be
modified by rebalance process. Since every IO fop is preceded
by a lookup, there are very less issues of stale layout. But
with enhancements of aggressive caching of stats in md-cache,
the lookup will reduce and expose the stale layout issue often.

Solution:
Since stale layout is already an issue on dht, there is already
a plan to fix this at the dht layer, but this fix is not currently
planned for any release. Until this fix comes out, we can have
a workaround where, the upcall will send a notification to md-cache
when a layout xattr is changed. As a part of layout change notification
the existing cache is invalidated and the next lookup will fetch the
latest layout.

This is not a foolproof solution as the window between the layout change
and the next lookup(after invalidation of stat), where there will be stale
layout. But until the final fix comes in, this reduces the stale layout
window.

Change-Id: Iacf871a38b35880c1fc0bc68fe7ce291265e71d4
BUG: 1369638
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15300
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

md-cache: Register the list of xattrs with cache-invalidation

2016-08-31T06:07:01+00:00

Issue:
md-cache caches a specified list of xattrs, and when cache invalidation
is enabled, it makes sense to recieve invalidation only when those xattrs
are modified by other clients. But the current implementation of upcall
is that, it will send invalidation when any of the on-disk xattrs is modified.

Solution:
md-cache sends a list of xattrs that it is interested in, to upcall by
issuing an ipc(). The challenge here is to make sure everytime a brick
goes offline and comes back up, the ipc() needs to be issued to the
bricks. Hence ipc() is sent from md-cache every time there is a
CHILD_UP/CHILD_MODIFIED event.

TODO:
There will be patches following, in cluster xlators, to implement ipc fop.

Change-Id: I6efcf3df474f5ce6eabd3d6694c00c7bd89bc25d
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15002
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Rajesh Joseph 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Prashanth Pai 
Reviewed-by: Raghavendra G

md-cache: Fix wrong cache time update for xattrs

2016-08-27T12:56:35+00:00

In md-cache, the cache has two times:
1. Time when the stat was last fetched for that inode
2. Time when the xattrs were last fetched for that inode. This
time should not be updated when only one of the xattrs is updated.
If, its updated when one of the cached xattr is changed, it can so
happen that the other xattrs have past their cache timeout, but are
still served from cache.

Solution:
Do not update the xattr cache time, when one of the xattrs being cached
is changed. With this, we may end up in cache timeout though it was
updated recently, but it is not a harm. The other way is to have
timeout for every xattr that is being cached. Its more complicated, and
may be not worth it, as we have lot of lookup fops, that are overloaded to
get all the xattrs.

Change-Id: Id77e547f403fc792348f1ea56b468b9260a5a34f
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15045
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

md-cache: Do not use features.cache-invalidation for both md-cache and upcall

2016-08-27T12:18:49+00:00

Currently, the volume set option features.cache-invalidation enables upcall
feature on server side and md-cache cache-invalidation on client side.
There are multiple problems that can arise from this:
1. The scenario when user wants to, enable upcall for nfs-ganesha setup,
   but do not want to enable md-cache cache-invalidation, as the
   nfs-clients have already cached the metadata and upcall is used to
   to invalidate the nfs-client cache. In this case, users should have
   a way of disabling md-cache invalidation without disabling upcall.

2. Upcall requires a op-version of GD_OP_VERSION_3_7_0, where as
   md-cache invalidation requires an op version of GD_OP_VERSION_3_9_0.
   Consider a setup where the servers are in op-version GD_OP_VERSION_3_7_0,
   and th clients are in op-version GD_OP_VERSION_3_9_0. if there is one
   single volume set option, user can enable this feature in this setup.
   But it can lead to stale xattr cache as the xattr invalidation was
   introduced in upcall only in release 3.8. Hence, we should not be
   able to enable md-cache invalidation, if all the servers and clients
   are not on opversion >= GD_OP_VERSION_3_9_0.

To solve the above mentioned issues, we have seperate volume options
for enabling md-cache invalidation and upcall. But this can lead to
issues when user enable md-cache invalidation and forgets to enable
upcall. Probably in the next release, these can be enables by default.

Change-Id: Ie70eff97fe12fcb623eec8f4f5861ac065bf483e
BUG: 1211863
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15314
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: soumya k 
Reviewed-by: Raghavendra G