glusterfs.git/tests/features, branch release-3.8-fb

cluster/afr: Handle gfid-less directories in heal flow

2017-07-12T17:04:32+00:00

Summary:
- Updates heal flow to handle case where a directory does not have a
  gfid assigned.  In this case we will remove _only_ empty directories
  in these cases such that the parent can re-gain consistency and files
  within can be correctly healed.
- Also adds a test for the case where a file does not have a gfid, this
  is already handles by the metadata heal flow, but tests were lacking
  for this code path.

Test Plan:
- prove -v tests/basic/shd_autofix_nogfid.t
- prove -v tests/basic/gfid_unsplit_shd.t

Reviewers: dph, moox, sshreyas

Reviewed By: sshreyas

Differential Revision: https://phabricator.fb.com/D2502067

Tasks: 8549168

Change-Id: I8dd3e6a6d62807cb38aafe597eced3d4b402351b
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/17750
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Jeff Darcy

Merge remote-tracking branch 'origin/release-3.8' into merge-3.8-again

2017-01-05T20:21:20+00:00

Change-Id: I844adf2aef161a44d446f8cd9b7ebcb224ee618a
Signed-off-by: Kevin Vigor

storage/posix: Add free space limits to bricks

2016-12-19T19:06:05+00:00

Summary:
- Add a configurable minimum free space for bricks, using the new
  options storage.min-free-disk (analagous to cluster.min-free-disk,
  and using the same units: either a percentage or an
  absolute number of bytes) and storage.freespace-check-interval
  (how frequently to check free space, in seconds).
- This is a cherry-pick of D2920210 to 3.8

Signed-off-by: Shreyas Siravara 

Change-Id: I4b87e421aad023e49b5972c6e61539670a818411
Reviewed-on: http://review.gluster.org/16176
Tested-by: Shreyas Siravara 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
Reviewed-by: Kevin Vigor 
CentOS-regression: Gluster Build System

features/locks: Add lock revocation functionality to posix locks translator

2016-12-09T17:06:18+00:00

Summary:
- Motivation: Prevents cluster instability by mis-behaving clients
causing bricks to OOM due to inode/entry lock pile-ups.
- Adds option to strip clients of entry/inode locks after N seconds
- Adds option to clear ALL locks should the revocation threshold get hit
- Adds option to clear all or granted locks should the max-blocked
threshold get hit (can be used in combination w/ revocation-clear-all).
- Options are:
features.locks-revocation-secs
features.locks-revocation-clear-all [on/off]
features.locks-revocation-max-blocked
- Adds monkey-locking option to ignore 1% of unlock requests (dev only)
features.locks-monkey-unlocking [on/off]
- Adds logging to indicate revocation event & reason

Test Plan:
First you will need TWO fuse mounts for this repro. Call them /mnt/patchy1 & /mnt/patchy2.

1. Enable monkey unlocking on the volume:
gluster vol set patchy features.locks-monkey-unlocking on

2. From the "patchy1", use DD or some other utility to begin writing to a file,
eventually the dd will hang due to the dropped unlocked requests. This now
simulates the broken client. Run:

for i in {1..1000};do dd if=/dev/zero of=/mnt/patchy1/testfile bs=1k count=10;done'

...this will eventually hang as the unlock request has been lost.

3. Goto another window and setup the mount "patchy2" @ /mnt/patchy2, and
observe that 'echo "hello" >> /mnt/patchy2/testfile" will hang due to the
inability of the client to take out the required lock.

4. Next, re-start the test this time enabling lock revocation; use a timeout of
2-5 seconds for testing:
'gluster vol set patchy features.locks-revocation-secs <2-5>'

5. Wait 2-5 seconds before executing step 3 above this time. Observe that this
time the access to the file will succeed, and the writes on patchy1 will
unblock until they hit another failed unlock request due to
"monkey-unlocking".

Change-Id: I814b9f635fec53834a26db634d1300d9a61057d8
Signed-off-by: Pranith Kumar K
Reviewed-on: http://review.gluster.org/14816
NetBSD-regression: NetBSD Build System
Reviewed-by: Krutika Dhananjay
CentOS-regression: Gluster Build System
Smoke: Gluster Build System
Reviewed-on: http://review.gluster.org/16086
Tested-by: Shreyas Siravara
Reviewed-by: Kevin Vigor

glusterd: clean up old port and allocate new one on every restart

2016-11-23T10:37:16+00:00

    Backport of http://review.gluster.org/#/c/15005/9.

GlusterD as of now was blindly assuming that the brick port which was
already allocated would be available to be reused and that assumption
is absolutely wrong.

Solution : On first attempt, we thought GlusterD should check if the
already allocated brick ports are free, if not allocate new port and
pass it to the daemon. But with that approach there is a possibility
that if PMAP_SIGNOUT is missed out, the stale port will be given back
to the clients where connection will keep on failing. Now given the
port allocation always start from base_port, if everytime a new port
has to be allocated for the daemons, the port range will still be
under control. So this fix tries to clean up old port using
pmap_registry_remove () if any and then goes for pmap_registry_alloc ()

This patch is being ported to 3.8 branch because, the brick process
blindly re-using old port, without registering with the pmap server,
causes snapd daemon to not start properly, even though snapd registers
with the pmap server. With this patch, all the brick processes and
snapd will register with the pmap server to either get the same port,
or a new port, and avoid port collision.

> Reviewed-on: http://review.gluster.org/15005
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Avra Sengupta 

(cherry picked from commit c3dee6d35326c6495591eb5bbf7f52f64031e2c4)

Change-Id: If54a055d01ab0cbc06589dc1191d8fc52eb2c84f
BUG: 1369766
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/15308
Tested-by: Avra Sengupta 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos

glusterd: volgen and volume set changes for leases

2016-06-13T11:23:13+00:00

Add leases xlator in volgen and also add corresponding volume set options

Backport of http://review.gluster.org/#/c/11722

Cherry picked from commit bc525b01002d7649be0ac28c5761c8ce92b6a658:
> Change-Id: Ic5de50cdb87eaf6a833e739bc7e08fecbeca3de3
> BUG: 1319992
> Signed-off-by: Poornima G 
> Reviewed-on: http://review.gluster.org/11722
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Atin Mukherjee 

Change-Id: Ic5de50cdb87eaf6a833e739bc7e08fecbeca3de3
BUG: 1342350
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/14626
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Atin Mukherjee 
Smoke: Gluster Build System

build: remove jbr and fdl, they is not ready for 3.8

2016-05-27T14:44:54+00:00

Change-Id: Ia24ad18c43d56a751988e562323ede26d7785848
BUG: 1317278
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/14519
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

features/locks: Implement mandatory locks

2016-05-03T09:24:58+00:00

Initial change to fix/enable the mandatory locking support in GlusterFS
as per the following design:

https://review.gluster.org/#/c/12014/

Accordingly 'locks.mandatory-locking' option is available as part of this
change which will accept one among the following values:

* off
* file
* forced
* optimal

See design doc for more details

> Reviewed-on: http://review.gluster.org/9768
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Poornima G 
> Reviewed-by: Raghavendra Talur 
> Reviewed-by: Rajesh Joseph 
> Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit 4517bf8dd6de310950cc5a612955aa3a2fddb57e)

Change-Id: I14c489b3f8af5ebcbfa155a03f0c175e9558ac46
BUG: 1332162
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/14149
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

WORM/Retention Translator: Implementation of file level WORM

2016-05-02T01:05:22+00:00

To activate the file level worm feature, the features.read-only and
features.worm options should be switched "off" on the volume and
the features.worm-file-level should be switched "on". Both read-only
and worm or worm-file-level cannot be switched "on" together. The
files which are created when the worm-file-level option is set on the
volume will have their own retention profile.

If both worm and worm-file-level are "on" at that time the worm
which is the volume level worm will have priority over file level
worm. If worm-file level is switched "off" after some time and the
read-only option is switched "on" then read-only will have priority.

The current implementation allows the users to manually transmit
a file to a WORM-Retained state by removing all the write bits of
the file using the chmod command. The file will have a retention
profile which contains the state of the file, mode of retention,
and the default retention time.

The file will be made WORM-Retained for a default of 120 seconds
during which it will be immutable and undeletable and it sets the
atime of the file to the time till which it is retained.
After that period if any fop request comes for that file, will
make the transition from WORM-Retained state to WORM state, where
the file will be immutable but deletable and, it will reset
the atime to the actual atime of the file. If a WORM file needs
to be made undeletable again, it can be done by using the chmod
command with all the write bits removed.

There are two modes of retention:
1. Relax: where the retention time of a WORM-Retained file can be
   increased or decreased.
2. Enterprise: where the retention time of a WORM-Retained file
   can be increased but not be decreased.
Whenever a utime change(touch -a, -t, ...)request comes for a
file it checks the mode of retention before setting the utimes.
This is done only if the file is WORM-Retained but for a WORM file
it will change the utimes.

Lazy auto commit:
Whenever a file gets created it will store the creation time of the
file or if a file already exists then any of the next unlink, link,
truncate or rename fops will set the current time as the start time
in an xattr. The next rename/unlink/truncate/link call will check for the
auto commit period and if is is expired, then it will automatically do
the state transition. If it is a normal file then it gets converted
to WORM-Retained state. If it is a WORM-Retained file and its retention
period is expired, then it gets converted to WORM state.

Added the volume set options for the WORM translator. It allows the users
to change the default values of auto-commit-period, default-retention-period,
retention-mode. To make use of the file-level WORM first we have to set the
'worm-file' option to 'on'. The files which are created when the worm-file
option is set on the volume will get WORM-Retained. Other files will work
as usual and will not be WORMed. The auto-commit-period, retention-mode,
and the default-retention-period values for the file will be set to the values
which are set on the volume when the file is created.

Added the tests to check the basic functionalities of the WORM/Retention feature.

Change-Id: I77bd9777f9395a944d76b5cc35a5b48a3c14d148
BUG: 1326308
Signed-off-by: karthik-us 
Reviewed-on: http://review.gluster.org/13429
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd: volume set changes for lock migration

2016-05-02T01:05:03+00:00

Change-Id: I48c6f9cdda47503615ba65882acd5eedf0a70c89
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14024
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System