glusterfs.git/xlators/mgmt/glusterd/src, branch v3.8rc1

glusterd: add defence mechanism to avoid brick port clashes

2016-05-10T07:11:33+00:00

Intro:
Currently glusterd maintain the portmap registry which contains ports that
are free to use between 49152 - 65535, this registry is initialized
once, and updated accordingly as an then when glusterd sees they are been
used.

Glusterd first checks for a port within the portmap registry and gets a FREE
port marked in it, then checks if that port is currently free using a connect()
function then passes it to brick process which have to bind on it.

Problem:
We see that there is a time gap between glusterd checking the port with
connect() and brick process actually binding on it. In this time gap it could
be so possible that any process would have occupied this port because of which
brick will fail to bind and exit.

Case 1:
To avoid the gluster client process occupying the port supplied by glusterd :

we have separated the client port map range with brick port map range more @
http://review.gluster.org/#/c/13998/

Case 2: (Handled by this patch)
To avoid the other foreign process occupying the port supplied by glusterd :

To handle above situation this patch implements a mechanism to return EADDRINUSE
error code to glusterd, upon which a new port is allocated and try to restart
the brick process with the newly allocated port.

Note: Incase of glusterd restarts i.e. runner_run_nowait() there is no way to
handle Case 2, becuase runner_run_nowait() will not wait to get the return/exit
code of the executed command (brick process). Hence as of now in such case,
we cannot know with what error the brick has failed to connect.

This patch also fix the runner_end() to perform some cleanup w.r.t
return values.

Backport of:
> Change-Id: Iec52e7f5d87ce938d173f8ef16aa77fd573f2c5e
> BUG: 1322805
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/14043
> Tested-by: Prasanna Kumar Kalever 
> Reviewed-by: Atin Mukherjee 
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Raghavendra G 

Change-Id: Id7d8351a0082b44310177e714edc0571ad0f7195
BUG: 1333711
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/14235
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

rpc: define client port range

2016-05-10T07:11:20+00:00

Problem:
when bind-insecure is 'off', all the clients bind to secure ports,
if incase all the secure ports exhaust the client will no more bind
to secure ports and tries gets a random port which is obviously insecure.

we have seen the client obtaining a port number in the range 49152-65535
which are actually reserved as part of glusterd's pmap_registry for bricks,
hence this will lead to port clashes between client and brick processes.

Solution:
If we can define different port ranges for clients incase where secure ports
exhaust, we can avoid the maximum port clashes with in gluster processes.

Still we are prone to have clashes with other non-gluster processes, but
the chances being very low, but that's a different story on its own,
which will be handled in upcoming patches.

Backportof:
> Change-Id: Ib5ce05991aa1290ccb17f6f04ffd65caf411feaf
> BUG: 1322805
> Signed-off-by: Prasanna Kumar Kalever 
> Reviewed-on: http://review.gluster.org/13998
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> Reviewed-by: Raghavendra G 

Change-Id: I2ab9608ddbefcdf5987d817c23dd066010148e19
BUG: 1333711
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/14234
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

glusterd/bitrot: Fix bitrot xlator_type

2016-05-08T18:43:18+00:00

Fix "voltype" value for bitrot in 'glusterd_volopt_map'.
Routines such as 'gd_is_boolean_option' will use
the value of "voltype" to load the shared
library and figure out data type of specific options.
If the value of "voltype" does not match with
the actual shared library name, it will break such
routines.

>Change-Id: I36af4bb3e7043cc87dd19aea1985f3167ff82c18
>BUG: 1332465
>Signed-off-by: Kotresh HR 
>Reviewed-on: http://review.gluster.org/14183
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Atin Mukherjee 
>Reviewed-by: Kaushal M 
>Smoke: Gluster Build System 
>CentOS-regression: Gluster Build System 
(cherry picked from commit 29254449b19697b93ebf376541ea5062b4a99fcb)

Change-Id: Iae7b00272d2d5231ce46944064badacb1f09bc3b
BUG: 1332864
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/14207
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: Niels de Vos

glusterd: remove-brick commit should not succeed when migration failed

2016-05-05T13:30:31+00:00

While remove a brick if the data migration was not successful,
remove-brick commit should not succeed as this can lead to
data loss.

Backport of http://review.gluster.org/#/c/12513/

> Change-Id: I1eac0ef775cc6910ece0e46ebb04051444d54393
> BUG: 1278325
> Signed-off-by: Sakshi Bansal 
> Reviewed-on: http://review.gluster.org/12513
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> Tested-by: Atin Mukherjee 

Change-Id: I1eac0ef775cc6910ece0e46ebb04051444d54393
BUG: 1332370
Signed-off-by: Sakshi Bansal 
Reviewed-on: http://review.gluster.org/14190
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

mgmt/glusterd: Fix op-version for mandatory-locking volume set option

2016-05-05T09:27:54+00:00

The following volume set option will be available with 3.8 release.

locks.mandatory-locking

This one-line change is to fix the wrongly set op-version for the
above mentioned volume set option.

> Reviewed-on: http://review.gluster.org/14180
> NetBSD-regression: NetBSD Build System 
> Smoke: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> CentOS-regression: Gluster Build System 

(cherry picked from commit 717e66a799b70c9b61ad56ab9d27915e265f9d3d)

Change-Id: Icc7d213f917f5f69e2ec84c685b1138ce2f86a90
BUG: 1332788
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/14195
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee

features/locks: Implement mandatory locks

2016-05-03T09:24:58+00:00

Initial change to fix/enable the mandatory locking support in GlusterFS
as per the following design:

https://review.gluster.org/#/c/12014/

Accordingly 'locks.mandatory-locking' option is available as part of this
change which will accept one among the following values:

* off
* file
* forced
* optimal

See design doc for more details

> Reviewed-on: http://review.gluster.org/9768
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Poornima G 
> Reviewed-by: Raghavendra Talur 
> Reviewed-by: Rajesh Joseph 
> Reviewed-by: Pranith Kumar Karampuri 

(cherry picked from commit 4517bf8dd6de310950cc5a612955aa3a2fddb57e)

Change-Id: I14c489b3f8af5ebcbfa155a03f0c175e9558ac46
BUG: 1332162
Signed-off-by: Anoop C S 
Reviewed-on: http://review.gluster.org/14149
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

WORM/Retention Translator: Implementation of file level WORM

2016-05-02T01:05:22+00:00

To activate the file level worm feature, the features.read-only and
features.worm options should be switched "off" on the volume and
the features.worm-file-level should be switched "on". Both read-only
and worm or worm-file-level cannot be switched "on" together. The
files which are created when the worm-file-level option is set on the
volume will have their own retention profile.

If both worm and worm-file-level are "on" at that time the worm
which is the volume level worm will have priority over file level
worm. If worm-file level is switched "off" after some time and the
read-only option is switched "on" then read-only will have priority.

The current implementation allows the users to manually transmit
a file to a WORM-Retained state by removing all the write bits of
the file using the chmod command. The file will have a retention
profile which contains the state of the file, mode of retention,
and the default retention time.

The file will be made WORM-Retained for a default of 120 seconds
during which it will be immutable and undeletable and it sets the
atime of the file to the time till which it is retained.
After that period if any fop request comes for that file, will
make the transition from WORM-Retained state to WORM state, where
the file will be immutable but deletable and, it will reset
the atime to the actual atime of the file. If a WORM file needs
to be made undeletable again, it can be done by using the chmod
command with all the write bits removed.

There are two modes of retention:
1. Relax: where the retention time of a WORM-Retained file can be
   increased or decreased.
2. Enterprise: where the retention time of a WORM-Retained file
   can be increased but not be decreased.
Whenever a utime change(touch -a, -t, ...)request comes for a
file it checks the mode of retention before setting the utimes.
This is done only if the file is WORM-Retained but for a WORM file
it will change the utimes.

Lazy auto commit:
Whenever a file gets created it will store the creation time of the
file or if a file already exists then any of the next unlink, link,
truncate or rename fops will set the current time as the start time
in an xattr. The next rename/unlink/truncate/link call will check for the
auto commit period and if is is expired, then it will automatically do
the state transition. If it is a normal file then it gets converted
to WORM-Retained state. If it is a WORM-Retained file and its retention
period is expired, then it gets converted to WORM state.

Added the volume set options for the WORM translator. It allows the users
to change the default values of auto-commit-period, default-retention-period,
retention-mode. To make use of the file-level WORM first we have to set the
'worm-file' option to 'on'. The files which are created when the worm-file
option is set on the volume will get WORM-Retained. Other files will work
as usual and will not be WORMed. The auto-commit-period, retention-mode,
and the default-retention-period values for the file will be set to the values
which are set on the volume when the file is created.

Added the tests to check the basic functionalities of the WORM/Retention feature.

Change-Id: I77bd9777f9395a944d76b5cc35a5b48a3c14d148
BUG: 1326308
Signed-off-by: karthik-us 
Reviewed-on: http://review.gluster.org/13429
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

glusterd: volume set changes for lock migration

2016-05-02T01:05:03+00:00

Change-Id: I48c6f9cdda47503615ba65882acd5eedf0a70c89
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14024
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/afr: Entry self-heal performance enhancements

2016-04-30T01:21:56+00:00

Change-Id: I52da41dff5619492b656c2217f4716a6cdadebe0
BUG: 1269461
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/12442
Reviewed-by: Pranith Kumar Karampuri 
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

glusterd: persist brickinfo->real_path

2016-04-29T16:17:22+00:00

Since real_path was not persisted and gets constructed at every glusterd
restart, glusterd will fail to come up if one of the brick's underlying file
system is crashed.

Solution is to construct real_path only once and get it persisted.

Change-Id: I97abc30372c1ffbbb2d43b716d7af09172147b47
BUG: 1330481
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/14075
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Kaushal M 
NetBSD-regression: NetBSD Build System