glusterfs.git/xlators/cluster/ec/src/ec.h, branch release-3.7

cluster/ec: Provide an option to enable/disable eager lock

2016-03-21T05:52:53+00:00

Problem: If a fop takes lock, and completes its operation,
it waits for 1 second before releasing the lock. However,
If ec find any lock contention within this time period,
it release the lock immediately before time expires. As we
take lock on first brick, for few operations, like read, it
might happen that discovery of lock contention might take
long time and can degrades the performance.

Solution: Provide an option to enable/disable eager lock.
If eager lock is disabled, lock will be released as soon
as fop completes.

gluster v set  disperse.eager-lock on
gluster v set  disperse.eager-lock off

master-
http://review.gluster.org/13605

Change-Id: I000985a787eba3c190fdcd5981dfbf04e64af166
BUG: 1318965
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/13773
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Pranith Kumar Karampuri

cluster/ec: Create this->itable in all cases

2016-01-31T02:27:07+00:00

Problem:
glfsheal operates based on mount's volfile which doesn't have iamshd flag due
to which this->itable is NULL, this leads to "inode not found" logs in glfsheal
logs.

Fix:
Ec only allocates itable with 10 inodes, so allocating this->itable in all
cases in init.

 >Change-Id: I01d3c05e93a17007a4716a2d6f392d2aa306a34b
 >BUG: 1294743
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/13112
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >(cherry picked from commit 080ddb79d8805253a7f4274606351570faae1add)

Change-Id: I4fb624fc26d47128221322da077d04b12add6452
Signed-off-by: Pranith Kumar K 
BUG: 1302943
Reviewed-on: http://review.gluster.org/13312
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

cluster/ec: Implement gfid-hash read-policy

2015-10-29T11:52:49+00:00

Add a policy in ec to performs reads from same bricks as long as they
are good. Based on the gfid of the file/directory it determines the
bricks to be considered for reading.

 >Change-Id: Ic97b5c54c086a28b5e07a330a4fd448551b49376
 >BUG: 1261260
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/12133
 >Tested-by: NetBSD Build System 
 >Tested-by: Gluster Build System 
 >Reviewed-by: Xavier Hernandez 

BUG: 1270705
Change-Id: Ibf0d21d7210125fa7aaa12b3f98bcdf7cd89ef02
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/12456
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System

cluster/ec: Make background healing optional behavior

2015-07-21T21:37:38+00:00

Provide options to control number of active background heal count and qlen.

 >Change-Id: Idc2419219d881f47e7d2e9bbc1dcdd999b372033
 >BUG: 1237381
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/11473
 >Reviewed-by: Xavier Hernandez 
 >Tested-by: Gluster Build System 

BUG: 1238476
Change-Id: I22ba902d9911195656db9e458c01b54cf0afcd7a
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/11680
Tested-by: Gluster Build System 
Reviewed-by: Xavier Hernandez

cluster/ec: Add throttling in background healing

2015-07-21T21:35:37+00:00

- 8 parallel heals can happen.
- 128 heals will wait for their turn
- Heals will be rejected if 128 heals are already waiting.

 >Change-Id: I2e99bf064db7bce71838ed9901a59ffd565ac390
 >BUG: 1237381
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/11471
 >Tested-by: Gluster Build System 
 >Reviewed-by: Xavier Hernandez 
 >Tested-by: NetBSD Build System 

BUG: 1238476
Change-Id: Id9625536197c1f5d6c3a9ed53f80bd9e6d14c507
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/11679
Tested-by: Gluster Build System 
Reviewed-by: Xavier Hernandez

cluster/ec: Fix all EIO errors in EC

2015-05-28T11:12:06+00:00

        Backport of http://review.gluster.org/10770
        Backport of http://review.gluster.org/10806
        Backport of http://review.gluster.org/10787
        Backport of http://review.gluster.org/10868
        Backport of http://review.gluster.com/10852

 - When a blocking lock is requested, lock request is succeeded even when
ec->fragment number of locks are acquired successfully in non-blocking locking
phase. This will lead to fop succeeding only on the bricks where the locks are
acquired, leading to the necessity of self-heals. To prevent these un-necessary
self-heals, if the remaining locks fail with EAGAIN in non-blocking lock phase
try blocking locking phase instead.

 -  Handle lookup failures while op in progress

 - cluster/ec: Correctly cleanup delayed locks
When a delayed lock is pending, a graph switch doesn't correctly
terminate it. This means that the update of version and size xattrs
is lost, causing EIO errors. This patch handles GF_EVENT_PARENT_DOWN
event to correctly finish pending udpdates before completing the
graph switch.

 - Fix use after free crash
ec_heal creates ec_fop_data but doesn't run ec_manager. ec_fop_data_allocate
adds this fop to ec->pending_fops, because ec_manager is not run on this heal
fop it is never removed from ec->pending_fops. When it is accessed after free
it leads to crash. It is better to not to add HEAL fops to ec->pending_fops
because we don't want graph switch to hang the mount because of a BIG
file/directory heal.

- Forced unlock when lock contention is detected
EC uses an eager lock mechanism to optimize multiple read/write
requests on the same entry or inode. This increases performance
but can have adverse results when other clients try to access the
same entry/inode. To solve this, this patch adds a functionality
to detect when this happens and force an earlier release to not
block other clients.

The method consists on requesting GF_GLUSTERFS_INODELK_COUNT and
GF_GLUSTERFS_ENTRYLK_COUNT for all fops that take a lock. When this
count is greater than one, the lock is marked to be released. All
fops already waiting for this lock will be executed normally before
releasing the lock, but new requests that also require it will be
blocked and restarted after the lock has been released and reacquired
again.

Another problem was that some operations did correctly lock the
parent of an entry when needed, but got the size and version xattrs
from the entry instead of the parent.

This patch solves this problem by binding all queries of size and
version to each lock and replacing all entrylk calls by inodelk ones
to remove concurrent updates on directory metadata.  This also allows
rename to correctly update source and destination directories.

BUG: 1225279
Change-Id: I02a6084b138dd38e018a462347cd9ce38610c7ef
Reviewed-on: http://review.gluster.org/10926
Tested-by: NetBSD Build System
Tested-by: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

cluster/ec: add separate versions for data/entry, metadata

2015-05-08T17:29:38+00:00

Adding 64 bits in "version" key of extended attributes. First 64 bits (Left)
represents Data version. Last 64 bits (right) represents Meta Data version.

Note: 3.7 and 3.6 version ec can't co-exist with this change because xattrop in
3.6 will fail with ERANGE as the buffer passed to it will be '8' bytes where as
the value will be 16 bytes in 3.7. Where as 3.7 version clients can work with
old version files. For upgrades we need to tell users to complete heals and
then upgrade

BUG: 1215265
Change-Id: Ib85114680cb7e75b8371c984d9f7b6401c1ffb93
Signed-off-by: Ashish Pandey 
Reviewed-on: http://review.gluster.org/10312
Reviewed-on: http://review.gluster.org/10626
Tested-by: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

cluster/ec: Fix readdir de-itransform

2015-04-11T09:11:15+00:00

Problem:
gf_deitransform returns the glbal client-id in the complete graph. So except
for the first disperse subvolume under dht, all the other disperse subvolumes
will return a client-id greater than ec->nodes, so readdir will always error
out in those subvolumes.

Fix:
Get the client subvolume whose client-id matches the client-id returned by
gf_deitransform of offset.

Change-Id: I26aa17504352d48d7ff14b390b62f49d7ab2d699
BUG: 1209113
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/10165
Tested-by: Gluster Build System 
Reviewed-by: Xavier Hernandez

libxlator: Change marker xattr handling interface

2015-03-25T18:10:40+00:00

- Changed the implementation of marker xattr handling to take just a
  function which populates important data that is different from
  default 'gauge' values and subvolumes where the call needs to be
  wound.
- Removed duplicate code I found while reading the code and moved it to
  cluster_marker_unwind. Removed unused structure members.
- Changed dht/afr/stripe implementations to follow the new implementation
- Implemented marker xattr handling for ec.

Change-Id: Ib0c3626fe31eb7c8aae841eabb694945bf23abd4
BUG: 1200372
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/9892
Tested-by: Gluster Build System 
Reviewed-by: Xavier Hernandez 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Ravishankar N 
Reviewed-by: Vijay Bellur

cluster/ec: Add self-heal-daemon command handlers

2015-03-09T22:36:31+00:00

This patch introduces the changes required in ec xlator to handle
index/full heal.

Index healer threads:
Ec xlator start an index healer thread per local brick. This thread keeps
waking up every minute to check if there are any files to be healed based on
the indices kept in index directory. Whenever child_up event comes, then also
this index healer thread wakes up and crawls the indices and triggers heal.
When self-heal-daemon is disabled on this particular volume then the healer
thread keeps waiting until it is enabled again to perform heals.

Full healer threads:
Ec xlator starts a full healer thread for the local subvolume provided by
glusterd to perform full crawl on the directory hierarchy to perform heals.
Once the crawl completes the thread exits if no more full heals are issued.

Changed xl-op prefix GF_AFR_OP to GF_SHD_OP to make it more generic.

Change-Id: Idf9b2735d779a6253717be064173dfde6f8f824b
BUG: 1177601
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/9787
Tested-by: Gluster Build System 
Reviewed-by: Ravishankar N 
Reviewed-by: Vijay Bellur