glusterfs.git/xlators/cluster/dht/src/dht-common.c, branch v3.7.0beta1

Cluster/DHT Mismatching gfid values in dht_local_t

2015-04-08T19:45:55+00:00

If multiple files with the same name but different gfids
exist on different subvolumes, dht_lookup_everywhere_cbk()
copies the gfid from the last received response into
local->gfid but does not update the local->stbuf structure.
dht_linkfile_create() uses the value in local->gfid, but
dht_linkfile_attr_heal() uses the one in local->stbuf, causing
a mismatch and failure while trying to heal the linkfile attrs.

Change-Id: I80d152be95b42d736c5d9182b955f42e374b82a5
BUG: 1205785
Signed-off-by: Nithya Balachandran 
Reviewed-on: http://review.gluster.org/9998
Reviewed-by: Susant Palai 
Reviewed-by: Shyamsundar Ranganathan 
Tested-by: Shyamsundar Ranganathan

dht : null dereference coverity fix

2015-04-08T19:19:38+00:00

Change-Id: I700e7ebdfe4929a6d74406ea081059bdddcf7a79
BUG: 789278
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9628
Reviewed-by: N Balachandran 
Tested-by: Gluster Build System 
Reviewed-by: Humble Devassy Chirammal 
Reviewed-by: Shyamsundar Ranganathan

cluster/dht: Unwind with proper op_ret

2015-04-06T13:27:13+00:00

1. Expected behavior of get_real_filename feature.

A getxattr on a existing dir with glusterfs.get_real_filename:
as key should result in one of the following things.

a. A value returned for that key having the real filename (a file whose
match is a case insensitive match to the filename passed in key).
b. op_ret = -1 and errno set to ENOENT meaning that no such file exists
under the specified dir in any case.
c. op_ret = -1 and errno set to ENODATA. This is a case assuming no
xlator interprets the glusterfs.get_real_filename key and it get
passed down to the posix xlator. Naturally, posix xlator would not
find any xattr with this key and would return ENODATA. This will be
interpreted specially by the caller as the feature not being supported
by underlying glusterfs.

2. What assumptions are wrong?
Initially the key used to be user.glusterfs.get_real_filename.
In that case, when posix xlator did a getxattr call it would have
received ENODATA as error. However, the key has now changed to
glusterfs.get_real_filename. This leads to a EOPNOTSUPP error instead.

Considering the above information, this is a rewrite of
get_real_filename logic in dht.

Change-Id: I012e9150047fc8563be91b0d112a368ac1cbf598
BUG: 1204140
Signed-off-by: Raghavendra Talur 
Reviewed-on: http://review.gluster.org/9956
Tested-by: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Shyamsundar Ranganathan

Avoid conflict between contrib/uuid and system uuid

2015-04-04T17:48:35+00:00

glusterfs relies on Linux uuid implementation, which
API is incompatible with most other systems's uuid. As
a result, libglusterfs has to embed contrib/uuid,
which is the Linux implementation, on non Linux systems.
This implementation is incompatible with systtem's
built in, but the symbols have the same names.

Usually this is not a problem because when we link
with -lglusterfs, libc's symbols are trumped. However
there is a problem when a program not linked with
-lglusterfs will dlopen() glusterfs component. In
such a case, libc's uuid implementation is already
loaded in the calling program, and it will be used
instead of libglusterfs's implementation, causing
crashes.

A possible workaround is to use pre-load libglusterfs
in the calling program (using LD_PRELOAD on NetBSD for
instance), but such a mechanism is not portable, nor
is it flexible. A much better approach is to rename
libglusterfs's uuid_* functions to gf_uuid_* to avoid
any possible conflict. This is what this change attempts.

BUG: 1206587
Change-Id: I9ccd3e13afed1c7fc18508e92c7beb0f5d49f31a
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/10017
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

Xlators : Fixed typos

2015-04-02T10:21:08+00:00

Change-Id: I948f85cb369206ee8ce8b8cd5e48cae9adb971c9
BUG: 1075417
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9529
Reviewed-by: Niels de Vos 
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: Humble Devassy Chirammal

libxlator: Change marker xattr handling interface

2015-03-25T18:10:40+00:00

- Changed the implementation of marker xattr handling to take just a
  function which populates important data that is different from
  default 'gauge' values and subvolumes where the call needs to be
  wound.
- Removed duplicate code I found while reading the code and moved it to
  cluster_marker_unwind. Removed unused structure members.
- Changed dht/afr/stripe implementations to follow the new implementation
- Implemented marker xattr handling for ec.

Change-Id: Ib0c3626fe31eb7c8aae841eabb694945bf23abd4
BUG: 1200372
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/9892
Tested-by: Gluster Build System 
Reviewed-by: Xavier Hernandez 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Ravishankar N 
Reviewed-by: Vijay Bellur

cluster/dht: Add tier translator.

2015-03-21T16:50:29+00:00

The tier translator shares most of DHT's code. It differs in how
subvolumes are chosen for I/Os, and how file migration (cache promotion
and demotion) is managed. That different functionality is split to either
DHT or tier logic according to the "tier_methods" structure.

A cache promotion and demotion thread is created in a manner
similar to the rebalance daemon. The thread operates a timing
wheel which periodically checks for promotion and demotion candidates
(files). Candidates are queued and then migrated. Candidates must exist on
the same node as the daemon and meet other critera per caching policies.

This patch has two authors (Dan Lambright and Joseph Fernandes). Dan
did the DHT changes and Joe wrote the cache policies. The fix depends on
DHT readidr changes and the database library which have been submitted
separately.  Header files in libglusterfs/src/gfdb should be reviewed in
patch 9683.

For more background and design see the feature page [1].

[1]
http://www.gluster.org/community/documentation/index.php/Features/data-classification

Change-Id: Icc26c517ccecf5c42aef039f5b9c6f7afe83e46c
BUG: 1194753
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9724
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

features/quota : Introducing inode quota

2015-03-19T01:24:12+00:00

==========================================================================
                             Inode quota
==========================================================================
= Currently, the only way to retrieve the number of files/objects in a   =
= directory or volume is to do a crawl of the entire directory/volume.   =
= This is expensive and is not scalable.                                 =
=                                                                        =
= The proposed mechanism will provide an easier alternative to determine =
= the count of files/objects in a directory or volume.                   =
=                                                                        =
= The new mechanism proposes to store count of objects/files as part of  =
= an extended attribute of a directory. Each directory's extended        =
= attribute value will indicate the number of files/objects present      =
= in a tree with the directory being considered as the root of the tree. =
=                                                                        =
= The count value can be accessed by performing a getxattr().            =
= Cluster translators like afr, dht and stripe will perform aggregation  =
= of count values from various bricks when getxattr() happens on the key =
= associated with file/object count.                                     =

A new interface is introduced:
------------------------------
        limit-objects  : limit the number of inodes at directory level
        list-objects   : list the directories where the limit is set
        remove-objects : remove the limit from the directory

==========================================================================

CLI COMMAND:
gluster volume quota  limit-objects   []

*  is a hard-limit for number of objects limitation for path ""
  If hard-limit is exceeded, creation of file/directory is no longer
permitted.

*  is a soft-limit for number of objects creation for path ""
  If soft-limit is exceeded, a warning is issued for each creation.

CLI COMMAND:
gluster volume quota  remove-objects [path]

==========================================================================

CLI COMMAND:
gluster volume quota  list-objects [path] ...

Sample output:
------------------
  Path                   Hard-limit Soft-limit   Used  Available
Soft-limit exceeded?
Hard-limit exceeded?
  ------------------------------------------------------------------------
--------------------------------------
  /dir                      10       80%          10       0
Yes
        Yes

==========================================================================

[root@snapshot-28 dir]# ls
a  b  file11  file12  file13  file14  file15  file16  file17
[root@snapshot-28 dir]# touch a1
touch: cannot touch `a1': Disk quota exceeded
* Nine files are created in directory "dir" and directory is included in
* the
count too. Hence the limit "10" is reached and further file creation
fails

==========================================================================

Note: We have also done some re-factoring in cli for volume name
validation. New function cli_validate_volname is created

==========================================================================

Change-Id: I1823497de4f790a2a20ebb1770293472ea33ee2b
BUG: 1190108
Signed-off-by: Sachin Pandit 
Signed-off-by: vmallika 
Reviewed-on: http://review.gluster.org/9769
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/dht: Change the subvolume encoding in d_off to be a "global"

2015-03-18T11:47:41+00:00

position in the graph rather than relative (local) to a particular
translator.

Encoding the volume in this way allows a single translator to manage
which brick is currently being scanned for directory entries. Using a
single translator minimizes allocated bits in the d_off. It also allows
multiple DHT translators in the same graph to have a common frame of
reference (the graph position) for which brick is being read. Multiple
DHT translators are needed for the Tiering feature.

The fix builds off a previous change (9332) which removed subvolume
encoding from AFR. The fix makes an equivalent change to the EC
translator.

More background can be found in fix 9332 and gluster-dev discussions [1].

DHT and AFR/EC are responsibile (as before) for choosing which brick to
enumerate directory entries in over the readdir lifecycle.

The client translator receiving the readdir fop encodes the dht_t. It
is referred to as the "leaf node" in the graph and corresponds to the
brick being scanned.

When DHT decodes the d_off, it translates the leaf node to a local
subvolume, which represents the next node in the graph leading to
the brick.

Tracking of leaf nodes is done in common utility functions. Leaf nodes
counts and positional information are updated on a graph switch.

[1] www.gluster.org/pipermail/gluster-devel/2015-January/043592.html

Change-Id: Iaf0ea86d7046b1ceadbad69d88707b243077ebc8
BUG: 1190734
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9688
Reviewed-by: Xavier Hernandez 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

dht : logically dead code removed

2015-03-03T14:40:40+00:00

CID :1124378
     1124401

Change-Id: Ib48e4a8d3fb12c4e0323a3946afb46eeb3926984
BUG: 789278
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9584
Tested-by: Gluster Build System 
Reviewed-by: N Balachandran 
Reviewed-by: Shyamsundar Ranganathan