glusterfs.git/xlators/cluster/dht/src/dht-layout.c, branch v3.7.7

dht: Adding log messages to the new logging framework

2015-07-27T07:21:48+00:00

        Backported from: http://review.gluster.org/10021

> Change-Id: Ib3bb61c5223f409c23c68100f3fe884918d2dc3f
> BUG: 1194640
> Reviewed-on: http://review.gluster.org/10021
> Reviewed-by: N Balachandran 
> Reviewed-by: Joseph Fernandes 
> Tested-by: Joseph Fernandes 
> Reviewed-by: Dan Lambright 
> Reviewed-by: Raghavendra G 
> Tested-by: Raghavendra G 
> Signed-off-by: arao 

BUG: 1217722
Change-Id: Ide79c6c1e6a466fb52f955c90a2b22711bec794a
Signed-off-by: arao 
Signed-off-by: Anusha Rao 
Reviewed-on: http://review.gluster.org/11350
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra G 
Tested-by: Gluster Build System

dht: make lookup-unhashed=auto do something actually useful

2015-05-10T04:55:09+00:00

The key concept here is to determine whether a directory is "clean" by
comparing its last-known-good topology to the current one for the
volume.  These are stored as "commit hashes" on the directory and the
volume root respectively.  The volume's commit hash changes whenever a
brick is added or removed, and a fix-layout is done.  A directory's
commit hash changes only when a full rebalance (not just fix-layout)
is done on it.  If all bricks are present and have a directory
commit hash that matches the volume commit hash, then we can assume
that every file is in its "proper" place. Therefore, if we look for
a file in that proper place and don't find it, we can assume it's not
on any other subvolume and *safely* skip the global (broadcast to all)
lookup.

Change-Id: Id6ce4593ba1f7daffa74cfab591cb45960629ae3
BUG: 1220064
Reviewed-on-master: http://review.gluster.org/#/c/7702/
Signed-off-by: Jeff Darcy 
Signed-off-by: Shyam 
Reviewed-on: http://review.gluster.org/10729
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Vijay Bellur

dht : coverity fixes

2015-04-07T10:52:05+00:00

CID : 1124352,1124365 (unchecked return value),
      1124377 ( logically dead code),
      1124511 (null dereference)

Change-Id: I61e029a078559cfe15d36bf0aa53418f6214e5cb
BUG: 789278
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9622
Tested-by: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: N Balachandran 
Reviewed-by: Vijay Bellur

Avoid conflict between contrib/uuid and system uuid

2015-04-04T17:48:35+00:00

glusterfs relies on Linux uuid implementation, which
API is incompatible with most other systems's uuid. As
a result, libglusterfs has to embed contrib/uuid,
which is the Linux implementation, on non Linux systems.
This implementation is incompatible with systtem's
built in, but the symbols have the same names.

Usually this is not a problem because when we link
with -lglusterfs, libc's symbols are trumped. However
there is a problem when a program not linked with
-lglusterfs will dlopen() glusterfs component. In
such a case, libc's uuid implementation is already
loaded in the calling program, and it will be used
instead of libglusterfs's implementation, causing
crashes.

A possible workaround is to use pre-load libglusterfs
in the calling program (using LD_PRELOAD on NetBSD for
instance), but such a mechanism is not portable, nor
is it flexible. A much better approach is to rename
libglusterfs's uuid_* functions to gf_uuid_* to avoid
any possible conflict. This is what this change attempts.

BUG: 1206587
Change-Id: I9ccd3e13afed1c7fc18508e92c7beb0f5d49f31a
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/10017
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

Xlators : Fixed typos

2015-04-02T10:21:08+00:00

Change-Id: I948f85cb369206ee8ce8b8cd5e48cae9adb971c9
BUG: 1075417
Signed-off-by: Manikandan Selvaganesh 
Reviewed-on: http://review.gluster.org/9529
Reviewed-by: Niels de Vos 
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: Humble Devassy Chirammal

cluster/dht: Change the subvolume encoding in d_off to be a "global"

2015-03-18T11:47:41+00:00

position in the graph rather than relative (local) to a particular
translator.

Encoding the volume in this way allows a single translator to manage
which brick is currently being scanned for directory entries. Using a
single translator minimizes allocated bits in the d_off. It also allows
multiple DHT translators in the same graph to have a common frame of
reference (the graph position) for which brick is being read. Multiple
DHT translators are needed for the Tiering feature.

The fix builds off a previous change (9332) which removed subvolume
encoding from AFR. The fix makes an equivalent change to the EC
translator.

More background can be found in fix 9332 and gluster-dev discussions [1].

DHT and AFR/EC are responsibile (as before) for choosing which brick to
enumerate directory entries in over the readdir lifecycle.

The client translator receiving the readdir fop encodes the dht_t. It
is referred to as the "leaf node" in the graph and corresponds to the
brick being scanned.

When DHT decodes the d_off, it translates the leaf node to a local
subvolume, which represents the next node in the graph leading to
the brick.

Tracking of leaf nodes is done in common utility functions. Leaf nodes
counts and positional information are updated on a graph switch.

[1] www.gluster.org/pipermail/gluster-devel/2015-January/043592.html

Change-Id: Iaf0ea86d7046b1ceadbad69d88707b243077ebc8
BUG: 1190734
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9688
Reviewed-by: Xavier Hernandez 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

testing: Switch to cmocka the successor of cmockery2

2015-03-05T21:51:37+00:00

This uses https://cmocka.org/ as the unit testing framework.

With this change, unit testing is made optional as well. We assume there
is no cmocka available while building. cmocka will be enabled by default
later on. For now, to build with cmocka run:

    $ ./configure --enable-cmocka

This change is based on the work of Andreas (replacing cmockery2 with
cmocka) and Kaleb (make cmockery2 an optional build dependency).

The only modifications I made, are additional #defines in unittest.h for
making sure the unit tests function as expected.

Change-Id: Iea4cbcdaf09996b49ffcf3680c76731459cb197e
BUG: 1067059
Merged-change: http://review.gluster.org/9762/
Signed-off-by: Andreas Schneider 
Signed-off-by: Kaleb S. KEITHLEY 
Signed-off-by: Niels de Vos 
Change-Id: Ia2e955481c102d5dce17695a9205395a6030e985
Reviewed-on: http://review.gluster.org/9738
Tested-by: Gluster Build System

cluster/dht: synchronize with other concurrent healers while healing layout.

2015-02-20T10:35:00+00:00

Current layout heal code assumes layout setting is idempotent. This
allowed multiple concurrent healers to set the layout without any
synchronization. However, this is not the case as different healers
can come up with different layout for same directory and making layout
setting non-idempotent. So, we bring in synchronization among healers
to
   1. Not to overwrite an ondisk well-formed layout.
   2. Refresh the in-memory layout with the ondisk layout if in-memory
   layout needs healing and ondisk layout is well formed.

This patch can synchronize
   1. among multiple healers.
   2. among multiple fix-layouts (which extends layout to consider
   added or removed brick)
   3. (but) not between healers and fix-layouts. So, the problem of
   in-memory stale layouts (not matching with layout ondisk), is not
   _completely_ fixed by this patch.

Signed-off-by: Raghavendra G 
Change-Id: Ia285f25e8d043bb3175c61468d0d11090acee539
BUG: 1176008
Reviewed-on: http://review.gluster.org/9302
Reviewed-by: N Balachandran

build: Support for unit tests using Cmockery2

2014-07-18T17:55:51+00:00

This patch will allow for developers to create unit tests for
their code.  Documentation has been added to the patch and
is available here:

doc/hacker-guide/en-US/markdown/unittest.md

Also, unit tests are run when RPM is created.

This patch is a replacement for http://review.gluster.org/#/c/7281
which removed unit test infrastucture from the repo due to multiple
conflicts.  Cmockery2 is now available in Fedora and EPEL, and soon
to be available in Debian and Ubuntu.  For all other operating
systems, please install from the source:

https://github.com/lpabon/cmockery2

BUG: 1067059

Change-Id: I1b36cb1f56fd10916f9bf535e8ad080a3358289f
Signed-off-by: Luis Pabón 
Reviewed-on: http://review.gluster.org/7538
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Vijay Bellur

cluster/dht: Do layout self healing of directory for nameless lookup

2014-06-17T12:39:22+00:00

Problem: Currently in the  nameless lookup code path, if at the
         end of the lookup, even if it detects that layout
         anamolies are there, layout healing will not be done as
         there is no code to heal it.
         So there can be race between mkdir and lookup.

         Assume mkdir is going on from some other mount point,
         Say, M1. Directories are created on some nodes but layout
         is not set yet.

         Now from M2, nameless lookup goes, lookup will be success
         full as the directory is present on some of the nodes, but
         it won't heal layout. Now if create goes after lookup fop,
         because layout is absent, file creation will fail.

Fix:     Included the code of layout self-heal in the nameless
         lookup path. At the end of lookup, layout will be computed
         as it would have been in the named lookup, but it will be
         set to those node only, where directory is present.
         So after that if create fop goes, the probabiliy to get the
         subvolume with proper hash-range is high now, so reduces
         the race window.

Other:  Whenever a directory is created, we have to choose a brick
        from which we start allocating layout in a circular fashion.
        To calculate this starting brick, I have changed the candidate
        from name of the directory to gfid of the directory

        But to compute where a given file belongs, we will still
        use the name of the file. Hash computed from the name of the
        file should belong to any one of the directory-hash-range

        Calculation of hash for a file is acting as a consumer and the
        setting of directory layout based on gfid is acting as a producer,
        which are independent from each other.

Change-Id: I3808c55082cd1b5c72d2c77cbbc063f55aa38bee
BUG: 1095888
Signed-off-by: Venkatesh Somyajulu 
Reviewed-on: http://review.gluster.org/7493
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra Bhat 
Reviewed-by: Vijay Bellur