glusterfs-quota.git/xlators/cluster, branch test

cluster/dht: mark linkfile creation/deletion as internal fop

2013-07-24T11:27:29+00:00

Currently dht creates/deletes linkfiles for various ops like
rename/linking and when layout changes. dht_linkfile_create
already sends a key GLUSTERFS_INTERNAL_FOP_KEY in dict to
identify this as an internal fop and not user based op.

Enhancing rename related links/unlinks to send this key too.
Marker/changelog or other xlators can now identify these as
internal fops and handle them accordingly

Change-Id: Ib1ca789e6dbce48703c55ad3f4f88f7cd2df3d06
BUG: 987428
Signed-off-by: shishir gowda 
Reviewed-on: http://review.gluster.org/5335
Reviewed-by: Amar Tumballi 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

storage/posix: implement batched fsync in a single thread

2013-07-23T13:11:12+00:00

Because of the extra fsync()s issued by AFR transaction, they
could potentially "clog" all the io-threads denying unrelated
operations from making progress.

This patch assigns a dedicated thread to issues fsyncs, as
an experimental feature to understand performance characteristics
with the approach.

As a basis, incoming individual fsync requests are grouped into
batches, falling in the same @batch-fsync-delay-usec window of
time. These windows can extend in practice, as processing of
the previous batch can take longer than @batch-fsync-delay-usec
while new requests are getting batched.

The feature support three modes (similar to the -S modes of fs_mark)

- syncfs: In this mode one syncfs() is issued per batch, instead
  of N fsync()s (one per file.)

- syncfs-single-fsync: In this mode one syncfs() is issued per
  batch (which, on Linux, guarantees the completion of write-out
  of dirty pages in the filesystem up to that point) and one single
  fsync() to synchronize or flush the controller/drive cache. This
  corresponds to -S 2 of fsmark.

- syncfs-reverse-fsync: In this mode, one syncfs() is issued per
  batch, and all the open files in that batch are fsync()'ed in
  the reverse order of the queue. This corresponds to -S 4 of
  fsmark.

- reverse-fsync: In this mode, no syncfs() is issued and all the
  files in the batch are fsync()'ed in the reverse order. This
  corresponds to -S 3 of fsmark.

Change-Id: Ia1e170a810c780c8d80e02cf910accc4170c4cd4
BUG: 927146
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/4746
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/afr: Handle parallel hardlinks self-heal

2013-07-23T07:33:39+00:00

Change-Id: Ieda11870c65edae500140b6c061f15a7b3f264f3
BUG: 986905
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/5370
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

fuse: auxiliary gfid mount support

2013-07-19T08:14:08+00:00

* files can be accessed directly through their gfid and not just
  through their paths. For eg., if the gfid of a file is
  f3142503-c75e-45b1-b92a-463cf4c01f99, that file can be accessed
  using /.gfid/f3142503-c75e-45b1-b92a-463cf4c01f99

  .gfid is a virtual directory used to seperate out the namespace
  for accessing files through gfid. This way, we do not conflict with
  filenames which can be qualified as uuids.

* A new file/directory/symlink can be created with a pre-specified
  gfid. A setxattr done on parent directory with fuse_auxgfid_newfile_args_t
  initialized with appropriate fields as value to key "glusterfs.gfid.newfile"
  results in the entry /bname whose gfid is set to args.gfid. The
  contents of the structure should be in network byte order.

  struct auxfuse_symlink_in {
        char     linkpath[]; /* linkpath is a null terminated string */
  } __attribute__ ((__packed__));

  struct auxfuse_mknod_in {
        unsigned int   mode;
        unsigned int   rdev;
        unsigned int   umask;
  } __attribute__ ((__packed__));

  struct auxfuse_mkdir_in {
        unsigned int   mode;
        unsigned int   umask;
  } __attribute__ ((__packed__));

  typedef struct {
        unsigned int  uid;
        unsigned int  gid;
        char          gfid[UUID_CANONICAL_FORM_LEN + 1]; /* a null terminated gfid string
                                                      * in canonical form.
                                                      */
        unsigned int  st_mode;
        char          bname[];     /* bname is a null terminated string */

        union {
                struct auxfuse_mkdir_in   mkdir;
                struct auxfuse_mknod_in   mknod;
                struct auxfuse_symlink_in symlink;
        } __attribute__ ((__packed__)) args;
  } __attribute__ ((__packed__)) fuse_auxgfid_newfile_args_t;

  An initial consumer of this feature would be geo-replication to
  create files on slave mount with same gfids as that on master.
  It will also help gsyncd to access files directly through their
  gfids. gsyncd in its newer version will be consuming a changelog
  (of master) containing operations on gfids and sync corresponding
  files to slave.

* Also, bring in support to heal gfids with a specific value.
  fuse-bridge sends across a gfid during a lookup, which storage
  translators assign to an inode (file/directory etc) if there is
  no gfid associated it. This patch brings in support
  to specify that gfid value from an application, instead of relying
  on random gfid generated by fuse-bridge.

  gfids can be healed through setxattr interface. setxattr should be
  done on parent directory. The key used is "glusterfs.gfid.heal"
  and the value should be the following structure whose contents
  should be in network byte order.

  typedef struct {
        char      gfid[UUID_CANONICAL_FORM_LEN + 1]; /* a null terminated gfid
                                                      * string in canonical form
                                                      */
        char      bname[]; /* a null terminated basename */
  } __attribute__((__packed__)) fuse_auxgfid_heal_args_t;

  This feature can be used for upgrading older geo-rep setups where gfids
  of files are different on master and slave to newer setups where they
  should be same. One can delete gfids on slave using setxattr -x and
  .glusterfs and issue stat on all the files with gfids from master.

Thanks to "Amar Tumballi"  and "Csaba Henk"
 for their inputs.

Signed-off-by: Raghavendra G 
Change-Id: Ie8ddc0fb3732732315c7ec49eab850c16d905e4e
BUG: 952029
Reviewed-on: http://review.gluster.com/#/c/4702
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi 
Reviewed-on: http://review.gluster.org/4702
Reviewed-by: Xavier Hernandez 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

dht: fix dht_discover_cbk doing a wrong layout set.

2013-07-18T05:49:38+00:00

with the sequence of operations are like below, we have issues
with current code (MP == mountpoint):
T0,MP1# mkdir /abcd (Succeeds on hash_subvol)
T1,MP2# mkdir /abcd (Gets EEXIST as dir exists in hash_subvol)
T2,MP2# mkdir /.gfid//xyz (lookup happens on abcd's
        gfid, calls dht_discover)
T3,MP1# (Completes mkdir(), goes to dir_selfheal to set the layouts).
T4,MP2# (dht_discover_cbk gets success for lookup as the entry
        existed, as layout is not yet written, it says normalize
        done, found holes).
T5,MP2# (as layout anomaly is not considered an issue in this patch,
        dht_layout_set happens on inode, with all xlators pointing
        to 0s)
T6,MP1# (completes mkdir call, inode has proper layouts)
T7,MP2# mkdir /.gfid//xyz fails with ENOENT
        (with log saying no subvol found for hash value of xyz.

Porting Amar's fix from down-stream beta branch.

Change-Id: Ibdc37ee614c96158a1330af19cad81a39bee2651
BUG: 982913
Original-author: Amar Tumballi 
Signed-off-by: shishir gowda 
Reviewed-on: http://review.gluster.org/5302
Reviewed-by: Amar Tumballi 
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

cluster/dht: Prevent dht_access from going into a loop.

2013-07-16T06:56:34+00:00

If access fails with ENOTCONN, do not wind to same subvol.
We wind to first-up-subvol if access fails with ENOTCONN.
In few cases, if dht has only 1 subvolume, and access fails with
ENOTCONN, we go into a infinite loop of winding to same subvol

The fix is to check if we previously wound to same subvol, and
fail if first-up-subvol is same.

Change-Id: Ib5d3ce7d33e8ea09147905a7df1ed280874fa549
BUG: 983431
Signed-off-by: shishir gowda 
Reviewed-on: http://review.gluster.org/5319
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

afr: customize client-pid=-1 xtime aggregation to tolerate a replica down

2013-07-15T08:24:19+00:00

Using the new 'pluggable policies' API of libxlator.

Change-Id: Ie7528182dff8fb42c6e8287a106d3057944df775
BUG: 847839
Original Author: Csaba Henk 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/4904
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

libxlator: implement pluggable aggregation policies

2013-07-15T08:23:53+00:00

The API is described in libxlator.h.

Behavior remains the same for this commit; this
is a preparatory step for per-translator customization
of aggregation.

Change-Id: I5d42923af59b2fd78e1ff59c12763875b57c5190
BUG: 847839
Original Author: Csaba Henk 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/4903
Tested-by: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Vijay Bellur

cluster/dht: node-uuid for directories winds to all subvolumes

2013-07-15T08:23:01+00:00

this works similar to pathinfo now except that the request is sent
to all subvolumes of dht. Underlying replica selects it's subvolume
in a round-robin fashion till one of them returns successfully.

Change-Id: Ie46c5f7090d04d8c2e487b209916ae6791e94624
BUG: 847839
Original Author: Venky Shankar 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/5225
Tested-by: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Vijay Bellur

cluster/*: get logic to calculate min() of the 'stime' xattr

2013-07-15T04:06:42+00:00

* in both distribute and replicate (ignoring stripe for now),
  add logic to calculate the min() of stime values.

* What is a 'stime' ? Why is this required:
  -  stime means 'slave xtime', mainly used to keep track of slave
  node's sync status when distributed geo-replication is used.
  Logic of calculating 'min()' for this stime is very important as
  in case of crashes/reboots/shutdown, we will have to 'restart'
  with crawling from stime time value from the mount point, which
  gives the 'min()' of all the bricks, which means, we don't miss
  syncing any files in the above cases.

Change-Id: I2be8d434326572be9d4986db665570a6181db1ee
BUG: 847839
Original Author: Amar Tumballi 
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/4893
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur