glusterfs.git/xlators/cluster/afr/src/pump.c, branch v3.5.0

synctask: minor enhancements

2013-08-28T22:52:24+00:00

- Enhance syncenv_new() to accept scaling parameters of syncproc.
  Previously the scaling parameters were hardcoded and decided at
  compile time.

- New API synctask_create() which returns the created synctask. This
  is similar to synctask_new which only returned the status of whether
  a synctask could be created or not.

  The meaning of NULL cbk in synctask_create() means the task is
  "joinable". Until synctask_join() is called on such a synctask,
  the task is not reaped and resources are not destroyed. The
  task would be in a zombie state after synctask_fn returns and
  before synctask_join() is called.

Change-Id: I368ec9037de9510d2ba951f0aad86aaf18d9a6b6
BUG: 986775
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/5365
Tested-by: Gluster Build System 
Reviewed-by: Brian Foster

cluster/afr: Let two data-self-heals compete in new domain

2013-07-04T04:35:30+00:00

Problem:
At the moment data-self-heal acquires locks in following
pattern. It takes full file lock then gets xattrs on files on both
replicas. Decides sources/sinks based on the xattrs. Now it acquires
lock from 0-128k then unlocks the full file lock. Syncs 0-128k range
from source to sink now acquires lock 128k+1 till 256k then unlocks
0-128k, syncs 128k+1 till 256k block... so on finally it takes full file
lock again then unlocks the final small range block.
It decrements pending counts and then unlocks the full file lock.

     This pattern of locks is chosen to avoid more than 1 self-heal
to be in progress. BUT if another self-heal tries to take a full
file lock while a self-heal is already in progress it will be put in
blocked queue, further inodelks from writes by the application will
also be put in blocked queue because of the way locks xlator grants
inodelks. So until the self-heal is complete writes are blocked.

Here is the code:
xlators/features/locks/src/inodelk.c - line 225
if (__blocked_lock_conflict (dom, lock) && !(__owner_has_lock (dom, lock))) {
         ret = -EAGAIN;
         if (can_block == 0)
                 goto out;

         gettimeofday (&lock->blkd_time, NULL);
         list_add_tail (&lock->blocked_locks, &dom->blocked_inodelks);
}

This leads to hangs in applications.

Fix:
Since we want to prevent two parallel self-heals. We let them compete
in a separate "domain". Lets call the domain on which the locks have
been taken on in previous approach as "data-domain".

In the new approach When a self-heal is triggered,
it acquires a full lock in the new domain "self-heal-domain".
    After this it performs data-self-heal using the locks in
    "data-domain" as before.
unlock the full file lock in "self-heal-domain"

With this approach, application's writevs don't have to wait
in pending queue when more than 1 self-heal is triggered.

Change-Id: Id79aef3dfa888945977fb9758374ac41c320d0d5
BUG: 967717
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/5100
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

pump: Set self-heal readdir size in pump

2013-04-04T00:33:15+00:00

Problem:
In Pump entry self-heal happens for each directory during the
first opendir using conservative merge. But in entry-self-heal
readdir is issued with '0' size. So entry self-heal is not
creating any files. After pump thinks entry self-heal is complete
it proceeds to heal each of the file in the directory it just
healed. Fortunately most of the times it chooses source-brick
in pump as read-child for readdir. This happens because readchild is the
subvolume on which lookup succeeds first. In pump lookup succeeds
faster in local process than on the destination brick process most
of the times. For all the entries pump finds in readdir it does a
lookup. During this lookup the entry on the destination brick is
created and healed. This is the reason why replace-brick
succeeds whenever read-child for the directory is chosen as the
source-brick.  Which is most of the times. When read-child is chosen
as the destination brick, readdir returns no entries so replace-brick
completes without syncing the whole data.

Fix:
Set readdir-size in pump so that entry self-heal happens with
64k size. This ensures that entry self-heal triggered from
opendir actually creates the files on the destination brick.

Change-Id: I65ea45d3c2735a9578f3aa34eff771b6563241ca
BUG: 909800
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/4712
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

cluster/afr: Empty string should not be default option val

2012-12-05T23:35:47+00:00

Glusterd does not allow empty string as default value. Changed
afr option values to disallow empty string as value.

Change-Id: I92a2d658907dbc6101e1139dd91f548acb5506f5
BUG: 859927
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/4271
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

cli: Changes and enhancements to XML output

2012-10-11T23:34:16+00:00

This patch contains several xml related changes which fix some bugs and
introduce xml output for commands which were missing it. These include,
* XML output for rebalance & remove-brick status
* XML output for replace-brick
* XML output for 'volume status all' in on xml document
* proper XML output for "volume {create|start|stop|delete}"
* type & status of a volume in 'volume info' is now given as a string as well

This patch also cleans up the '#if (HAVE_LIB_XML)' sections from the code-base,
so that it is not littered around.

Change-Id: I5bb022adf0fedf7e3ead92b4b79bfa02b0b5fef5
BUG: 828131
Signed-off-by: Kaushal M 
Reviewed-on: http://review.gluster.org/3869
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

libglusterfs/dict: make 'dict_t' a opaque object

2012-09-06T07:34:15+00:00

* ie, don't dereference dict_t pointer, instead use APIs everywhere
* other than dict_t only 'data_t' should be the valid export from dict.h

* added 'dict_foreach_fnmatch()' API
* changed dict_lookup() to use data_t, instead of data_pair_t

Change-Id: I400bb0dd55519a7c5d2a107e67c8e7a7207228dc
Signed-off-by: Amar Tumballi 
BUG: 850917
Reviewed-on: http://review.gluster.org/3829
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

remove useless if-before-free (and free-like) functions

2012-07-13T21:03:42+00:00

See comments in http://bugzilla.redhat.com/839925 for
the code to perform this change.

Signed-off-by: Jim Meyering 
BUG: 839925
Change-Id: I10e4ecff16c3749fe17c2831c516737e08a3205a
Reviewed-on: http://review.gluster.com/3661
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

license: dual license under GPLV2 and LGPLV3+

2012-05-11T05:06:16+00:00

Note that the license was not changed in any of the following:
	.../argp-standalone/...
        .../booster/...
        .../cli/...
        .../contrib/...
        .../extras/...
        .../glusterfsd/...
        .../glusterfs-hadoop/...
        .../mod_clusterfs/...
        .../scheduler/...
        .../swift/...

The license was not changed in any of the non-building xlators. The
license was not changed in any of the xlators that seemed — to me — to
be clearly server-side only, e.g. protocol/server

Note too that copyright was changed along with the license; I did
not change the copyright in files where the license did not change.

If you find any errors or ommissions please don't hesitate to let me know.

The complete list of files with the license change is:
	libglusterfs/src/byte-order.h
	libglusterfs/src/call-stub.c
	libglusterfs/src/call-stub.h
	libglusterfs/src/checksum.c
	libglusterfs/src/checksum.h
	libglusterfs/src/circ-buff.c
	libglusterfs/src/circ-buff.h
	libglusterfs/src/common-utils.c
	libglusterfs/src/common-utils.h
	libglusterfs/src/compat-errno.c
	libglusterfs/src/compat-errno.h
	libglusterfs/src/compat.c
	libglusterfs/src/compat.h
	libglusterfs/src/daemon.c
	libglusterfs/src/daemon.h
	libglusterfs/src/defaults.c
	libglusterfs/src/defaults.h
	libglusterfs/src/dict.c
	libglusterfs/src/dict.h
	libglusterfs/src/event-history.c
	libglusterfs/src/event-history.h
	libglusterfs/src/event.c
	libglusterfs/src/event.h
	libglusterfs/src/fd-lk.c
	libglusterfs/src/fd-lk.h
	libglusterfs/src/fd.c
	libglusterfs/src/fd.h
	libglusterfs/src/gf-dirent.c
	libglusterfs/src/gf-dirent.h
	libglusterfs/src/globals.c
	libglusterfs/src/globals.h
	libglusterfs/src/glusterfs.h
	libglusterfs/src/graph-print.c
	libglusterfs/src/graph-utils.h
	libglusterfs/src/graph.c
	libglusterfs/src/hashfn.c
	libglusterfs/src/hashfn.h
	libglusterfs/src/iatt.h
	libglusterfs/src/inode.c
	libglusterfs/src/inode.h
	libglusterfs/src/iobuf.c
	libglusterfs/src/iobuf.h
	libglusterfs/src/latency.c
	libglusterfs/src/latency.h
	libglusterfs/src/list.h
	libglusterfs/src/lkowner.h
	libglusterfs/src/locking.h
	libglusterfs/src/logging.c
	libglusterfs/src/logging.h
	libglusterfs/src/mem-pool.c
	libglusterfs/src/mem-pool.h
	libglusterfs/src/mem-types.h
	libglusterfs/src/options.c
	libglusterfs/src/options.h
	libglusterfs/src/rbthash.c
	libglusterfs/src/rbthash.h
	libglusterfs/src/run.c
	libglusterfs/src/run.h
	libglusterfs/src/scheduler.c
	libglusterfs/src/scheduler.h
	libglusterfs/src/stack.c
	libglusterfs/src/stack.h
	libglusterfs/src/statedump.c
	libglusterfs/src/statedump.h
	libglusterfs/src/syncop.c
	libglusterfs/src/syncop.h
	libglusterfs/src/syscall.c
	libglusterfs/src/syscall.h
	libglusterfs/src/timer.c
	libglusterfs/src/timer.h
	libglusterfs/src/trie.c
	libglusterfs/src/trie.h
	libglusterfs/src/xlator.c
	libglusterfs/src/xlator.h
	libglusterfsclient/src/libglusterfsclient-dentry.c
	libglusterfsclient/src/libglusterfsclient-internals.h
	libglusterfsclient/src/libglusterfsclient.c
	libglusterfsclient/src/libglusterfsclient.h
	rpc/rpc-lib/src/auth-glusterfs.c
	rpc/rpc-lib/src/auth-null.c
	rpc/rpc-lib/src/auth-unix.c
	rpc/rpc-lib/src/protocol-common.h
	rpc/rpc-lib/src/rpc-clnt.c
	rpc/rpc-lib/src/rpc-clnt.h
	rpc/rpc-lib/src/rpc-transport.c
	rpc/rpc-lib/src/rpc-transport.h
	rpc/rpc-lib/src/rpcsvc-auth.c
	rpc/rpc-lib/src/rpcsvc-common.h
	rpc/rpc-lib/src/rpcsvc.c
	rpc/rpc-lib/src/rpcsvc.h
	rpc/rpc-lib/src/xdr-common.h
	rpc/rpc-lib/src/xdr-rpc.c
	rpc/rpc-lib/src/xdr-rpc.h
	rpc/rpc-lib/src/xdr-rpcclnt.c
	rpc/rpc-lib/src/xdr-rpcclnt.h
	rpc/rpc-transport/rdma/src/name.c
	rpc/rpc-transport/rdma/src/name.h
	rpc/rpc-transport/rdma/src/rdma.c
	rpc/rpc-transport/rdma/src/rdma.h
	rpc/rpc-transport/socket/src/name.c
	rpc/rpc-transport/socket/src/name.h
	rpc/rpc-transport/socket/src/socket.c
	rpc/rpc-transport/socket/src/socket.h
	xlators/cluster/afr/src/afr-common.c
	xlators/cluster/afr/src/afr-dir-read.c
	xlators/cluster/afr/src/afr-dir-read.h
	xlators/cluster/afr/src/afr-dir-write.c
	xlators/cluster/afr/src/afr-dir-write.h
	xlators/cluster/afr/src/afr-inode-read.c
	xlators/cluster/afr/src/afr-inode-read.h
	xlators/cluster/afr/src/afr-inode-write.c
	xlators/cluster/afr/src/afr-inode-write.h
	xlators/cluster/afr/src/afr-lk-common.c
	xlators/cluster/afr/src/afr-mem-types.h
	xlators/cluster/afr/src/afr-open.c
	xlators/cluster/afr/src/afr-self-heal-algorithm.c
	xlators/cluster/afr/src/afr-self-heal-algorithm.h
	xlators/cluster/afr/src/afr-self-heal-common.c
	xlators/cluster/afr/src/afr-self-heal-common.h
	xlators/cluster/afr/src/afr-self-heal-data.c
	xlators/cluster/afr/src/afr-self-heal-entry.c
	xlators/cluster/afr/src/afr-self-heal-metadata.c
	xlators/cluster/afr/src/afr-self-heal.h
	xlators/cluster/afr/src/afr-self-heald.c
	xlators/cluster/afr/src/afr-self-heald.h
	xlators/cluster/afr/src/afr-transaction.c
	xlators/cluster/afr/src/afr-transaction.h
	xlators/cluster/afr/src/afr.c
	xlators/cluster/afr/src/afr.h
	xlators/cluster/afr/src/pump.c
	xlators/cluster/afr/src/pump.h
	xlators/cluster/dht/src/dht-common.c
	xlators/cluster/dht/src/dht-common.h
	xlators/cluster/dht/src/dht-diskusage.c
	xlators/cluster/dht/src/dht-hashfn.c
	xlators/cluster/dht/src/dht-helper.c
	xlators/cluster/dht/src/dht-inode-read.c
	xlators/cluster/dht/src/dht-inode-write.c
	xlators/cluster/dht/src/dht-layout.c
	xlators/cluster/dht/src/dht-linkfile.c
	xlators/cluster/dht/src/dht-mem-types.h
	xlators/cluster/dht/src/dht-rebalance.c
	xlators/cluster/dht/src/dht-rename.c
	xlators/cluster/dht/src/dht-selfheal.c
	xlators/cluster/dht/src/dht.c
	xlators/cluster/dht/src/nufa.c
	xlators/cluster/dht/src/switch.c
	xlators/cluster/stripe/src/stripe-helpers.c
	xlators/cluster/stripe/src/stripe-mem-types.h
	xlators/cluster/stripe/src/stripe.c
	xlators/cluster/stripe/src/stripe.h
	xlators/features/index/src/index-mem-types.h ¹
	xlators/features/index/src/index.c ¹
	xlators/features/index/src/index.h ¹
	xlators/performance/io-cache/src/io-cache.c
	xlators/performance/io-cache/src/io-cache.h
	xlators/performance/io-cache/src/ioc-inode.c
	xlators/performance/io-cache/src/ioc-mem-types.h
	xlators/performance/io-cache/src/page.c
	xlators/performance/io-threads/src/io-threads.c
	xlators/performance/io-threads/src/io-threads.h
	xlators/performance/io-threads/src/iot-mem-types.h
	xlators/performance/md-cache/src/md-cache-mem-types.h
	xlators/performance/md-cache/src/md-cache.c
	xlators/performance/quick-read/src/quick-read-mem-types.h
	xlators/performance/quick-read/src/quick-read.c
	xlators/performance/quick-read/src/quick-read.h
	xlators/performance/read-ahead/src/page.c
	xlators/performance/read-ahead/src/read-ahead-mem-types.h
	xlators/performance/read-ahead/src/read-ahead.c
	xlators/performance/read-ahead/src/read-ahead.h
	xlators/performance/symlink-cache/src/symlink-cache.c
	xlators/performance/write-behind/src/write-behind-mem-types.h
	xlators/performance/write-behind/src/write-behind.c
	xlators/protocol/auth/addr/src/addr.c ¹
	xlators/protocol/auth/login/src/login.c ¹
	xlators/protocol/client/src/client-callback.c
	xlators/protocol/client/src/client-handshake.c
	xlators/protocol/client/src/client-helpers.c
	xlators/protocol/client/src/client-lk.c
	xlators/protocol/client/src/client-mem-types.h
	xlators/protocol/client/src/client.c
	xlators/protocol/client/src/client.h
	xlators/protocol/client/src/client3_1-fops.c

¹ Copyright only, license reverted to original

Change-Id: If560e826c61b6b26f8b9af7bed6e4bcbaeba31a8
BUG: 820551
Signed-off-by: Kaleb KEITHLEY 
Reviewed-on: http://review.gluster.com/3304
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/pump: Fixed fd leaks in pump directory traversal logic

2012-05-03T22:58:48+00:00

Change-Id: Ie54c8c9b19be6824445ada6bbcd4983e90a7c5c4
BUG: 818593
Signed-off-by: Raghavendra Bhat 
Reviewed-on: http://review.gluster.com/3267
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

pump: Removed extra dict_unref in pump_command_reply

2012-04-25T11:06:48+00:00

afr_local_cleanup is already performing the dict_unref on local->dict.

Change-Id: I07bcb100db0a0b96b3a698c444f43f96065746fb
BUG: 804905
Signed-off-by: Krishnan Parthasarathi 
Reviewed-on: http://review.gluster.com/3223
Tested-by: Gluster Build System 
Reviewed-by: Amar Tumballi