glusterfs.git/libglusterfs/src/xlator.c, branch v3.7.9

core : Use correct path in dlopen for socket.so

2015-12-28T05:36:49+00:00

Backport of http://review.gluster.org/12656

This patch fixes the path for socket.so file while loading the so dynamically.
Also for config.memory-accounting & config.transport voltype is changed to
glusterd to fix the warning message coming from xlator_volopt_dynload

Change-Id: I0f7964814586f2018d4922b23c683f4e1eb3098e
BUG: 1283833
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/12656
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos 
Reviewed-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/12670
Tested-by: NetBSD Build System

libglusterfs:Porting log messages to new framework

2015-09-08T07:56:55+00:00

	Backport of http://review.gluster.org/11896

Cherry picked from 038dfe57cf0c5944b0392332dbf5a00bb1208150
>Change-Id: I8625b7dc8941720cc7a864b8fddbcc7b4c485fcd
>BUG: 1252836
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/11896
>Tested-by: NetBSD Build System 
>Tested-by: Gluster Build System 
>Reviewed-by: Manikandan Selvaganesh 
>Reviewed-by: Niels de Vos 

Change-Id: I8625b7dc8941720cc7a864b8fddbcc7b4c485fcd
BUG: 1258769
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/12076
Reviewed-by: Manikandan Selvaganesh 
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Niels de Vos

defaults,globals,iobuf,latency,logging,options,xlator/libglusterfs : porting to a new logging framework

2015-07-13T13:10:58+00:00

        Backport of http://review.gluster.org/10826

Cherry picked from ef374774b8727b2f76915a1f88b26c83c77dde43
>Change-Id: If6a55186cddc3d1c4d22e3d56b45358b84feeb49
>BUG: 1194640
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/10826
>Tested-by: NetBSD Build System 
>Reviewed-by: Pranith Kumar Karampuri 

Change-Id: If6a55186cddc3d1c4d22e3d56b45358b84feeb49
BUG: 1217722
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/11405
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Niels de Vos

core: use reference counting for mem_acct structures

2015-05-09T21:27:36+00:00

When freeing memory, our memory-accounting code expects to be able to
dereference from the (previously) allocated block to its owning
translator.  However, as we have already found once in option
validation and twice in logging, that translator might itself have
been freed and the dereference attempt causes on of our daemons to
crash with SIGSEGV.  This patch attempts to fix that as follows:

 * We no longer embed a struct mem_acct directly in a struct xlator,
   but instead allocate it separately.

 * Allocated memory blocks now contain a pointer to the mem_acct
   instead of the xlator.

 * The mem_acct structure contains a reference count, manipulated in
   both the normal and translator allocate/free code using atomic
   increments and decrements.

 * Because it's now a separate structure, we can defer freeing the
   mem_acct until its reference count reaches zero (either way).

 * Some unit tests were disabled, because they embedded their own
   copies of the implementation for what they were supposedly testing.
   Life's too short to spend time fixing tests that seem designed to
   impede progress by requiring a certain implementation as well as
   behavior.

Change-Id: Id929b11387927136f78626901729296b6c0d0fd7
BUG: 1219026
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/10417
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Niels de Vos 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-on: http://review.gluster.org/10723
Tested-by: NetBSD Build System

Avoid conflict between contrib/uuid and system uuid

2015-04-04T17:48:35+00:00

glusterfs relies on Linux uuid implementation, which
API is incompatible with most other systems's uuid. As
a result, libglusterfs has to embed contrib/uuid,
which is the Linux implementation, on non Linux systems.
This implementation is incompatible with systtem's
built in, but the symbols have the same names.

Usually this is not a problem because when we link
with -lglusterfs, libc's symbols are trumped. However
there is a problem when a program not linked with
-lglusterfs will dlopen() glusterfs component. In
such a case, libc's uuid implementation is already
loaded in the calling program, and it will be used
instead of libglusterfs's implementation, causing
crashes.

A possible workaround is to use pre-load libglusterfs
in the calling program (using LD_PRELOAD on NetBSD for
instance), but such a mechanism is not portable, nor
is it flexible. A much better approach is to rename
libglusterfs's uuid_* functions to gf_uuid_* to avoid
any possible conflict. This is what this change attempts.

BUG: 1206587
Change-Id: I9ccd3e13afed1c7fc18508e92c7beb0f5d49f31a
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/10017
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

core : free up mem_acct.rec in xlator_destroy

2015-03-30T12:40:19+00:00

Problem:
We've observed that glusterd was OOM killed after some minutes when volume set
command was run in a loop.

Analysis:

Initially the suspection was in glusterd code, but a deep dive into the codebase
revealed that while validating all the options as part of graph reconfiguration
at the time of freeing up the xlator object its one of the member mem_acct is
left over which causes memory leak.

Solution:

Free up xlator's mem_acct.rec in xlator_destroy ()

Change-Id: Ie9e7267e1ac4ab7b8af6e4d7c6660dfe99b4d641
BUG: 1201203
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/9862
Reviewed-by: Niels de Vos 
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Raghavendra Bhat 
Reviewed-by: Kaleb KEITHLEY

features/bit-rot: Implementation of bit-rot xlator

2015-03-24T17:55:32+00:00

This is the "Signer" -- responsible for signing files with their
checksums upon last file descriptor close (last release()).
The event notification facility provided by the changelog xlator
is made use of.

Moreover, checksums are as of now SHA256 hash of the object data
and is the only available hash at this point of time. Therefore,
there is no special "what hash to use" type check, although it's
does not take much to add various hashing algorithms to sign
objects with. Signatures are stored in extended attributes of the
objects along with the the type of hashing used to calculate the
signature. This makes thing future proof when other hash types
are added. The signature  infrastructure is provided by bitrot
stub: a little piece of code that sits over the POSIX xlator
providing interfaces to "get or set" objects signature and it's
staleness.

Since objects are signed upon receiving release() notification,
pre-existing data which are "never" modified would never be
signed. To counter this, an initial crawler thread is spawned
The crawler scans the entire brick for objects that are unsigned
or "missed" signing due to the server going offline (node reboots,
crashes, etc..) and triggers an explicit sign. This would also
sign objects when bit-rot is enabled for a volume and/or after
upgrade.

Change-Id: I1d9a98bee6cad1c39c35c53c8fb0fc4bad2bf67b
BUG: 1170075
Original-Author: Raghavendra Bhat 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/9711
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

Quota/marker : Support for inode quota

2015-03-18T04:07:04+00:00

Currently, the only way to retrieve the number of files/objects in a
directory or volume is to do a crawl of the entire directory/volume.
This is expensive and is not scalable.

The new mechanism proposes to store count of objects/files as part of
an extended attribute of a directory. Each directory's extended
attribute value will indicate the number of files/objects present
in a tree with the directory being considered as the root of the tree.

Currently file usage is accounted in marker by doing multiple FOPs
like setting and getting xattrs. Doing this with STACK WIND and
UNWIND can be harder to debug as involves multiple callbacks.
In this code we are replacing current mechanism with syncop approach
as syncop code is much simpler to follow and help us implement inode
quota in an organized way.

Change-Id: Ibf366fbe07037284e89a241ddaff7750fc8771b4
BUG: 1188636
Signed-off-by: vmallika 
Signed-off-by: Sachin Pandit 
Reviewed-on: http://review.gluster.org/9567
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

every/where: add GF_FOP_IPC for inter-translator communication

2015-03-17T14:02:15+00:00

Several features - e.g. encryption, erasure codes, or NSR - involve
multiple cooperating translators which sometimes need a "private" means
of communication amongst themselves.  Historically we've used virtual or
synthetic xattrs, but that's not very elegant and clutters up the
getxattr/setxattr path which must also handle real xattr requests.  This
new fop should address that.

The only argument is an int32_t "op" which should be recognized by the
target translator.  It is recommended that translators using these
feature follow some convention regarding the ops that they define, to
avoid conflicts.  Using a hash of the target translator's type string as
a base for a series of ops would probably be a good start.  Any other
information can be passed in both directions using xdata.

The default behavior for this fop, as with any other, is to pass through
to FIRST_CHILD.  That makes use of this fop "transparent" to other
translators that were written before it existed, but it also means that
it only really works with pass-through translators.  If a routing
translator (such as DHT) or a fan-out translator (such as AFR) is
involved, the IPC might not reach its intended destination unless those
translators are modified to forward IPC fops along all paths.

If an IPC gets all the way to storage/posix it is considered an error,
much like an uncaught exception.  We don't actually *do* anything in
that case, but we do log it send back an EOPNOTSUPP error.  This makes
the "unrecognized opcode" condition distinguishable from the "no IPC
support" condition (which would yield an RPC error instead) so clients
can probe for the presence of a handler for their own favorite opcode
and either use that or use old-school xattrs depending on the result.

BUG: 1158628
Signed-off-by: Venky Shankar 
Signed-off-by: Jeff Darcy 
Change-Id: I84af1b17babe5b30ec03ecf027ae37d09b873968
Reviewed-on: http://review.gluster.org/8812
Reviewed-by: Vijay Bellur

xlator: avoiding possibility of a crash if (xl->ctx) is NULL.

2015-03-16T03:01:23+00:00

Change-Id: I41acd9970bef04bb16cd4d8532a84a95d5fb642a
BUG: 1199003
Signed-off-by: Humble Devassy Chirammal .
Reviewed-on: http://review.gluster.org/9810
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra Bhat 
Reviewed-by: Vijay Bellur