glusterfs.git/libglusterfs/src/dict.c, branch v3.7.9

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

call-stub,circ-buff,client_t,compat,dict/libglusterfs : Porting to a new logging framework

2015-07-09T07:11:04+00:00

        Backport of http://review.gluster.org/10828

Cherry picked from a1e32fbcfbfaf6e4c63e140b3b90a80dc748a269
>Change-Id: Ie7d180e0ab2fed1270d66504606d1b2522884020
>BUG: 1194640
>Signed-off-by: Mohamed Ashiq 
>Reviewed-on: http://review.gluster.org/10828
>Tested-by: NetBSD Build System 
>Tested-by: Gluster Build System 
>Reviewed-by: Pranith Kumar Karampuri 

Change-Id: Ie7d180e0ab2fed1270d66504606d1b2522884020
BUG: 1217722
Signed-off-by: Mohamed Ashiq 
Reviewed-on: http://review.gluster.org/11406
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

cluster/ec: Fix dictionary compare function

2015-05-08T22:03:16+00:00

If both dicts are NULL then equal. If one of the dicts is NULL but the other
has only ignorable keys then also they are equal. If both dicts are non-null
then check if for each non-ignorable key, values are same or not.  value_ignore
function is used to skip comparing values for the keys which must be present in
both the dictionaries but the value could be different.

geo-rep's stime xattr doesn't need to be present in list xattr but when
getxattr comes on stime xattr even if there aren't enough responses with the
xattr we should still give out an answer which is maximum of the stimes
available.

Change-Id: I8de2ceaa2db785b797f302f585d88e73b154167d
BUG: 1216303
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/10078
Reviewed-on: http://review.gluster.org/10690
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System

features/bit-rot: Implementation of bit-rot xlator

2015-03-24T17:55:32+00:00

This is the "Signer" -- responsible for signing files with their
checksums upon last file descriptor close (last release()).
The event notification facility provided by the changelog xlator
is made use of.

Moreover, checksums are as of now SHA256 hash of the object data
and is the only available hash at this point of time. Therefore,
there is no special "what hash to use" type check, although it's
does not take much to add various hashing algorithms to sign
objects with. Signatures are stored in extended attributes of the
objects along with the the type of hashing used to calculate the
signature. This makes thing future proof when other hash types
are added. The signature  infrastructure is provided by bitrot
stub: a little piece of code that sits over the POSIX xlator
providing interfaces to "get or set" objects signature and it's
staleness.

Since objects are signed upon receiving release() notification,
pre-existing data which are "never" modified would never be
signed. To counter this, an initial crawler thread is spawned
The crawler scans the entire brick for objects that are unsigned
or "missed" signing due to the server going offline (node reboots,
crashes, etc..) and triggers an explicit sign. This would also
sign objects when bit-rot is enabled for a volume and/or after
upgrade.

Change-Id: I1d9a98bee6cad1c39c35c53c8fb0fc4bad2bf67b
BUG: 1170075
Original-Author: Raghavendra Bhat 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/9711
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/dht: Add tier translator.

2015-03-21T16:50:29+00:00

The tier translator shares most of DHT's code. It differs in how
subvolumes are chosen for I/Os, and how file migration (cache promotion
and demotion) is managed. That different functionality is split to either
DHT or tier logic according to the "tier_methods" structure.

A cache promotion and demotion thread is created in a manner
similar to the rebalance daemon. The thread operates a timing
wheel which periodically checks for promotion and demotion candidates
(files). Candidates are queued and then migrated. Candidates must exist on
the same node as the daemon and meet other critera per caching policies.

This patch has two authors (Dan Lambright and Joseph Fernandes). Dan
did the DHT changes and Joe wrote the cache policies. The fix depends on
DHT readidr changes and the database library which have been submitted
separately.  Header files in libglusterfs/src/gfdb should be reviewed in
patch 9683.

For more background and design see the feature page [1].

[1]
http://www.gluster.org/community/documentation/index.php/Features/data-classification

Change-Id: Icc26c517ccecf5c42aef039f5b9c6f7afe83e46c
BUG: 1194753
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/9724
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

features/index: Add option to track specific xattrs for xattrop64

2015-02-25T05:57:53+00:00

This enables trusted.ec.dirty to be tracked in index

Change-Id: Ief1619110859f6f9ccee3da229f0688b73e2124b
BUG: 1177601
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/9602
Reviewed-by: Krutika Dhananjay 
Tested-by: Gluster Build System

dict: Remove the redundant hash calculation when the hash size is 1

2014-11-26T08:10:15+00:00

Currently the dict is created with hash size 1, i.e. there is
only one hash bucket and the calculation of hash decomes redundant.

Change-Id: Id70aea0d798902494ebb6d82955d97d591bc73d2
BUG: 789278
Signed-off-by: Poornima Gurusiddaiah 
Reviewed-on: http://review.gluster.org/8211
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra Talur 
Reviewed-by: Vijay Bellur

dict: Write dict_foreach_* in form of dict_foreach_match

2014-11-25T17:50:30+00:00

Change-Id: Iaa3454f7f3b6516660b1976bea63e39ea7795f8f
BUG: 1164051
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/9121
Reviewed-by: Jeff Darcy 
Reviewed-by: Krishnan Parthasarathi 
Tested-by: Gluster Build System

features/marker: Filter internal xattrs in lookup

2014-11-11T11:33:41+00:00

Afr should ignore quota-size-key as part of self-heal
but should heal quota-limit key.

Change-Id: Ic0b06bd20a563a00d6bfdc2dc5a76c661e533ecb
BUG: 1161106
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/9061
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra G 
Reviewed-by: Vijay Bellur

glusterd: statedump support

2014-10-15T10:02:40+00:00

Although glusterd currently has statedump support but it doesn't dump its
context information. Implementing glusterd_dump_priv function to export per-node
glusterd information would be useful for debugging bugs. Once implemented, we
could enhance sos-report to fetch this information. This would potentially
reduce our time to root cause and data needed for debugability can be dumped
gradually.

Following is the main items of the dump list targeted in this patch :

    * Supported max/min op-version and current op-version
    * Information about peer list
    * Information about peer list involved while a transaction is going on
      (xaction_peers)
    * option dictionary in glusterd_conf_t
    * mgmt_v3_lock in glusterd_conf_t
    * List of connected clients
    * uuid of glusterd
    * A section of rpc related information like live connections and their
      statistics

There are couple of issues which were found during implementation and testing
phase:
 - xaction_peers of glusterd_conf_t was not initialized in init because of which
   traversing through this list head was crashing when there was no active
   transaction
 - gf_free was not setting the typestr to NULL if the the alloc count becomes 0
   for a mem-type earlier allocated.

Change-Id: Ic9bce2d57682fc1771cd2bc6af0b7316ecbc761f
BUG: 1139682
Signed-off-by: Atin Mukherjee 
Reviewed-on: http://review.gluster.org/8665
Tested-by: Gluster Build System 
Reviewed-by: Kaushal M