glusterfs.git/libglusterfs/src/gfdb, branch v3.7.15

tier/libgfdb: Ordering query results from libgfdb

2016-04-26T21:51:05+00:00

When querying we will order the query result to get
the hotest or the coldest files in the queried list
so that these files are migrated first.

Now here we are giving priority to the write heat(time and counters),
as it requires complex queries to have a composite
ordering of write and read + it has it impact on performance.

Backport of http://review.gluster.org/13607

> Change-Id: I2e0415dcfad4218b42c68fc5c2ed8d1f075ce9ea
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/13607
> Smoke: Gluster Build System 
> Tested-by: Joseph Fernandes
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: If5fad07f8d0f50016b10e256803abd5266cd708f
BUG: 1323017
Reviewed-on: http://review.gluster.org/13881
Smoke: Gluster Build System 
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright

tier/gfdb : Round-Robin read of query files

2016-02-14T14:23:17+00:00

1. Each brick on a host will get a separate query file.
2. While reading query record from these query files we
   read them in a Round-Robin manner.
3. When an error occurs during migration we rename it to
   query file with an time stamp and .err extension for
   better debugging.

Backport of http://review.gluster.org/13293

> Change-Id: I27c4285d24fd695d2d5cbd9fd7db3879d277ecc8
> BUG: 1302772
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/13293
> Smoke: Gluster Build System 
> Tested-by: N Balachandran 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: Dan Lambright 
> CentOS-regression: Gluster Build System 

Change-Id: I9448f19c10a82fff9b1b3c9387c9026adc0742f1
BUG: 1306514
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/13427
Smoke: Gluster Build System 
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright

all: reduce "inline" usage

2016-01-18T09:02:34+00:00

There are three kinds of inline functions: plain inline, extern inline,
and static inline.  All three have been removed from .c files, except
those in "contrib" which aren't our problem.  Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone.  Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort.  This part was easy to do automatically without (as far as I can
tell) any ill effect.

In the process, several pieces of dead code were flagged by the
compiler, and were removed.

backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155,
http://review.gluster.org/11769, BUG: 1245331

Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44
BUG: 1283302
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/12646
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

ctr/sql: Providing for vol set for sqlcachesize and sqlWALsize and skip recording path

2015-12-22T22:45:20+00:00

1. Providing vol set option for cache size and wal autocheck point
   so that performance can be tuned.
2. Removed recording of file path in the db. Trimming database columns.
   Path need not be stored in the db, as PARGFID, GFID, Basename is suffice
   to derive the path during migration.

Backport of http://review.gluster.org/12972

> Change-Id: I2cb590451a6d244bc91fe66c6dbffe2c2059dfb8
> BUG: 1293034
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12972
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Tested-by: NetBSD Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: Ia1c109983ec6ce75ed27b8c08f454f5b6283c31d
BUG: 1293659
Reviewed-on: http://review.gluster.org/13067
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/libgfdb/sql: Correcting logic in sql query for read

2015-12-05T00:46:51+00:00

Correcting query in the  where clause

from "a & b | c"

to "a & (b | c)"

where "a" is the condition to join the gf_file_tb and gf_flink_tb
through gfids

"b" is the condition for the write heat
and "c" is the condition for read heat

Backport of http://review.gluster.org/12823

> Change-Id: I99226d82b0efb68fbef3a40f02b215bb2b4370d6
> BUG: 1286656
> Signed-off-by: Joseph Fernandes 
> Signed-off-by: Dan Lambright 
> Reviewed-on: http://review.gluster.org/12823
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
Signed-off-by: Joseph Fernandes 

Change-Id: I70533c8589a6cd67e8737e110af0a23abed9013d
BUG: 1287583
Reviewed-on: http://review.gluster.org/12868
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Correcting rename logic

2015-11-26T17:19:35+00:00

Problem: When a file with old_file_name and GFID_1 is renamed with a new_file_name
which already exists and with GFID_2, this is what happens in linux internaly.
   a. "new_file_name" is unlinked for GFID_2
   b. a hardlink "new_file_name" is created to GFID_1
   c. "old_file_name" hardlink is unlinked for GFID_2.
Well this is all internal to linux, and gluster just issues a rename system call
at POSIX layer. But CTR Xlator doesn't delete the entries corresponding to the
"new_file_name" and GFID_2. Thus leaving the stale entry in the DB.
The following are the implications.
a. Promotion are tried on these stale entries which will fail and show
false results in the status of migration,
b. GFID_2 Files with 2 hardlinks, which will have only one hardlink
after the rename will not be promoted or demoted as the DB shows 2 entries.

Solution: Delete the older database entry for the replaced hardlink

Backport of http://review.gluster.org/12711

> Change-Id: I4eafa0872253e29ff1f0bec4283bcfc579ecf0e2
> BUG: 1284090
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12711
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: Ic35348303ec21f9bd19f20a48f3141449349668b
BUG: 1285688
Reviewed-on: http://review.gluster.org/12762
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Providing option to record or ignore metadata heat

2015-11-16T15:42:42+00:00

Currently we heat up a file for both data and metadata write.
Here we provide a ctr xlator option called "ctr-record-metadata-heat"
were the admin can decide on recording metadata heat i.e heatup a
file on metadata writes or not.
Metadata data operation are
a. setattr: explicit changing of atime/mtime using utimes,
            changing of posix permissions of the file
b. rename: Renaming a file,
c. unlink, link: adding or deleting hardlinks
d. xattrs: setting or removal of xattrs.

NOTE: atime, mtime and ctime change through writev, readv, truncate, mknod
and create will not be considered here as these fops are data and primary
metadata fops.

Defaultly "ctr-record-metadata-heat" is off. Admin can
switch it on using gluster volume set command.

Backport of http://review.gluster.org/12540

> Change-Id: I91157509255dd5cb429cda2b6d4f64582e155e7b
> BUG: 1279166
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12540
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: I986c319f0cc337b0692a1dd02f71254e786afac4
BUG: 1282315
Reviewed-on: http://review.gluster.org/12582
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/libgfdb: Extending log level flexibity in libgfdb

2015-11-12T01:48:33+00:00

Extending log level flexibity to relevant fops and operations
This is an extension to
http://review.gluster.org/#/c/12491/

Backport of http://review.gluster.org/#/c/12567/

> Change-Id: I33b2f7732f89f52569fb99baa692c7e3bb4c7ab1
> BUG: 1277352
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12567
> Tested-by: NetBSD Build System 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 
Signed-off-by: Joseph Fernandes 

Change-Id: Idcde04a65775ce98367967faf8a53b0f07ff022e
BUG: 1279059
Reviewed-on: http://review.gluster.org/12569
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/libgfdb: Replacing ASCII query file with binary

2015-11-08T13:22:02+00:00

Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.

Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.

The on disk format of query record is as follows,

+---------------------------------------------------------------------------+
| Length of serialized query record |       Serialized Query Record         |
+---------------------------------------------------------------------------+
             4 bytes                     Length of serialized query record
                                                      |
                                                      |
     -------------------------------------------------|
     |
     |
     V
   Serialized Query Record Format:
   +---------------------------------------------------------------------------+
   | GFID |  Link count   |    |.....                      | FOOTER |
   +---------------------------------------------------------------------------+
     16 B        4 B         Link Length                                  4 B
                                |                                          |
                                |                                          |
   -----------------------------|                                          |
   |                                                                       |
   |                                                                       |
   V                                                                       |
   Each  will be serialized as                                  |
   +-----------------------------------------------+                       |
   | PGID | BASE_NAME_LENGTH |      BASE_NAME      |                       |
   +-----------------------------------------------+                       |
     16 B       4 B             BASE_NAME_LENGTH                           |
                                                                           |
                                                                           |
   ------------------------------------------------------------------------|
   |
   |
   V
   FOOTER is a magic number 0xBAADF00D indicating the end of the record.
   This also serves as a serialized schema validator.

Backport of http://review.gluster.org/#/c/12354/

> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes 
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran 
> Tested-by: Gluster Build System 
> Reviewed-by: Dan Lambright 
> Tested-by: Dan Lambright 

Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/ctr: Ignore CTR Lookup heal insert errors

2015-11-07T18:32:33+00:00

CTR doesnt read from the DB, so to make sure that file records are
created it does a heal during a lookup. It remembers the decision in
the inode context cache and retrys periodically. When the volume is
restarted it looses all the inode cache from the previous time and CTR
lookup heals tries the heal again, but this time it finds that the records
are already there from sql and logs this error, and remembers this until the
volume is restarted or inode is flushed out of inode cache of the brick.

Solution: the log levels should be reduced to trace for this case and
customers need not see this.

Backport of http://review.gluster.org/#/c/12491/

Change-Id: I5f11e83c306ae912a90f12220f62bb7563b8eb96
BUG: 1279059
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/12534
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright