glusterfs.git/xlators/cluster/ec/src/ec-helpers.c, branch v7.6 cluster/ec: Don't enqueue an entry if it is already healing 2019-03-27T11:14:48+00:00 Ashish Pandey aspandey@redhat.com 2018-11-28T05:52:52+00:00 313dcefe7a62bd16cd794040df068f9bec9c6927 Problem: 1 - heal-wait-qlength is by default 128. If shd is disabled and we need to heal files, client side heal is needed. If we access these files that will trigger the heal. However, it has been observed that a file will be enqueued multiple times in the heal wait queue, which in turn causes queue to be filled and prevent other files to be enqueued. 2 - While a file is going through healing and a write fop from mount comes on that file, it sends write on all the bricks including healing one. At the end it updates version and size on all the bricks. However, it does not unset dirty flag on all the bricks, even if this write fop was successful on all the bricks. After healing completion this dirty flag remain set and never gets cleaned up if SHD is disabled. Solution: 1 - If an entry is already in queue or going through heal process, don't enqueue next client side request to heal the same file. 2 - Unset dirty on all the bricks at the end if fop has succeeded on all the bricks even if some of the bricks are going through heal. Change-Id: Ia61ffe230c6502ce6cb934425d55e2f40dd1a727 updates: bz#1593224 Signed-off-by: Ashish Pandey <aspandey@redhat.com> 
Problem:
1 - heal-wait-qlength is by default 128. If shd is disabled
and we need to heal files, client side heal is needed.
If we access these files that will trigger the heal.
However, it has been observed that a file will be enqueued
multiple times in the heal wait queue, which in turn causes
queue to be filled and prevent other files to be enqueued.

2 - While a file is going through healing and a write fop from
mount comes on that file, it sends write on all the bricks including
healing one. At the end it updates version and size on all the
bricks. However, it does not unset dirty flag on all the bricks,
even if this write fop was successful on all the bricks.
After healing completion this dirty flag remain set and never
gets cleaned up if SHD is disabled.

Solution:
1 - If an entry is already in queue or going through heal process,
don't enqueue next client side request to heal the same file.

2 - Unset dirty on all the bricks at the end if fop has succeeded on
all the bricks even if some of the bricks are going through heal.

Change-Id: Ia61ffe230c6502ce6cb934425d55e2f40dd1a727
updates: bz#1593224
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
libglusterfs: Move devel headers under glusterfs directory 2018-12-05T21:47:04+00:00 ShyamsundarR srangana@redhat.com 2018-11-29T19:08:06+00:00 20ef211cfa5b5fcc437484a879fdc5d4c66bbaf5 libglusterfs devel package headers are referenced in code using include semantics for a program, this while it works can be better especially when dealing with out of tree xlator builds or in general out of tree devel package usage. Towards this, the following changes are done, - moved all devel headers under a glusterfs directory - Included these headers using system header notation <> in all code outside of libglusterfs - Included these headers using own program notation "" within libglusterfs This change although big, is just moving around the headers and making it correct when including these headers from other sources. This helps us correctly include libglusterfs includes without namespace conflicts. Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b Updates: bz#1193929 Signed-off-by: ShyamsundarR <srangana@redhat.com> 
libglusterfs devel package headers are referenced in code using
include semantics for a program, this while it works can be better
especially when dealing with out of tree xlator builds or in
general out of tree devel package usage.

Towards this, the following changes are done,
- moved all devel headers under a glusterfs directory
- Included these headers using system header notation <> in all
code outside of libglusterfs
- Included these headers using own program notation "" within
libglusterfs

This change although big, is just moving around the headers and
making it correct when including these headers from other sources.

This helps us correctly include libglusterfs includes without
namespace conflicts.

Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b
Updates: bz#1193929
Signed-off-by: ShyamsundarR <srangana@redhat.com>
Land part 2 of clang-format changes 2018-09-12T12:22:45+00:00 Gluster Ant bugzilla-bot@gluster.org 2018-09-12T12:22:45+00:00 e16868dede6455cab644805af6fe1ac312775e13 Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4 Signed-off-by: Nigel Babu <nigelb@redhat.com> 
Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu <nigelb@redhat.com>
multiple files: remove unndeeded memset() 2018-08-29T17:31:11+00:00 Yaniv Kaul ykaul@redhat.com 2018-08-24T17:59:30+00:00 cbe133c832fd9907619a1231aa3d9dda1330ae74 This is a squash of multiple commits: contrib/fuse-lib/misc.c: remove unneeded memset() All flock variables are properly set, no need to memset it. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I8e0512c5a88daadb0e587f545fdb9b32ca8858a2 libglusterfs/src/{client_t|fd|inode|stack}.c: remove some memset() I don't think there's a need for any of them. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I2be9ccc3a5cb5da51a92af73488cdabd1c527f59 libglusterfs/src/xlator.c: remove unneeded memset() All xl->mem_acct members are properly set, no need to memset it. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I7f264cd47e7a06255a3f3943c583de77ae8e3147 xlators/cluster/afr/src/afr-self-heal-common.c: remove unneeded memset() Since we are going over the whole array anyway, initialize it properly, to either 1 or 0. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: Ied4210388976b6a7a2e91cc3de334534d6fef201 xlators/cluster/dht/src/dht-common.c: remove unneeded memset() Since we are going over the whole array anyway it is initialized properly. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: Idc436d2bd0563b6582908d7cbebf9dbc66a42c9a xlators/cluster/ec/src/ec-helpers.c: remove unneeded memset() Since we are going over the whole array anyway it is initialized properly. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I81bf971f7fcecb4599e807d37f426f55711978fa xlators/mgmt/glusterd/src/glusterd-volgen.c: remove some memset() I don't think there's a need for any of them. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I476ea59ba53546b5153c269692cd5383da81ce2d xlators/mgmt/glusterd/src/glusterd-geo-rep.c: read() in 4K blocks The current 1K seems small. 4K is usually better (in Linux). Also remove a memset() that I don't think is needed between reads. Only compile-tested! Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I5fb7950c92d282948376db14919ad12e589eac2b xlators/storage/posix/src/posix-{gfid-path|inode-fd-ops}.c: remove memset() before sys_*xattr() functions. I don't see a reason to memset the array sent to the functions sys_llistxattr(), sys_lgetxattr(), sys_lgetxattr(), sys_flistxattr(), sys_fgetxattr(). (Note: it's unclear to me why we are calling sys_*txattr() functions with XATTR_VAL_BUF_SIZE-1 size instead of XATTR_VAL_BUF_SIZE ). Only compile-tested! Change-Id: Ief2103b56ba6c71e40ed343a93684eef6b771346 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com> 
This is a squash of multiple commits:

contrib/fuse-lib/misc.c: remove unneeded memset()

All flock variables are properly set, no need to memset it.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: I8e0512c5a88daadb0e587f545fdb9b32ca8858a2

libglusterfs/src/{client_t|fd|inode|stack}.c: remove some memset()

I don't think there's a need for any of them.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: I2be9ccc3a5cb5da51a92af73488cdabd1c527f59

libglusterfs/src/xlator.c: remove unneeded memset()

All xl->mem_acct members are properly set,
no need to memset it.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: I7f264cd47e7a06255a3f3943c583de77ae8e3147

xlators/cluster/afr/src/afr-self-heal-common.c: remove unneeded memset()

Since we are going over the whole array anyway, initialize it
properly, to either 1 or 0.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: Ied4210388976b6a7a2e91cc3de334534d6fef201

xlators/cluster/dht/src/dht-common.c: remove unneeded memset()

Since we are going over the whole array anyway it is initialized
properly.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: Idc436d2bd0563b6582908d7cbebf9dbc66a42c9a

xlators/cluster/ec/src/ec-helpers.c: remove unneeded memset()

Since we are going over the whole array anyway it is initialized
properly.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: I81bf971f7fcecb4599e807d37f426f55711978fa

xlators/mgmt/glusterd/src/glusterd-volgen.c: remove some memset()

I don't think there's a need for any of them.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: I476ea59ba53546b5153c269692cd5383da81ce2d

xlators/mgmt/glusterd/src/glusterd-geo-rep.c: read() in 4K blocks

The current 1K seems small. 4K is usually better (in Linux).

Also remove a memset() that I don't think is needed between reads.

Only compile-tested!

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

Change-Id: I5fb7950c92d282948376db14919ad12e589eac2b

xlators/storage/posix/src/posix-{gfid-path|inode-fd-ops}.c: remove memset()
before sys_*xattr() functions.

I don't see a reason to memset the array sent to the functions
sys_llistxattr(), sys_lgetxattr(), sys_lgetxattr(), sys_flistxattr(),
sys_fgetxattr().

(Note: it's unclear to me why we are calling sys_*txattr() functions with
XATTR_VAL_BUF_SIZE-1 size instead of XATTR_VAL_BUF_SIZE ).

Only compile-tested!

Change-Id: Ief2103b56ba6c71e40ed343a93684eef6b771346
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
multiple files: move from strlen() to sizeof() 2018-08-25T17:10:11+00:00 Yaniv Kaul ykaul@redhat.com 2018-08-21T15:56:24+00:00 14ee864f8b573d1bccf4f586450947ba4aa690a9 {ec-heal|ec-combine|ec-helpers|ec-inode-read}.c For const strings, just do compile time size calc instead of runtime. Compile-tested only! Change-Id: If92ba0a7a20f64b898d01c6e3b6708190ca93e04 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com> 
{ec-heal|ec-combine|ec-helpers|ec-inode-read}.c

For const strings, just do compile time size calc instead of runtime.

Compile-tested only!

Change-Id: If92ba0a7a20f64b898d01c6e3b6708190ca93e04
updates: bz#1193929
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
cluster/ec: OpenFD heal implementation for EC 2018-01-05T06:55:44+00:00 Sunil Kumar Acharya sheggodu@redhat.com 2017-03-23T07:20:41+00:00 78d67da17356b48cf1d5a6595764650d5b200ba7 Existing EC code doesn't try to heal the OpenFD to avoid unnecessary healing of the data later. Fix implements the healing of open FDs before carrying out file operations on them by making an attempt to open the FDs on required up nodes. BUG: 1431955 Change-Id: Ib696f59c41ffd8d5678a484b23a00bb02764ed15 Signed-off-by: Sunil Kumar Acharya <sheggodu@redhat.com> 
Existing EC code doesn't try to heal the OpenFD to
avoid unnecessary healing of the data later.

Fix implements the healing of open FDs before
carrying out file operations on them by making an
attempt to open the FDs on required up nodes.

BUG: 1431955
Change-Id: Ib696f59c41ffd8d5678a484b23a00bb02764ed15
Signed-off-by: Sunil Kumar Acharya <sheggodu@redhat.com>
cluster/ec: Keep last written strip in in-memory cache 2017-11-10T22:15:37+00:00 Ashish Pandey aspandey@redhat.com 2017-09-18T08:37:31+00:00 a87abbd42e8b02deabbdfe290b16ed0d2f2e4c45 Problem: Consider an EC volume with configuration 4 + 2. The stripe size for this would be 512 * 4 = 2048. That means, 2048 bytes of user data stored in one stripe. Let's say 2048 + 512 = 2560 bytes are already written on this volume. 512 Bytes would be in second stripe. Now, if there are sequential writes with offset 2560 and of size 1 Byte, we have to read the whole stripe, encode it with 1 Byte and then again have to write it back. Next, write with offset 2561 and size of 1 Byte will again READ-MODIFY-WRITE the whole stripe. This is causing bad performance because of lots of READ request travelling over the network. There are some tools and scenario's where such kind of load is coming and users are not aware of that. Example: fio and zip Solution: One possible solution to deal with this issue is to keep last stripe in memory. This way, we need not to read it again and we can save READ fop going over the network. Considering the above example, we have to keep last 2048 bytes (maximum) in memory per file. Change-Id: I3f95e6fc3ff81953646d374c445a40c6886b0b85 BUG: 1471753 Signed-off-by: Ashish Pandey <aspandey@redhat.com> 
Problem:
Consider an EC volume with configuration  4 + 2.
The stripe size for this would be 512 * 4 = 2048.
That means, 2048 bytes of user data stored in one
stripe. Let's say 2048 + 512 = 2560 bytes are
already written on this volume. 512 Bytes would
be in second stripe. Now, if there are sequential
writes with offset 2560 and of size 1 Byte, we have
to read the whole stripe, encode it with 1 Byte and
then again have to write it back. Next, write with
offset 2561 and size of 1 Byte will again
READ-MODIFY-WRITE the whole stripe. This is causing
bad performance because of lots of READ request
travelling over the network.

There are some tools and scenario's where such kind
of load is coming and users are not aware of that.
Example: fio and zip

Solution:
One possible solution to deal with this issue is to
keep last stripe in memory. This way, we need not to
read it again and we can save READ fop going over the
network. Considering the above example, we have to
keep last 2048 bytes (maximum) in memory per file.

Change-Id: I3f95e6fc3ff81953646d374c445a40c6886b0b85
BUG: 1471753
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
cluster/ec: add functions for stripe alignment 2017-10-13T08:17:27+00:00 Xavier Hernandez jahernan@redhat.com 2017-10-06T08:39:58+00:00 3dce15e10c263e8e071b26046568e0a171a3153d This patch removes old functions to align offsets and sizes to stripe size boundaries and adds new ones to offer more possibilities. The new functions are: * ec_adjust_offset_down() Aligns a given offset to a multiple of the stripe size equal or smaller than the initial one. It returns the size of the gap between the aligned offset and the given one. * ec_adjust_offset_up() Aligns a given offset to a multiple of the stripe size equal or greater than the initial one. It returns the size of the skipped region between the given offset and the aligned one. If an overflow happens, the returned valid has negative sign (but correct value) and the offset is set to the maximum value (not aligned). * ec_adjust_size_down() Aligns the given size to a multiple of the stripe size equal or smaller than the initial one. It returns the size of the missed region between the aligned size and the given one. * ec_adjust_size_up() Aligns the given size to a multiple of the stripe size equal or greater than the initial one. It returns the size of the gap between the given size and the aligned one. If an overflow happens, the returned value has negative sign (but correct value) and the size is set to the maximum value (not aligned). These functions have been defined in ec-helpers.h as static inline since they are very small and compilers can optimize them (specially the 'scale' argument). Change-Id: I4c91009ad02f76c73772034dfde27ee1c78a80d7 Signed-off-by: Xavier Hernandez <jahernan@redhat.com> 
This patch removes old functions to align offsets and sizes
to stripe size boundaries and adds new ones to offer more
possibilities.

The new functions are:

 * ec_adjust_offset_down()
     Aligns a given offset to a multiple of the stripe size
     equal or smaller than the initial one. It returns the
     size of the gap between the aligned offset and the given
     one.

 * ec_adjust_offset_up()
     Aligns a given offset to a multiple of the stripe size
     equal or greater than the initial one. It returns the
     size of the skipped region between the given offset and
     the aligned one. If an overflow happens, the returned
     valid has negative sign (but correct value) and the
     offset is set to the maximum value (not aligned).

 * ec_adjust_size_down()
     Aligns the given size to a multiple of the stripe size
     equal or smaller than the initial one. It returns the
     size of the missed region between the aligned size and
     the given one.

 * ec_adjust_size_up()
     Aligns the given size to a multiple of the stripe size
     equal or greater than the initial one. It returns the
     size of the gap between the given size and the aligned
     one. If an overflow happens, the returned value has
     negative sign (but correct value) and the size is set
     to the maximum value (not aligned).

These functions have been defined in ec-helpers.h as static
inline since they are very small and compilers can optimize
them (specially the 'scale' argument).

Change-Id: I4c91009ad02f76c73772034dfde27ee1c78a80d7
Signed-off-by: Xavier Hernandez <jahernan@redhat.com>
cluster/ec: FORWARD_NULL coverity fix 2017-09-29T15:09:36+00:00 Akarsha Rai akrai@redhat.com 2017-09-26T10:09:28+00:00 687908a364efcfb5ed69ff3dd0a8ef5b1551e88e Problem: ctx pointer could be NULL Solution: Updated the code to verify ctx pointer BUG: 789278 Change-Id: I25e07a07c6ebe2f630c99ba3aa9a61656fbaa981 Signed-off-by: Akarsha Rai <akrai@redhat.com> 
Problem: ctx pointer could be NULL

Solution: Updated the code to verify ctx pointer

BUG: 789278
Change-Id: I25e07a07c6ebe2f630c99ba3aa9a61656fbaa981
Signed-off-by: Akarsha Rai <akrai@redhat.com>
core: fix spelling errors 2017-06-02T11:50:43+00:00 Kaleb S. KEITHLEY kkeithle@redhat.com 2017-06-01T10:56:22+00:00 07fd39479bdd6502f7781894be06eb66aaa8ef10 fixes for various minor spelling errors and typos Reported-by: Patrick Matthäi <pmatthaei@debian.org> Change-Id: Ic1be36f82e3d822bbdc9559878bd79520fc0fcd5 BUG: 1457808 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: https://review.gluster.org/17442 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Niels de Vos <ndevos@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.org> 
fixes for various minor spelling errors and typos

Reported-by: Patrick Matthäi <pmatthaei@debian.org>
Change-Id: Ic1be36f82e3d822bbdc9559878bd79520fc0fcd5
BUG: 1457808
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: https://review.gluster.org/17442
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Smoke: Gluster Build System <jenkins@build.gluster.org>