glusterfs.git/xlators/mgmt, branch v7dev glusterd: get-state command should not fail if any brick is gone bad 2019-02-05T14:40:25+00:00 Sanju Rakonde srakonde@redhat.com 2019-02-04T09:37:14+00:00 90922d20f55e26b23bfab0fbc4e179e305c38037 Problem: get-state command will error out, if any of the underlying brick(s) of volume(s) in the cluster go bad. It is expected that get-state command should not error out, but should generate an output successfully. Solution: In glusterd_get_state(), a statfs call is made on the brick path for every bricks of the volumes to calculate the total and free memory available. If any of statfs call fails on any brick, we should not error out and should report total memory and free memory of that brick as 0. This patch also handles a statfs failure scenario in glusterd_store_retrieve_bricks(). fixes: bz#1672205 Change-Id: Ia9e8a1d8843b65949d72fd6809bd21d39b31ad83 Signed-off-by: Sanju Rakonde <srakonde@redhat.com> 
Problem: get-state command will error out, if any of the underlying
brick(s) of volume(s) in the cluster go bad.

It is expected that get-state command should not error out, but
should generate an output successfully.

Solution: In glusterd_get_state(), a statfs call is made on the
brick path for every bricks of the volumes to calculate the total
and free memory available. If any of statfs call fails on any
brick, we should not error out and should report total memory and free
memory of that brick as 0.

This patch also handles a statfs failure scenario in
glusterd_store_retrieve_bricks().

fixes: bz#1672205

Change-Id: Ia9e8a1d8843b65949d72fd6809bd21d39b31ad83
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
glusterd: manage upgrade to current master 2019-02-04T03:47:15+00:00 Amar Tumballi amarts@redhat.com 2018-12-17T08:46:21+00:00 ec05f3a21f44e1fd5e089b7a0fffd5265b67cdfc Scenarios tested: * Upgrade the node when there are stripe / tiering and regular type of volumes are present. - All volumes are started fine (as the change was not on brick volfile) - For tier, the functionality may not even work, as changetimerecorder is not present. - 'gluster volume info' properly shows as 'NOT SUPPORTED' for stripe and tier type of volume. * Upgrade in a rolling upgrade scenario, where an old version is able to connect to higher master. - on a normal volume, if the volfile-server was new, the newer client volfiles needed to have utime xlator conditionally. - with this one change, all other changes seem to work fine. Change-Id: Ib2d3b69dafa02b2c695a735b13c1aa70aba07cb8 updates: bz#1635688 Signed-off-by: Amar Tumballi <amarts@redhat.com> 
Scenarios tested:

* Upgrade the node when there are stripe / tiering and regular
type of volumes are present.
  - All volumes are started fine (as the change was not on brick volfile)
  - For tier, the functionality may not even work, as changetimerecorder
    is not present.
  - 'gluster volume info' properly shows as 'NOT SUPPORTED' for stripe and
    tier type of volume.

* Upgrade in a rolling upgrade scenario, where an old version is
able to connect to higher master.
  - on a normal volume, if the volfile-server was new, the newer client
    volfiles needed to have utime xlator conditionally.
  - with this one change, all other changes seem to work fine.

Change-Id: Ib2d3b69dafa02b2c695a735b13c1aa70aba07cb8
updates: bz#1635688
Signed-off-by: Amar Tumballi <amarts@redhat.com>
features/sdfs: disable by default 2019-01-29T13:54:22+00:00 Amar Tumballi amarts@redhat.com 2019-01-28T13:00:24+00:00 829337ed3971a53086f1562d826e79d4f3e3ed39 With the feature enabled, some of the performance testing results, specially those which create millions of small files, got approximately 4x regression compared to version before enabling this. On master without this patch: 765 creates/sec On master with this patch : 3380 creates/sec Also there seems to be regression caused by this in 'ls -l' workload. On master without this patch: 3030 files/sec On master with this patch : 16610 files/sec This is a feature added to handle multiple clients parallely operating (specially those which race for file creates with same name) on a single namespace/directory. Considering that is < 3% of Gluster's usecase right now, it makes sense to disable the feature by default, so we don't penalize the default users who doesn't bother about this usecase. Also note that the client side translators, specially, distribute, replicate and disperse already handle the issue upto 99.5% of the cases without SDFS, so it makes sense to keep the feature disabled by default. Credits: Shyamsunder <srangana@redhat.com> for running the tests and getting the numbers. Change-Id: Iec49ce1d82e621e9db25eb633fcb1d932e74f4fc Updates: bz#1670031 Signed-off-by: Amar Tumballi <amarts@redhat.com> 
With the feature enabled, some of the performance testing results,
specially those which create millions of small files, got approximately
4x regression compared to version before enabling this.

On master without this patch:  765 creates/sec
On master with this patch   : 3380 creates/sec

Also there seems to be regression caused by this in 'ls -l' workload.

On master without this patch:  3030 files/sec
On master with this patch   : 16610 files/sec

This is a feature added to handle multiple clients parallely operating
(specially those which race for file creates with same name) on a single
namespace/directory. Considering that is < 3% of Gluster's usecase right
now, it makes sense to disable the feature by default, so we don't
penalize the default users who doesn't bother about this usecase.
Also note that the client side translators, specially, distribute,
replicate and disperse already handle the issue upto 99.5% of the cases
without SDFS, so it makes sense to keep the feature disabled by default.

Credits: Shyamsunder <srangana@redhat.com> for running the tests and
getting the numbers.

Change-Id: Iec49ce1d82e621e9db25eb633fcb1d932e74f4fc
Updates: bz#1670031
Signed-off-by: Amar Tumballi <amarts@redhat.com>
rpc: use address-family option from vol file 2019-01-22T13:47:19+00:00 Milind Changire mchangir@redhat.com 2019-01-22T06:40:59+00:00 b6c417785e54620331ae35d6971fe8bef98b4619 This patch helps enable IPv6 connections in the cluster. The default address-family is IPv4 without using this option explicitly. When address-family is set to "inet6" in the /etc/glusterfs/glusterd.vol file, the mount command-line also needs to have -o xlator-option="transport.address-family=inet6" added to it. This option also gets added to the brick command-line. Snapshot and gfapi use-cases should also use this option to pass in the inet6 address-family. Change-Id: I97db91021af27bacb6d7578e33ea4817f66d7270 fixes: bz#1635863 Signed-off-by: Milind Changire <mchangir@redhat.com> 
This patch helps enable IPv6 connections in the cluster.
The default address-family is IPv4 without using this option explicitly.

When address-family is set to "inet6" in the /etc/glusterfs/glusterd.vol
file, the mount command-line also needs to have
-o xlator-option="transport.address-family=inet6" added to it.

This option also gets added to the brick command-line.
Snapshot and gfapi use-cases should also use this option to pass in the
inet6 address-family.

Change-Id: I97db91021af27bacb6d7578e33ea4817f66d7270
fixes: bz#1635863
Signed-off-by: Milind Changire <mchangir@redhat.com>
locks/fencing: Add a security knob for fencing 2019-01-22T05:23:44+00:00 Susant Palai spalai@redhat.com 2019-01-18T11:56:36+00:00 3c556353cd1dde0593096c9e9e11b877403971f0 There is a low level security issue with fencing since one client can preempt another client's lock. This patch does not completely eliminate the issue of a client misbehaving, but certainly it adds a security layer for default use cases that does not need fencing. Change-Id: I55cd15f2ed1ae0f2556e3d27a2ef4bc10fdada1c updates: #466 Signed-off-by: Susant Palai <spalai@redhat.com> 
There is a low level security issue with fencing since one client
can preempt another client's lock.

This patch does not completely eliminate the issue of a client
misbehaving, but certainly it adds a security layer for default use cases
that does not need fencing.

Change-Id: I55cd15f2ed1ae0f2556e3d27a2ef4bc10fdada1c
updates: #466
Signed-off-by: Susant Palai <spalai@redhat.com>
glusterd: Avoid dict_leak in __glusterd_handle_cli_uuid_get function 2019-01-22T05:12:50+00:00 Mohit Agrawal moagrawal@redhat.com 2019-01-21T12:23:12+00:00 4deeab02e3d15bd266f24d0f7b28f0e5401fa950 Change-Id: Iefe08b136044495f6fa2b092c9e8c833efee1400 fixes: bz#1667905 Signed-off-by: Mohit Agrawal <moagrawal@redhat.com> 
Change-Id: Iefe08b136044495f6fa2b092c9e8c833efee1400
fixes: bz#1667905
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
glusterd: Resolve memory leak in get-state command 2019-01-21T11:10:58+00:00 Mohit Agrawal moagrawal@redhat.com 2019-01-21T03:42:30+00:00 5903111ad21cb937258c0fda24ea7dec466347b4 In gluster get-state volumeoptions command there was some amount of leak observed. This fix resolves the identified leaks. Change-Id: Ibde5743d1136fa72c531d48bb1b0b5da0c0b82a1 fixes: bz#1667779 Signed-off-by: Mohit Agrawal <moagrawal@redhat.com> 
In gluster get-state volumeoptions command there was some amount of leak
observed. This fix resolves the identified leaks.

Change-Id: Ibde5743d1136fa72c531d48bb1b0b5da0c0b82a1
fixes: bz#1667779
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
core: Resolve dict_leak at the time of destroying graph 2019-01-14T04:01:54+00:00 Mohit Agrawal moagrawal@redhat.com 2018-11-30T10:37:39+00:00 211521f039bb5c883ef444577b5962bad9e18be1 Problem: In gluster code some of the places it call's get_new_dict to create a dictionary without taking reference so at the time of dict_unref it has become a leak Solution: To resolve the same call dict_new instead of get_new_dict updates bz#1650403 Change-Id: I3ccbbf5af07079a4fa09aad2cd0458c8625b2f06 Signed-off-by: Mohit Agrawal <moagrawal@redhat.com> 
Problem: In gluster code some of the places it call's get_new_dict
         to create a dictionary without taking reference so at the time
         of dict_unref it has become a leak

Solution: To resolve the same call dict_new instead of get_new_dict

updates bz#1650403
Change-Id: I3ccbbf5af07079a4fa09aad2cd0458c8625b2f06
Signed-off-by: Mohit Agrawal <moagrawal@redhat.com>
glusterd: fix crash 2019-01-13T12:50:46+00:00 Sanju Rakonde srakonde@redhat.com 2019-01-10T10:47:39+00:00 741f652769bc074fe85da1485e0e16df6e6766f1 Problem: running "gluster get-state glusterd odir /get-state" resulted in glusterd crash. Cause: In the above command output directory has been specified without "/" at the end. If "/" is not given at the end, "/" will be added to path using "strcat", so the added character "/" is not having memory allocated. When tried to free, glusterd will crash as"/" has no memory allocated. Solution: Instead of concatenating "/" to output directory, add it to output filename. Change-Id: I5dc00a71e46fbef4d07fe99ae23b36fb60dec1c2 fixes: bz#1665038 Signed-off-by: Sanju Rakonde <srakonde@redhat.com> 
Problem: running "gluster get-state glusterd odir /get-state"
resulted in glusterd crash.

Cause: In the above command output directory has been specified
without "/" at the end. If "/" is not given at the end, "/" will
be added to path using "strcat", so the added character "/" is
not having memory allocated. When tried to free, glusterd will
crash as"/" has no memory allocated.

Solution: Instead of concatenating "/" to output directory, add
it to output filename.

Change-Id: I5dc00a71e46fbef4d07fe99ae23b36fb60dec1c2
fixes: bz#1665038
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>
glusterd: rebalance status should display information of localhost 2019-01-04T12:43:46+00:00 Sanju Rakonde srakonde@redhat.com 2019-01-04T10:19:53+00:00 95ec7391d8db26b933e542694add2b419dddf89f Problem: https://review.gluster.org/#/c/glusterfs/+/21762/ has migrated rebalance commands from op-sm framework to mgmt v3 framework. In a heterogenous cluster, if rebalance commands follow op-sm framework, localhost information is not displayed in the output of "gluster v rebalance <volname> status". Cause: Previously without https://review.gluster.org/#/c/glusterfs/+/21762/ rebalance commands were following op-sm framework. In glusterd_volume_rebalance_use_rsp_dict() current_index variable keeps track of number/count of peers in trusted storage pool. In op-sm, glusterd_volume_rebalance_use_rsp_dict() will be called only for the peers. So the current index should start from 2 assuming local host as node 1. With the above patch, rebalance commands are following mgmt v3 framework. In mgmt v3, glusterd_volume_rebalance_use_rsp_dict() is called for all nodes. For localhost it is called from brick-op function and for peers it is called from brick-op call back function. So the current index value should start from 1. https://review.gluster.org/#/c/glusterfs/+/21762/ has changed the value of current index to 1. Because of this, In heterogenous cluster, local host's information is overwritten by one of the peers information. And rebalance status will not display localhost's information in the output. Solution: assign a value to current index based on a op-version check. Change-Id: I2dfba1f007e908cf160acc4a4a5d8ef672572e4d fixes: bz#1663243 Signed-off-by: Sanju Rakonde <srakonde@redhat.com> 
Problem:
https://review.gluster.org/#/c/glusterfs/+/21762/ has migrated
rebalance commands from op-sm framework to mgmt v3 framework.

In a heterogenous cluster, if rebalance commands follow op-sm
framework, localhost information is not displayed in the
output of "gluster v rebalance <volname> status".

Cause:
Previously without https://review.gluster.org/#/c/glusterfs/+/21762/
rebalance commands were following op-sm framework.
In glusterd_volume_rebalance_use_rsp_dict() current_index variable
keeps track of number/count of peers in trusted storage pool.
In op-sm, glusterd_volume_rebalance_use_rsp_dict() will be called
only for the peers. So the current index should start from 2
assuming local host as node 1.

With the above patch, rebalance commands are following mgmt v3
framework. In mgmt v3, glusterd_volume_rebalance_use_rsp_dict()
is called for all nodes. For localhost it is called from
brick-op function and for peers it is called from brick-op
call back function. So the current index value should start
from 1.

https://review.gluster.org/#/c/glusterfs/+/21762/ has changed the
value of current index to 1. Because of this, In heterogenous cluster,
local host's information is overwritten by one of the peers information.
And rebalance status will not display localhost's information in
the output.

Solution: assign a value to current index based on a op-version
check.

Change-Id: I2dfba1f007e908cf160acc4a4a5d8ef672572e4d
fixes: bz#1663243
Signed-off-by: Sanju Rakonde <srakonde@redhat.com>