glusterfs.git/tests/bugs, branch v3.5.1beta2

rpc: implement server.manage-gids for group resolving on the bricks

2014-05-23T08:34:02+00:00

The new volume option 'server.manage-gids' can be enabled in
environments where a user belongs to more than the current absolute
maximum of 93 groups. This option triggers the following behavior:

1. The AUTH_GLUSTERFS structure sent by GlusterFS clients (fuse, nfs or
   libgfapi) will contain only one (1) auxiliary group, instead of
   a full list. This reduces network usage and prevents problems in
   encoding the AUTH_GLUSTERFS structure which should fit in 400 bytes.
2. The single group in the RPC Calls received by the server is replaced
   by resolving the groups server-side. Permission checks and similar in
   lower xlators are applied against the full list of groups where the
   user belongs to, and not the single auxiliary group that the client
   sent.

Cherry picked from commit 2fd499d148fc8865c77de8b2c73fe0b7e1737882:
> BUG: 1053579
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7501
> Tested-by: Gluster Build System 
> Reviewed-by: Santosh Pradhan 
> Reviewed-by: Harshavardhana 
> Reviewed-by: Anand Avati 

Change-Id: I9e540de13e3022f8b63ff893ecba511129a47b91
BUG: 1096425
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7830
Tested-by: Gluster Build System 
Reviewed-by: Santosh Pradhan

rpc: warn and truncate grouplist if RPC/AUTH can not hold everything

2014-05-22T13:02:21+00:00

The GlusterFS protocol currently uses AUTH_GLUSTERFS_V2 in the RPC/AUTH
header. This header contains the uid, gid and auxiliary groups of the
user/process that accesses the Gluster Volume.

The AUTH_GLUSTERFS_V2 structure allows up to 65535 auxiliary groups to
be passed on. Unfortunately, the RPC/AUTH header is limited to 400 bytes
by the RPC specification: http://tools.ietf.org/html/rfc5531#section-8.2

In order to not cause complete failures on the client-side when trying
to encode a AUTH_GLUSTERFS_V2 that would result in more than 400 bytes,
we can calculate the expected size of the other elements:

    1 | pid
    1 | uid
    1 | gid
    1 | groups_len
   XX | groups_val (GF_MAX_AUX_GROUPS=65535)
    1 | lk_owner_len
   YY | lk_owner_val (GF_MAX_LOCK_OWNER_LEN=1024)
  ----+-------------------------------------------
    5 | total xdr-units

  one XDR-unit is defined as BYTES_PER_XDR_UNIT = 4 bytes
  MAX_AUTH_BYTES = 400 is the maximum, this is 100 xdr-units.
  XX + YY can be 95 to fill the 100 xdr-units.

  Note that the on-wire protocol has tighter requirements than the
  internal structures. It is possible for xlators to use more groups and
  a bigger lk_owner than that can be sent by a GlusterFS-client.

This change prevents overflows when allocating the RPC/AUTH header. Two
new macros are introduced to calculate the number of groups that fit in
the RPC/AUTH header, when taking the size of the lk_owner in account. In
case the list of groups exceeds the maximum possible, only the first
groups are passed over the RPC/GlusterFS protocol to the bricks.
A warning is added to the logs, so that most system administrators will
get informed.

The reducing of the number of groups is not a new inventions. The
RPC/AUTH header (AUTH_SYS or AUTH_UNIX) that NFS uses has a limit of 16
groups. Most, if not all, NFS-clients will reduce any bigger number of
groups to 16. (nfs.server-aux-gids can be used to workaround the limit
of 16 groups, but the Gluster NFS-server will be limited to a maximum of
93 groups, or fewer in case the lk_owner structure contains more items.)

Cherry picked from commit 8235de189845986a535d676b1fd2c894b9c02e52:
> BUG: 1053579
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/7202
> Tested-by: Gluster Build System 
> Reviewed-by: Harshavardhana 
> Reviewed-by: Santosh Pradhan 
> Reviewed-by: Vijay Bellur 

Change-Id: I8410e59d0fd246d601b54b961d3ae9cb5a858c10
BUG: 1096425
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7829
Reviewed-by: Lalatendu Mohanty 
Tested-by: Gluster Build System

posix: if brick-uid or brick-gid is not specified, do not set

2014-05-09T14:36:02+00:00

Current code would set owner uid/gid explicitly to 0/0 on start
even if none was specified. Fix it.

Cherry picked from commit 8bdc329:
> Change-Id: I72dec9e79c51bd1eb3af5334c42b7c23b01d0258
> BUG: 1040275
> Signed-off-by: Anand Avati 
> Reviewed-on: http://review.gluster.org/6476
> Tested-by: Gluster Build System 
> Tested-by: Lukáš Bezdička 
> Reviewed-by: Krishnan Parthasarathi 
> Reviewed-by: Vijay Bellur 

Change-Id: Ie0396c1a4e6e0979ea9c855d33db963544a75c42
BUG: 1095971
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7720
Tested-by: Gluster Build System

write-behind: track filesize when doing extending writes

2014-05-09T14:33:39+00:00

A program that calls mmap() on a newly created sparse file, may receive
a SIGBUS signal. If SIGBUS is not handled, a segmentation fault will
occur and the program will exit.

A bug in the write-behind translator can cause the creation of a sparse
file created with open(), seek(), write() to be cached. The last write()
may not be sent to the server, until write-behind deems this necessary.

* open(.., O_TRUNC, ...)/creat() the file, it is 0 bytes big
* seek() into the file, use offset 31
* write() 1 byte to the file
* the range from byte 0-30 are unwritten so called 'sparse'

The following illustration tries to capture this:

    Legend:
    [ = start of file
    _ = unallocated/unwritten bytes
    # = allocated bytes in the file
    ] = end of file

    [_______________#]
     |              |
     '- byte 0      '- byte 31

Without this change, reading from byte 0-30 will return an error, and
reading the same area through an mmap()'d pointer will trigger a SIGBUS.
Reading from this range did not trigger the outstanding write() to be
flushed. The brick that receives the read() (translated over the network
from mmap()) does not know that the file has been extended, and returns
-EINVAL. This error gets transported back from the brick to the
glusterfs-fuse client, and translated by the Linux kernel/VFS into
SIGBUS triggered by mmap().

In order to solve this, a new attribute to the wb_inode structure is
introduced; the current size of the file. All FOPs that can modify the
size, are expected to update wb_inode->size. This makes it possible for
extending writes with an offset bigger than EOF to mark the unwritten
area as modified/pending.

Cherry picked from commit b0515e2a4a08b657ef7e9715fb8c6222c700e78c:
> Change-Id: If5ba6646732e6be26568541ea9b12852a5d0b988
> BUG: 1058663
> Signed-off-by: Niels de Vos 
> Reviewed-on: http://review.gluster.org/6835
> Tested-by: Gluster Build System 
> Reviewed-by: Raghavendra G 
> Reviewed-by: Anand Avati 

Change-Id: Ic73288202c6cb37d8c066628e9c489c6fade49bd
BUG: 1071191
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/7166
Tested-by: Gluster Build System

cluster/afr: Unable to self heal symbolic links

2014-05-08T13:31:35+00:00

Problem:
Under the entry self heal, readlink is done at the
source and sink. When readlink is done at the sink,
because link is not present at the sink, afr expects
ENOENT. AFR translator takes decisions for new link
creation based on ENOENT but server translator is modified
to return ESTALE because of which afr xlator is not able
to heal.

Fix:
The check for inode absence at server includes ESTALE as
well.

Change-Id: I9218da214ed44f7219570ad9dae298d6b5cbded9
BUG: 1046624
Signed-off-by: Venkatesh Somyajulu 
Reviewed-on: http://review.gluster.org/6600
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

storage/posix: do not dereference gfid symlinks before posix_handle_mkdir_hashes()

2014-05-05T13:38:03+00:00

Whenever a new directory is created, its corresponding gfid file must
also be created. This was done first calling MAKE_HANDLE_PATH() to get
the path of the gfid file, then calling posix_handle_mkdir_hashes() to
create the parent directories of the gfid, and finally creating the
soft-link.

In normal circumstances, the gfid we want to create won't exist and
MAKE_HANDLE_PATH() will return a simple path to the new gfid. However if
the volume is damaged and a self-heal is running, it is possible that we
try to create an already existing gfid. In this case, MAKE_HANDLE_PATH()
will return a path to the directory instead of the path to the gfid.

To solve this problem, every time a path to a gfid is needed, a call to
MAKE_HANDLE_ABSPATH() is made instead of the call to MAKE_HANDLE_PATH().

BUG: 859581
Change-Id: I84405bf04562e647fc02445f45358e9451f9b479
Signed-off-by: Xavier Hernandez 
Reviewed-on: http://review.gluster.org/6736
Tested-by: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY 
Reviewed-by: Raghavendra Bhat 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Niels de Vos

Tests: Changes to tests for glfs-heal implementation.

2014-04-28T16:40:16+00:00

volume heal  info does not depend on self-heald
anymore so removed one complete test tests/bugs/bug-880898.t
and also removed specific tests related to that in
tests/basic/self-heald.t.

Change-Id: I2e5cd3641b86cad63af41c9564d57ab6ebb60742
BUG: 1039544
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/6515
Tested-by: Gluster Build System 
Reviewed-by: Ravishankar N 
Reviewed-by: Vijay Bellur

storage/posix: perform chmod after chown.

2014-02-17T09:51:03+00:00

Problem:
When a replica brick is added to a volume, set-user-ID and set-group-ID
permission bits of files are not set correctly in the new brick. The issue
is in the posix_setattr() call where we do a chmod followed by a chown.

But according to the man pages for chown:
When the owner or group of an executable file are changed by an unprivileged
user the S_ISUID and S_ISGID mode bits are cleared.  POSIX does not specify
whether this also  should  happen  when  root does the chown().

Fix:
Swap the chmod and chown calls in posix_setattr()

BUG: 1058797
Change-Id: Id2fbd8394cf6faf669f414775409f20f46009f2b
Signed-off-by: Ravishankar N 
Reviewed-on: http://review.gluster.org/6988
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/marker: Filter quota xattrs on file as well

2014-02-03T04:26:13+00:00

Problem:
Quota contributions of a file/directory are tracked by quota
xlator using xattrs on the file. Quota allows these xattrs to be
healed as part of metadata self-heal. This leads to
wrong quota calculations on this brick after self-heal because
quota xattrs don't represent the actual contributions on the
brick anymore.

Fix:
Don't let self-heal of this xattr happen as part of self-heal
by filtering quota xattrs on file in listxattr.

Change-Id: Iea68a116595ba271e58c6fdcc3dd21c7bb55ebb3
BUG: 1035576
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/6374
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/6812

mgmt/glusterd: make sure quota enforcer has established connection with quotad before marking quota as enabled.

2014-01-29T06:46:35+00:00

without this patch there is a window of time when quota is marked as
enabled in quota-enforcer, but connection to quotad wouldn't have been
established. Any checklimit done during this period can result in a
failed fop because of unavailability of quotad.

Change-Id: I0d509fabc434dd55ce9ec59157123524197fcc80
Signed-off-by: Raghavendra G 
BUG: 969461
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/6572
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/6820