glusterfs.git/xlators/mount, branch v3.11.0

Halo Replication feature for AFR translator

2017-05-08T05:37:07+00:00

	Backport of https://review.gluster.org/16177
		    https://review.gluster.org/17174

Merged both these patches to make sure IPV6 changes don't make it to 3.11 at all.

Summary:
Halo Geo-replication is a feature which allows Gluster or NFS clients to write
locally to their region (as defined by a latency "halo" or threshold if you
like), and have their writes asynchronously propagate from their origin to the
rest of the cluster.  Clients can also write synchronously to the cluster
simply by specifying a halo-latency which is very large (e.g. 10seconds) which
will include all bricks.

In other words, it allows clients to decide at mount time if they desire
synchronous or asynchronous IO into a cluster and the cluster can support both
of these modes to any number of clients simultaneously.

There are a few new volume options due to this feature:
  halo-shd-latency:  The threshold below which self-heal daemons will
  consider children (bricks) connected.

  halo-nfsd-latency: The threshold below which NFS daemons will consider
  children (bricks) connected.

  halo-latency: The threshold below which all other clients will
  consider children (bricks) connected.

  halo-min-replicas: The minimum number of replicas which are to
  be enforced regardless of latency specified in the above 3 options.
  If the number of children falls below this threshold the next
  best (chosen by latency) shall be swapped in.

New FUSE mount options:
  halo-latency & halo-min-replicas: As descripted above.

This feature combined with multi-threaded SHD support (D1271745) results in
some pretty cool geo-replication possibilities.

Operational Notes:
- Global consistency is gaurenteed for synchronous clients, this is provided by
  the existing entry-locking mechanism.
- Asynchronous clients on the other hand and merely consistent to their region.
  Writes & deletes will be protected via entry-locks as usual preventing
  concurrent writes into files which are undergoing replication.  Read operations
  on the other hand should never block.
- Writes are allowed from _any_ region and propagated from the origin to all
  other regions.  The take away from this is care should be taken to ensure
  multiple writers do not write the same files resulting in a gfid split-brain
  which will require resolution via split-brain policies (majority, mtime &
  size).  Recommended method for preventing this is using the nfs-auth feature to
  define which region for each share has RW permissions, tiers not in the origin
  region should have RO perms.

TODO:
- Synchronous clients (including the SHD) should choose clients from their own
  region as preferred sources for reads.  Most of the plumbing is in place for
  this via the child_latency array.
- Better GFID split brain handling & better dent type split brain handling
  (i.e. create a trash can and move the offending files into it).
- Tagging in addition to latency as a means of defining which children you wish
  to synchronously write to

Test Plan:
- The usual suspects, clang, gcc w/ address sanitizer & valgrind
- Prove tests

Reviewers: jackl, dph, cjh, meyering

Reviewed By: meyering

Subscribers: ethanr

Differential Revision: https://phabricator.fb.com/D1272053

Tasks: 4117827

 >Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
 >BUG: 1428061
 >Signed-off-by: Kevin Vigor 
 >Reviewed-on: http://review.gluster.org/16099
 >Reviewed-on: https://review.gluster.org/16177
 >Tested-by: Pranith Kumar Karampuri 
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Pranith Kumar Karampuri 

BUG: 1448416
Change-Id: I694a9ab429722da538da171ec528406e77b5e6d1
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17192
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaushal M

fuse: enhance fusedump to include timestamp and a signature

2017-04-30T08:36:25+00:00

(Also referred to as "fusedump v2".)

Change-Id: I837944024efd1b9055c2f5f91bd5723ef350e688
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/16422
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
Tested-by: Raghavendra G 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

mount/fuse: Replace GF_LOG_OCCASIONALLY with gf_log() to report fop failure at all times

2017-04-30T08:35:47+00:00

Change-Id: Ibd8e1c6172812951092ff6097ba4bed943051b7c
BUG: 1440051
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17086
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Bhat

fuse: clean up mount flag processing

2017-04-27T17:38:48+00:00

In general, when one invokes a mount helper program -- basically
anything that mounts something based on its command line, so thinking of
mount(8), mount. or fusermount, but also of FUSE servers in
general, including glusterfs -- the command line arguments that are to
affect mount(2) are mapped to a bitmask called the mount flags, which is
passed to mount(2), so that the kernel can interpret the flag bits and
adjusts properties of the mount accordingly.

There is a traditional syntax for this mechanism as implemented in
mount(8): one passes "-ocomma,separated,mount,options" and the
individual option name strings are mapped to flag bits in mount(8).

FUSE further explores this idea and typically the FUSE server command
lines allow further option names to be used in the "-ooption,name,list"
which are then separated from the kernel sanctioned option names (to
which we'll refer as "system mount options") and are passed to a
platform specific lower level fuse mount helper interface.

The separation of system mount option names and FUSE specific option
names is also platform specific, so the general mount interface
function, which in case of glusterfs is gf_fuse_mount(), should abstract
this away.

Therefore we change the signature of this function from

        int gf_fuse_mount (const char *mountpoint, char *fsname,
                           unsigned long mountflags, char *mnt_param,
                           pid_t *mtab_pid, int status_fd);

to

        int gf_fuse_mount (const char *mountpoint, char *fsname,
                           char *mnt_param, pid_t *mtab_pid,
                           int status_fd);

and deal with flag extraction in platform specific mount code. Note that
the sole purpose of the mountflags argument was to indicate read-only
mounting. The other system mount option names were expected to reside in
the comma-separated mnt_param string, but they were not properly
processed (see the referred BUG). With the new gf_fuse_mount signature
read-only mounting is to be indicated as a "ro" component in mnt_param.

- For Darwin, which has a dedicated, separate gf_fuse_mount
  implementation, gf_fuse_mount was ignoring mountflags, so only the
  signature had to to be adjusted. However, as bonus, we gain read-only
  support for Darwin, which was missing so far, given that it was
  indicated via the ignored mountflags. Darwin's low level mount helper
  relies on the "ro" component of the option string, which agrees with
  the new calling convention of gf_fuse_mount.

- On Linux, system mount option name handling (apart from the
  distinguished read-only option) used to have the inadvertent side
  effect of adding "nosuid,nodev" as indicated in BUG; since
  Ia89d975d1e27fcfa5ab2036ba546aa8fa0d2d1b0 this side effect is removed,
  but system mount option name handling was left broken (passing system
  mount options other than "ro" fails to mount).

- On other platforms, system mount option name handling is broken
  (expect for the distinguished read-only option).

As of this change, in the general (non-Darwin) implementation of
gf_fuse_mount we take care of proper separation of system mount names
and their conversion to mount flags. For Linux, we adopt the conversion
table from FUSE upstream. For other systems we just provide a best
effort to support those system mount options which are understood across
all Unices (nosuid,nodev,noatime,noexec,ro). (This can be improved later
to provide proper plaform support.)

BUG: 1297182
Change-Id: I5d10b5df46feba7a02bf5bf1018db69e6b52260a
Signed-off-by: Csaba Henk 
Reviewed-on: https://review.gluster.org/16313
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Amar Tumballi 
Tested-by: Amar Tumballi

fuse-bridge: fuse_getattr(): send root lookup() only in case of failure.

2017-04-07T17:22:05+00:00

this will make sure we don't fail in any current cases, and also will
enable the performance better.

Change-Id: Ia421e1913e1b00f0730a004bf7c84bf7e2a62636
BUG: 1437780
Signed-off-by: Amar Tumballi 
Reviewed-on: https://review.gluster.org/16945
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

core: run many bricks within one glusterfsd process

2017-01-31T00:13:58+00:00

This patch adds support for multiple brick translator stacks running
in a single brick server process.  This reduces our per-brick memory usage by
approximately 3x, and our appetite for TCP ports even more.  It also creates
potential to avoid process/thread thrashing, and to improve QoS by scheduling
more carefully across the bricks, but realizing that potential will require
further work.

Multiplexing is controlled by the "cluster.brick-multiplex" global option.  By
default it's off, and bricks are started in separate processes as before.  If
multiplexing is enabled, then *compatible* bricks (mostly those with the same
transport options) will be started in the same process.

Change-Id: I45059454e51d6f4cbb29a4953359c09a408695cb
BUG: 1385758
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/14763
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

fuse: Fix a possible resource leak under GF_SOLARIS_HOST_OS

2017-01-17T20:53:35+00:00

in fuse-helpers.c

Change-Id: Ie367a6dec2a0d5848631b19ebbe39ceafa954a60
BUG: 1412918
Signed-off-by: Saurabh Badhwar 
Reviewed-on: http://review.gluster.org/16395
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Saravanakumar Arumugam 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

fuse: fix memory leak in setxattr

2017-01-12T19:03:09+00:00

If there's some failed check in setxattr of mount/fuse before
actually starting the operation, a fuse_state_t structure is
leaked.

This fix correctly releases allocated resources in case of
error.

Change-Id: I8b1cda67a613c13b6bc38947352e2ccfccf96a1d
BUG: 1412174
Signed-off-by: Xavier Hernandez 
Reviewed-on: http://review.gluster.org/16380
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

mount/fuse: Fix the place where graph-switch event is logged

2017-01-05T06:07:53+00:00

Change-Id: I3c8577b87db02a2a6ce6159e7d04cf58a2bda0c1
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/16302
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G

fuse: fix fuse dumping for FUSE_WRITE

2016-09-23T05:34:05+00:00

Data coming with FUSE_WRITE requests are arranged
with a special alignment, cf. 15d85ff1. fuse_dumper()
was not aware of this and didn't dump the proper
reqion for FUSE_WRITE.

BUG: 1377427
Signed-off-by: Csaba Henk 
Change-Id: I36255ca3336e95be6e2d256c8199761ddec41869
Reviewed-on: http://review.gluster.org/15525
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System