glusterfs.git/tests, branch release-3.8-fb glusterd: fix missing/renamed options 2017-10-02T19:23:14+00:00 Jeff Darcy jdarcy@fb.com 2017-10-02T19:23:14+00:00 8de2ec1b8ddecc9ba01bd918b6b5c208561b22fe Change-Id: I2ca0298ee9d166f58b8730256ea76a04e547ce5d Signed-off-by: Jeff Darcy <jdarcy@fb.com> 
Change-Id: I2ca0298ee9d166f58b8730256ea76a04e547ce5d
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
self-heal: fix automatic split-brain resolution options 2017-09-27T23:03:35+00:00 Jeff Darcy jdarcy@fb.com 2017-09-27T20:18:25+00:00 d9f9be442808ed13135f560698a2f95fe66282a5 Differential Revision: https://phabricator.intern.facebook.com/D5927193 Change-Id: Ife04c8738b9ee721e7be9bc843b2f6d54bbb468e 
Differential Revision: https://phabricator.intern.facebook.com/D5927193

Change-Id: Ife04c8738b9ee721e7be9bc843b2f6d54bbb468e
Replace namespace/io-stats/io-threads with 3.6-fb versions 2017-09-15T20:47:01+00:00 Jeff Darcy jdarcy@fb.com 2017-09-15T13:59:01+00:00 8dfdecf220d1c9365e1f8d6af9ead5e48c61e2eb This rolls up multiple patches related to namespace identificaton and throttling/QoS. This primarily includes the following, all by Michael Goulet <mgoulet@fb.com>. io-threads: Add weighted round robin queueing by namespace https://phabricator.facebook.com/D5615269 io-threads: Add per-namespaces queue sizes to IO_THREADS_QUEUE_SIZE_KEY https://phabricator.facebook.com/D5683162 io-threads: Implement better slot allocation algorithm https://phabricator.facebook.com/D5683186 io-threads: Only enable weighted queueing on bricks https://phabricator.facebook.com/D5700062 io-threads: Update queue sizes on drain https://phabricator.facebook.com/D5704832 Fix parsing (-1) as default NS weight https://phabricator.facebook.com/D5723383 Parts of the following patches have also been applied to satisfy dependencies. io-throttling: Calculate moving averages and throttle offending hosts https://phabricator.fb.com/D2516161 Shreyas Siravara <sshreyas@fb.com> Hook up ODS logging for FUSE clients. https://phabricator.facebook.com/D3963376 Kevin Vigor <kvigor@fb.com> Add the flag --skip-nfsd-start to skip the NFS daemon stating, even if it is enabled https://phabricator.facebook.com/D4575368 Alex Lorca <alexlorca@fb.com> There are also some "standard" changes: dealing with code that moved, reindenting to comply with Gluster coding standards, gf_uuid_xxx, etc. This patch *does* revert some changes which have occurred upstream since 3.6; these will be re-applied as apppropriate on top of this new base. Change-Id: I69024115da7a60811e5b86beae781d602bdb558d Signed-off-by: Jeff Darcy <jdarcy@fb.com> 
This rolls up multiple patches related to namespace identificaton and
throttling/QoS.  This primarily includes the following, all by Michael
Goulet <mgoulet@fb.com>.

  io-threads: Add weighted round robin queueing by namespace
  https://phabricator.facebook.com/D5615269

  io-threads: Add per-namespaces queue sizes to IO_THREADS_QUEUE_SIZE_KEY
  https://phabricator.facebook.com/D5683162

  io-threads: Implement better slot allocation algorithm
  https://phabricator.facebook.com/D5683186

  io-threads: Only enable weighted queueing on bricks
  https://phabricator.facebook.com/D5700062

  io-threads: Update queue sizes on drain
  https://phabricator.facebook.com/D5704832

  Fix parsing (-1) as default NS weight
  https://phabricator.facebook.com/D5723383

Parts of the following patches have also been applied to satisfy
dependencies.

  io-throttling: Calculate moving averages and throttle offending hosts
  https://phabricator.fb.com/D2516161
  Shreyas Siravara <sshreyas@fb.com>

  Hook up ODS logging for FUSE clients.
  https://phabricator.facebook.com/D3963376
  Kevin Vigor <kvigor@fb.com>

  Add the flag --skip-nfsd-start to skip the NFS daemon stating, even if
  it is enabled
  https://phabricator.facebook.com/D4575368
  Alex Lorca <alexlorca@fb.com>

There are also some "standard" changes: dealing with code that moved,
reindenting to comply with Gluster coding standards, gf_uuid_xxx, etc.

This patch *does* revert some changes which have occurred upstream since
3.6; these will be re-applied as apppropriate on top of this new base.

Change-Id: I69024115da7a60811e5b86beae781d602bdb558d
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Disable brick daemon from incorrect brick directory 2017-09-13T17:50:36+00:00 krad krad@fb.com 2017-07-18T23:41:49+00:00 8a3de0b4ca841cc2405b7e60ecf70f8eca62b800 Summary: Currently the bricks can open any mount directory from the given volume. This patch adds a provision to prevent bricks from opening brick directories that aren't created for them. This will help with operating gluster on large scale. We add a new xattr GF_XATTR_BRICK_NAME to the brick directory. When we start a brick daemon, we make sure the path on disk matches with the config provided. For backward compatibility, we ignore if there is no value for GF_XATTR_BRICK_NAME and set the current brick daemon's path as value. We ignore GF_XATTR_BRICK_NAME during healing and reset GF_XATTR_BRICK_NAME on brick replace. Test Plan: Run fb-smoke Reviewers: jdarcy, sshreyas Reviewed By: sshreyas Differential Revision: https://phabricator.intern.facebook.com/D5448921 Porting note: disabled some checks to deal with the snapshot case Change-Id: I98e62033dfd07f30ad3b99ac003ce94c8d935e5f Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18275 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
Summary:
Currently the bricks can open any mount directory from the given volume. This patch adds a provision to prevent
bricks from opening brick directories that aren't created for them. This will help with operating gluster on large
scale.

We add a new xattr GF_XATTR_BRICK_NAME to the brick directory. When we start a brick daemon, we make sure the path on
disk matches with the config provided. For backward compatibility, we ignore if there is no value for
GF_XATTR_BRICK_NAME and set the current brick daemon's path as value.

We ignore GF_XATTR_BRICK_NAME during healing and reset GF_XATTR_BRICK_NAME on brick replace.

Test Plan: Run fb-smoke

Reviewers: jdarcy, sshreyas

Reviewed By: sshreyas

Differential Revision: https://phabricator.intern.facebook.com/D5448921

Porting note: disabled some checks to deal with the snapshot case

Change-Id: I98e62033dfd07f30ad3b99ac003ce94c8d935e5f
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18275
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Remediation for XFS/DIO corruption problem. 2017-09-12T21:35:20+00:00 Jeff Darcy jdarcy@fb.com 2017-05-18T17:22:12+00:00 cf8a4d17fe9133f050560c4d655255a003185fe5 This adds a new volume option, shd-validate-data. When set, the self-heal code will fetch checksums for regular files along with all the usual xattrs. If the file seems OK but the checksums show a data mismatch, and if there is only one replica that's out of step with the others, then we modify the source/sink calculations to force a heal from one of the agreeing replicas to the odd one out. Combined with a tool to put files into the self-heal index (being developed separately), this provides a very rudimentary kind of scrubbing functionality. Validation is now conditional on the "trusted.glusterfs.validate-status" xattr having the specific value of "suspect" to avoid redoing validation (which is expensive) as we find the same file in multiple bricks' indices. When we decide to take action, we update this xattr to "clean" for copies that were in the majority and "repaired" for the odd one out that gets clobbered. We also copy the about-to-be-clobbered copy into an "orphans" directory to facilitate analysis of corruption patterns. The data goes into ${GFID}.data there, while ${GFID}.link is a symlink to the file's old location. Porting note: this is several internal squashed together ("See Also") Differential Revision: https://phabricator.intern.facebook.com/D5092983 See Also: https://phabricator.intern.facebook.com/D5126974 See Also: https://phabricator.intern.facebook.com/D5127427 See Also: https://phabricator.intern.facebook.com/D5132804 See Also: https://phabricator.intern.facebook.com/D5209185 See Also: https://phabricator.intern.facebook.com/D5370353 Change-Id: Ie0ae18b368c408a5e47d0bf03ebac80b87b70aa9 Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18269 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
This adds a new volume option, shd-validate-data. When set, the self-heal code
will fetch checksums for regular files along with all the usual xattrs. If the
file seems OK but the checksums show a data mismatch, and if there is only one
replica that's out of step with the others, then we modify the source/sink
calculations to force a heal from one of the agreeing replicas to the odd one
out. Combined with a tool to put files into the self-heal index (being developed
separately), this provides a very rudimentary kind of scrubbing functionality.

Validation is now conditional on the "trusted.glusterfs.validate-status" xattr
having the specific value of "suspect" to avoid redoing validation (which is
expensive) as we find the same file in multiple bricks' indices. When we decide
to take action, we update this xattr to "clean" for copies that were in the
majority and "repaired" for the odd one out that gets clobbered. We also copy
the about-to-be-clobbered copy into an "orphans" directory to facilitate
analysis of corruption patterns. The data goes into ${GFID}.data there, while
${GFID}.link is a symlink to the file's old location.

Porting note: this is several internal squashed together ("See Also")

Differential Revision: https://phabricator.intern.facebook.com/D5092983
See Also: https://phabricator.intern.facebook.com/D5126974
See Also: https://phabricator.intern.facebook.com/D5127427
See Also: https://phabricator.intern.facebook.com/D5132804
See Also: https://phabricator.intern.facebook.com/D5209185
See Also: https://phabricator.intern.facebook.com/D5370353

Change-Id: Ie0ae18b368c408a5e47d0bf03ebac80b87b70aa9
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18269
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
nfs: Enable multi-core epoll support in gNFSd 2017-09-12T15:28:24+00:00 Shreyas Siravara sshreyas@fb.com 2016-03-30T22:35:15+00:00 4e880ed12b989b36387ea79179b5f14d8cee9491 Summary: - Enables multi-core epoll support in the nfs daemon. - Option can be turned on using: gluster volume set <volname> nfs.event-threads <numthreads> Test Plan: Prove test! Reviewers: kvigor, rwareing Reviewed By: rwareing Subscribers: dld, moox, dph Differential Revision: https://phabricator.fb.com/D3117966 Change-Id: Ie8a7b1ba04b0e83f5ec7a09f9d181fe59be479ca Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18266 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
Summary:
- Enables multi-core epoll support in the nfs daemon.
- Option can be turned on using:
  gluster volume set <volname> nfs.event-threads <numthreads>

Test Plan: Prove test!

Reviewers: kvigor, rwareing

Reviewed By: rwareing

Subscribers: dld, moox, dph

Differential Revision: https://phabricator.fb.com/D3117966

Change-Id: Ie8a7b1ba04b0e83f5ec7a09f9d181fe59be479ca
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18266
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
event: Idle connection management 2017-09-12T15:12:21+00:00 Shreyas Siravara sshreyas@fb.com 2016-03-28T21:17:53+00:00 14e24da1eb59a85fe99c22bafd8641ca2b75a923 Summary: - This diff adds support for detecting and tracking idle client connections. - It allows *service translators* (server, nfs) to opt-in to detect and close idle client connections. - Right now it explicitly restricts the service to NFS as a safety. Here are the debug logs when a client connection gets closed: [2016-03-29 17:27:06.154232] W [socket.c:2426:socket_timeout_handler] 0-socket: Shutting down idle client connection (idle=20s,fd=20,conn=[2401:db00:11:d0af:face:0:3:0:957]->[2401:db00:11:d0af:face:0:3:0:2049])! [2016-03-29 17:27:06.154292] D [event-epoll.c:655:__event_epoll_timeout_slot] 0-epoll: Connection on slot->fd=9 was idle for 20 seconds! [2016-03-29 17:27:06.163282] D [socket.c:629:__socket_rwv] 0-socket.nfs-server: EOF on socket [2016-03-29 17:27:06.163298] D [socket.c:2474:socket_event_handler] 0-transport: disconnecting now [2016-03-29 17:27:06.163316] D [event-epoll.c:614:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=9 from gen=4 to slot->gen=5, fd=20, slot->fd=20 Test Plan: - Used stuck NFS mounts to create idle clients and unstuck them. Reviewers: kvigor, rwareing Reviewed By: rwareing Subscribers: dld, moox, dph Differential Revision: https://phabricator.fb.com/D3112099 Change-Id: Ic06c89e03f87daabab7f07f892390edd1a1fcc20 Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18265 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
Summary:
- This diff adds support for detecting and tracking idle client connections.
- It allows *service translators* (server, nfs) to opt-in to detect and close idle client connections.
- Right now it explicitly restricts the service to NFS as a safety.

Here are the debug logs when a client connection gets closed:

  [2016-03-29 17:27:06.154232] W [socket.c:2426:socket_timeout_handler] 0-socket: Shutting down idle client connection (idle=20s,fd=20,conn=[2401:db00:11:d0af:face:0:3:0:957]->[2401:db00:11:d0af:face:0:3:0:2049])!
  [2016-03-29 17:27:06.154292] D [event-epoll.c:655:__event_epoll_timeout_slot] 0-epoll: Connection on slot->fd=9 was idle for 20 seconds!
  [2016-03-29 17:27:06.163282] D [socket.c:629:__socket_rwv] 0-socket.nfs-server: EOF on socket
  [2016-03-29 17:27:06.163298] D [socket.c:2474:socket_event_handler] 0-transport: disconnecting now
  [2016-03-29 17:27:06.163316] D [event-epoll.c:614:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=9 from gen=4 to slot->gen=5, fd=20, slot->fd=20

Test Plan: - Used stuck NFS mounts to create idle clients and unstuck them.

Reviewers: kvigor, rwareing

Reviewed By: rwareing

Subscribers: dld, moox, dph

Differential Revision: https://phabricator.fb.com/D3112099

Change-Id: Ic06c89e03f87daabab7f07f892390edd1a1fcc20
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18265
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
cluster/afr: Add additional test coverage for unsplit flows 2017-09-11T20:41:06+00:00 Richard Wareing rwareing@fb.com 2015-12-17T20:25:01+00:00 60b35dbfa42a65d81a18efda2776c0e733c4e769 Summary: - Adds test coverage for unsplitting via SHD Test Plan: - Run prove -v tests/bugs/fb2506544* (https://phabricator.fb.com/P56056659) Reviewers: moox, dld, dph, sshreyas Reviewed By: sshreyas Differential Revision: https://phabricator.fb.com/D2770524 Porting note: also added fb*.t tests to test_env. Change-Id: Iac28b595194925a45e62b6438611c9bade58b30b Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18261 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
Summary: - Adds test coverage for unsplitting via SHD

Test Plan: - Run prove -v tests/bugs/fb2506544* (https://phabricator.fb.com/P56056659)

Reviewers: moox, dld, dph, sshreyas

Reviewed By: sshreyas

Differential Revision: https://phabricator.fb.com/D2770524

Porting note: also added fb*.t tests to test_env.

Change-Id: Iac28b595194925a45e62b6438611c9bade58b30b
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18261
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
clusters/afr: Move root entry heal flow to SHD 2017-09-11T15:50:22+00:00 Richard Wareing rwareing@fb.com 2015-12-03T20:36:20+00:00 20ca645344e4af89a53b8d304063a6dd6dc5100a Summary: - Improves upon D2387001 by moving the "forced" root gfid heal to the SHDs - Removed code which forced NFSd/FUSE clients through the entry heal for the root GFID, this will make them spin up just as fast as prior to D2387001 (i.e. instantly) Porting note: mostly inapplicable in 3.8, only one non-test change survived Test Plan: - Must pass tests/bugs/fb8149516.t Reviewers: dph, moox, sshreyas Reviewed By: sshreyas Differential Revision: https://phabricator.fb.com/D2722239 Change-Id: I35f5827df6ead1bb0ff886ca0adabb2add2e7163 Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18259 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
Summary:
- Improves upon D2387001 by moving the "forced" root gfid heal to the SHDs
- Removed code which forced NFSd/FUSE clients through the entry heal for
  the root GFID, this will make them spin up just as fast as prior to D2387001 (i.e. instantly)

Porting note: mostly inapplicable in 3.8, only one non-test change survived

Test Plan: - Must pass tests/bugs/fb8149516.t

Reviewers: dph, moox, sshreyas

Reviewed By: sshreyas

Differential Revision: https://phabricator.fb.com/D2722239

Change-Id: I35f5827df6ead1bb0ff886ca0adabb2add2e7163
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18259
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
cluster/afr: AFR2 Discovery entry heal flow should only happen on root gfid 2017-09-08T23:47:02+00:00 Richard Wareing rwareing@fb.com 2015-09-23T05:55:40+00:00 8250b0bf29d5423bf6014b45b43a22c6aace5576 Summary: - Prevents entry self-heal flow from happening on non-root GFIDs Test Plan: - Run prove -v tests/bugs/fb8149516.t Reviewers: dph, moox, sshreyas Reviewed By: sshreyas Differential Revision: https://phabricator.fb.com/D2470622 Change-Id: Id8559f2cfeb6e1e5c26dc1571854c0fbc0b59e08 Signed-off-by: Jeff Darcy <jdarcy@fb.com> Reviewed-on: https://review.gluster.org/18250 Reviewed-by: Jeff Darcy <jeff@pl.atyp.us> Tested-by: Jeff Darcy <jeff@pl.atyp.us> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> 
Summary: - Prevents entry self-heal flow from happening on non-root GFIDs

Test Plan: - Run prove -v tests/bugs/fb8149516.t

Reviewers: dph, moox, sshreyas

Reviewed By: sshreyas

Differential Revision: https://phabricator.fb.com/D2470622

Change-Id: Id8559f2cfeb6e1e5c26dc1571854c0fbc0b59e08
Signed-off-by: Jeff Darcy <jdarcy@fb.com>
Reviewed-on: https://review.gluster.org/18250
Reviewed-by: Jeff Darcy <jeff@pl.atyp.us>
Tested-by: Jeff Darcy <jeff@pl.atyp.us>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>