diff options
author | raghavendra talur <raghavendra.talur@gmail.com> | 2015-08-20 15:09:31 +0530 |
---|---|---|
committer | Humble Devassy Chirammal <humble.devassy@gmail.com> | 2015-08-31 02:27:22 -0700 |
commit | 9e9e3c5620882d2f769694996ff4d7e0cf36cc2b (patch) | |
tree | 3a00cbd0cc24eb7df3de9b2eeeb8d42ee9175f88 /done/GlusterFS 3.5/Brick Failure Detection.md | |
parent | f6055cdb4dedde576ed8ec55a13814a69dceefdc (diff) |
Create basic directory structure
All new features specs go into in_progress directory.
Once signed off, it should be moved to done directory.
For now,
This change moves all the Gluster 4.0 feature specs to
in_progress. All other specs are under done/release-version.
More cleanup required will be done incrementally.
Change-Id: Id272d301ba8c434cbf7a9a966ceba05fe63b230d
BUG: 1206539
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/11969
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Tested-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Diffstat (limited to 'done/GlusterFS 3.5/Brick Failure Detection.md')
-rw-r--r-- | done/GlusterFS 3.5/Brick Failure Detection.md | 151 |
1 files changed, 151 insertions, 0 deletions
diff --git a/done/GlusterFS 3.5/Brick Failure Detection.md b/done/GlusterFS 3.5/Brick Failure Detection.md new file mode 100644 index 0000000..9952698 --- /dev/null +++ b/done/GlusterFS 3.5/Brick Failure Detection.md @@ -0,0 +1,151 @@ +Feature +------- + +Brick Failure Detection + +Summary +------- + +This feature attempts to identify storage/file system failures and +disable the failed brick without disrupting the remainder of the node's +operation. + +Owners +------ + +Vijay Bellur with help from Niels de Vos (or the other way around) + +Current status +-------------- + +Currently, if the underlying storage or file system failure happens, a +brick process will continue to function. In some cases, a brick can hang +due to failures in the underlying system. Due to such hangs in brick +processes, applications running on glusterfs clients can hang. + +Detailed Description +-------------------- + +Detecting failures on the filesystem that a brick uses makes it possible +to handle errors that are caused from outside of the Gluster +environment. + +There have been hanging brick processes when the underlying storage of a +brick went unavailable. A hanging brick process can still use the +network and repond to clients, but actual I/O to the storage is +impossible and can cause noticible delays on the client side. + +Benefit to GlusterFS +-------------------- + +Provide better detection of storage subsytem failures and prevent bricks +from hanging. + +Scope +----- + +### Nature of proposed change + +Add a health-checker to the posix xlator that periodically checks the +status of the filesystem (implies checking of functional +storage-hardware). + +### Implications on manageability + +When a brick process detects that the underlaying storage is not +responding anymore, the process will exit. There is no automated way +that the brick process gets restarted, the sysadmin will need to fix the +problem with the storage first. + +After correcting the storage (hardware or filesystem) issue, the +following command will start the brick process again: + + # gluster volume start <VOLNAME> force + +### Implications on presentation layer + +None + +### Implications on persistence layer + +None + +### Implications on 'GlusterFS' backend + +None + +### Modification to GlusterFS metadata + +None + +### Implications on 'glusterd' + +'glusterd' can detect that the brick process has exited, +`gluster volume status` will show that the brick process is not running +anymore. System administrators checking the logs should be able to +triage the cause. + +How To Test +----------- + +The health-checker thread that is part of each brick process will get +started automatically when a volume has been started. Verifying its +functionality can be done in different ways. + +On virtual hardware: + +- disconnect the disk from the VM that holds the brick + +On real hardware: + +- simulate a RAID-card failure by unplugging the card or cables + +On a system that uses LVM for the bricks: + +- use device-mapper to load an error-table for the disk, see [this + description](http://review.gluster.org/5176). + +On any system (writing to random offsets of the block device, more +difficult to trigger): + +1. cause corruption on the filesystem that holds the brick +2. read contents from the brick, hoping to hit the corrupted area +3. the filsystem should abort after hitting a bad spot, the + health-checker should notice that shortly afterwards + +User Experience +--------------- + +No more hanging brick processes when storage-hardware or the filesystem +fails. + +Dependencies +------------ + +Posix translator, not available for the BD-xlator. + +Documentation +------------- + +The health-checker is enabled by default and runs a check every 30 +seconds. This interval can be changed per volume with: + + # gluster volume set <VOLNAME> storage.health-check-interval <SECONDS> + +If `SECONDS` is set to 0, the health-checker will be disabled. + +For further details refer: +<https://forge.gluster.org/glusterfs-core/glusterfs/blobs/release-3.5/doc/features/brick-failure-detection.md> + +Status +------ + +glusterfs-3.4 and newer include a health-checker for the posix xlator, +which was introduced with [bug +971774](https://bugzilla.redhat.com/971774): + +- [posix: add a simple + health-checker](http://review.gluster.org/5176)? + +Comments and Discussion +----------------------- |