summaryrefslogtreecommitdiffstats
path: root/doc
diff options
context:
space:
mode:
authorkshithijiyer <kshithij.ki@gmail.com>2019-06-05 19:40:29 +0530
committerAmar Tumballi <amarts@redhat.com>2019-06-08 05:45:35 +0000
commit2eaf8e846afd71c30f2a6ff6f863a39b1145b8b6 (patch)
tree333b3494bf767ab195717fe4e8cdf48398681a7f /doc
parent2ff76fa45c53b8e291cf70c98b28800f3ed5f6fc (diff)
Fixing formatting errors in markdown files
There are a lot of fromatting error is markdown files peresent under /doc directiory of the project. Fixing formatting errors and sending a patch. Fixes: bz#1718273 Change-Id: I08f938088bbaaafddf634f73616ea0dbfe7aedf3 Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
Diffstat (limited to 'doc')
-rw-r--r--doc/debugging/analyzing-regression-cores.md43
-rw-r--r--doc/debugging/gfid-to-path.md43
-rw-r--r--doc/debugging/split-brain.md75
-rw-r--r--doc/debugging/statedump.md77
-rw-r--r--doc/developer-guide/Language-Bindings.md1
-rw-r--r--doc/developer-guide/Using-Gluster-Test-Framework.md1
-rw-r--r--doc/developer-guide/afr-locks-evolution.md6
-rw-r--r--doc/developer-guide/afr-self-heal-daemon.md2
-rw-r--r--doc/developer-guide/bd-xlator.md494
-rw-r--r--doc/developer-guide/brickmux-thread-reduction.md8
-rw-r--r--doc/developer-guide/coding-standard.md28
-rw-r--r--doc/developer-guide/datastructure-inode.md61
-rw-r--r--doc/developer-guide/datastructure-iobuf.md36
-rw-r--r--doc/developer-guide/datastructure-mem-pool.md8
-rw-r--r--doc/developer-guide/dirops-transactions-in-dht.md3
-rw-r--r--doc/developer-guide/network_compression.md20
-rw-r--r--doc/developer-guide/syncop.md2
-rw-r--r--doc/developer-guide/translator-development.md2
-rw-r--r--doc/features/ctime.md32
19 files changed, 474 insertions, 468 deletions
diff --git a/doc/debugging/analyzing-regression-cores.md b/doc/debugging/analyzing-regression-cores.md
index cbbb387794d..5e10f41c6eb 100644
--- a/doc/debugging/analyzing-regression-cores.md
+++ b/doc/debugging/analyzing-regression-cores.md
@@ -1,36 +1,35 @@
-This document explains how to analyze core-dumps obtained from regression
-machines, with examples.
-1) Download the core-tarball and extract it.
-2) 'cd' into directory where the tarball is extracted.
-~~~
+# Analyzing Regression Cores
+This document explains how to analyze core-dumps obtained from regression machines, with examples.
+1. Download the core-tarball and extract it.
+2. `cd` into directory where the tarball is extracted.
+```
[sh]# pwd
/home/user/Downloads
[sh]# ls
build build-install-20150625_05_42_39.tar.bz2 lib64 usr
-~~~
-3) Determine the core file you need to examine. There can be more than one core file.
-You can list them from './build/install/cores' directory.
-~~~
+```
+3. Determine the core file you need to examine. There can be more than one core file. You can list them from './build/install/cores' directory.
+```
[sh]# ls build/install/cores/
core.9341 liblist.txt liblist.txt.tmp
-~~~
+```
In case you are unsure which binary generated the core-file, executing 'file' command on it will help.
-~~~
+```
[sh]# file ./build/install/cores/core.9341
./build/install/cores/core.9341: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'
-~~~
-As seen, the core file was generated by glusterfsd binary, and path to it is provided (/build/install/sbin/glusterfsd).
-4) Now, run the following command on the core:
-~~~
+```
+As seen, the core file was generated by glusterfsd binary, and path to it is provided (/build/install/sbin/glusterfsd).
+
+4. Now, run the following command on the core:
+```
gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.xxx' <target, say ./build/install/sbin/glusterd>
In this case,
gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.9341' ./build/install/sbin/glusterfsd
-~~~
-5) You can cross check if all shared libraries are available and loaded by using 'info sharedlibrary' command from
-inside gdb.
-6) Once verified, usual gdb commands based on requirement can be used to debug the core.
-'bt' or 'backtrace' from gdb of core used in examples:
-~~~
+```
+5. You can cross check if all shared libraries are available and loaded by using 'info sharedlibrary' command from inside gdb.
+6. Once verified, usual gdb commands based on requirement can be used to debug the core.
+ `bt` or `backtrace` from gdb of core used in examples:
+```
Core was generated by `/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f512a54e625 in raise () from ./lib64/libc.so.6
@@ -52,4 +51,4 @@ Program terminated with signal SIGABRT, Aborted.
#12 0x00007f512a55f8f0 in ?? () from ./lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
(gdb)
-~~~
+```
diff --git a/doc/debugging/gfid-to-path.md b/doc/debugging/gfid-to-path.md
index 09c459e52c8..49e9aa09a3f 100644
--- a/doc/debugging/gfid-to-path.md
+++ b/doc/debugging/gfid-to-path.md
@@ -1,37 +1,37 @@
-#Convert GFID to Path
+# Convert GFID to Path
GlusterFS internal file identifier (GFID) is a uuid that is unique to each
file across the entire cluster. This is analogous to inode number in a
normal filesystem. The GFID of a file is stored in its xattr named
`trusted.gfid`.
-####Special mount using [gfid-access translator][1]:
-~~~
+#### Special mount using [gfid-access translator][1]:
+```
mount -t glusterfs -o aux-gfid-mount vm1:test /mnt/testvol
-~~~
+```
Assuming, you have `GFID` of a file from changelog (or somewhere else).
For trying this out, you can get `GFID` of a file from mountpoint:
-~~~
+```
getfattr -n glusterfs.gfid.string /mnt/testvol/dir/file
-~~~
+```
---
-###Get file path from GFID (Method 1):
+### Get file path from GFID (Method 1):
**(Lists hardlinks delimited by `:`, returns path as seen from mountpoint)**
-####Turn on build-pgfid option
-~~~
+#### Turn on build-pgfid option
+```
gluster volume set test build-pgfid on
-~~~
+```
Read virtual xattr `glusterfs.ancestry.path` which contains the file path
-~~~
+```
getfattr -n glusterfs.ancestry.path -e text /mnt/testvol/.gfid/<GFID>
-~~~
+```
**Example:**
-~~~
+```
[root@vm1 glusterfs]# ls -il /mnt/testvol/dir/
total 1
10610563327990022372 -rw-r--r--. 2 root root 3 Jul 17 18:05 file
@@ -46,28 +46,27 @@ glusterfs.gfid.string="11118443-1894-4273-9340-4b212fa1c0e4"
getfattr: Removing leading '/' from absolute path names
# file: mnt/testvol/.gfid/11118443-1894-4273-9340-4b212fa1c0e4
glusterfs.ancestry.path="/dir/file:/dir/file3"
-~~~
+```
---
-###Get file path from GFID (Method 2):
+### Get file path from GFID (Method 2):
**(Does not list all hardlinks, returns backend brick path)**
-~~~
+```
getfattr -n trusted.glusterfs.pathinfo -e text /mnt/testvol/.gfid/<GFID>
-~~~
+```
**Example:**
-~~~
+```
[root@vm1 glusterfs]# getfattr -n trusted.glusterfs.pathinfo -e text /mnt/testvol/.gfid/11118443-1894-4273-9340-4b212fa1c0e4
getfattr: Removing leading '/' from absolute path names
# file: mnt/testvol/.gfid/11118443-1894-4273-9340-4b212fa1c0e4
trusted.glusterfs.pathinfo="(<DISTRIBUTE:test-dht> <POSIX(/mnt/brick-test/b):vm1:/mnt/brick-test/b/dir//file3>)"
-~~~
+```
---
-###Get file path from GFID (Method 3):
+### Get file path from GFID (Method 3):
https://gist.github.com/semiosis/4392640
---
-####References and links:
+#### References and links:
[posix: placeholders for GFID to path conversion](http://review.gluster.org/5951)
-[1]: https://github.com/gluster/glusterfs/blob/master/doc/features/gfid-access.md
diff --git a/doc/debugging/split-brain.md b/doc/debugging/split-brain.md
index b0d938e26bc..6b122c40551 100644
--- a/doc/debugging/split-brain.md
+++ b/doc/debugging/split-brain.md
@@ -1,33 +1,36 @@
-Steps to recover from File split-brain.
-======================================
-
-Quick Start:
-============
-1. Get the path of the file that is in split-brain:
-> It can be obtained either by
-> a) The command `gluster volume heal info split-brain`.
-> b) Identify the files for which file operations performed
- from the client keep failing with Input/Output error.
-
-2. Close the applications that opened this file from the mount point.
+# Steps to recover from File split-brain
+This document contains steps to recover from a file split-brain.
+## Quick Start:
+### Step 1. Get the path of the file that is in split-brain:
+It can be obtained either by
+1. The command `gluster volume heal info split-brain`.
+2. Identify the files for which file operations performed from the client keep failing with Input/Output error.
+
+### Step 2. Close the applications that opened this file from the mount point.
In case of VMs, they need to be powered-off.
-3. Decide on the correct copy:
-> This is done by observing the afr changelog extended attributes of the file on
+### Step 3. Decide on the correct copy:
+This is done by observing the afr changelog extended attributes of the file on
the bricks using the getfattr command; then identifying the type of split-brain
(data split-brain, metadata split-brain, entry split-brain or split-brain due to
gfid-mismatch); and finally determining which of the bricks contains the 'good copy'
of the file.
-> `getfattr -d -m . -e hex <file-path-on-brick>`.
+```
+getfattr -d -m . -e hex <file-path-on-brick>
+```
+
It is also possible that one brick might contain the correct data while the
other might contain the correct metadata.
-4. Reset the relevant extended attribute on the brick(s) that contains the
-'bad copy' of the file data/metadata using the setfattr command.
-> `setfattr -n <attribute-name> -v <attribute-value> <file-path-on-brick>`
+### Step 4. Reset the relevant extended attribute on the brick(s) that contains the 'bad copy' of the file data/metadata using the setfattr command.
+```
+setfattr -n <attribute-name> -v <attribute-value> <file-path-on-brick>
+```
-5. Trigger self-heal on the file by performing lookup from the client:
-> `ls -l <file-path-on-gluster-mount>`
+### Step 5. Trigger self-heal on the file by performing lookup from the client:
+```
+ls -l <file-path-on-gluster-mount>
+```
Detailed Instructions for steps 3 through 5:
===========================================
@@ -36,13 +39,15 @@ afr changelog extended attributes.
Execute `getfattr -d -m . -e hex <file-path-on-brick>`
-* Example:
+Example:
+```
[root@store3 ~]# getfattr -d -e hex -m. brick-a/file.txt
\#file: brick-a/file.txt
security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000
trusted.afr.vol-client-2=0x000000000000000000000000
trusted.afr.vol-client-3=0x000000000200000000000000
trusted.gfid=0x307a5c9efddd4e7c96e94fd4bcdcbd1b
+```
The extended attributes with `trusted.afr.<volname>-client-<subvolume-index>`
are used by afr to maintain changelog of the file.The values of the
@@ -51,10 +56,11 @@ client (fuse or nfs-server) processes. When the glusterfs client modifies a file
or directory, the client contacts each brick and updates the changelog extended
attribute according to the response of the brick.
-'subvolume-index' is nothing but (brick number - 1) in
+`subvolume-index` is nothing but (brick number - 1) in
`gluster volume info <volname>` output.
-* Example:
+Example:
+```
[root@pranithk-laptop ~]# gluster volume info vol
Volume Name: vol
Type: Distributed-Replicate
@@ -71,6 +77,7 @@ attribute according to the response of the brick.
brick-f: pranithk-laptop:/gfs/brick-f
brick-g: pranithk-laptop:/gfs/brick-g
brick-h: pranithk-laptop:/gfs/brick-h
+```
In the example above:
```
@@ -91,12 +98,15 @@ present in all the other bricks in it's replica set as seen by that brick.
In the example volume given above, all files in brick-a will have 2 entries,
one for itself and the other for the file present in it's replica pair, i.e.brick-b:
+```
trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for itself (brick-a)
trusted.afr.vol-client-1=0x000000000000000000000000 -->changelog for brick-b as seen by brick-a
-
+```
Likewise, all files in brick-b will have:
+```
trusted.afr.vol-client-0=0x000000000000000000000000 -->changelog for brick-a as seen by brick-b
trusted.afr.vol-client-1=0x000000000000000000000000 -->changelog for itself (brick-b)
+```
The same can be extended for other replica pairs.
@@ -122,7 +132,8 @@ When a file split-brain happens it could be either data split-brain or
meta-data split-brain or both. When a split-brain happens the changelog of the
file would be something like this:
-* Example:(Lets consider both data, metadata split-brain on same file).
+Example:(Lets consider both data, metadata split-brain on same file).
+```
[root@pranithk-laptop vol]# getfattr -d -m . -e hex /gfs/brick-?/a
getfattr: Removing leading '/' from absolute path names
\#file: gfs/brick-a/a
@@ -133,10 +144,11 @@ trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
trusted.afr.vol-client-0=0x000003b00000000100000000
trusted.afr.vol-client-1=0x000000000000000000000000
trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
+```
-###Observations:
+### Observations:
-####According to changelog extended attributes on file /gfs/brick-a/a:
+#### According to changelog extended attributes on file /gfs/brick-a/a:
The first 8 digits of trusted.afr.vol-client-0 are all
zeros (0x00000000................), and the first 8 digits of
trusted.afr.vol-client-1 are not all zeros (0x000003d7................).
@@ -149,7 +161,7 @@ trusted.afr.vol-client-1 are not all zeros (0x........00000001........).
So the changelog on /gfs/brick-a/a implies that some metadata operations succeeded
on itself but failed on /gfs/brick-b/a.
-####According to Changelog extended attributes on file /gfs/brick-b/a:
+#### According to Changelog extended attributes on file /gfs/brick-b/a:
The first 8 digits of trusted.afr.vol-client-0 are not all
zeros (0x000003b0................), and the first 8 digits of
trusted.afr.vol-client-1 are all zeros (0x00000000................).
@@ -205,6 +217,7 @@ Hence execute
`setfattr -n trusted.afr.vol-client-1 -v 0x000003d70000000000000000 /gfs/brick-a/a`
Thus after the above operations are done, the changelogs look like this:
+```
[root@pranithk-laptop vol]# getfattr -d -m . -e hex /gfs/brick-?/a
getfattr: Removing leading '/' from absolute path names
\#file: gfs/brick-a/a
@@ -216,7 +229,7 @@ trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
trusted.afr.vol-client-0=0x000000000000000100000000
trusted.afr.vol-client-1=0x000000000000000000000000
trusted.gfid=0x80acdbd886524f6fbefa21fc356fed57
-
+```
Triggering Self-heal:
---------------------
@@ -243,9 +256,9 @@ needs to be removed.The gfid-link files are present in the .glusterfs folder
in the top-level directory of the brick. If the gfid of the file is
0x307a5c9efddd4e7c96e94fd4bcdcbd1b (the trusted.gfid extended attribute got
from the getfattr command earlier),the gfid-link file can be found at
-> /gfs/brick-a/.glusterfs/30/7a/307a5c9efddd4e7c96e94fd4bcdcbd1b
+`/gfs/brick-a/.glusterfs/30/7a/307a5c9efddd4e7c96e94fd4bcdcbd1b`
-####Word of caution:
+#### Word of caution:
Before deleting the gfid-link, we have to ensure that there are no hard links
to the file present on that brick. If hard-links exist,they must be deleted as
well.
diff --git a/doc/debugging/statedump.md b/doc/debugging/statedump.md
index 9939576e270..9d594320ddc 100644
--- a/doc/debugging/statedump.md
+++ b/doc/debugging/statedump.md
@@ -1,21 +1,30 @@
-#Statedump
+# Statedump
Statedump is a file generated by glusterfs process with different data structure state which may contain the active inodes, fds, mempools, iobufs, memory allocation stats of different types of datastructures per xlator etc.
-##How to generate statedump
-We can find the directory where statedump files are created using 'gluster --print-statedumpdir' command.
+## How to generate statedump
+We can find the directory where statedump files are created using `gluster --print-statedumpdir` command.
Create that directory if not already present based on the type of installation.
Lets call this directory `statedump-directory`.
-We can generate statedump using 'kill -USR1 <pid-of-gluster-process>'.
+We can generate statedump using `kill -USR1 <pid-of-gluster-process>`.
gluster-process is nothing but glusterd/glusterfs/glusterfsd process.
There are also commands to generate statedumps for brick processes/nfs server/quotad
-For bricks: `gluster volume statedump <volname>`
+For bricks:
+```
+gluster volume statedump <volname>
+```
-For nfs server: `gluster volume statedump <volname> nfs`
+For nfs server:
+```
+gluster volume statedump <volname> nfs
+```
-For quotad: `gluster volume statedump <volname> quotad`
+For quotad:
+```
+gluster volume statedump <volname> quotad
+```
For brick-processes files will be created in `statedump-directory` with name of the file as `hyphenated-brick-path.<pid>.dump.timestamp`. For all other processes it will be `glusterdump.<pid>.dump.timestamp`.
@@ -24,21 +33,21 @@ processes could have used the `SIGUSR1` signal already for other purposes.
To generate statedump for the processes, using libgfapi, below command can be
executed from one of the nodes in the gluster cluster to which the libgfapi
application is connected to.
-
- gluster volume statedump <volname> client <hostname>:<process id>
-
+```
+gluster volume statedump <volname> client <hostname>:<process id>
+```
The statedumps can be found in the `statedump-directory`, the name of the
statedumps being `glusterdump.<pid>.dump.timestamp`. For a process there can be
multiple such files created depending on the number of times the volume is
accessed by the process (related to the number of `glfs_init()` calls).
-##How to read statedump
+## How to read statedump
We shall see snippets of each type of statedump.
First and last lines of the file have starting and ending time of writing the statedump file. Times will be in UTC timezone.
mallinfo return status is printed in the following format. Please read man mallinfo for more information about what each field means.
-###Mallinfo
+### Mallinfo
```
[mallinfo]
mallinfo_arena=100020224 /* Non-mmapped space allocated (bytes) */
@@ -53,7 +62,7 @@ mallinfo_fordblks=3310112 /* Total free space (bytes) */
mallinfo_keepcost=133712 /* Top-most, releasable space (bytes) */
```
-###Data structure allocation stats
+### Data structure allocation stats
For every xlator data structure memory per translator loaded in the call-graph is displayed in the following format:
For xlator with name: glusterfs
@@ -74,7 +83,7 @@ max_num_allocs=3 #Maximum number of active allocations at any point in the life
total_allocs=7 #Number of times this data is allocated in the life of the process.
```
-###Mempools
+### Mempools
Mempools are optimization to reduce the number of allocations of a data type. If we create a mem-pool of lets say 1024 elements for a data-type, new elements will be allocated from heap using syscalls like calloc, only if all the 1024 elements in the pool are in active use.
@@ -94,7 +103,7 @@ cur-stdalloc=0 #Denotes the number of allocations made from heap once cold-count
max-stdalloc=0 #Maximum number of allocations from heap that are in active use at any point in the life of the process.
```
-###Iobufs
+### Iobufs
```
[iobuf.global]
iobuf_pool=0x1f0d970 #The memory pool for iobufs
@@ -105,7 +114,7 @@ iobuf_pool.arena_cnt=8 #Total number of arenas in the pool
iobuf_pool.request_misses=0 #The number of iobufs that were stdalloc'd (as they exceeded the default max page size provided by iobuf_pool).
```
-There are 3 lists of arenas
+There are 3 lists of arenas:
1. Arena list: arenas allocated during iobuf pool creation and the arenas that are in use(active_cnt != 0) will be part of this list.
2. Purge list: arenas that can be purged(no active iobufs, active_cnt == 0).
@@ -142,7 +151,7 @@ arena.6.active_iobuf.2.ptr=0x7fdb92189000
At any given point in time if there are lots of filled arenas then that could be a sign of iobuf leaks.
-###Call stack
+### Call stack
All the fops received by gluster are handled using call-stacks. Call stack contains the information about uid/gid/pid etc of the process that is executing the fop. Each call-stack contains different call-frames per xlator which handles that fop.
```
@@ -157,7 +166,7 @@ op=LOOKUP #Fop
type=1 #Type of the op i.e. FOP/MGMT-OP
cnt=9 #Number of frames in this stack.
```
-###Call-frame
+### Call-frame
Each frame will have information about which xlator the frame belongs to, what is the function it wound to/from and will be unwind to. It also mentions if the unwind happened or not. If we observe hangs in the system and want to find out which xlator is causing it. Take a statedump and see what is the final xlator which is yet to be unwound.
```
@@ -172,7 +181,7 @@ wind_to=priv->children[i]->fops->lookup
unwind_to=afr_lookup_cbk #Parent xlator function to which unwind happened
```
-###History of operations in Fuse
+### History of operations in Fuse
Fuse maintains history of operations that happened in fuse.
@@ -188,7 +197,7 @@ TIME=2014-07-09 16:44:57.523394
message=[0] fuse_getattr_resume: 4591, STAT, path: (/iozone.tmp), gfid: (3afb4968-5100-478d-91e9-76264e634c9f)
```
-###Xlator configuration
+### Xlator configuration
```
[cluster/replicate.r2-replicate-0] #Xlator type, name information
child_count=2 #Number of children to the xlator
@@ -208,7 +217,7 @@ favorite_child=-1
wait_count=1
```
-###Graph/inode table
+### Graph/inode table
```
[active graph - 1]
@@ -220,7 +229,7 @@ conn.1.bound_xl./data/brick01a/homegfs.lru_size=183 #Number of inodes present in
conn.1.bound_xl./data/brick01a/homegfs.purge_size=0 #Number of inodes present in purge list
```
-###Inode
+### Inode
```
[conn.1.bound_xl./data/brick01a/homegfs.active.324] #324th inode in active inode list
gfid=e6d337cf-97eb-44b3-9492-379ba3f6ad42 #Gfid of the inode
@@ -239,7 +248,7 @@ ia_type=2
Ref by xl:.fuse=1
Ref by xl:.patchy-client-0=-1
```
-###Inode context
+### Inode context
For each inode per xlator some context could be stored. This context can also be printed in the statedump. Here is the inode ctx of locks xlator
```
[xlator.features.locks.homegfs-locks.inode]
@@ -256,12 +265,12 @@ lock-dump.domain.domain=homegfs-replicate-0 #Domain name where entry/data operat
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=11141120, len=131072, pid = 18446744073709551615, owner=080b1ada117f0000, client=0xb7fc30, connection-id=compute-30-029.com-3505-2014/06/29-14:46:12:477358-homegfs-client-0-0-1, granted at Sun Jun 29 11:10:36 2014 #Active lock information
```
-##FAQ
-###How to debug Memory leaks using statedump?
+## FAQ
+### How to debug Memory leaks using statedump?
-####Using memory accounting feature:
+#### Using memory accounting feature:
-`https://bugzilla.redhat.com/show_bug.cgi?id=1120151` is one of the bugs which was debugged using statedump to see which data-structure is leaking. Here is the process used to find what the leak is using statedump. According to the bug the observation is that the process memory usage is increasing whenever one of the bricks is wiped in a replicate volume and a `full` self-heal is invoked to heal the contents. Statedump of the process is taken using kill -USR1 `<pid-of-gluster-self-heal-daemon>`.
+[Bug 1120151](https://bugzilla.redhat.com/show_bug.cgi?id=1120151) is one of the bugs which was debugged using statedump to see which data-structure is leaking. Here is the process used to find what the leak is using statedump. According to the bug the observation is that the process memory usage is increasing whenever one of the bricks is wiped in a replicate volume and a `full` self-heal is invoked to heal the contents. Statedump of the process is taken using `kill -USR1 <pid-of-gluster-self-heal-daemon>`.
```
grep -w num_allocs glusterdump.5225.dump.1405493251
num_allocs=77078
@@ -284,10 +293,10 @@ grep of the statedump revealed too many allocations for the following data-types
3. gf_common_mt_mem_pool.
After checking afr-code for allocations with tag `gf_common_mt_char` found `data-self-heal` code path does not free one such allocated memory. `gf_common_mt_mem_pool` suggests that there is a leak in pool memory. `replicate-0:dict_t`, `glusterfs:data_t` and `glusterfs:data_pair_t` pools are using lot of memory, i.e. cold_count is `0` and too many allocations. Checking source code of dict.c revealed that `key` in `dict` is allocated with `gf_common_mt_char` i.e. `2.` tag and value is created using gf_asprintf which in-turn uses `gf_common_mt_asprintf` i.e. `1.`. Browsing the code for leak in self-heal code paths lead to a line which over-writes a variable with new dictionary even when it was already holding a reference to another dictionary. After fixing these leaks, ran the same test to verify that none of the `num_allocs` are increasing even after healing 10,000 files directory hierarchy in statedump of self-heal daemon.
-Please check http://review.gluster.org/8316 for more info about patch/code.
+Please check this [patch](http://review.gluster.org/8316) for more info about the fix.
-####Debugging leaks in memory pools:
-Statedump output of memory pools was used to test and verify the fixes to https://bugzilla.redhat.com/show_bug.cgi?id=1134221. On code analysis, dict_t objects were found to be leaking (in terms of not being unref'd enough number of times, during name self-heal. The test involved creating 100 files on plain replicate volume, removing them from one of the bricks's backend, and then triggering lookup on them from the mount point. Statedump of the mount process was taken before executing the test case and after it, after compiling glusterfs with -DDEBUG flags (to have cold count set to 0 by default).
+#### Debugging leaks in memory pools:
+Statedump output of memory pools was used to test and verify the fixes to [Bug 1134221](https://bugzilla.redhat.com/show_bug.cgi?id=1134221). On code analysis, dict_t objects were found to be leaking (in terms of not being unref'd enough number of times, during name self-heal. The test involved creating 100 files on plain replicate volume, removing them from one of the brick's backend, and then triggering lookup on them from the mount point. Statedump of the mount process was taken before executing the test case and after it, after compiling glusterfs with -DDEBUG flags (to have cold count set to 0 by default).
Statedump output of the fuse mount process before the test case was executed:
@@ -319,7 +328,7 @@ cur-stdalloc=214
max-stdalloc=220
```
-Here, with cold count being 0 by default, cur-stdalloc indicated the number of dict_t objects that were allocated in heap using mem_get(), and yet to be freed using mem_put() (refer to https://github.com/gluster/glusterfs/blob/master/doc/data-structures/mem-pool.md for more details on how mempool works). After the test case (name selfheal of 100 files), there was a rise in the cur-stdalloc value (from 14 to 214) for dict_t.
+Here, with cold count being 0 by default, `cur-stdalloc` indicated the number of `dict_t` objects that were allocated in heap using `mem_get()`, and yet to be freed using `mem_put()` (refer to this [page](https://github.com/gluster/glusterfs/blob/master/doc/data-structures/mem-pool.md) for more details on how mempool works). After the test case (name selfheal of 100 files), there was a rise in the cur-stdalloc value (from 14 to 214) for `dict_t`.
After these leaks were fixed, glusterfs was again compiled with -DDEBUG flags, and the same steps were performed again and statedump was taken before and after executing the test case, of the mount. This was done to ascertain the validity of the fix. And the following are the results:
@@ -353,8 +362,8 @@ max-stdalloc=119
```
The value of cur-stdalloc remained 14 before and after the test, indicating that the fix indeed does what it's supposed to do.
-###How to debug hangs because of frame-loss?
-`https://bugzilla.redhat.com/show_bug.cgi?id=994959` is one of the bugs where statedump was helpful in finding where the frame was lost. Here is the process used to find where the hang is using statedump.
+### How to debug hangs because of frame-loss?
+[Bug 994959](https://bugzilla.redhat.com/show_bug.cgi?id=994959) is one of the bugs where statedump was helpful in finding where the frame was lost. Here is the process used to find where the hang is using statedump.
When the hang was observed, statedumps are taken for all the processes. On mount's statedump the following stack is shown:
```
[global.callpool.stack.1.frame.1]
@@ -402,4 +411,4 @@ unwind_to=qr_readdirp_cbk
```
`unwind_to` shows that call was unwound to `afr_readdirp_cbk` from client xlator.
Inspecting that function revealed that afr is not unwinding the stack when fop failed.
-Check http://review.gluster.org/5531 for more info about patch/code changes.
+Check this [patch](http://review.gluster.org/5531) for more info about the fix.
diff --git a/doc/developer-guide/Language-Bindings.md b/doc/developer-guide/Language-Bindings.md
index 33f2e58504c..951f5fae2f6 100644
--- a/doc/developer-guide/Language-Bindings.md
+++ b/doc/developer-guide/Language-Bindings.md
@@ -1,3 +1,4 @@
+# Language Bindings
GlusterFS 3.4 introduced the libgfapi client API for C programs. This
page lists bindings to the libgfapi C library from other languages.
diff --git a/doc/developer-guide/Using-Gluster-Test-Framework.md b/doc/developer-guide/Using-Gluster-Test-Framework.md
index 96fa9247e84..d2bb1c391da 100644
--- a/doc/developer-guide/Using-Gluster-Test-Framework.md
+++ b/doc/developer-guide/Using-Gluster-Test-Framework.md
@@ -1,3 +1,4 @@
+# USing Gluster Test Framwork
Description
-----------
diff --git a/doc/developer-guide/afr-locks-evolution.md b/doc/developer-guide/afr-locks-evolution.md
index 7d2a136d871..2dabbcfeb13 100644
--- a/doc/developer-guide/afr-locks-evolution.md
+++ b/doc/developer-guide/afr-locks-evolution.md
@@ -32,10 +32,10 @@ AFR makes use of locks xlator extensively:
* For Entry self-heal, it is `entrylk(NULL name, parent inode)`. Specifying NULL for the name takes full lock on the directory referred to by the inode.
* For data self-heal, there is a bit of history as to how locks evolved:
-###Initial version (say version 1) :
+### Initial version (say version 1) :
There was no concept of selfheal daemon (shd). Only client lookups triggered heals. so AFR always took `inodelk(0,0,DATA_DOMAIN)` for healing. The issue with this approach was that when heal was in progress, I/O from clients was blocked .
-###version 2:
+### version 2:
shd was introduced. We needed to allow I/O to go through when heal was going,provided the ranges did not overlap. To that extent, the following approach was adopted:
+ 1.shd takes (full inodelk in DATA_DOMAIN). Thus client FOPS are blocked and cannot modify changelog-xattrs
@@ -79,7 +79,7 @@ It modifies data but the FOP succeeds only on brick 2. writev returns success, a
and thus goes ahead and copies stale 128Kb from brick 1 to brick2. Thus as far as application is concerned, `writev` returned success but bricks have stale data.
What needs to be done is `writev` must return success only if it succeeded on atleast one source brick (brick b1 in this case). Otherwise The heal still happens in reverse direction but as far as the application is concerned, it received an error.
-###Note on lock **domains**
+### Note on lock **domains**
We have used conceptual names in this document like DATA_DOMAIN/ METADATA_DOMAIN/ SELF_HEAL_DOMAIN. In the code, these are mapped to strings that are based on the AFR xlator name like so:
DATA_DOMAIN --->"vol_name-replicate-n"
diff --git a/doc/developer-guide/afr-self-heal-daemon.md b/doc/developer-guide/afr-self-heal-daemon.md
index b85ddd1c856..65940d420b7 100644
--- a/doc/developer-guide/afr-self-heal-daemon.md
+++ b/doc/developer-guide/afr-self-heal-daemon.md
@@ -39,7 +39,7 @@ When a client (mount) performs an operation on the file, the index xlator presen
and removes it in post-op phase if the operation is successful. Thus if an entry is present inside the .glusterfs/indices/xattrop/ directory when there is no I/O
happening on the file, it means the file needs healing (or atleast an examination if the brick crashed after the post-op completed but just before the removal of the hardlink).
-####Index heal steps:
+#### Index heal steps:
<pre><code>
In shd process of *each node* {
opendir +readdir (.glusterfs/indices/xattrop/)
diff --git a/doc/developer-guide/bd-xlator.md b/doc/developer-guide/bd-xlator.md
index 1771fb6e24b..f087efbe361 100644
--- a/doc/developer-guide/bd-xlator.md
+++ b/doc/developer-guide/bd-xlator.md
@@ -1,8 +1,8 @@
-#Block device translator
+# Block device translator
Block device translator (BD xlator) is a translator added to GlusterFS which provides block backend for GlusterFS. This replaces the existing bd_map translator in GlusterFS that provided similar but very limited functionality. GlusterFS expects the underlying brick to be formatted with a POSIX compatible file system. BD xlator changes that and allows for having bricks that are raw block devices like LVM which needn’t have any file systems on them. Hence with BD xlator, it becomes possible to build a GlusterFS volume comprising of bricks that are logical volumes (LV).
-##bd
+## bd
BD xlator maps underlying LVs to files and hence the LVs appear as files to GlusterFS clients. Though BD volume externally appears very similar to the usual Posix volume, not all operations are supported or possible for the files on a BD volume. Only those operations that make sense for a block device are supported and the exact semantics are described in subsequent sections.
@@ -20,18 +20,18 @@ BD xlator was mainly developed to use block devices directly as VM images when G
Though BD xlator is primarily intended to be used with block devices, it does provide full Posix xlator compatibility for files that are created on BD volume but are not backed by or mapped to a block device. Such files which don’t have a block device mapping exist on the Posix directory that is specified during BD volume creation. BD xlator is available from GlusterFS-3.5 release.
-###Compiling BD translator
+### Compiling BD translator
BD xlator needs lvm2 development library. –enable-bd-xlator option can be used with `./configure` script to explicitly enable BD translator. The following snippet from the output of configure script shows that BD xlator is enabled for compilation.
-#####GlusterFS configure summary
+##### GlusterFS configure summary
Block Device xlator : yes
-###Creating a BD volume
+### Creating a BD volume
BD supports hosting of both linear LV and thin LV within the same volume. However seperate examples are provided below. As noted above, the prerequisite for a BD volume is VG which is created from a loop device here, but it can be any other device too.
@@ -39,270 +39,259 @@ BD supports hosting of both linear LV and thin LV within the same volume. Howeve
* Creating BD volume with linear LV backend
* Create a loop device
-
-
- [root@node ~]# dd if=/dev/zero of=bd-loop count=1024 bs=1M
-
- [root@node ~]# losetup /dev/loop0 bd-loop
-
-
+```
+[root@node ~]# dd if=/dev/zero of=bd-loop count=1024 bs=1M
+[root@node ~]# losetup /dev/loop0 bd-loop
+```
* Prepare a brick by creating a VG
-
- [root@node ~]# pvcreate /dev/loop0
-
- [root@node ~]# vgcreate bd-vg /dev/loop0
-
-
+```
+[root@node ~]# pvcreate /dev/loop0
+[root@node ~]# vgcreate bd-vg /dev/loop0
+```
* Create the BD volume
* Create a POSIX directory first
-
-
- [root@node ~]# mkdir /bd-meta
+```
+[root@node ~]# mkdir /bd-meta
+```
It is recommended that this directory is created on an LV in the brick VG itself so that both data and metadata live together on the same device.
* Create and mount the volume
-
- [root@node ~]# gluster volume create bd node:/bd-meta?bd-vg force
-
+```
+[root@node ~]# gluster volume create bd node:/bd-meta?bd-vg force
+```
The general syntax for specifying the brick is `host:/posix-dir?volume-group-name` where “?” is the separator.
-
-
-
- [root@node ~]# gluster volume start bd
- [root@node ~]# gluster volume info bd
- Volume Name: bd
- Type: Distribute
- Volume ID: cb042d2a-f435-4669-b886-55f5927a4d7f
- Status: Started
- Xlator 1: BD
- Capability 1: offload_copy
- Capability 2: offload_snapshot
- Number of Bricks: 1
- Transport-type: tcp
- Bricks:
- Brick1: node:/bd-meta
- Brick1 VG: bd-vg
-
-
-
- [root@node ~]# mount -t glusterfs node:/bd /mnt
+```
+[root@node ~]# gluster volume start bd
+[root@node ~]# gluster volume info bd
+Volume Name: bd
+Type: Distribute
+Volume ID: cb042d2a-f435-4669-b886-55f5927a4d7f
+Status: Started
+Xlator 1: BD
+Capability 1: offload_copy
+Capability 2: offload_snapshot
+Number of Bricks: 1
+Transport-type: tcp
+Bricks:
+Brick1: node:/bd-meta
+Brick1 VG: bd-vg
+
+[root@node ~]# mount -t glusterfs node:/bd /mnt
+```
* Create a file that is backed by an LV
-
- [root@node ~]# ls /mnt
-
- [root@node ~]#
+```
+[root@node ~]# ls /mnt
+```
Since the volume is empty now, so is the underlying VG.
-
- [root@node ~]# lvdisplay bd-vg
- [root@node ~]#
+```
+[root@node ~]# lvdisplay bd-vg
+```
Creating a file that is mapped to an LV is a 2 step operation. First the file should be created on the mount point and a specific extended attribute should be set to map the file to LV.
-
- [root@node ~]# touch /mnt/lv
- [root@node ~]# setfattr -n “user.glusterfs.bd” -v “lv” /mnt/lv
+```
+[root@node ~]# touch /mnt/lv
+[root@node ~]# setfattr -n “user.glusterfs.bd” -v “lv” /mnt/lv
+```
Now an LV got created in the VG brick and the file /mnt/lv maps to this LV. Any read/write to this file ends up as read/write to the underlying LV.
- [root@node ~]# lvdisplay bd-vg
- — Logical volume —
- LV Path /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287
- LV Name 6ff0f25f-2776-4d19-adfb-df1a3cab8287
- VG Name bd-vg
- LV UUID PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6
- LV Write Access read/write
- LV Creation host, time node, 2013-11-26 16:15:45 +0530
- LV Status available
- open 0
- LV Size 4.00 MiB
- Current LE 1
- Segments 1
- Allocation inherit
- Read ahead sectors 0
- Block device 253:6
+```
+[root@node ~]# lvdisplay bd-vg
+— Logical volume —
+LV Path /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287
+LV Name 6ff0f25f-2776-4d19-adfb-df1a3cab8287
+VG Name bd-vg
+LV UUID PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6
+LV Write Access read/write
+LV Creation host, time node, 2013-11-26 16:15:45 +0530
+LV Status available
+open 0
+LV Size 4.00 MiB
+Current LE 1
+Segments 1
+Allocation inherit
+Read ahead sectors 0
+Block device 253:6
+```
The file gets created with default LV size which is 1 LE which is 4MB in this case.
-
- [root@node ~]# ls -lh /mnt/lv
- -rw-r–r–. 1 root root 4.0M Nov 26 16:15 /mnt/lv
-
-truncate can be used to set the required file size.
-
- [root@node ~]# truncate /mnt/lv -s 256M
- [root@node ~]# lvdisplay bd-vg
- — Logical volume —
- LV Path /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287
- LV Name 6ff0f25f-2776-4d19-adfb-df1a3cab8287
- VG Name bd-vg
- LV UUID PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6
- LV Write Access read/write
- LV Creation host, time node, 2013-11-26 16:15:45 +0530
- LV Status available
- # open 0
- LV Size 256.00 MiB
- Current LE 64
- Segments 1
- Allocation inherit
- Read ahead sectors 0
- Block device 253:6
-
-
- [root@node ~]# ls -lh /mnt/lv
- -rw-r–r–. 1 root root 256M Nov 26 16:15 /mnt/lv
-
- currently LV size has been set to 256
+```
+[root@node ~]# ls -lh /mnt/lv
+-rw-r–r–. 1 root root 4.0M Nov 26 16:15 /mnt/lv
+```
+
+Truncate can be used to set the required file size.
+
+```
+[root@node ~]# trunc ate /mnt/lv -s 256M
+[root@node ~]# lvdisplay bd-vg
+— Logical volume —
+LV Path /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287
+LV Name 6ff0f25f-2776-4d19-adfb-df1a3cab8287
+VG Name bd-vg
+LV UUID PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6
+LV Write Access read/write
+LV Creation host, time node, 2013-11-26 16:15:45 +0530
+LV Status available
+# open 0
+LV Size 256.00 MiB
+Current LE 64
+Segments 1
+Allocation inherit
+Read ahead sectors 0
+Block device 253:6
+```
+
+```
+[root@node ~]# ls -lh /mnt/lv
+-rw-r–r–. 1 root root 256M Nov 26 16:15 /mnt/lv
+```
+
+Currently LV size has been set to 256.
The size of the file/LV can be specified during creation/mapping time itself like this:
-
- setfattr -n “user.glusterfs.bd” -v “lv:256MB” /mnt/lv
-
+```
+setfattr -n “user.glusterfs.bd” -v “lv:256MB” /mnt/lv
+```
2. Creating BD volume with thin LV backend
* Create a loop device
-
-
- [root@node ~]# dd if=/dev/zero of=bd-loop-thin count=1024 bs=1M
-
- [root@node ~]# losetup /dev/loop0 bd-loop-thin
-
+```
+[root@node ~]# dd if=/dev/zero of=bd-loop-thin count=1024 bs=1M
+[root@node ~]# losetup /dev/loop0 bd-loop-thin
+```
* Prepare a brick by creating a VG and thin pool
-
-
- [root@node ~]# pvcreate /dev/loop0
-
- [root@node ~]# vgcreate bd-vg-thin /dev/loop0
-
+```
+[root@node ~]# pvcreate /dev/loop0
+[root@node ~]# vgcreate bd-vg-thin /dev/loop0
+```
* Create a thin pool
-
-
- [root@node ~]# lvcreate –thin bd-vg-thin -L 1000M
-
- Rounding up size to full physical extent 4.00 MiB
- Logical volume “lvol0″ created
-
-lvdisplay shows the thin pool
-
- [root@node ~]# lvdisplay bd-vg-thin
- — Logical volume —
- LV Name lvol0
- VG Name bd-vg-thin
- LV UUID HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn
- LV Write Access read/write
- LV Creation host, time node, 2013-11-26 16:39:06 +0530
- LV Pool transaction ID 0
- LV Pool metadata lvol0_tmeta
- LV Pool data lvol0_tdata
- LV Pool chunk size 64.00 KiB
- LV Zero new blocks yes
- LV Status available
- # open 0
- LV Size 1000.00 MiB
- Allocated pool data 0.00%
- Allocated metadata 0.88%
- Current LE 250
- Segments 1
- Allocation inherit
- Read ahead sectors auto
- Block device 253:9
+```
+[root@node ~]# lvcreate –thin bd-vg-thin -L 1000M
+```
+
+Rounding up size to full physical extent 4.00 MiB.
+Logical volume “lvol0″ created.
+
+lvdisplay shows the thin pool.
+```
+[root@node ~]# lvdisplay bd-vg-thin
+— Logical volume —
+LV Name lvol0
+VG Name bd-vg-thin
+LV UUID HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn
+LV Write Access read/write
+LV Creation host, time node, 2013-11-26 16:39:06 +0530
+LV Pool transaction ID 0
+LV Pool metadata lvol0_tmeta
+LV Pool data lvol0_tdata
+LV Pool chunk size 64.00 KiB
+LV Zero new blocks yes
+LV Status available
+# open 0
+LV Size 1000.00 MiB
+Allocated pool data 0.00%
+Allocated metadata 0.88%
+Current LE 250
+Segments 1
+Allocation inherit
+Read ahead sectors auto
+Block device 253:9
+```
* Create the BD volume
* Create a POSIX directory first
-
-
- [root@node ~]# mkdir /bd-meta-thin
+```
+[root@node ~]# mkdir /bd-meta-thin
+```
* Create and mount the volume
-
- [root@node ~]# gluster volume create bd-thin node:/bd-meta-thin?bd-vg-thin force
-
- [root@node ~]# gluster volume start bd-thin
-
-
- [root@node ~]# gluster volume info bd-thin
- Volume Name: bd-thin
- Type: Distribute
- Volume ID: 27aa7eb0-4ffa-497e-b639-7cbda0128793
- Status: Started
- Xlator 1: BD
- Capability 1: thin
- Capability 2: offload_copy
- Capability 3: offload_snapshot
- Number of Bricks: 1
- Transport-type: tcp
- Bricks:
- Brick1: node:/bd-meta-thin
- Brick1 VG: bd-vg-thin
-
-
- [root@node ~]# mount -t glusterfs node:/bd-thin /mnt
+```
+[root@node ~]# gluster volume create bd-thin node:/bd-meta-thin?bd-vg-thin force
+
+[root@node ~]# gluster volume start bd-thin
+
+[root@node ~]# gluster volume info bd-thin
+Volume Name: bd-thin
+Type: Distribute
+Volume ID: 27aa7eb0-4ffa-497e-b639-7cbda0128793
+Status: Started
+Xlator 1: BD
+Capability 1: thin
+Capability 2: offload_copy
+Capability 3: offload_snapshot
+Number of Bricks: 1
+Transport-type: tcp
+Bricks:
+Brick1: node:/bd-meta-thin
+Brick1 VG: bd-vg-thin
+
+[root@node ~]# mount -t glusterfs node:/bd-thin /mnt
+```
* Create a file that is backed by a thin LV
-
-
- [root@node ~]# ls /mnt
-
- [root@node ~]#
+```
+[root@node ~]# ls /mnt
+```
Creating a file that is mapped to a thin LV is a 2 step operation. First the file should be created on the mount point and a specific extended attribute should be set to map the file to a thin LV.
+```
+[root@node ~]# touch /mnt/thin-lv
- [root@node ~]# touch /mnt/thin-lv
-
- [root@node ~]# setfattr -n “user.glusterfs.bd” -v “thin:256MB” /mnt/thin-lv
-
+[root@node ~]# setfattr -n “user.glusterfs.bd” -v “thin:256MB” /mnt/thin-lv
+```
Now /mnt/thin-lv is a thin provisioned file that is backed by a thin LV and size has been set to 256.
-
- [root@node ~]# lvdisplay bd-vg-thin
- — Logical volume —
- LV Name lvol0
- VG Name bd-vg-thin
- LV UUID HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn
- LV Write Access read/write
- LV Creation host, time node, 2013-11-26 16:39:06 +0530
- LV Pool transaction ID 1
- LV Pool metadata lvol0_tmeta
- LV Pool data lvol0_tdata
- LV Pool chunk size 64.00 KiB
- LV Zero new blocks yes
- LV Status available
- # open 0
- LV Size 000.00 MiB
- Allocated pool data 0.00%
- Allocated metadata 0.98%
- Current LE 250
- Segments 1
- Allocation inherit
- Read ahead sectors auto
- Block device 253:9
-
-
-
-
- — Logical volume —
- LV Path dev/bd-vg-thin/081b01d1-1436-4306-9baf-41c7bf5a2c73
- LV Name 081b01d1-1436-4306-9baf-41c7bf5a2c73
- VG Name bd-vg-thin
- LV UUID coxpTY-2UZl-9293-8H2X-eAZn-wSp6-csZIeB
- LV Write Access read/write
- LV Creation host, time node, 2013-11-26 16:43:19 +0530
- LV Pool name lvol0
- LV Status available
- # open 0
- LV Size 256.00 MiB
- Mapped size 0.00%
- Current LE 64
- Segments 1
- Allocation inherit
- Read ahead sectors auto
- Block device 253:10
-
+```
+[root@node ~]# lvdisplay bd-vg-thin
+— Logical volume —
+LV Name lvol0
+VG Name bd-vg-thin
+LV UUID HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn
+LV Write Access read/write
+LV Creation host, time node, 2013-11-26 16:39:06 +0530
+LV Pool transaction ID 1
+LV Pool metadata lvol0_tmeta
+LV Pool data lvol0_tdata
+LV Pool chunk size 64.00 KiB
+LV Zero new blocks yes
+LV Status available
+# open 0
+LV Size 000.00 MiB
+Allocated pool data 0.00%
+Allocated metadata 0.98%
+Current LE 250
+Segments 1
+Allocation inherit
+Read ahead sectors auto
+Block device 253:9
+
+— Logical volume —
+LV Path dev/bd-vg-thin/081b01d1-1436-4306-9baf-41c7bf5a2c73
+LV Name 081b01d1-1436-4306-9baf-41c7bf5a2c73
+VG Name bd-vg-thin
+LV UUID coxpTY-2UZl-9293-8H2X-eAZn-wSp6-csZIeB
+LV Write Access read/write
+LV Creation host, time node, 2013-11-26 16:43:19 +0530
+LV Pool name lvol0
+LV Status available
+# open 0
+LV Size 256.00 MiB
+Mapped size 0.00%
+Current LE 64
+Segments 1
+Allocation inherit
+Read ahead sectors auto
+Block device 253:10
+```
@@ -310,7 +299,7 @@ Now /mnt/thin-lv is a thin provisioned file that is backed by a thin LV and size
As can be seen from above, creation of a file resulted in creation of a thin LV in the brick.
-###Improvisation on BD translator:
+### Improvisation on BD translator:
First version of BD xlator ( block backend) had few limitations such as
@@ -356,42 +345,40 @@ is mapped to BD.
Usage:
Server side:
-
- [root@host1 ~]# gluster volume create bdvol host1:/storage/vg1_info?vg1 host2:/storage/vg2_info?vg2
+```
+[root@host1 ~]# gluster volume create bdvol host1:/storage/vg1_info?vg1 host2:/storage/vg2_info?vg2
+```
It creates a distributed gluster volume 'bdvol' with Volume Group vg1
using posix brick /storage/vg1_info in host1 and Volume Group vg2 using
/storage/vg2_info in host2.
-
-
- [root@host1 ~]# gluster volume start bdvol
+```
+[root@host1 ~]# gluster volume start bdvol
+```
Client side:
-
- [root@node ~]# mount -t glusterfs host1:/bdvol /media
- [root@node ~]# touch /media/posix
+```
+[root@node ~]# mount -t glusterfs host1:/bdvol /media
+[root@node ~]# touch /media/posix
+```
It creates regular posix file 'posix' in either host1:/vg1 or host2:/vg2 brick
-
- [root@node ~]# mkdir /media/image
-
- [root@node ~]# touch /media/image/lv1
-
+```
+[root@node ~]# mkdir /media/image
+[root@node ~]# touch /media/image/lv1
+```
It also creates regular posix file 'lv1' in either host1:/vg1 or
host2:/vg2 brick
-
- [root@node ~]# setfattr -n "user.glusterfs.bd" -v "lv" /media/image/lv1
-
- [root@node ~]#
-
+```
+[root@node ~]# setfattr -n "user.glusterfs.bd" -v "lv" /media/image/lv1
+```
Above setxattr results in creating a new LV in corresponding brick's VG
and it sets 'user.glusterfs.bd' with value 'lv:<default-extent-size''
-
-
- [root@node ~]# truncate -s5G /media/image/lv1
-
+```
+[root@node ~]# truncate -s5G /media/image/lv1
+```
It results in resizig LV 'lv1'to 5G
@@ -417,12 +404,14 @@ features can be exploited.
BD xlator exports capability information through gluster volume info (and --xml) output. For eg:
`snip of gluster volume info output for a BD based volume`
-
+ ```
Xlator 1: BD
Capability 1: thin
+```
`snip of gluster volume info --xml output for a BD based volume`
+```
<xlators>
<xlator>
<name>BD</name>
@@ -431,7 +420,7 @@ BD xlator exports capability information through gluster volume info (and --xml)
</capabilities>
</xlator>
</xlators>
-
+```
But this capability information should also exposed through some other means so that a host
which is not part of Gluster peer could also avail this capabilities.
@@ -465,5 +454,4 @@ For example, GlusterFS block driver in qemu need to get the capability list so t
-Parts of this documentation were originally published here
-#http://raobharata.wordpress.com/2013/11/27/glusterfs-block-device-translator/
+Parts of this documentation were originally published [here](http://raobharata.wordpress.com/2013/11/27/glusterfs-block-device-translator/).
diff --git a/doc/developer-guide/brickmux-thread-reduction.md b/doc/developer-guide/brickmux-thread-reduction.md
index 83462c06b50..7d76e8ff579 100644
--- a/doc/developer-guide/brickmux-thread-reduction.md
+++ b/doc/developer-guide/brickmux-thread-reduction.md
@@ -1,4 +1,4 @@
-#Resource usage reduction in brick multiplexing
+# Resource usage reduction in brick multiplexing
Each brick is regresented with a graph of translators in a brick process.
Each translator in the graph has its own set of threads and mem pools
@@ -24,7 +24,7 @@ threads:
4. aio-thread
- changelog and bitrot threads(even when the features are not enabled)
-##io-threads
+## io-threads
io-threads should be made global to the process, having 16+ threads for
each brick does not make sense. But io-thread translator is loaded in
@@ -38,7 +38,7 @@ This poses performance issues due to lock-contention in the io-threds
layer. This also shall be addressed by having multiple locks instead of
one global lock for io-threads.
-##Posix threads
+## Posix threads
Most of the posix threads execute tasks in a timely manner, hence it can be
replaced with a timer whose handler register a task to synctask framework, once
the task is complete, the timer is registered again. With this we can eliminate
@@ -56,7 +56,7 @@ api in synctask that can cancel all the tasks from a given translator.
For future, this will be replced to use global thread-pool(once implemented).
-##Changelog and bitrot threads
+## Changelog and bitrot threads
In the initial implementation, the threads are not created if the feature is
not enabled. We need to share threads across changelog instances if we plan
diff --git a/doc/developer-guide/coding-standard.md b/doc/developer-guide/coding-standard.md
index a8aa49cc2da..031c6c0da99 100644
--- a/doc/developer-guide/coding-standard.md
+++ b/doc/developer-guide/coding-standard.md
@@ -6,23 +6,23 @@ Before you get started
Before starting with other part of coding standard, install `clang-format`
On Fedora:
-
+```
$ dnf install clang
-
+```
On debian/Ubuntu:
-
+```
$ apt-get install clang
-
+```
Once you are done with all the local changes, you need to run below set of commands,
before submitting the patch for review.
-
- $ git add $file # if any
- $ git commit -a -s -m "commit message"
- $ git show --pretty="format:" --name-only | grep -v "contrib/" | egrep "*\.[ch]$" | xargs clang-format -i
- $ git diff # see if there are any changes
- $ git commit -a --amend # get the format changes done
- $ ./submit-for-review.sh
-
+```
+$ git add $file # if any
+$ git commit -a -s -m "commit message"
+$ git show --pretty="format:" --name-only | grep -v "contrib/" | egrep "*\.[ch]$" | xargs clang-format -i
+$ git diff # see if there are any changes
+$ git commit -a --amend # get the format changes done
+$ ./submit-for-review.sh
+```
Structure definitions should have a comment per member
@@ -620,8 +620,8 @@ If a value isn't supposed/expected to change, there's no cost to adding a
### Avoid global variables (including 'static' auto variables)
Almost all state in Gluster is contextual and should be contained in the
-appropriate structure reflecting its scope (e.g. call\_frame\_t, call\_stack\_t,
-xlator\_t, glusterfs\_ctx\_t). With dynamic loading and graph switches in play,
+appropriate structure reflecting its scope (e.g. `call\_frame\_t`, `call\_stack\_t`,
+`xlator\_t`, `glusterfs\_ctx\_t`). With dynamic loading and graph switches in play,
each global requires careful consideration of when it should be initialized or
reinitialized, when it might _accidentally_ be reinitialized, when its value
might become stale, and so on. A few global variables are needed to serve as
diff --git a/doc/developer-guide/datastructure-inode.md b/doc/developer-guide/datastructure-inode.md
index a340ab9ca8e..45d7a941e5f 100644
--- a/doc/developer-guide/datastructure-inode.md
+++ b/doc/developer-guide/datastructure-inode.md
@@ -1,6 +1,6 @@
-#Inode and dentry management in GlusterFS:
+# Inode and dentry management in GlusterFS:
-##Background
+## Background
Filesystems internally refer to files and directories via inodes. Inodes
are unique identifiers of the entities stored in a filesystem. Whenever an
application has to operate on a file/directory (read/modify), the filesystem
@@ -41,11 +41,10 @@ struct _inode_table {
};
```
-#Life-cycle
+# Life-cycle
```
-
inode_table_new (size_t lru_limit, xlator_t *xl)
-
+```
This is a function which allocates a new inode table. Usually the top xlators in
the graph such as protocol/server (for bricks), fuse and nfs (for fuse and nfs
mounts) and libgfapi do inode managements. Hence they are the ones which will
@@ -59,11 +58,8 @@ new inode table.
Thus an allocated inode table is destroyed only when the filesystem daemon is
killed or unmounted.
-```
-
-#what it contains.
-```
+# what it contains.
Inode table in glusterfs mainly contains a hash table for maintaining inodes.
In general a file/directory is considered to be existing if there is a
corresponding inode present in the inode table. If a inode for a file/directory
@@ -76,21 +72,21 @@ size of the hash table (as of now it is hard coded to 14057. The hash value of
a inode is calculated using its gfid).
Apart from the hash table, inode table also maintains 3 important list of inodes
-1) Active list:
+1. Active list:
Active list contains all the active inodes (i.e inodes which are currently part
of some fop).
-2) Lru list:
+2. Lru list:
Least recently used inodes list. A limit can be set for the size of the lru
list. For bricks it is 16384 and for clients it is infinity.
-3) Purge list:
+3. Purge list:
List of all the inodes which have to be purged (i.e inodes which have to be
deleted from the inode table due to unlink/rmdir/forget).
And at last it also contains the mem-pool for allocating inodes, dentries so
that frequent malloc/calloc and free of the data structures can be avoided.
-```
-#Data structure (inode)
+
+# Data structure (inode)
```
struct _inode {
inode_table_t *table; /* the table this inode belongs to */
@@ -108,7 +104,7 @@ struct _inode {
struct _inode_ctx *_ctx; /* place holder for keeping the
information about the inode by different xlators */
};
-
+```
As said above, inodes are internal way of identifying the files/directories. A
inode uniquely represents a file/directory. A new inode is created whenever a
create/mkdir/symlink/mknod operations are performed. Apart from that a new inode
@@ -128,9 +124,9 @@ inodes are those inodes whose refcount is greater than zero. Whenever some
operation comes on a file/directory, and the resolver tries to find the inode
for it, it increments the refcount of the inode before returning the inode. The
refcount of an inode can be incremented by calling the below function
-
+```
inode_ref (inode_t *inode)
-
+```
Any xlator which wants to operate on a inode as part of some fop (or wants the
inode in the callback), should hold a ref on the inode.
Once the fop is completed before sending the reply of the fop to the above
@@ -139,18 +135,18 @@ zero, it is removed from the active inodes list and put into LRU list maintained
by the inode table. Thus in short if some fop is happening on a file/directory,
the corresponding inode will be in the active list or it will be in the LRU
list.
-```
-#Life Cycle
+
+# Life Cycle
A new inode is created whenever a new file/directory/symlink is created OR a
successful lookup of an existing entry is done. The xlators which does inode
management (as of now protocol/server, fuse, nfs, gfapi) will perform inode_link
operation upon successful lookup or successful creation of a new entry.
-
+```
inode_link (inode_t *inode, inode_t *parent, const char *name,
struct iatt *buf);
-
+```
inode_link actually adds the inode to the inode table (to be precise it adds
the inode to the hash table maintained by the inode table. The hash value is
calculated based on the gfid). Copies the gfid to the inode (the gfid is
@@ -160,7 +156,7 @@ A inode is removed from the inode table and eventually destroyed when unlink
or rmdir operation is performed on a file/directory, or the the lru limit of
the inode table has been exceeded.
-#Data structure (dentry)
+# Data structure (dentry)
```
struct _dentry {
@@ -170,22 +166,22 @@ struct _dentry {
char *name; /* name of the directory entry */
inode_t *parent; /* directory of the entry */
};
-
+```
A dentry is the presence of an entry for a file/directory within its parent
directory. A dentry usually points to the inode to which it belongs to. In
glusterfs a dentry contains the following fields.
-1) a hook using which it can add itself to the list of
+1. a hook using which it can add itself to the list of
the dentries maintained by the inode to which it points to.
-2) A hash table pointer.
-3) Pointer to the inode to which it belongs to.
-4) Name of the dentry
-5) Pointer to the inode of the parent directory in which the dentry is present
+2. A hash table pointer.
+3. Pointer to the inode to which it belongs to.
+4. Name of the dentry
+5. Pointer to the inode of the parent directory in which the dentry is present
A new dentry is created when a new file/directory/symlink is created or a hard
link to an existing file is created.
-
+```
__dentry_create (inode_t *inode, inode_t *parent, const char *name);
-
+```
A dentry holds a refcount on the parent
directory so that the parent inode is never removed from the active inode's list
and put to the lru list (If the lru limit of the lru list is exceeded, there is
@@ -212,15 +208,14 @@ deleted due to file removal or lru limit being exceeded the inode is retired
purge list maintained by the inode table), the nlookup count is set to 0 via
inode_forget api. The inode table, then prunes all the inodes from the purge
list by destroying the inode contexts maintained by each xlator.
-
+```
unlinking of the dentry is done via inode_unlink;
void
inode_unlink (inode_t *inode, inode_t *parent, const char *name);
-
+```
If the inode has multiple hard links, then the unlink operation performed by
the application results just in the removal of the dentry with the name provided
by the application. For the inode to be removed, all the dentries of the inode
should be unlinked.
-```
diff --git a/doc/developer-guide/datastructure-iobuf.md b/doc/developer-guide/datastructure-iobuf.md
index 5f521f1485f..03604e3672c 100644
--- a/doc/developer-guide/datastructure-iobuf.md
+++ b/doc/developer-guide/datastructure-iobuf.md
@@ -1,6 +1,6 @@
-#Iobuf-pool
-##Datastructures
-###iobuf
+# Iobuf-pool
+## Datastructures
+### iobuf
Short for IO Buffer. It is one allocatable unit for the consumers of the IOBUF
API, each unit hosts @page_size(defined in arena structure) bytes of memory. As
initial step of processing a fop, the IO buffer passed onto GlusterFS by the
@@ -28,7 +28,7 @@ struct iobuf {
};
```
-###iobref
+### iobref
There may be need of multiple iobufs for a single fop, like in vectored read/write.
Hence multiple iobufs(default 16) are encapsulated under one iobref.
```
@@ -40,7 +40,7 @@ struct iobref {
int used; /* number of iobufs added to this iobref */
};
```
-###iobuf_arenas
+### iobuf_arenas
One region of memory MMAPed from the operating system. Each region MMAPs
@arena_size bytes of memory, and hosts @arena_size / @page_size IOBUFs.
The same sized iobufs are grouped into one arena, for sanity of access.
@@ -77,7 +77,7 @@ struct iobuf_arena {
};
```
-###iobuf_pool
+### iobuf_pool
Pool of Iobufs. As there may be many Io buffers required by the filesystem,
a pool of iobufs are preallocated and kept, if these preallocated ones are
exhausted only then the standard malloc/free is called, thus improving the
@@ -139,8 +139,8 @@ arenas in the purge list are destroyed only if there is atleast one arena in
(e.g: If there is an arena (page_size=128KB, count=32) in purge list, this arena
is destroyed(munmap) only if there is an arena in 'arenas' list with page_size=128KB).
-##APIs
-###iobuf_get
+## APIs
+### iobuf_get
```
struct iobuf *iobuf_get (struct iobuf_pool *iobuf_pool);
@@ -149,7 +149,7 @@ Creates a new iobuf of the default page size(128KB hard coded as of yet).
Also takes a reference(increments ref count), hence no need of doing it
explicitly after getting iobuf.
-###iobuf_get2
+### iobuf_get2
```
struct iobuf * iobuf_get2 (struct iobuf_pool *iobuf_pool, size_t page_size);
@@ -179,7 +179,7 @@ if (requested iobuf size > Max iobuf size in the pool(1MB as of yet))
Also takes a reference(increments ref count), hence no need of doing it
explicitly after getting iobuf.
-###iobuf_ref
+### iobuf_ref
```
struct iobuf *iobuf_ref (struct iobuf *iobuf);
@@ -188,7 +188,7 @@ struct iobuf *iobuf_ref (struct iobuf *iobuf);
xlator/function/, its a good practice to take a reference so that iobuf is not
deleted by the allocator.
-###iobuf_unref
+### iobuf_unref
```
void iobuf_unref (struct iobuf *iobuf);
```
@@ -203,33 +203,33 @@ Unreference the iobuf, if the ref count is zero iobuf is considered free.
Every iobuf_ref should have a corresponding iobuf_unref, and also every
iobuf_get/2 should have a correspondning iobuf_unref.
-###iobref_new
+### iobref_new
```
struct iobref *iobref_new ();
```
Creates a new iobref structure and returns its pointer.
-###iobref_ref
+### iobref_ref
```
struct iobref *iobref_ref (struct iobref *iobref);
```
Take a reference on the iobref.
-###iobref_unref
+### iobref_unref
```
void iobref_unref (struct iobref *iobref);
```
Decrements the reference count of the iobref. If the ref count is 0, then unref
all the iobufs(iobuf_unref) in the iobref, and destroy the iobref.
-###iobref_add
+### iobref_add
```
int iobref_add (struct iobref *iobref, struct iobuf *iobuf);
```
Adds the given iobuf into the iobref, it takes a ref on the iobuf before adding
it, hence explicit iobuf_ref is not required if adding to the iobref.
-###iobref_merge
+### iobref_merge
```
int iobref_merge (struct iobref *to, struct iobref *from);
```
@@ -239,13 +239,13 @@ on all the iobufs added to the 'to' iobref. Hence iobref_unref should be
performed both on 'from' and 'to' iobrefs (performing iobref_unref only on 'to'
will not free the iobufs and may result in leak).
-###iobref_clear
+### iobref_clear
```
void iobref_clear (struct iobref *iobref);
```
Unreference all the iobufs in the iobref, and also unref the iobref.
-##Iobuf Leaks
+## Iobuf Leaks
If all iobuf_refs/iobuf_new do not have correspondning iobuf_unref, then the
iobufs are not freed and recurring execution of such code path may lead to huge
memory leaks. The easiest way to identify if a memory leak is caused by iobufs
diff --git a/doc/developer-guide/datastructure-mem-pool.md b/doc/developer-guide/datastructure-mem-pool.md
index c71aa2a8ddd..225567cbf9f 100644
--- a/doc/developer-guide/datastructure-mem-pool.md
+++ b/doc/developer-guide/datastructure-mem-pool.md
@@ -1,5 +1,5 @@
-#Mem-pool
-##Background
+# Mem-pool
+## Background
There was a time when every fop in glusterfs used to incur cost of allocations/de-allocations for every stack wind/unwind between xlators because stack/frame/*_localt_t in every wind/unwind was allocated and de-allocated. Because of all these system calls in the fop path there was lot of latency and the worst part is that most of the times the number of frames/stacks active at any time wouldn't cross a threshold. So it was decided that this threshold number of frames/stacks would be allocated in the beginning of the process only once. Get one of them from the pool of stacks/frames whenever `STACK_WIND` is performed and put it back into the pool in `STACK_UNWIND`/`STACK_DESTROY` without incurring any extra system calls. The data structures are allocated only when threshold number of such items are in active use i.e. pool is in complete use.% increase in the performance once this was added to all the common data structures (inode/fd/dict etc) in xlators throughout the stack was tremendous.
## Data structure
@@ -27,7 +27,7 @@ will be served from here until all the elements in the pool are in use i.e. cold
};
```
-##Life-cycle
+## Life-cycle
```
mem_pool_new (data_type, unsigned long count)
@@ -120,5 +120,5 @@ mem_pool_destroy (struct mem_pool *pool)
Deletes this pool from the `global_list` maintained by `glusterfs-ctx` and frees all the memory allocated in `mem_pool_new`.
-###How to pick pool-size
+### How to pick pool-size
This varies from work-load to work-load. Create the mem-pool with some random size and run the work-load. Take the statedump after the work-load is complete. In the statedump if `max_alloc` is always less than `cold_count` may be reduce the size of the pool closer to `max_alloc`. On the otherhand if there are lots of `pool-misses` then increase the `pool_size` by `max_stdalloc` to achieve better 'hit-rate' of the pool.
diff --git a/doc/developer-guide/dirops-transactions-in-dht.md b/doc/developer-guide/dirops-transactions-in-dht.md
index 83f63be3f45..909a97001aa 100644
--- a/doc/developer-guide/dirops-transactions-in-dht.md
+++ b/doc/developer-guide/dirops-transactions-in-dht.md
@@ -1,3 +1,4 @@
+# dirops transactions in dht
Need for transactions during operations on directories arise from two
basic design elements of DHT:
@@ -269,4 +270,4 @@ And the examples are:
* Both _dst_ and _dst/dst_ have same gfid - _gfid-src_. As observed earlier, symptom might be directory listing being incomplete
- mkdir (dst) vs renamedir ("src", "dst")
- rmdir (src) vs renamedir ("src", "dst")
- - rmdir (dst) vs renamedir ("src", "dst") \ No newline at end of file
+ - rmdir (dst) vs renamedir ("src", "dst")
diff --git a/doc/developer-guide/network_compression.md b/doc/developer-guide/network_compression.md
index 7327591ef63..1222a765276 100644
--- a/doc/developer-guide/network_compression.md
+++ b/doc/developer-guide/network_compression.md
@@ -1,9 +1,9 @@
-#On-Wire Compression + Decompression
+# On-Wire Compression + Decompression
The 'compression translator' compresses and decompresses data in-flight
between client and bricks.
-###Working
+### Working
When a writev call occurs, the client compresses the data before sending it to
brick. On the brick, compressed data is decompressed. Similarly, when a readv
call occurs, the brick compresses the data before sending it to client. On the
@@ -19,7 +19,7 @@ During normal operation, this is the format of data sent over wire:
The trailer contains the CRC32 checksum and length of original uncompressed
data. This is used for validation.
-###Usage
+### Usage
Turning on compression xlator:
@@ -27,7 +27,7 @@ Turning on compression xlator:
gluster volume set <vol_name> network.compression on
~~~
-###Configurable parameters (optional)
+### Configurable parameters (optional)
**Compression level**
~~~
@@ -35,10 +35,10 @@ gluster volume set <vol_name> network.compression.compression-level 8
~~~
~~~
-0 : no compression
-1 : best speed
-9 : best compression
--1 : default compression
+ 0 : no compression
+ 1 : best speed
+ 9 : best compression
+-1 : default compression
~~~
**Minimum file size**
@@ -55,7 +55,7 @@ Other less frequently used parameters include `network.compression.mem-level`
and `network.compression.window-size`. More details can about these options
can be found by running `gluster volume set help` command.
-###Known Issues and Limitations
+### Known Issues and Limitations
* Compression translator cannot work with striped volumes.
* Mount point hangs when writing a file with write-behind xlator turned on. To
@@ -65,7 +65,7 @@ set`performance.strict-write-ordering` to on.
distribute volumes. This limitation is caused by AFR not being able to
propagate xdata. This issue has been fixed in glusterfs versions > 3.5
-###TODO
+### TODO
Although zlib offers high compression ratio, it is very slow. We can make the
translator pluggable to add support for other compression methods such as
[lz4 compression](https://code.google.com/p/lz4/)
diff --git a/doc/developer-guide/syncop.md b/doc/developer-guide/syncop.md
index 4e30451f30e..bcc8bd08e01 100644
--- a/doc/developer-guide/syncop.md
+++ b/doc/developer-guide/syncop.md
@@ -1,4 +1,4 @@
-#syncop framework
+# syncop framework
A coroutines-based, cooperative multi-tasking framework.
## Topics
diff --git a/doc/developer-guide/translator-development.md b/doc/developer-guide/translator-development.md
index 3bf7e153354..8f67bb1cae9 100644
--- a/doc/developer-guide/translator-development.md
+++ b/doc/developer-guide/translator-development.md
@@ -472,7 +472,7 @@ hello
Now let's interrupt the process and see where we are.
```
-^C
+
Program received signal SIGINT, Interrupt.
0x0000003a0060b3dc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
diff --git a/doc/features/ctime.md b/doc/features/ctime.md
index 6c6cad90055..74a77abed4b 100644
--- a/doc/features/ctime.md
+++ b/doc/features/ctime.md
@@ -1,25 +1,25 @@
-#Consistent time attributes in gluster across replica/distribute
+# Consistent time attributes in gluster across replica/distribute
-####Problem:
+#### Problem:
Traditionally gluster has been using time attributes (ctime, atime, mtime) of files/dirs from bricks. The problem with this approach is that, it is not consisteant across replica and distribute bricks. And applications which depend on it breaks as replica might not always return time attributes from same brick.
Tar especially gives "file changed as we read it" whenever it detects ctime differences when stat is served from different bricks. The way we have been trying to solve it is to serve the stat structures from same brick in afr, max-time in dht. But it doesn't avoid the problem completely. Because there is no way to change ctime at the moment(lutimes() only allows mtime, atime), there is little we can do to make sure ctimes match after self-heals/xattr updates/rebalance.
-####Solution Proposed:
+#### Solution Proposed:
Store time attribues (ctime, mtime, atime) as an xattr of the file. The xattr is updated based
on the fop. If a filesystem fop changes only mtime and ctime, update only those in xattr for
that file.
-####Design Overview:
+#### Design Overview:
1) As part of each fop, top layer will generate a time stamp and pass it to the down along
with other information
-> - This will bring a dependency for NTP synced clients along with servers
-> - There can be a diff in time if the fop stuck in the xlator for various reason,
+ - This will bring a dependency for NTP synced clients along with servers
+ - There can be a diff in time if the fop stuck in the xlator for various reason,
for ex: because of locks.
2) On the server, posix layer stores the value in the memory (inode ctx) and will sync the data periodically to the disk as an extended attr
-> - Of course sync call also will force it. And fop comes for an inode which is not linked, we do the sync immediately.
+ - Of course sync call also will force it. And fop comes for an inode which is not linked, we do the sync immediately.
3) Each time when inodes are created or initialized it read the data from disk and store in inode ctx.
@@ -29,19 +29,19 @@ for ex: because of locks.
6) File ops that changes the parent directory attr time need to be consistent across all the distributed directories across the subvolumes. (for eg: a create call will change ctime and mtime of parent dir)
-> - This has to handle separately because we only send the fop to the hashed subvolume.
-> - We can asynchronously send the timeupdate setattr fop to the other subvoumes and change the values for parent directory if the file fops is successful on hashed subvolume.
-> - This will have a window where the times are inconsistent across dht subvolume (Please provide your suggestions)
+ - This has to handle separately because we only send the fop to the hashed subvolume.
+ - We can asynchronously send the timeupdate setattr fop to the other subvoumes and change the values for parent directory if the file fops is successful on hashed subvolume.
+ - This will have a window where the times are inconsistent across dht subvolume (Please provide your suggestions)
7) Currently we have couple of mount options for time attributes like noatime, relatime , nodiratime etc. But we are not explicitly handled those options even if it is given as mount option when gluster mount.
-####Implementation Overview:
+#### Implementation Overview:
This features involves changes in following xlators.
-> - utime xlator
-> - posix xlator
+ - utime xlator
+ - posix xlator
-#####utime xlator:
+##### utime xlator:
This is a new client side xlator which does following tasks.
1. It will generate a time stamp and passes it down in frame->root->ctime and over the network.
@@ -50,7 +50,7 @@ This is a new client side xlator which does following tasks.
Patches:
1. https://review.gluster.org/#/c/19857/
-#####posix xlator:
+##### posix xlator:
Following tasks are done in posix xlator:
1. Provides APIs to set and get the xattr from backend. It also caches the xattr in inode context. During get, it updates time attributes stored in xattr into iatt structure.
@@ -61,7 +61,7 @@ Following tasks are done in posix xlator:
2. https://review.gluster.org/#/c/19795/
3. https://review.gluster.org/#/c/19796/
-####Pending Work:
+#### Pending Work:
1. Handling of time related mount options (noatime, realatime,etc)
2. flag based create (depending on flags in open, create behaviour might change)
3. Changes in dht for direcotory sync acrosss multiple subvolumes