diff options
| author | Humble Devassy Chirammal <hchiramm@redhat.com> | 2015-03-30 12:21:05 +0530 |
|---|---|---|
| committer | Kaleb KEITHLEY <kkeithle@redhat.com> | 2015-03-30 05:51:12 -0700 |
| commit | 8907f67ba215172b01a7018adcbb063fcc4570e9 (patch) | |
| tree | 68cf6557990ae4215926068625a7b5e0e1374882 /doc/developer-guide | |
| parent | e3bd2387a5973df4548fe4a62b5dfc227a2bdc64 (diff) | |
doc: restructure developer docs to new layout
The developer oriented information is scattered in source
and its very difficult to identify which are those.
With this patch subdirs are created under developer-guide
which will be the parent for developer notes. The changes
suggested in http://review.gluster.org/#/c/8827/ are also
included in this patch.
Change-Id: I4c8510d52c49f4066225f72cac8f97f087d6c70c
BUG: 1206539
Signed-off-by: Humble Devassy Chirammal <hchiramm@redhat.com>
Reviewed-on: http://review.gluster.org/10038
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Lalatendu Mohanty <lmohanty@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Diffstat (limited to 'doc/developer-guide')
20 files changed, 3705 insertions, 0 deletions
diff --git a/doc/developer-guide/adding-fops.md b/doc/developer-guide/adding-fops.md new file mode 100644 index 00000000000..3f72ed3e23a --- /dev/null +++ b/doc/developer-guide/adding-fops.md @@ -0,0 +1,18 @@ +Adding a new FOP +================ + +Steps to be followed when adding a new FOP to GlusterFS: + +1. Edit `glusterfs.h` and add a `GF_FOP_*` constant. +2. Edit `xlator.[ch]` and: + * add the new prototype for fop and callback. + * edit `xlator_fops` structure. +3. Edit `xlator.c` and add to fill_defaults. +4. Edit `protocol.h` and add struct necessary for the new FOP. +5. Edit `defaults.[ch]` and provide default implementation. +6. Edit `call-stub.[ch]` and provide stub implementation. +7. Edit `common-utils.c` and add to gf_global_variable_init(). +8. Edit client-protocol and add your FOP. +9. Edit server-protocol and add your FOP. +10. Implement your FOP in any translator for which the default implementation + is not sufficient. diff --git a/doc/developer-guide/afr/afr-locks-evolution.md b/doc/developer-guide/afr/afr-locks-evolution.md new file mode 100644 index 00000000000..7d2a136d871 --- /dev/null +++ b/doc/developer-guide/afr/afr-locks-evolution.md @@ -0,0 +1,91 @@ +History of locking in AFR +-------------------------- + +GlusterFS has **locks** translator which provides the following internal locking operations called `inodelk`, `entrylk` which are used by afr to achieve synchronization of operations on files or directories that conflict with each other. + +`Inodelk` gives the facility for translators in GlusterFS to obtain range (denoted by tuple with **offset**, **length**) locks in a given **domain** for an inode. +Full file lock is denoted by the tuple (offset: `0`, length: `0`) i.e. length `0` is considered as infinity. + +`Entrylk` enables translators of GlusterFS to obtain locks on `name` in a given **domain** for an inode, typically a directory. + +The **locks** translator provides both *blocking* and *nonblocking* variants and of these locks. + + +AFR makes use of locks xlator extensively: + +1)For FOPS (from clients) +----------------------- +* Data transactions take inode locks on data domain, Let's refer to this domain name as DATA_DOMAIN. + + So locking for writes would be something like this:`inodelk(offset,length, DATA_DOMAIN)` + For truncating a file to zero, it would be `inodelk(0,0,DATA_DOMAIN)` + +* Metadata transactions (chown/chmod) also take inode locks but on a special range on metadata domain, + i.e.`(LLONG_MAX-1 , 0, METADATA_DOMAIN).` + +* Entry transactions (create, mkdir, rmdir,unlink, symlink, link,rename) take entrylk on `(name, parent inode)`. + + +2)For self heal: +------------- +* For Metadata self-heal, it is the same. i.e.`inodelk(LLONG_MAX-1 , 0, METADATA_DOMAIN)`. +* For Entry self-heal, it is `entrylk(NULL name, parent inode)`. Specifying NULL for the name takes full lock on the directory referred to by the inode. +* For data self-heal, there is a bit of history as to how locks evolved: + +###Initial version (say version 1) : +There was no concept of selfheal daemon (shd). Only client lookups triggered heals. so AFR always took `inodelk(0,0,DATA_DOMAIN)` for healing. The issue with this approach was that when heal was in progress, I/O from clients was blocked . + +###version 2: +shd was introduced. We needed to allow I/O to go through when heal was going,provided the ranges did not overlap. To that extent, the following approach was adopted: + ++ 1.shd takes (full inodelk in DATA_DOMAIN). Thus client FOPS are blocked and cannot modify changelog-xattrs ++ 2.shd inspects xattrs to determine source/sink ++ 3.shd takes a chunk inodelk(0-128kb) again in DATA_DOMAIN (locks xlator allows overlapping locks if lock owner is the same). ++ 4.unlock full lock ++ 5.heal ++ 6.take next chunk lock(129-256kb) ++ 7.unlock 1st chunk lock, heal the second chunk and so on. + + +Thus after 4, any client FOP could write to regions that was not currently under heal. The exception was truncate (to size 0) because it needs full file lock and will always block because some chunk is always under lock by the shd until heal completes. + +Another issue was that 2 shds could run in parallel. Say SHD1 and SHD2 compete for step 1. Let SHD1 win. It proceeds and completes step 4. Now SHD2 also succeeds in step 1, continues all steps. Thus at the end both shds will decrement the changelog leading to negative values in it) + +### version 3 +To prevent parallel self heals, another domain was introduced, let us call it SELF_HEAL_DOMAIN. With this domain, the following approach was adopted and is **the approach currently in use**: + ++ 1.shd takes (full inodelk on SELF_HEAL_DOMAIN) ++ 2.shd takes (full inodelk on DATA_DOMAIN) ++ 3.shd inspects xattrs to determine source/sink ++ 4.unlock full lock on DATA_DOMAIN ++ 5.take chunk lock(0-128kb) on DATA_DOMAIN ++ 6.heal ++ 7.take next chunk lock(129-256kb) on DATA_DOMAIN ++ 8.unlock 1st chunk lock, heal and so on. ++ 9.Finally release full lock on SELF_HEAL_DOMAIN + +Thus until one shd completes step 9, another shd cannot start step 1, solving the problem of simultaneous heals. +Note that the issue of truncate (to zero) FOP hanging still remains. +Also there are multiple network calls involved in this scheme. (lock,heal(ie read+write), unlock) per chunk. i.e 4 calls per chunk. + +### version 4 (ToDo) +Some improvements that need to be made in version 3: +* Reduce network calls using piggy backing. +* After taking chunk lock and healing, we need to unlock the lock before locking the next chunk. This gives a window for any pending truncate FOPs to succeed. If truncate succeeds, the heal of next chunk will fail (read returns zero) +and heal is stopped. *BUT* there is **yet another** issue: + +* shd does steps 1 to 4. Let's assume source is brick b1, sink is brick b2 . i.e xattrs are (0,1) and (0,0) on b1 and b2 respectively. Now before shd takes (0-128kb) lock, a client FOP takes it. +It modifies data but the FOP succeeds only on brick 2. writev returns success, and the attrs now read (0,1) (1,0). SHD takes over and heals. It had observed (0,1),(0,0) earlier +and thus goes ahead and copies stale 128Kb from brick 1 to brick2. Thus as far as application is concerned, `writev` returned success but bricks have stale data. +What needs to be done is `writev` must return success only if it succeeded on atleast one source brick (brick b1 in this case). Otherwise The heal still happens in reverse direction but as far as the application is concerned, it received an error. + +###Note on lock **domains** +We have used conceptual names in this document like DATA_DOMAIN/ METADATA_DOMAIN/ SELF_HEAL_DOMAIN. In the code, these are mapped to strings that are based on the AFR xlator name like so: + +DATA_DOMAIN --->"vol_name-replicate-n" + +METADATA_DOMAIN --->"vol_name-replicate-n:metadata" + +SELF_HEAL_DOMAIN -->"vol_name-replicate-n:self-heal" + +where vol_name is the name of the volume and 'n' is the replica subvolume index (starting from 0). diff --git a/doc/developer-guide/afr/afr.md b/doc/developer-guide/afr/afr.md new file mode 100644 index 00000000000..566573a4e26 --- /dev/null +++ b/doc/developer-guide/afr/afr.md @@ -0,0 +1,191 @@ +cluster/afr translator +====================== + +Locking +------- + +Before understanding replicate, one must understand two internal FOPs: + +### `GF_FILE_LK` + +This is exactly like `fcntl(2)` locking, except the locks are in a +separate domain from locks held by applications. + +### `GF_DIR_LK (loc_t *loc, char *basename)` + +This allows one to lock a name under a directory. For example, +to lock /mnt/glusterfs/foo, one would use the call: + +``` +GF_DIR_LK ({loc_t for "/mnt/glusterfs"}, "foo") +``` + +If one wishes to lock *all* the names under a particular directory, +supply the basename argument as `NULL`. + +The locks can either be read locks or write locks; consult the +function prototype for more details. + +Both these operations are implemented by the features/locks (earlier +known as posix-locks) translator. + +Basic design +------------ + +All FOPs can be classified into four major groups: + +### inode-read + +Operations that read an inode's data (file contents) or metadata (perms, etc.). + +access, getxattr, fstat, readlink, readv, stat. + +### inode-write + +Operations that modify an inode's data or metadata. + +chmod, chown, truncate, writev, utimens. + +### dir-read + +Operations that read a directory's contents or metadata. + +readdir, getdents, checksum. + +### dir-write + +Operations that modify a directory's contents or metadata. + +create, link, mkdir, mknod, rename, rmdir, symlink, unlink. + +Some of these make a subgroup in that they modify *two* different entries: +link, rename, symlink. + +### Others + +Other operations. + +flush, lookup, open, opendir, statfs. + +Algorithms +---------- + +Each of the four major groups has its own algorithm: + +### inode-read, dir-read + +1. Send a request to the first child that is up: + * if it fails: + * try the next available child + * if we have exhausted all children: + * return failure + +### inode-write + + All operations are done in parallel unless specified otherwise. + +1. Send a ``GF_FILE_LK`` request on all children for a write lock on the + appropriate region + (for metadata operations: entire file (0, 0) for writev: + (offset, offset+size of buffer)) + * If a lock request fails on a child: + * unlock all children + * try to acquire a blocking lock (`F_SETLKW`) on each child, serially. + If this fails (due to `ENOTCONN` or `EINVAL`): + Consider this child as dead for rest of transaction. +2. Mark all children as "pending" on all (alive) children (see below for +meaning of "pending"). + * If it fails on any child: + * mark it as dead (in transaction local state). +3. Perform operation on all (alive) children. + * If it fails on any child: + * mark it as dead (in transaction local state). +4. Unmark all successful children as not "pending" on all nodes. +5. Unlock region on all (alive) children. + +### dir-write + + The algorithm for dir-write is same as above except instead of holding + `GF_FILE_LK` locks we hold a GF_DIR_LK lock on the name being operated upon. + In case of link-type calls, we hold locks on both the operand names. + +"pending" +--------- + +The "pending" number is like a journal entry. A pending entry is an +array of 32-bit integers stored in network byte-order as the extended +attribute of an inode (which can be a directory as well). + +There are three keys corresponding to three types of pending operations: + +### `AFR_METADATA_PENDING` + +There are some metadata operations pending on this inode (perms, ctime/mtime, +xattr, etc.). + +### `AFR_DATA_PENDING` + +There is some data pending on this inode (writev). + +### `AFR_ENTRY_PENDING` + +There are some directory operations pending on this directory +(create, unlink, etc.). + +Self heal +--------- + +* On lookup, gather extended attribute data: + * If entry is a regular file: + * If an entry is present on one child and not on others: + * create entry on others. + * If entries exist but have different metadata (perms, etc.): + * consider the entry with the highest `AFR_METADATA_PENDING` number as + definitive and replicate its attributes on children. + * If entry is a directory: + * Consider the entry with the highest `AFR_ENTRY_PENDING` number as + definitive and replicate its contents on all children. + * If any two entries have non-matching types (i.e., one is file and + other is directory): + * Announce to the user via log that a split-brain situation has been + detected, and do nothing. +* On open, gather extended attribute data: + * Consider the file with the highest `AFR_DATA_PENDING` number as + the definitive one and replicate its contents on all other + children. + +During all self heal operations, appropriate locks must be held on all +regions/entries being affected. + +Inode scaling +------------- + +Inode scaling is necessary because if a situation arises where an inode number +is returned for a directory (by lookup) which was previously the inode number +of a file (as per FUSE's table), then FUSE gets horribly confused (consult a +FUSE expert for more details). + +To avoid such a situation, we distribute the 64-bit inode space equally +among all children of replicate. + +To illustrate: + +If c1, c2, c3 are children of replicate, they each get 1/3 of the available +inode space: + +------------- -- -- -- -- -- -- -- -- -- -- -- --- +Child: c1 c2 c3 c1 c2 c3 c1 c2 c3 c1 c2 ... +Inode number: 1 2 3 4 5 6 7 8 9 10 11 ... +------------- -- -- -- -- -- -- -- -- -- -- -- --- + +Thus, if lookup on c1 returns an inode number "2", it is scaled to "4" +(which is the second inode number in c1's space). + +This way we ensure that there is never a collision of inode numbers from +two different children. + +This reduction of inode space doesn't really reduce the usability of +replicate since even if we assume replicate has 1024 children (which would be a +highly unusual scenario), each child still has a 54-bit inode space: +$2^{54} \sim 1.8 \times 10^{16}$, which is much larger than any real +world requirement. diff --git a/doc/developer-guide/afr/self-heal-daemon.md b/doc/developer-guide/afr/self-heal-daemon.md new file mode 100644 index 00000000000..1fd8f08062a --- /dev/null +++ b/doc/developer-guide/afr/self-heal-daemon.md @@ -0,0 +1,91 @@ +Self-Heal Daemon +================ +The self-heal daemon (shd) is a glusterfs process that is responsible for healing files in a replicate/ disperse gluster volume. +Every server (brick) node of the volume runs one instance of the shd. So even if one node contains replicate/ disperse bricks of +multiple volumes, it would be healed by the same shd. + +This document only describes how the shd works for replicate (AFR) volumes. + +The shd is launched by glusterd when the volume starts (only if the volume includes a replicate configuration). The graph +of the shd process in every node contains the following: The io-stats which is the top most xlator, its children being the +replicate xlators (subvolumes) of *only* the bricks present in that particular node node, and finally *all* the client xlators that are the children of the replicate xlators. + +The shd does two types of self-heal crawls: Index heal and Full heal. For both these types of crawls, the basic idea is the same: +For each file encountered while crawling, perform metadata, data and entry heals under appropriate locks. +* An overview of how each of these heals is performed is detailed in the 'Self-healing' section of *doc/features/afr-v1.md* +* The different file locks which the shd takes for each of these heals is detailed in *doc/code/xlators/cluster/afr/afr-locks-evolution.md* + +Metadata heal refers to healing extended attributes, mode and permissions of a file or directory. +Data heal refers to healing the file contents. +Entry self-heal refers to healing entries inside a directory. + +Index heal +========== +The index heal is done: + a) Every 600 seconds (can be changed via the `cluster.heal-timeout` volume option) + b) When it is explicitly triggered via the `gluster vol heal <VOLNAME>` command + c) Whenever a replica brick that was down comes back up. + +Only one heal can be in progress at one time, irrespective of reason why it was triggered. If another heal is triggered before the first one completes, it will be queued. +Only one heal can be queued while the first one is running. If an Index heal is queued, it can be overridden by queuing a Full heal and not vice-versa. Also, before processing +each entry in index heal, a check is made if a full heal is queued. If it is, then the index heal is aborted so that the full heal can proceed. + +In index heal, each shd reads the entries present inside .glusterfs/indices/xattrop folder and triggers heal on each entry with appropriate locks. +The .glusterfs/indices/xattrop directory contains a base entry of the name "xattrop-<virtual-gfid-string>". All other entries are hardlinks to the base entry. The +*names* of the hardlinks are the gfid strings of the files that may need heal. + +When a client (mount) performs an operation on the file, the index xlator present in each brick process adds the hardlinks in the pre-op phase of the FOP's transaction +and removes it in post-op phase if the operation is successful. Thus if an entry is present inside the .glusterfs/indices/xattrop directory when there is no I/O +happening on the file, it means the file needs healing (or atleast an examination if the brick crashed after the post-op completed but just before the removal of the hardlink). + +####Index heal steps: +<pre><code> +In shd process of *each node* { + opendir +readdir (.glusterfs/indices/xattrop) + for each entry inside it { + self_heal_entry() //Explained below. + } +} +</code></pre> + +<pre><code> +self_heal_entry() { + Call syncop_lookup(replicae subvolume) which eventually does { + take appropriate locks + determine source and sinks from AFR changelog xattrs + perform whatever heal is needed (any of metadata, data and entry heal in that order) + clear changelog xattrs and hardlink inside .glusterfs/indices/xattrop + } +} +</code></pre> + +Note: +* If the gfid hardlink is present in the .glusterfs/indices/xattrop of both replica bricks, then each shd will try to heal the file but only one of them will be able to proceed due to the self-heal domain lock. + +* While processing entries inside .glusterfs/indices/xattrop, if shd encounters an entry whose parent is yet to be healed, it will skip it and it will be picked up in the next crawl. + +* If a file is in data/ metadata split-brain, it will not be healed. + +* If a directory is in entry split-brain, a conservative merge will be performed, wherein after the merge, the entries of the directory will be a union of the entries in the replica pairs. + +Full heal +========= +A full heal is triggered by running `gluster vol heal <VOLNAME> full`. This command is usually run in disk replacement scenarios where the entire data is to be copied from one of the healthy bricks of the replica to the brick that was just replaced. + +Unlike the index heal which runs on the shd of every node in a replicate subvolume, the full heal is run only on the shd of one node per replicate subvolume: the node having the highest UUID. +i.e In a 2x2 volume made of 4 nodes N1, N2, N3 and N4, If UUID of N1>N2 and UUID N4 >N3, then the full crawl is carried out by the shds of N1 and N4.(Node UUID can be found in `/var/lib/glusterd/glusterd.info`) + +The full heal steps are almost identical to the index heal, except the heal is performed on each replica starting from the root of the volume: +<pre><code> +In shd process of *highest UUID node per replica* { + opendir +readdir ("/") + for each entry inside it { + self_heal_entry() + if (entry == directory) { + /* Recurse*/ + again opendir+readdir (directory) followed by self_heal_entry() of each entry. + } + + } +} +</code></pre> diff --git a/doc/developer-guide/bd-xlator.md b/doc/developer-guide/bd-xlator.md new file mode 100644 index 00000000000..1771fb6e24b --- /dev/null +++ b/doc/developer-guide/bd-xlator.md @@ -0,0 +1,469 @@ +#Block device translator + +Block device translator (BD xlator) is a translator added to GlusterFS which provides block backend for GlusterFS. This replaces the existing bd_map translator in GlusterFS that provided similar but very limited functionality. GlusterFS expects the underlying brick to be formatted with a POSIX compatible file system. BD xlator changes that and allows for having bricks that are raw block devices like LVM which needn’t have any file systems on them. Hence with BD xlator, it becomes possible to build a GlusterFS volume comprising of bricks that are logical volumes (LV). + +##bd + +BD xlator maps underlying LVs to files and hence the LVs appear as files to GlusterFS clients. Though BD volume externally appears very similar to the usual Posix volume, not all operations are supported or possible for the files on a BD volume. Only those operations that make sense for a block device are supported and the exact semantics are described in subsequent sections. + +While Posix volume takes a file system directory as brick, BD volume needs a volume group (VG) as brick. In the usual use case of BD volume, a file created on BD volume will result in an LV being created in the brick VG. In addition to a VG, BD volume also needs a file system directory that should be specified at the volume creation time. This directory is necessary for supporting the notion of directories and directory hierarchy for the BD volume. Metadata about LVs (size, mapping info) is stored in this directory. + +BD xlator was mainly developed to use block devices directly as VM images when GlusterFS is used as storage for KVM virtualization. Some of the salient points of BD xlator are + +* Since BD supports file level snapshots and clones by leveraging the snapshot and clone capabilities of LVM, it can be used to fully off-load snapshot and cloning operations from QEMU to the storage (GlusterFS) itself. + +* BD understands dm-thin LVs and hence can support files that are backed by thinly provisioned LVs. This capability of BD xlator translates to having thinly provisioned raw VM images. + +* BD enables thin LVs from a thin pool to be used from multiple nodes that have visibility to GlusterFS BD volume. Thus thin pool can be used as a VM image repository allowing access/visibility to it from multiple nodes. + +* BD supports true zerofill by using BLKZEROOUT ioctl on underlying block devices. Thus BD allows SCSI WRITESAME to be used on underlying block device if the device supports it. + +Though BD xlator is primarily intended to be used with block devices, it does provide full Posix xlator compatibility for files that are created on BD volume but are not backed by or mapped to a block device. Such files which don’t have a block device mapping exist on the Posix directory that is specified during BD volume creation. BD xlator is available from GlusterFS-3.5 release. + +###Compiling BD translator + +BD xlator needs lvm2 development library. –enable-bd-xlator option can be used with `./configure` script to explicitly enable BD translator. The following snippet from the output of configure script shows that BD xlator is enabled for compilation. + + +#####GlusterFS configure summary + + … + Block Device xlator : yes + + +###Creating a BD volume + +BD supports hosting of both linear LV and thin LV within the same volume. However seperate examples are provided below. As noted above, the prerequisite for a BD volume is VG which is created from a loop device here, but it can be any other device too. + + +* Creating BD volume with linear LV backend + +* Create a loop device + + + [root@node ~]# dd if=/dev/zero of=bd-loop count=1024 bs=1M + + [root@node ~]# losetup /dev/loop0 bd-loop + + +* Prepare a brick by creating a VG + + [root@node ~]# pvcreate /dev/loop0 + + [root@node ~]# vgcreate bd-vg /dev/loop0 + + +* Create the BD volume + +* Create a POSIX directory first + + + [root@node ~]# mkdir /bd-meta + +It is recommended that this directory is created on an LV in the brick VG itself so that both data and metadata live together on the same device. + + +* Create and mount the volume + + [root@node ~]# gluster volume create bd node:/bd-meta?bd-vg force + + +The general syntax for specifying the brick is `host:/posix-dir?volume-group-name` where “?” is the separator. + + + + [root@node ~]# gluster volume start bd + [root@node ~]# gluster volume info bd + Volume Name: bd + Type: Distribute + Volume ID: cb042d2a-f435-4669-b886-55f5927a4d7f + Status: Started + Xlator 1: BD + Capability 1: offload_copy + Capability 2: offload_snapshot + Number of Bricks: 1 + Transport-type: tcp + Bricks: + Brick1: node:/bd-meta + Brick1 VG: bd-vg + + + + [root@node ~]# mount -t glusterfs node:/bd /mnt + +* Create a file that is backed by an LV + + [root@node ~]# ls /mnt + + [root@node ~]# + +Since the volume is empty now, so is the underlying VG. + + [root@node ~]# lvdisplay bd-vg + [root@node ~]# + +Creating a file that is mapped to an LV is a 2 step operation. First the file should be created on the mount point and a specific extended attribute should be set to map the file to LV. + + [root@node ~]# touch /mnt/lv + [root@node ~]# setfattr -n “user.glusterfs.bd” -v “lv” /mnt/lv + +Now an LV got created in the VG brick and the file /mnt/lv maps to this LV. Any read/write to this file ends up as read/write to the underlying LV. + + [root@node ~]# lvdisplay bd-vg + — Logical volume — + LV Path /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287 + LV Name 6ff0f25f-2776-4d19-adfb-df1a3cab8287 + VG Name bd-vg + LV UUID PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6 + LV Write Access read/write + LV Creation host, time node, 2013-11-26 16:15:45 +0530 + LV Status available + open 0 + LV Size 4.00 MiB + Current LE 1 + Segments 1 + Allocation inherit + Read ahead sectors 0 + Block device 253:6 + +The file gets created with default LV size which is 1 LE which is 4MB in this case. + + [root@node ~]# ls -lh /mnt/lv + -rw-r–r–. 1 root root 4.0M Nov 26 16:15 /mnt/lv + +truncate can be used to set the required file size. + + [root@node ~]# truncate /mnt/lv -s 256M + [root@node ~]# lvdisplay bd-vg + — Logical volume — + LV Path /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287 + LV Name 6ff0f25f-2776-4d19-adfb-df1a3cab8287 + VG Name bd-vg + LV UUID PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6 + LV Write Access read/write + LV Creation host, time node, 2013-11-26 16:15:45 +0530 + LV Status available + # open 0 + LV Size 256.00 MiB + Current LE 64 + Segments 1 + Allocation inherit + Read ahead sectors 0 + Block device 253:6 + + + [root@node ~]# ls -lh /mnt/lv + -rw-r–r–. 1 root root 256M Nov 26 16:15 /mnt/lv + + currently LV size has been set to 256 + +The size of the file/LV can be specified during creation/mapping time itself like this: + + setfattr -n “user.glusterfs.bd” -v “lv:256MB” /mnt/lv + +2. Creating BD volume with thin LV backend + +* Create a loop device + + + [root@node ~]# dd if=/dev/zero of=bd-loop-thin count=1024 bs=1M + + [root@node ~]# losetup /dev/loop0 bd-loop-thin + + +* Prepare a brick by creating a VG and thin pool + + + [root@node ~]# pvcreate /dev/loop0 + + [root@node ~]# vgcreate bd-vg-thin /dev/loop0 + + +* Create a thin pool + + + [root@node ~]# lvcreate –thin bd-vg-thin -L 1000M + + Rounding up size to full physical extent 4.00 MiB + Logical volume “lvol0″ created + +lvdisplay shows the thin pool + + [root@node ~]# lvdisplay bd-vg-thin + — Logical volume — + LV Name lvol0 + VG Name bd-vg-thin + LV UUID HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn + LV Write Access read/write + LV Creation host, time node, 2013-11-26 16:39:06 +0530 + LV Pool transaction ID 0 + LV Pool metadata lvol0_tmeta + LV Pool data lvol0_tdata + LV Pool chunk size 64.00 KiB + LV Zero new blocks yes + LV Status available + # open 0 + LV Size 1000.00 MiB + Allocated pool data 0.00% + Allocated metadata 0.88% + Current LE 250 + Segments 1 + Allocation inherit + Read ahead sectors auto + Block device 253:9 + +* Create the BD volume + +* Create a POSIX directory first + + + [root@node ~]# mkdir /bd-meta-thin + +* Create and mount the volume + + [root@node ~]# gluster volume create bd-thin node:/bd-meta-thin?bd-vg-thin force + + [root@node ~]# gluster volume start bd-thin + + + [root@node ~]# gluster volume info bd-thin + Volume Name: bd-thin + Type: Distribute + Volume ID: 27aa7eb0-4ffa-497e-b639-7cbda0128793 + Status: Started + Xlator 1: BD + Capability 1: thin + Capability 2: offload_copy + Capability 3: offload_snapshot + Number of Bricks: 1 + Transport-type: tcp + Bricks: + Brick1: node:/bd-meta-thin + Brick1 VG: bd-vg-thin + + + [root@node ~]# mount -t glusterfs node:/bd-thin /mnt + +* Create a file that is backed by a thin LV + + + [root@node ~]# ls /mnt + + [root@node ~]# + +Creating a file that is mapped to a thin LV is a 2 step operation. First the file should be created on the mount point and a specific extended attribute should be set to map the file to a thin LV. + + [root@node ~]# touch /mnt/thin-lv + + [root@node ~]# setfattr -n “user.glusterfs.bd” -v “thin:256MB” /mnt/thin-lv + +Now /mnt/thin-lv is a thin provisioned file that is backed by a thin LV and size has been set to 256. + + [root@node ~]# lvdisplay bd-vg-thin + — Logical volume — + LV Name lvol0 + VG Name bd-vg-thin + LV UUID HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn + LV Write Access read/write + LV Creation host, time node, 2013-11-26 16:39:06 +0530 + LV Pool transaction ID 1 + LV Pool metadata lvol0_tmeta + LV Pool data lvol0_tdata + LV Pool chunk size 64.00 KiB + LV Zero new blocks yes + LV Status available + # open 0 + LV Size 000.00 MiB + Allocated pool data 0.00% + Allocated metadata 0.98% + Current LE 250 + Segments 1 + Allocation inherit + Read ahead sectors auto + Block device 253:9 + + + + + — Logical volume — + LV Path dev/bd-vg-thin/081b01d1-1436-4306-9baf-41c7bf5a2c73 + LV Name 081b01d1-1436-4306-9baf-41c7bf5a2c73 + VG Name bd-vg-thin + LV UUID coxpTY-2UZl-9293-8H2X-eAZn-wSp6-csZIeB + LV Write Access read/write + LV Creation host, time node, 2013-11-26 16:43:19 +0530 + LV Pool name lvol0 + LV Status available + # open 0 + LV Size 256.00 MiB + Mapped size 0.00% + Current LE 64 + Segments 1 + Allocation inherit + Read ahead sectors auto + Block device 253:10 + + + + + +As can be seen from above, creation of a file resulted in creation of a thin LV in the brick. + + +###Improvisation on BD translator: + +First version of BD xlator ( block backend) had few limitations such as + +* Creation of directories not supported +* Supports only single brick +* Does not use extended attributes (and client gfid) like posix xlator +* Creation of special files (symbolic links, device nodes etc) not + supported + +Basic limitation of not allowing directory creation was blocking +oVirt/VDSM to consume BD xlator as part of Gluster domain since VDSM +creates multi-level directories when GlusterFS is used as storage +backend for storing VM images. + +To overcome these limitations a new BD xlator with following +improvements are implemented. + +* New hybrid BD xlator that handles both regular files and block device + files +* The volume will have both POSIX and BD bricks. Regular files are + created on POSIX bricks, block devices are created on the BD brick (VG) +* BD xlator leverages exiting POSIX xlator for most POSIX calls and + hence sits above the POSIX xlator +* Block device file is differentiated from regular file by an extended + attribute +* The xattr 'user.glusterfs.bd' (BD_XATTR) plays a role in mapping a + posix file to Logical Volume (LV). +* When a client sends a request to set BD_XATTR on a posix file, a new + LV is created and mapped to posix file. So every block device will + have a representative file in POSIX brick with 'user.glusterfs.bd' + (BD_XATTR) set. +* Here after all operations on this file results in LV related + operations. + +For example, opening a file that has BD_XATTR set results in opening +the LV block device, reading results in reading the corresponding LV +block device. + +When BD xlator gets request to set BD_XATTR via setxattr call, it +creates a LV and information about this LV is placed in the xattr of the +posix file. xattr "user.glusterfs.bd" used to identify that posix file +is mapped to BD. + +Usage: +Server side: + + [root@host1 ~]# gluster volume create bdvol host1:/storage/vg1_info?vg1 host2:/storage/vg2_info?vg2 + +It creates a distributed gluster volume 'bdvol' with Volume Group vg1 +using posix brick /storage/vg1_info in host1 and Volume Group vg2 using +/storage/vg2_info in host2. + + + [root@host1 ~]# gluster volume start bdvol + +Client side: + + [root@node ~]# mount -t glusterfs host1:/bdvol /media + [root@node ~]# touch /media/posix + +It creates regular posix file 'posix' in either host1:/vg1 or host2:/vg2 brick + + [root@node ~]# mkdir /media/image + + [root@node ~]# touch /media/image/lv1 + + +It also creates regular posix file 'lv1' in either host1:/vg1 or +host2:/vg2 brick + + [root@node ~]# setfattr -n "user.glusterfs.bd" -v "lv" /media/image/lv1 + + [root@node ~]# + + +Above setxattr results in creating a new LV in corresponding brick's VG +and it sets 'user.glusterfs.bd' with value 'lv:<default-extent-size'' + + + [root@node ~]# truncate -s5G /media/image/lv1 + + +It results in resizig LV 'lv1'to 5G + +New BD xlator code is placed in `xlators/storage/bd` directory. + +Also add volume-uuid to the VG so that same VG cannot be used for other +bricks/volumes. After deleting a gluster volume, one has to manually +remove the associated tag using vgchange <vg-name> --deltag +`<trusted.glusterfs.volume-id:<volume-id>>` + + +#### Exposing volume capabilities + +With multiple storage translators (posix and bd) being supported in GlusterFS, it becomes +necessary to know the volume type so that user can issue appropriate calls that are relevant +only to the a given volume type. Hence there needs to be a way to expose the type of +the storage translator of the volume to the user. + +BD xlator is capable of providing server offloaded file copy, server/storage offloaded +zeroing of a file etc. This capabilities should be visible to the client/user, so that these +features can be exploited. + +BD xlator exports capability information through gluster volume info (and --xml) output. For eg: + +`snip of gluster volume info output for a BD based volume` + + Xlator 1: BD + Capability 1: thin + +`snip of gluster volume info --xml output for a BD based volume` + + <xlators> + <xlator> + <name>BD</name> + <capabilities> + <capability>thin</capability> + </capabilities> + </xlator> + </xlators> + +But this capability information should also exposed through some other means so that a host +which is not part of Gluster peer could also avail this capabilities. + +* Type + +BD translator supports both regular files and block device, i,e., one can create files on +GlusterFS volume backed by BD translator and this file could end up as regular posix file or +a logical volume (block device) based on the user''s choice. User can do a setxattr on the +created file to convert it to a logical volume. + +Users of BD backed volume like QEMU would like to know that it is working with BD type of volume +so that it can issue an additional setxattr call after creating a VM image on GlusterFS backend. +This is necessary to ensure that the created VM image is backed by LV instead of file. + +There are different ways to expose this information (BD type of volume) to user. +One way is to export it via a `getxattr` call. That said, When a client issues getxattr("volume_type") +on a root gfid, bd xlator will return 1 implying its BD xlator. But posix xlator will return ENODATA +and client code can interpret this as posix xlator. Also capability list can be returned via +getxattr("caps") for root gfid. + +* Capabilities + +BD xlator supports new features such as server offloaded file copy, thin provisioned VM images etc. + +There is no standard way of exploiting these features from client side (such as syscall +to exploit server offloaded copy). So these features need to be exported to the client so that +they can be used. BD xlator latest version exports these capabilities information through +gluster volume info (and --xml) output. But if a client is not part of GlusterFS peer +it can''t run volume info command to get the list of capabilities of a given GlusterFS volume. +For example, GlusterFS block driver in qemu need to get the capability list so that these features are used. + + + +Parts of this documentation were originally published here +#http://raobharata.wordpress.com/2013/11/27/glusterfs-block-device-translator/ diff --git a/doc/developer-guide/coding-standard.md b/doc/developer-guide/coding-standard.md new file mode 100644 index 00000000000..368c5553464 --- /dev/null +++ b/doc/developer-guide/coding-standard.md @@ -0,0 +1,402 @@ +GlusterFS Coding Standards +========================== + +Structure definitions should have a comment per member +------------------------------------------------------ + +Every member in a structure definition must have a comment about its +purpose. The comment should be descriptive without being overly verbose. + +*Bad:* + +``` +gf_lock_t lock; /* lock */ +``` + +*Good:* + +``` +DBTYPE access_mode; /* access mode for accessing + * the databases, can be + * DB_HASH, DB_BTREE + * (option access-mode <mode>) + */ +``` + +Declare all variables at the beginning of the function +------------------------------------------------------ + +All local variables in a function must be declared immediately after the +opening brace. This makes it easy to keep track of memory that needs to be freed +during exit. It also helps debugging, since gdb cannot handle variables +declared inside loops or other such blocks. + +Always initialize local variables +--------------------------------- + +Every local variable should be initialized to a sensible default value +at the point of its declaration. All pointers should be initialized to NULL, +and all integers should be zero or (if it makes sense) an error value. + + +*Good:* + +``` +int ret = 0; +char *databuf = NULL; +int _fd = -1; +``` + +Initialization should always be done with a constant value +---------------------------------------------------------- + +Never use a non-constant expression as the initialization value for a variable. + + +*Bad:* + +``` +pid_t pid = frame->root->pid; +char *databuf = malloc (1024); +``` + +Validate all arguments to a function +------------------------------------ + +All pointer arguments to a function must be checked for `NULL`. +A macro named `VALIDATE` (in `common-utils.h`) +takes one argument, and if it is `NULL`, writes a log message and +jumps to a label called `err` after setting op_ret and op_errno +appropriately. It is recommended to use this template. + + +*Good:* + +``` +VALIDATE(frame); +VALIDATE(this); +VALIDATE(inode); +``` + +Never rely on precedence of operators +------------------------------------- + +Never write code that relies on the precedence of operators to execute +correctly. Such code can be hard to read and someone else might not +know the precedence of operators as accurately as you do. + +*Bad:* + +``` +if (op_ret == -1 && errno != ENOENT) +``` + +*Good:* + +``` +if ((op_ret == -1) && (errno != ENOENT)) +``` + +Use exactly matching types +-------------------------- + +Use a variable of the exact type declared in the manual to hold the +return value of a function. Do not use an ``equivalent'' type. + + +*Bad:* + +``` +int len = strlen (path); +``` + +*Good:* + +``` +size_t len = strlen (path); +``` + +Never write code such as `foo->bar->baz`; check every pointer +------------------------------------------------------------- + +Do not write code that blindly follows a chain of pointer +references. Any pointer in the chain may be `NULL` and thus +cause a crash. Verify that each pointer is non-null before following +it. + +Check return value of all functions and system calls +---------------------------------------------------- + +The return value of all system calls and API functions must be checked +for success or failure. + +*Bad:* + +``` +close (fd); +``` + +*Good:* + +``` +op_ret = close (_fd); +if (op_ret == -1) { + gf_log (this->name, GF_LOG_ERROR, + "close on file %s failed (%s)", real_path, + strerror (errno)); + op_errno = errno; + goto out; +} +``` + + +Gracefully handle failure of malloc +----------------------------------- + +GlusterFS should never crash or exit due to lack of memory. If a +memory allocation fails, the call should be unwound and an error +returned to the user. + +*Use result args and reserve the return value to indicate success or failure:* + +The return value of every functions must indicate success or failure (unless +it is impossible for the function to fail --- e.g., boolean functions). If +the function needs to return additional data, it must be returned using a +result (pointer) argument. + +*Bad:* + +``` +int32_t dict_get_int32 (dict_t *this, char *key); +``` + +*Good:* + +``` +int dict_get_int32 (dict_t *this, char *key, int32_t *val); +``` + +Always use the `n' versions of string functions +----------------------------------------------- + +Unless impossible, use the length-limited versions of the string functions. + +*Bad:* + +``` +strcpy (entry_path, real_path); +``` + +*Good:* + +``` +strncpy (entry_path, real_path, entry_path_len); +``` + +No dead or commented code +------------------------- + +There must be no dead code (code to which control can never be passed) or +commented out code in the codebase. + +Only one unwind and return per function +--------------------------------------- + +There must be only one exit out of a function. `UNWIND` and return +should happen at only point in the function. + +Function length or Keep functions small +--------------------------------------- + +We live in the UNIX-world where modules do one thing and do it well. +This rule should apply to our functions also. If a function is very long, try splitting it +into many little helper functions. The question is, in a coding +spree, how do we know a function is long and unreadable. One rule of +thumb given by Linus Torvalds is that, a function should be broken-up +if you have 4 or more levels of indentation going on for more than 3-4 +lines. + +*Example for a helper function:* +``` +static int +same_owner (posix_lock_t *l1, posix_lock_t *l2) +{ + return ((l1->client_pid == l2->client_pid) && + (l1->transport == l2->transport)); +} +``` + +Defining functions as static +---------------------------- + +Define internal functions as static only if you're +very sure that there will not be a crash(..of any kind..) emanating in +that function. If there is even a remote possibility, perhaps due to +pointer derefering, etc, declare the function as non-static. This +ensures that when a crash does happen, the function name shows up the +in the back-trace generated by libc. However, doing so has potential +for polluting the function namespace, so to avoid conflicts with other +components in other parts, ensure that the function names are +prepended with a prefix that identify the component to which it +belongs. For eg. non-static functions in io-threads translator start +with iot_. + +Ensure function calls wrap around after 80-columns +-------------------------------------------------- + +Place remaining arguments on the next line if needed. + +Functions arguments and function definition +------------------------------------------- + +Place all the arguments of a function definition on the same line +until the line goes beyond 80-cols. Arguments that extend beyind +80-cols should be placed on the next line. + +Style issues +------------ + +### Brace placement + +Use K&R/Linux style of brace placement for blocks. + +*Good:* + +``` +int some_function (...) +{ + if (...) { + /* ... */ + } else if (...) { + /* ... */ + } else { + /* ... */ + } + + do { + /* ... */ + } while (cond); +} +``` + +### Indentation + +Use *eight* spaces for indenting blocks. Ensure that your +file contains only spaces and not tab characters. You can do this +in Emacs by selecting the entire file (`C-x h`) and +running `M-x untabify`. + +To make Emacs indent lines automatically by eight spaces, add this +line to your `.emacs`: + +``` +(add-hook 'c-mode-hook (lambda () (c-set-style "linux"))) +``` + +### Comments + +Write a comment before every function describing its purpose (one-line), +its arguments, and its return value. Mention whether it is an internal +function or an exported function. + +Write a comment before every structure describing its purpose, and +write comments about each of its members. + +Follow the style shown below for comments, since such comments +can then be automatically extracted by doxygen to generate +documentation. + +*Good:* + +``` +/** +* hash_name -hash function for filenames +* @par: parent inode number +* @name: basename of inode +* @mod: number of buckets in the hashtable +* +* @return: success: bucket number +* failure: -1 +* +* Not for external use. +*/ +``` + +### Indicating critical sections + +To clearly show regions of code which execute with locks held, use +the following format: + +``` +pthread_mutex_lock (&mutex); +{ + /* code */ +} +pthread_mutex_unlock (&mutex); +``` + +*A skeleton fop function:* + +This is the recommended template for any fop. In the beginning come +the initializations. After that, the `success' control flow should be +linear. Any error conditions should cause a `goto` to a single +point, `out`. At that point, the code should detect the error +that has occurred and do appropriate cleanup. + +``` +int32_t +sample_fop (call_frame_t *frame, xlator_t *this, ...) +{ + char * var1 = NULL; + int32_t op_ret = -1; + int32_t op_errno = 0; + DIR * dir = NULL; + struct posix_fd * pfd = NULL; + + VALIDATE_OR_GOTO (frame, out); + VALIDATE_OR_GOTO (this, out); + + /* other validations */ + + dir = opendir (...); + + if (dir == NULL) { + op_errno = errno; + gf_log (this->name, GF_LOG_ERROR, + "opendir failed on %s (%s)", loc->path, + strerror (op_errno)); + goto out; + } + + /* another system call */ + if (...) { + op_errno = ENOMEM; + gf_log (this->name, GF_LOG_ERROR, + "out of memory :("); + goto out; + } + + /* ... */ + + out: + if (op_ret == -1) { + + /* check for all the cleanup that needs to be + done */ + + if (dir) { + closedir (dir); + dir = NULL; + } + + if (pfd) { + FREE (pfd->path); + FREE (pfd); + pfd = NULL; + } + } + + STACK_UNWIND (frame, op_ret, op_errno, fd); + return 0; +} +``` diff --git a/doc/developer-guide/coding-standard.pdf b/doc/developer-guide/coding-standard.pdf Binary files differnew file mode 100644 index 00000000000..bc9cb56202c --- /dev/null +++ b/doc/developer-guide/coding-standard.pdf diff --git a/doc/developer-guide/coding-standard.tex b/doc/developer-guide/coding-standard.tex new file mode 100644 index 00000000000..80c11041e60 --- /dev/null +++ b/doc/developer-guide/coding-standard.tex @@ -0,0 +1,384 @@ +\documentclass{article}[12pt] +\usepackage{color} + +\begin{document} + + +\hrule +\begin{center}\textbf{\Large{GlusterFS Coding Standards}}\end{center} +\begin{center}\textbf{\large{\textcolor{red}{Z} Research}}\end{center} +\begin{center}{July 14, 2008}\end{center} +\hrule + +\vspace{8ex} + +\section*{$\bullet$ Structure definitions should have a comment per member} + +Every member in a structure definition must have a comment about its +purpose. The comment should be descriptive without being overly verbose. + +\vspace{2ex} +\textsl{Bad}: + +\begin{verbatim} + gf_lock_t lock; /* lock */ +\end{verbatim} + +\textsl{Good}: + +\begin{verbatim} + DBTYPE access_mode; /* access mode for accessing + * the databases, can be + * DB_HASH, DB_BTREE + * (option access-mode <mode>) + */ +\end{verbatim} + +\section*{$\bullet$ Declare all variables at the beginning of the function} +All local variables in a function must be declared immediately after the +opening brace. This makes it easy to keep track of memory that needs to be freed +during exit. It also helps debugging, since gdb cannot handle variables +declared inside loops or other such blocks. + +\section*{$\bullet$ Always initialize local variables} +Every local variable should be initialized to a sensible default value +at the point of its declaration. All pointers should be initialized to NULL, +and all integers should be zero or (if it makes sense) an error value. + +\vspace{2ex} + +\textsl{Good}: + +\begin{verbatim} + int ret = 0; + char *databuf = NULL; + int _fd = -1; +\end{verbatim} + +\section*{$\bullet$ Initialization should always be done with a constant value} +Never use a non-constant expression as the initialization value for a variable. + +\vspace{2ex} + +\textsl{Bad}: + +\begin{verbatim} + pid_t pid = frame->root->pid; + char *databuf = malloc (1024); +\end{verbatim} + +\section*{$\bullet$ Validate all arguments to a function} +All pointer arguments to a function must be checked for \texttt{NULL}. +A macro named \texttt{VALIDATE} (in \texttt{common-utils.h}) +takes one argument, and if it is \texttt{NULL}, writes a log message and +jumps to a label called \texttt{err} after setting op\_ret and op\_errno +appropriately. It is recommended to use this template. + +\vspace{2ex} + +\textsl{Good}: + +\begin{verbatim} + VALIDATE(frame); + VALIDATE(this); + VALIDATE(inode); +\end{verbatim} + +\section*{$\bullet$ Never rely on precedence of operators} +Never write code that relies on the precedence of operators to execute +correctly. Such code can be hard to read and someone else might not +know the precedence of operators as accurately as you do. +\vspace{2ex} + +\textsl{Bad}: + +\begin{verbatim} + if (op_ret == -1 && errno != ENOENT) +\end{verbatim} + +\textsl{Good}: + +\begin{verbatim} + if ((op_ret == -1) && (errno != ENOENT)) +\end{verbatim} + +\section*{$\bullet$ Use exactly matching types} +Use a variable of the exact type declared in the manual to hold the +return value of a function. Do not use an ``equivalent'' type. + +\vspace{2ex} + +\textsl{Bad}: + +\begin{verbatim} + int len = strlen (path); +\end{verbatim} + +\textsl{Good}: + +\begin{verbatim} + size_t len = strlen (path); +\end{verbatim} + +\section*{$\bullet$ Never write code such as \texttt{foo->bar->baz}; check every pointer} +Do not write code that blindly follows a chain of pointer +references. Any pointer in the chain may be \texttt{NULL} and thus +cause a crash. Verify that each pointer is non-null before following +it. + +\section*{$\bullet$ Check return value of all functions and system calls} +The return value of all system calls and API functions must be checked +for success or failure. + +\vspace{2ex} +\textsl{Bad}: + +\begin{verbatim} + close (fd); +\end{verbatim} + +\textsl{Good}: + +\begin{verbatim} + op_ret = close (_fd); + if (op_ret == -1) { + gf_log (this->name, GF_LOG_ERROR, + "close on file %s failed (%s)", real_path, + strerror (errno)); + op_errno = errno; + goto out; + } +\end{verbatim} + + +\section*{$\bullet$ Gracefully handle failure of malloc} +GlusterFS should never crash or exit due to lack of memory. If a +memory allocation fails, the call should be unwound and an error +returned to the user. + +\section*{$\bullet$ Use result args and reserve the return value to indicate success or failure} +The return value of every functions must indicate success or failure (unless +it is impossible for the function to fail --- e.g., boolean functions). If +the function needs to return additional data, it must be returned using a +result (pointer) argument. + +\vspace{2ex} +\textsl{Bad}: + +\begin{verbatim} + int32_t dict_get_int32 (dict_t *this, char *key); +\end{verbatim} + +\textsl{Good}: + +\begin{verbatim} + int dict_get_int32 (dict_t *this, char *key, int32_t *val); +\end{verbatim} + +\section*{$\bullet$ Always use the `n' versions of string functions} +Unless impossible, use the length-limited versions of the string functions. + +\vspace{2ex} +\textsl{Bad}: + +\begin{verbatim} + strcpy (entry_path, real_path); +\end{verbatim} + +\textsl{Good}: + +\begin{verbatim} + strncpy (entry_path, real_path, entry_path_len); +\end{verbatim} + +\section*{$\bullet$ No dead or commented code} +There must be no dead code (code to which control can never be passed) or +commented out code in the codebase. + +\section*{$\bullet$ Only one unwind and return per function} +There must be only one exit out of a function. \texttt{UNWIND} and return +should happen at only point in the function. + +\section*{$\bullet$ Function length or Keep functions small} +We live in the UNIX-world where modules do one thing and do it well. +This rule should apply to our functions also. If a function is very long, try splitting it +into many little helper functions. The question is, in a coding +spree, how do we know a function is long and unreadable. One rule of +thumb given by Linus Torvalds is that, a function should be broken-up +if you have 4 or more levels of indentation going on for more than 3-4 +lines. + +\vspace{2ex} +\textsl{Example for a helper function}: +\begin{verbatim} + static int + same_owner (posix_lock_t *l1, posix_lock_t *l2) + { + return ((l1->client_pid == l2->client_pid) && + (l1->transport == l2->transport)); + } +\end{verbatim} + +\section*{$\bullet$ Defining functions as static} +Define internal functions as static only if you're +very sure that there will not be a crash(..of any kind..) emanating in +that function. If there is even a remote possibility, perhaps due to +pointer derefering, etc, declare the function as non-static. This +ensures that when a crash does happen, the function name shows up the +in the back-trace generated by libc. However, doing so has potential +for polluting the function namespace, so to avoid conflicts with other +components in other parts, ensure that the function names are +prepended with a prefix that identify the component to which it +belongs. For eg. non-static functions in io-threads translator start +with iot\_. + +\section*{$\bullet$ Ensure function calls wrap around after +80-columns.} +Place remaining arguments on the next line if needed. + +\section*{$\bullet$ Functions arguments and function definition} +Place all the arguments of a function definition on the same line +until the line goes beyond 80-cols. Arguments that extend beyind +80-cols should be placed on the next line. + +\section*{Style issues} + +\subsection*{Brace placement} +Use K\&R/Linux style of brace placement for blocks. + +\textsl{Example}: +\begin{verbatim} + int some_function (...) + { + if (...) { + /* ... */ + } else if (...) { + /* ... */ + } else { + /* ... */ + } + + do { + /* ... */ + } while (cond); + } +\end{verbatim} + +\subsection*{Indentation} +Use \textbf{eight} spaces for indenting blocks. Ensure that your +file contains only spaces and not tab characters. You can do this +in Emacs by selecting the entire file (\texttt{C-x h}) and +running \texttt{M-x untabify}. + +To make Emacs indent lines automatically by eight spaces, add this +line to your \texttt{.emacs}: + +\begin{verbatim} + (add-hook 'c-mode-hook (lambda () (c-set-style "linux"))) +\end{verbatim} + +\subsection*{Comments} +Write a comment before every function describing its purpose (one-line), +its arguments, and its return value. Mention whether it is an internal +function or an exported function. + +Write a comment before every structure describing its purpose, and +write comments about each of its members. + +Follow the style shown below for comments, since such comments +can then be automatically extracted by doxygen to generate +documentation. + +\textsl{Example}: +\begin{verbatim} +/** + * hash_name -hash function for filenames + * @par: parent inode number + * @name: basename of inode + * @mod: number of buckets in the hashtable + * + * @return: success: bucket number + * failure: -1 + * + * Not for external use. + */ +\end{verbatim} + +\subsection*{Indicating critical sections} +To clearly show regions of code which execute with locks held, use +the following format: + +\begin{verbatim} + pthread_mutex_lock (&mutex); + { + /* code */ + } + pthread_mutex_unlock (&mutex); +\end{verbatim} + +\section*{A skeleton fop function} +This is the recommended template for any fop. In the beginning come +the initializations. After that, the `success' control flow should be +linear. Any error conditions should cause a \texttt{goto} to a single +point, \texttt{out}. At that point, the code should detect the error +that has occurred and do appropriate cleanup. + +\begin{verbatim} +int32_t +sample_fop (call_frame_t *frame, xlator_t *this, ...) +{ + char * var1 = NULL; + int32_t op_ret = -1; + int32_t op_errno = 0; + DIR * dir = NULL; + struct posix_fd * pfd = NULL; + + VALIDATE_OR_GOTO (frame, out); + VALIDATE_OR_GOTO (this, out); + + /* other validations */ + + dir = opendir (...); + + if (dir == NULL) { + op_errno = errno; + gf_log (this->name, GF_LOG_ERROR, + "opendir failed on %s (%s)", loc->path, + strerror (op_errno)); + goto out; + } + + /* another system call */ + if (...) { + op_errno = ENOMEM; + gf_log (this->name, GF_LOG_ERROR, + "out of memory :("); + goto out; + } + + /* ... */ + + out: + if (op_ret == -1) { + + /* check for all the cleanup that needs to be + done */ + + if (dir) { + closedir (dir); + dir = NULL; + } + + if (pfd) { + FREE (pfd->path); + FREE (pfd); + pfd = NULL; + } + } + + STACK_UNWIND (frame, op_ret, op_errno, fd); + return 0; +} +\end{verbatim} + +\end{document} diff --git a/doc/developer-guide/daemon-management-framework.md b/doc/developer-guide/daemon-management-framework.md new file mode 100644 index 00000000000..592192e665d --- /dev/null +++ b/doc/developer-guide/daemon-management-framework.md @@ -0,0 +1,38 @@ + +How to introduce new daemons using daemon management framework +============================================================== +Glusterd manages GlusterFS daemons providing services like NFS, Proactive +self-heal, Quota, User servicable snapshots etc. Following are some of the +aspects that come under daemon management. + +Data members & functions of different management objects + +- **Connection Management** + - unix domain sockets based channel for internal communication + - rpc connection for the communication + - frame timeout value for UDS + - Methods - notify + - init, connect, termination, disconnect APIs can be invoked using the + connection management object + +- **Process Management** + - Name of the process + - pidfile to detect if the daemon is running + - loggging directory, log file, volfile, volfileserver & volfileid + - init, stop APIs can be invoked using the process management object + +- **Service Management** + - connection object + - process object + - online status + - Methods - manager, start, stop which can be abstracted as a common methods + or specific to service requirements + - init API can be invoked using the service management object + + The above structures defines the skeleton of the daemon management framework. + Introduction of new daemons in GlusterFS needs to inherit these properties. Any + requirement specific to a daemon needs to be implemented in its own service + (for eg : snapd defines its own type glusterd_snapdsvc_t using glusterd_svc_t + and snapd specific data). New daemons will need to have its own service specific + code written in glusterd-<feature>-svc.h{c} and need to reuse the existing + framework. diff --git a/doc/developer-guide/data-structures/inode.md b/doc/developer-guide/data-structures/inode.md new file mode 100644 index 00000000000..a340ab9ca8e --- /dev/null +++ b/doc/developer-guide/data-structures/inode.md @@ -0,0 +1,226 @@ +#Inode and dentry management in GlusterFS: + +##Background +Filesystems internally refer to files and directories via inodes. Inodes +are unique identifiers of the entities stored in a filesystem. Whenever an +application has to operate on a file/directory (read/modify), the filesystem +maps that file/directory to the right inode and start referring to that inode +whenever an operation has to be performed on the file/directory. + +In GlusterFS a new inode gets created whenever a new file/directory is created +OR when a successful lookup is done on a file/directory for the first time. +Inodes in GlusterFS are maintained by the inode table which gets initiated when +the filesystem daemon is started (both for the brick process as well as the +mount process). Below are some important data structures for inode management. + +## Data-structure (inode-table) +``` +struct _inode_table { + pthread_mutex_t lock; + size_t hashsize; /* bucket size of inode hash and dentry hash */ + char *name; /* name of the inode table, just for gf_log() */ + inode_t *root; /* root directory inode, with inode + number and gfid 1 */ + xlator_t *xl; /* xlator to be called to do purge and + the xlator which maintains the inode table*/ + uint32_t lru_limit; /* maximum LRU cache size */ + struct list_head *inode_hash; /* buckets for inode hash table */ + struct list_head *name_hash; /* buckets for dentry hash table */ + struct list_head active; /* list of inodes currently active (in an fop) */ + uint32_t active_size; /* count of inodes in active list */ + struct list_head lru; /* list of inodes recently used. + lru.next most recent */ + uint32_t lru_size; /* count of inodes in lru list */ + struct list_head purge; /* list of inodes to be purged soon */ + uint32_t purge_size; /* count of inodes in purge list */ + + struct mem_pool *inode_pool; /* memory pool for inodes */ + struct mem_pool *dentry_pool; /* memory pool for dentrys */ + struct mem_pool *fd_mem_pool; /* memory pool for fd_t */ + int ctxcount; /* number of slots in inode->ctx */ +}; +``` + +#Life-cycle +``` + +inode_table_new (size_t lru_limit, xlator_t *xl) + +This is a function which allocates a new inode table. Usually the top xlators in +the graph such as protocol/server (for bricks), fuse and nfs (for fuse and nfs +mounts) and libgfapi do inode managements. Hence they are the ones which will +allocate a new inode table by calling the above function. + +Each xlator graph in glusterfs maintains an inode table. So in fuse clients, +whenever there is a graph change due to add brick/remove brick or +addition/removal of some other xlators, a new graph is created which creates a +new inode table. + +Thus an allocated inode table is destroyed only when the filesystem daemon is +killed or unmounted. + +``` + +#what it contains. +``` + +Inode table in glusterfs mainly contains a hash table for maintaining inodes. +In general a file/directory is considered to be existing if there is a +corresponding inode present in the inode table. If a inode for a file/directory +cannot be found in the inode table, glusterfs tries to resolve it by sending a +lookup on the entry for which the inode is needed. If lookup is successful, then +a new inode correponding to the entry is added to the hash table present in the +inode table. Thus an inode present in the hash-table means, its an existing +file/directory within the filesystem. The inode table also contains the hash +size of the hash table (as of now it is hard coded to 14057. The hash value of +a inode is calculated using its gfid). + +Apart from the hash table, inode table also maintains 3 important list of inodes +1) Active list: +Active list contains all the active inodes (i.e inodes which are currently part +of some fop). +2) Lru list: +Least recently used inodes list. A limit can be set for the size of the lru +list. For bricks it is 16384 and for clients it is infinity. +3) Purge list: +List of all the inodes which have to be purged (i.e inodes which have to be +deleted from the inode table due to unlink/rmdir/forget). + +And at last it also contains the mem-pool for allocating inodes, dentries so +that frequent malloc/calloc and free of the data structures can be avoided. +``` + +#Data structure (inode) +``` +struct _inode { + inode_table_t *table; /* the table this inode belongs to */ + uuid_t gfid; /* unique identifier of the inode */ + gf_lock_t lock; + uint64_t nlookup; + uint32_t fd_count; /* Open fd count */ + uint32_t ref; /* reference count on this inode */ + ia_type_t ia_type; /* what kind of file */ + struct list_head fd_list; /* list of open files on this inode */ + struct list_head dentry_list; /* list of directory entries for this inode */ + struct list_head hash; /* hash table pointers */ + struct list_head list; /* active/lru/purge */ + + struct _inode_ctx *_ctx; /* place holder for keeping the + information about the inode by different xlators */ +}; + +As said above, inodes are internal way of identifying the files/directories. A +inode uniquely represents a file/directory. A new inode is created whenever a +create/mkdir/symlink/mknod operations are performed. Apart from that a new inode +is created upon the successful fresh lookup of a file/directory. Say the +filesystem contained some file "a" within root and the filesystem was +unmounted. Now when glusterfs is mounted and some operation is perfomed on "/a", +glusterfs tries to get the inode for the entry "a" with parent inode as +root. But, since glusterfs just came up, it will not be able to find the inode +for "a" and will send a lookup on "/a". If the lookup operation succeeds (i.e. +the root of glusterfs contains an entry called "a"), then a new inode for "/a" +is created and added to the inode table. + +Depending upon the situation, an inode can be in one of the 3 lists maintained +by the inode table. If some fop is happening on the inode, then the inode will +be present in the active inodes list maintained by the inode table. Active +inodes are those inodes whose refcount is greater than zero. Whenever some +operation comes on a file/directory, and the resolver tries to find the inode +for it, it increments the refcount of the inode before returning the inode. The +refcount of an inode can be incremented by calling the below function + +inode_ref (inode_t *inode) + +Any xlator which wants to operate on a inode as part of some fop (or wants the +inode in the callback), should hold a ref on the inode. +Once the fop is completed before sending the reply of the fop to the above +layers , the inode has to be unrefed. When the refcount of an inode becomes +zero, it is removed from the active inodes list and put into LRU list maintained +by the inode table. Thus in short if some fop is happening on a file/directory, +the corresponding inode will be in the active list or it will be in the LRU +list. +``` + +#Life Cycle + +A new inode is created whenever a new file/directory/symlink is created OR a +successful lookup of an existing entry is done. The xlators which does inode +management (as of now protocol/server, fuse, nfs, gfapi) will perform inode_link +operation upon successful lookup or successful creation of a new entry. + +inode_link (inode_t *inode, inode_t *parent, const char *name, + struct iatt *buf); + +inode_link actually adds the inode to the inode table (to be precise it adds +the inode to the hash table maintained by the inode table. The hash value is +calculated based on the gfid). Copies the gfid to the inode (the gfid is +present in the iatt structure). Creates a dentry with the new name. + +A inode is removed from the inode table and eventually destroyed when unlink +or rmdir operation is performed on a file/directory, or the the lru limit of +the inode table has been exceeded. + +#Data structure (dentry) +``` + +struct _dentry { + struct list_head inode_list; /* list of dentries of inode */ + struct list_head hash; /* hash table pointers */ + inode_t *inode; /* inode of this directory entry */ + char *name; /* name of the directory entry */ + inode_t *parent; /* directory of the entry */ +}; + +A dentry is the presence of an entry for a file/directory within its parent +directory. A dentry usually points to the inode to which it belongs to. In +glusterfs a dentry contains the following fields. +1) a hook using which it can add itself to the list of +the dentries maintained by the inode to which it points to. +2) A hash table pointer. +3) Pointer to the inode to which it belongs to. +4) Name of the dentry +5) Pointer to the inode of the parent directory in which the dentry is present + +A new dentry is created when a new file/directory/symlink is created or a hard +link to an existing file is created. + +__dentry_create (inode_t *inode, inode_t *parent, const char *name); + +A dentry holds a refcount on the parent +directory so that the parent inode is never removed from the active inode's list +and put to the lru list (If the lru limit of the lru list is exceeded, there is +a chance of parent inode being destroyed. To avoid it, the dentries hold a +reference to the parent inode). A dentry is removed whenevern a unlink/rmdir +is perfomed on a file/directory. Or when the lru limit has been exceeded, the +oldest inodes are purged out of the inode table, during which all the dentries +of the inode are removed. + +Whenever a unlink/rmdir comes on a file/directory, the corresponding inode +should be removed from the inode table. So upon unlink/rmdir, the inode will +be moved to the purge list maintained by the inode table and from there it is +destroyed. To be more specific, if a inode has to be destroyed, its refcount +and nlookup count both should become 0. For refcount to become 0, the inode +should not be part of any fop (there should not be any open fds). Or if the +inode belongs to a directory, then there should not be any fop happening on the +directory and it should not contain any dentries within it. For nlookup count to +become zero, a forget has to be sent on the inode with nlookup count set to 0 as +an argument. For fuse clients, forget is sent by the kernel itself whenever a +unlink/rmdir is performed. But for brick processes, upon unlink/rmdir, the +protocol/server itself has to do inode_forget. Whenever the inode has to be +deleted due to file removal or lru limit being exceeded the inode is retired +(i.e. all the dentries of the inode are deleted and the inode is moved to the +purge list maintained by the inode table), the nlookup count is set to 0 via +inode_forget api. The inode table, then prunes all the inodes from the purge +list by destroying the inode contexts maintained by each xlator. + +unlinking of the dentry is done via inode_unlink; + +void +inode_unlink (inode_t *inode, inode_t *parent, const char *name); + +If the inode has multiple hard links, then the unlink operation performed by +the application results just in the removal of the dentry with the name provided +by the application. For the inode to be removed, all the dentries of the inode +should be unlinked. +``` + diff --git a/doc/developer-guide/data-structures/iobuf.md b/doc/developer-guide/data-structures/iobuf.md new file mode 100644 index 00000000000..5f521f1485f --- /dev/null +++ b/doc/developer-guide/data-structures/iobuf.md @@ -0,0 +1,259 @@ +#Iobuf-pool +##Datastructures +###iobuf +Short for IO Buffer. It is one allocatable unit for the consumers of the IOBUF +API, each unit hosts @page_size(defined in arena structure) bytes of memory. As +initial step of processing a fop, the IO buffer passed onto GlusterFS by the +other applications (FUSE VFS/ Applications using gfapi) is copied into GlusterFS +space i.e. iobufs. Hence Iobufs are mostly allocated/deallocated in Fuse, gfapi, +protocol xlators, and also in performance xlators to cache the IO buffers etc. +``` +struct iobuf { + union { + struct list_head list; + struct { + struct iobuf *next; + struct iobuf *prev; + }; + }; + struct iobuf_arena *iobuf_arena; + + gf_lock_t lock; /* for ->ptr and ->ref */ + int ref; /* 0 == passive, >0 == active */ + + void *ptr; /* usable memory region by the consumer */ + + void *free_ptr; /* in case of stdalloc, this is the + one to be freed not the *ptr */ +}; +``` + +###iobref +There may be need of multiple iobufs for a single fop, like in vectored read/write. +Hence multiple iobufs(default 16) are encapsulated under one iobref. +``` +struct iobref { + gf_lock_t lock; + int ref; + struct iobuf **iobrefs; /* list of iobufs */ + int alloced; /* 16 by default, grows as required */ + int used; /* number of iobufs added to this iobref */ +}; +``` +###iobuf_arenas +One region of memory MMAPed from the operating system. Each region MMAPs +@arena_size bytes of memory, and hosts @arena_size / @page_size IOBUFs. +The same sized iobufs are grouped into one arena, for sanity of access. + +``` +struct iobuf_arena { + union { + struct list_head list; + struct { + struct iobuf_arena *next; + struct iobuf_arena *prev; + }; + }; + + size_t page_size; /* size of all iobufs in this arena */ + size_t arena_size; /* this is equal to + (iobuf_pool->arena_size / page_size) + * page_size */ + size_t page_count; + + struct iobuf_pool *iobuf_pool; + + void *mem_base; + struct iobuf *iobufs; /* allocated iobufs list */ + + int active_cnt; + struct iobuf active; /* head node iobuf + (unused by itself) */ + int passive_cnt; + struct iobuf passive; /* head node iobuf + (unused by itself) */ + uint64_t alloc_cnt; /* total allocs in this pool */ + int max_active; /* max active buffers at a given time */ +}; + +``` +###iobuf_pool +Pool of Iobufs. As there may be many Io buffers required by the filesystem, +a pool of iobufs are preallocated and kept, if these preallocated ones are +exhausted only then the standard malloc/free is called, thus improving the +performance. Iobuf pool is generally one per process, allocated during +glusterfs_ctx_t init (glusterfs_ctx_defaults_init), currently the preallocated +iobuf pool memory is freed on process exit. Iobuf pool is globally accessible +across GlusterFs, hence iobufs allocated by any xlator can be accessed by any +other xlators(unless iobuf is not passed). +``` +struct iobuf_pool { + pthread_mutex_t mutex; + size_t arena_size; /* size of memory region in + arena */ + size_t default_page_size; /* default size of iobuf */ + + int arena_cnt; + struct list_head arenas[GF_VARIABLE_IOBUF_COUNT]; + /* array of arenas. Each element of the array is a list of arenas + holding iobufs of particular page_size */ + + struct list_head filled[GF_VARIABLE_IOBUF_COUNT]; + /* array of arenas without free iobufs */ + + struct list_head purge[GF_VARIABLE_IOBUF_COUNT]; + /* array of of arenas which can be purged */ + + uint64_t request_misses; /* mostly the requests for higher + value of iobufs */ +}; +``` +~~~ +The default size of the iobuf_pool(as of yet): +1024 iobufs of 128Bytes = 128KB +512 iobufs of 512Bytes = 256KB +512 iobufs of 2KB = 1MB +128 iobufs of 8KB = 1MB +64 iobufs of 32KB = 2MB +32 iobufs of 128KB = 4MB +8 iobufs of 256KB = 2MB +2 iobufs of 1MB = 2MB +Total ~13MB +~~~ +As seen in the datastructure iobuf_pool has 3 arena lists. + +- arenas: +The arenas allocated during iobuf_pool create, are part of this list. This list +also contains arenas that are partially filled i.e. contain few active and few +passive iobufs (passive_cnt !=0, active_cnt!=0 except for initially allocated +arenas). There will be by default 8 arenas of the sizes mentioned above. +- filled: +If all the iobufs in the arena are filled(passive_cnt = 0), the arena is moved +to the filled list. If any of the iobufs from the filled arena is iobuf_put, +then the arena moves back to the 'arenas' list. +- purge: +If there are no active iobufs in the arena(active_cnt = 0), the arena is moved +to purge list. iobuf_put() triggers destruction of the arenas in this list. The +arenas in the purge list are destroyed only if there is atleast one arena in +'arenas' list, that way there won't be spurious mmap/unmap of buffers. +(e.g: If there is an arena (page_size=128KB, count=32) in purge list, this arena +is destroyed(munmap) only if there is an arena in 'arenas' list with page_size=128KB). + +##APIs +###iobuf_get + +``` +struct iobuf *iobuf_get (struct iobuf_pool *iobuf_pool); +``` +Creates a new iobuf of the default page size(128KB hard coded as of yet). +Also takes a reference(increments ref count), hence no need of doing it +explicitly after getting iobuf. + +###iobuf_get2 + +``` +struct iobuf * iobuf_get2 (struct iobuf_pool *iobuf_pool, size_t page_size); +``` +Creates a new iobuf of a specified page size, if page_size=0 default page size +is considered. +``` +if (requested iobuf size > Max iobuf size in the pool(1MB as of yet)) + { + Perform standard allocation(CALLOC) of the requested size and + add it to the list iobuf_pool->arenas[IOBUF_ARENA_MAX_INDEX]. + } + else + { + -Round the page size to match the stndard sizes in iobuf pool. + (eg: if 3KB is requested, it is rounded to 8KB). + -Select the arena list corresponding to the rounded size + (eg: select 8KB arena) + If the selected arena has passive count > 0, then return the + iobuf from this arena, set the counters(passive/active/etc.) + appropriately. + else the arena is full, allocate new arena with rounded size + and standard page numbers and add to the arena list + (eg: 128 iobufs of 8KB is allocated). + } +``` +Also takes a reference(increments ref count), hence no need of doing it +explicitly after getting iobuf. + +###iobuf_ref + +``` +struct iobuf *iobuf_ref (struct iobuf *iobuf); +``` + Take a reference on the iobuf. If using an iobuf allocated by some other +xlator/function/, its a good practice to take a reference so that iobuf is not +deleted by the allocator. + +###iobuf_unref +``` +void iobuf_unref (struct iobuf *iobuf); +``` +Unreference the iobuf, if the ref count is zero iobuf is considered free. + +``` + -Delete the iobuf, if allocated from standard alloc and return. + -set the active/passive count appropriately. + -if passive count > 0 then add the arena to 'arena' list. + -if active count = 0 then add the arena to 'purge' list. +``` +Every iobuf_ref should have a corresponding iobuf_unref, and also every +iobuf_get/2 should have a correspondning iobuf_unref. + +###iobref_new +``` +struct iobref *iobref_new (); +``` +Creates a new iobref structure and returns its pointer. + +###iobref_ref +``` +struct iobref *iobref_ref (struct iobref *iobref); +``` +Take a reference on the iobref. + +###iobref_unref +``` +void iobref_unref (struct iobref *iobref); +``` +Decrements the reference count of the iobref. If the ref count is 0, then unref +all the iobufs(iobuf_unref) in the iobref, and destroy the iobref. + +###iobref_add +``` +int iobref_add (struct iobref *iobref, struct iobuf *iobuf); +``` +Adds the given iobuf into the iobref, it takes a ref on the iobuf before adding +it, hence explicit iobuf_ref is not required if adding to the iobref. + +###iobref_merge +``` +int iobref_merge (struct iobref *to, struct iobref *from); +``` +Adds all the iobufs in the 'from' iobref to the 'to' iobref. Merge will not +cause the delete of the 'from' iobref, therefore it will result in another ref +on all the iobufs added to the 'to' iobref. Hence iobref_unref should be +performed both on 'from' and 'to' iobrefs (performing iobref_unref only on 'to' +will not free the iobufs and may result in leak). + +###iobref_clear +``` +void iobref_clear (struct iobref *iobref); +``` +Unreference all the iobufs in the iobref, and also unref the iobref. + +##Iobuf Leaks +If all iobuf_refs/iobuf_new do not have correspondning iobuf_unref, then the +iobufs are not freed and recurring execution of such code path may lead to huge +memory leaks. The easiest way to identify if a memory leak is caused by iobufs +is to take a statedump. If the statedump shows a lot of filled arenas then it is +a sure sign of leak. Refer doc/debugging/statedump.md for more details. + +If iobufs are leaking, the next step is to find where the iobuf_unref went +missing. There is no standard/easy way of debugging this, code reading and logs +are the only ways. If there is a liberty to reproduce the memory leak at will, +then logs(gf_callinginfo) in iobuf_ref/unref might help. +TODO: A easier way to debug iobuf leaks. diff --git a/doc/developer-guide/data-structures/mem-pool.md b/doc/developer-guide/data-structures/mem-pool.md new file mode 100644 index 00000000000..c71aa2a8ddd --- /dev/null +++ b/doc/developer-guide/data-structures/mem-pool.md @@ -0,0 +1,124 @@ +#Mem-pool +##Background +There was a time when every fop in glusterfs used to incur cost of allocations/de-allocations for every stack wind/unwind between xlators because stack/frame/*_localt_t in every wind/unwind was allocated and de-allocated. Because of all these system calls in the fop path there was lot of latency and the worst part is that most of the times the number of frames/stacks active at any time wouldn't cross a threshold. So it was decided that this threshold number of frames/stacks would be allocated in the beginning of the process only once. Get one of them from the pool of stacks/frames whenever `STACK_WIND` is performed and put it back into the pool in `STACK_UNWIND`/`STACK_DESTROY` without incurring any extra system calls. The data structures are allocated only when threshold number of such items are in active use i.e. pool is in complete use.% increase in the performance once this was added to all the common data structures (inode/fd/dict etc) in xlators throughout the stack was tremendous. + +## Data structure +``` +struct mem_pool { + struct list_head list; /*Each member in the mempool is element padded with a doubly-linked-list + ptr of mempool + is-in +-use info. This list is used to add the element to the list of free members in the mem-pool*/ + int hot_count;/*number of mempool elements that are in active use*/ + int cold_count;/*number of mempool elements that are not in use. If a new allocation is required it +will be served from here until all the elements in the pool are in use i.e. cold-count becomes 0.*/ + gf_lock_t lock;/*synchronization mechanism*/ + unsigned long padded_sizeof_type;/*Each mempool element is padded with a doubly-linked-list + ptr of mempool + is-in +-use info to operate the pool of elements, this size is the element-size after padding*/ + void *pool;/*Starting address of pool*/ + void *pool_end;/*Ending address of pool*/ +/* If an element address is in the range between pool, pool_end addresses then it is alloced from the pool otherwise it is 'calloced' this is very useful for functions like 'mem_put'*/ + int real_sizeof_type;/* size of just the element without any padding*/ + uint64_t alloc_count; /*Number of times this type of data is allocated through out the life of this process. This may include calloced elements as well*/ + uint64_t pool_misses; /*Number of times the element had to be allocated from heap because all elements from the pool are in active use.*/ + int max_alloc; /*Maximum number of elements from the pool in active use at any point in the life of the process. This does *not* include calloced elements*/ + int curr_stdalloc;/*Number of elements that are allocated from heap at the moment because the pool is in completed use. It should be '0' when pool is not in complete use*/ + int max_stdalloc;/*Maximum number of allocations from heap after the pool is completely used that are in active use at any point in the life of the process.*/ + char *name; /*Contains xlator-name:data-type as a string + struct list_head global_list;/*This is used to insert it into the global_list of mempools maintained in 'glusterfs-ctx' +}; +``` + +##Life-cycle +``` +mem_pool_new (data_type, unsigned long count) + +This is a macro which expands to mem_pool_new_fn (sizeof (data_type), count, string-rep-of-data_type) + +struct mem_pool * +mem_pool_new_fn (unsigned long sizeof_type, unsigned long count, char *name) + +Padded-element: + ---------------------------------------- +|list-ptr|mem-pool-address|in-use|Element| + ---------------------------------------- + ``` + +This function allocates the `mem-pool` structure and sets up the pool for use. +`name` parameter above is the `string` containing type of the datatype. This `name` is appended to `xlator-name + ':'` so that it can be easily identified in things like statedump. `count` is the number of elements that need to be allocated. `sizeof_type` is the size of each element. Ideally `('sizeof_type'*'count')` should be the size of the total pool. But to manage the pool using `mem_get`/`mem_put` (will be explained after this section) each element needs to be padded in the front with a `('list', 'mem-pool-address', 'in_use')`. So the actual size of the pool it allocates will be `('padded_sizeof_type'*'count')`. Why these extra elements are needed will be evident after understanding how `mem_get` and `mem_put` are implemented. In this function it just initializes all the `list` structures in front of each element and adds them to the `mem_pool->list` which represent the list of `cold` elements which can be allocated whenever `mem_get` is called on this mem_pool. It remembers mem_pool's start and end addresses in `mem_pool->pool`, `mem_pool->pool_end` respectively. Initializes `mem_pool->cold_count` to `count` and `mem_pool->hot_count` to `0`. This mem-pool will be added to the list of `global_list` maintained in `glusterfs-ctx` + + +``` +void* mem_get (struct mem_pool *mem_pool) + +Initial-list before mem-get +---------------- +| Pool | +| ----------- | ---------------------------------------- ---------------------------------------- +| | pool-list | |<---> |list-ptr|mem-pool-address|in-use|Element|<--->|list-ptr|mem-pool-address|in-use|Element| +| ----------- | ---------------------------------------- ---------------------------------------- +---------------- + +list after mem-get from the pool +---------------- +| Pool | +| ----------- | ---------------------------------------- +| | pool-list | |<--->|list-ptr|mem-pool-address|in-use|Element| +| ----------- | ---------------------------------------- +---------------- + +List when the pool is full: + ---------------- +| Pool | extra element that is allocated +| ----------- | ---------------------------------------- +| | pool-list | | |list-ptr|mem-pool-address|in-use|Element| +| ----------- | ---------------------------------------- + ---------------- +``` + +This function is similar to `malloc()` but it gives memory of type `element` of this pool. When this function is called it increments `mem_pool->alloc_count`, checks if there are any free elements in the pool that can be returned by inspecting `mem_pool->cold_count`. If `mem_pool->cold_count` is non-zero then it means there are elements in the pool which are not in active use. It deletes one element from the list of free elements and decrements `mem_pool->cold_count` and increments `mem_pool->hot_count` to indicate there is one more element in active use. Updates `mem_pool->max_alloc` accordingly. Sets `element->in_use` in the padded memory to `1`. Sets `element->mem_pool` address to this mem_pool also in the padded memory(It is useful for mem_put). Returns the address of the memory after the padded boundary to the caller of this function. In the cases where all the elements in the pool are in active use it `callocs` the element with padded size and sets mem_pool address in the padded memory. To indicate the pool-miss and give useful accounting information of the pool-usage it increments `mem_pool->pool_misses`, `mem_pool->curr_stdalloc`. Updates `mem_pool->max_stdalloc` accordingly. + +``` +void* mem_get0 (struct mem_pool *mem_pool) +``` +Just like `calloc` is to `malloc`, `mem_get0` is to `mem_get`. It memsets the memory to all '0' before returning the element. + + +``` +void mem_put (void *ptr) + +list before mem-put from the pool + ---------------- +| Pool | +| ----------- | ---------------------------------------- +| | pool-list | |<--->|list-ptr|mem-pool-address|in-use|Element| +| ----------- | ---------------------------------------- + ---------------- + +list after mem-put to the pool + ---------------- +| Pool | +| ----------- | ---------------------------------------- ---------------------------------------- +| | pool-list | |<---> |list-ptr|mem-pool-address|in-use|Element|<--->|list-ptr|mem-pool-address|in-use|Element| +| ----------- | ---------------------------------------- ---------------------------------------- + ---------------- + +If mem_put is putting an element not from pool then it is just freed so +no change to the pool + ---------------- +| Pool | +| ----------- | +| | pool-list | | +| ----------- | + ---------------- +``` + +This function is similar to `free()`. Remember that ptr passed to this function is the address of the element, so this function gets the ptr to its head of the padding in front of it. If this memory falls in bettween `mem_pool->pool`, `mem_pool->pool_end` then the memory is part of the 'pool' memory that is allocated so it does some sanity checks to see if the memory is indeed head of the element by checking if `in_use` is set to `1`. It resets `in_use` to `0`. It gets the mem_pool address stored in the padded region and adds this element to the list of free elements. Decreases `mem_pool->hot_count` increases `mem_pool->cold_count`. In the case where padded-element address does not fall in the range of `mem_pool->pool`, `mem_pool->pool_end` it just frees the element and decreases `mem_pool->curr_stdalloc`. + +``` +void +mem_pool_destroy (struct mem_pool *pool) +``` +Deletes this pool from the `global_list` maintained by `glusterfs-ctx` and frees all the memory allocated in `mem_pool_new`. + + +###How to pick pool-size +This varies from work-load to work-load. Create the mem-pool with some random size and run the work-load. Take the statedump after the work-load is complete. In the statedump if `max_alloc` is always less than `cold_count` may be reduce the size of the pool closer to `max_alloc`. On the otherhand if there are lots of `pool-misses` then increase the `pool_size` by `max_stdalloc` to achieve better 'hit-rate' of the pool. diff --git a/doc/developer-guide/gfapi-symbol-versions/gfapi-symbol-versions.md b/doc/developer-guide/gfapi-symbol-versions/gfapi-symbol-versions.md new file mode 100644 index 00000000000..e4f4fe9f052 --- /dev/null +++ b/doc/developer-guide/gfapi-symbol-versions/gfapi-symbol-versions.md @@ -0,0 +1,270 @@ + +## Symbol Versions and SO_NAMEs + + In general, adding new APIs to a shared library does not require that +symbol versions be used or the the SO_NAME be "bumped." These actions +are usually reserved for when a major change is introduced, e.g. many +APIs change or a signficant change in the functionality occurs. + + Over the normal lifetime of a When a new API is added, the library is +recompiled, consumers of the new API are able to do so, and existing, +legacy consumers of the original API continue as before. If by some +chance an old copy of the library is installed on a system, it's unlikely +that most applications will be affected. New applications that use the +new API will incur a run-time error terminate. + + Bumping the SO_NAME, i.e. changing the shared lib's file name, e.g. +from libfoo.so.0 to libfoo.so.1, which also changes the ELF SO_NAME +attribute inside the file, works a little differently. libfoo.so.0 +contains only the old APIs. libfoo.so.1 contains both the old and new +APIs. Legacy software that was linked with libfoo.so.0 continues to work +as libfoo.so.0 is usually left installed on the system. New software that +uses the new APIs is linked with libfoo.so.1, and works as long as +long as libfoo.so.1 is installed on the system. Accidentally (re)installing +libfoo.so.0 doesn't break new software as long as reinstalling doesn't +erase libfoo.so.1. + + Using symbol versions is somewhere in the middle. The shared library +file remains libfoo.so.0 forever. Legacy APIs may or may not have an +associated symbol version. New APIs may or may not have an associated +symbol version either. In general symbol versions are reserved for APIs +that have changed. Either the function's signature has changed, i.e. the +return time or the number of paramaters, and/or the parameter types have +changed. Another reason for using symbol versions on an API is when the +behaviour or functionality of the API changes dramatically. As with a +library that doesn't use versioned symbols, old and new applications +either find or don't find the versioned symbols they need. If the versioned +symbol doesn't exist in the installed library, the application incurs a +run-time error and terminates. + + GlusterFS wanted to keep tight control over the APIs in libgfapi. +Originally bumping the SO_NAME was considered, and GlusterFS-3.6.0 was +released with libgfapi.so.7. Not only was "7" a mistake (it should have +been "6"), but it was quickly pointed out that many dependent packages +that use libgfapi would be forced to be recompiled/relinked. Thus no +packages of 3.6.0 were ever released and 3.6.1 was quickly released with +libgfapi.so.0, but with symbol versions. There's no strong technical +reason for either; the APIs have not changed, only new APIs have been +added. It's merely being done in anticipation that some APIs might change +sometime in the future. + + Enough about that now, let's get into the nitty gritty—— + +## Adding new APIs + +### Adding a public API. + + This is the default, and the easiest thing to do. Public APIs have +declarations in either glfs.h, glfs-handles.h, or, at your discretion, +in a new header file intended for consumption by other developers. + +Here's what you need to do to add a new public API: + ++ Write the declaration, e.g. in glfs.h: + +```C + int glfs_dtrt (const char *volname, void *stuff) __THROW +``` + ++ Write the definition, e.g. in glfs-dtrt.c: + +```C + int + pub_glfs_dtrt (const char *volname, void *stuff) + { + ... + return 0; + } +``` + ++ Add the symbol version magic for ELF, gnu toolchain to the definition. + + following the definition of your new function in glfs-dtrtops.c, add a + line like this: + +```C + GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_dtrt, 3.7.0) +``` + + The whole thing should look like: + +```C + int + pub_glfs_dtrt (const char *volname, void *stuff) + { + ... + } + GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_dtrt, 3.7.0); +``` + + In this example, 3.7.0 refers to the Version the symbol will first + appear in. There's nothing magic about it, it's just a string token. + The current versions we have are 3.4.0, 3.4.2, 3.5.0, 3.5.1, and 3.6.0. + They are to be considered locked or closed. You can not, must not add + any new APIs and use these versions. Most new APIs will use 3.7.0. If + you add a new API appearing in 3.6.2 (and mainline) then you would use + 3.6.2. + ++ Add the symbol version magic for OS X to the declaration. + + following the declaration in glfs.h, add a line like this: + +```C + GFAPI_PUBLIC(glfs_dtrt, 3.7.0) +``` + + The whole thing should look like: + +```C + int glfs_dtrt (const char *volname, void *stuff) __THROW + GFAPI_PUBLIC(glfs_dtrt, 3.7.0); +``` + + The version here must match the version associated with the definition. + ++ Add the new API to the ELF, gnu toolchain link map file, gfapi.map + + Most new public APIs will probably be added to a new section that + looks like this: + +``` + GFAPI_3.7.0 { + global: + glfs_dtrt; + } GFAPI_PRIVATE_3.7.0; +``` + + if you're adding your new API to, e.g. 3.6.2, it'll look like this: + +``` + GFAPI_3.6.2 { + global: + glfs_dtrt; + } GFAPI_3.6.0; +``` + + and you must change the +``` + GFAPI_PRIVATE_3.7.0 { ...} GFAPI_3.6.0; +``` + section to: +``` + GFAPI_PRIVATE_3.7.0 { ...} GFAPI_3.6.2; +``` + ++ Add the new API to the OS X alias list file, gfapi.aliases. + + Most new APIs will use a line that looks like this: + +```C + _pub_glfs_dtrt _glfs_dtrt$GFAPI_3.7.0 +``` + + if you're adding your new API to, e.g. 3.6.2, it'll look like this: + +```C + _pub_glfs_dtrt _glfs_dtrt$GFAPI_3.6.2 +``` + +And that's it. + + +### Adding a private API. + + If you're thinking about adding a private API that isn't declared in +one of the header files, then you should seriously rethink what you're +doing and figure out how to put it in libglusterfs instead. + +If that hasn't convinced you, follow the instructions above, but use the +_PRIVATE versions of macros, symbol versions, and aliases. If you're 1337 +enough to ignore this advice, then you're 1337 enough to figure out how +to do it. + + +## Changing an API. + +### Changing a public API. + + There are two ways an API might change, 1) its signature has changed, or +2) its new functionality or behavior is substantially different than the +old. An APIs signature consists of the function return type, and the number +and/or type of its parameters. E.g. the original API: + +```C + int glfs_dtrt (const char *volname, void *stuff); +``` + +and the changed API: + +```C + void *glfs_dtrt (const char *volname, glfs_t *ctx, void *stuff); +``` + + One way to avoid a change like this, and which is preferable in many +ways, is to leave the legacy glfs_dtrt() function alone, document it as +deprecated, and simply add a new API, e.g. glfs_dtrt2(). Practically +speaking, that's effectively what we'll be doing anyway, the difference +is only that we'll use a versioned symbol to do it. + + On the assumption that adding a new API is undesirable for some reason, +perhaps the use of glfs_gnu() is just so pervasive that we really don't +want to add glfs_gnu2(). + ++ change the declaration in glfs.h: + +```C + glfs_t *glfs_gnu (const char *volname, void *stuff) __THROW + GFAPI_PUBLIC(glfs_gnu, 3.7.0); +```` + +Note that there is only the single, new declaration. + ++ change the old definition of glfs_gnu() in glfs.c: + +```C + struct glfs * + pub_glfs_gnu340 (const char * volname) + { + ... + } + GFAPI_SYMVER_PUBLIC(glfs_gnu340, glfs_gnu, 3.4.0); +``` + ++ create the new definition of glfs_gnu in glfs.c: + +```C + struct glfs * + pub_glfs_gnu (const char * volname, void *stuff) + { + ... + } + GFAPI_SYMVER_PUBLIC_DEFAULT(glfs_gnu, 3.7.0); +``` + ++ Add the new API to the ELF, gnu toolchain link map file, gfapi.map + +``` + GFAPI_3.7.0 { + global: + glfs_gnu; + } GFAPI_PRIVATE_3.7.0; +``` + ++ Update the OS X alias list file, gfapi.aliases, for both versions: + +Change the old line: +```C + _pub_glfs_gnu _glfs_gnu$GFAPI_3.4.0 +``` +to: +```C + _pub_glfs_gnu340 _glfs_gnu$GFAPI_3.4.0 +``` + +Add a new line: +```C + _pub_glfs_gnu _glfs_gnu$GFAPI_3.7.0 +``` + ++ Lastly, change all gfapi internal calls glfs_gnu to the new API. + diff --git a/doc/developer-guide/network_compression.md b/doc/developer-guide/network_compression.md new file mode 100644 index 00000000000..7327591ef63 --- /dev/null +++ b/doc/developer-guide/network_compression.md @@ -0,0 +1,71 @@ +#On-Wire Compression + Decompression + +The 'compression translator' compresses and decompresses data in-flight +between client and bricks. + +###Working +When a writev call occurs, the client compresses the data before sending it to +brick. On the brick, compressed data is decompressed. Similarly, when a readv +call occurs, the brick compresses the data before sending it to client. On the +client, the compressed data is decompressed. Thus, the amount of data sent over +the wire is minimized. Compression/Decompression is done using Zlib library. + +During normal operation, this is the format of data sent over wire: + +~~~ +<compressed-data> + trailer(8 bytes) +~~~ + +The trailer contains the CRC32 checksum and length of original uncompressed +data. This is used for validation. + +###Usage + +Turning on compression xlator: + +~~~ +gluster volume set <vol_name> network.compression on +~~~ + +###Configurable parameters (optional) + +**Compression level** +~~~ +gluster volume set <vol_name> network.compression.compression-level 8 +~~~ + +~~~ +0 : no compression +1 : best speed +9 : best compression +-1 : default compression +~~~ + +**Minimum file size** + +~~~ +gluster volume set <vol_name> network.compression.min-size 50 +~~~ + +Data is compressed only when its size exceeds the above value in bytes. + +**Other paramaters** + +Other less frequently used parameters include `network.compression.mem-level` +and `network.compression.window-size`. More details can about these options +can be found by running `gluster volume set help` command. + +###Known Issues and Limitations + +* Compression translator cannot work with striped volumes. +* Mount point hangs when writing a file with write-behind xlator turned on. To +overcome this, turn off `performance.write-behind` entirely OR +set`performance.strict-write-ordering` to on. +* For glusterfs versions <= 3.5, compression traslator can ONLY work with pure +distribute volumes. This limitation is caused by AFR not being able to +propagate xdata. This issue has been fixed in glusterfs versions > 3.5 + +###TODO +Although zlib offers high compression ratio, it is very slow. We can make the +translator pluggable to add support for other compression methods such as +[lz4 compression](https://code.google.com/p/lz4/) diff --git a/doc/developer-guide/posix.md b/doc/developer-guide/posix.md new file mode 100644 index 00000000000..84c813e55a2 --- /dev/null +++ b/doc/developer-guide/posix.md @@ -0,0 +1,59 @@ +storage/posix translator +======================== + +Notes +----- + +### `SET_FS_ID` + +This is so that all filesystem checks are done with the user's +uid/gid and not GlusterFS's uid/gid. + +### `MAKE_REAL_PATH` + +This macro concatenates the base directory of the posix volume +('option directory') with the given path. + +### `need_xattr` in lookup + +If this flag is passed, lookup returns a xattr dictionary that contains +the file's create time, the file's contents, and the version number +of the file. + +This is a hack to increase small file performance. If an application +wants to read a small file, it can finish its job with just a lookup +call instead of a lookup followed by read. + +### `getdents`/`setdents` + +These are used by unify to set and get directory entries. + +### `ALIGN_BUF` + +Macro to align an address to a page boundary (4K). + +### `priv->export_statfs` + +In some cases, two exported volumes may reside on the same +partition on the server. Sending statvfs info for both +the volumes will lead to erroneous df output at the client, +since free space on the partition will be counted twice. + +In such cases, user can disable exporting statvfs info +on one of the volumes by setting this option. + +### `xattrop` + +This fop is used by replicate to set version numbers on files. + +### `getxattr`/`setxattr` hack to read/write files + +A key, `GLUSTERFS_FILE_CONTENT_STRING`, is handled in a special way by +`getxattr`/`setxattr`. A getxattr with the key will return the entire +content of the file as the value. A `setxattr` with the key will write +the value as the entire content of the file. + +### `posix_checksum` + +This calculates a simple XOR checksum on all entry names in a +directory that is used by unify to compare directory contents. diff --git a/doc/developer-guide/rpc-for-glusterfs.changes-done.txt b/doc/developer-guide/rpc-for-glusterfs.changes-done.txt new file mode 100644 index 00000000000..6bbbca78826 --- /dev/null +++ b/doc/developer-guide/rpc-for-glusterfs.changes-done.txt @@ -0,0 +1,18 @@ +This document serves as a basic coding standard/practise for further +developments after proper protocol layer is implemented. + +With this release we are bringing abstraction based on xlator driven +operation and protocol driven operation. ie, all the client side (fuse) +operations are xlator driven operations and will come with 'op' value +taken from 'libglusterfs/'. + +All the server protocol driven operations are driven by which ever +version of protocol is used. + +All the currently implemented fops will remain, and 'getspec' being generated +by top level and passes through translator graph, is treated as an 'fop'. + +All new 'gluster' and 'glusterd' related calls will be _mgmt_ calls instead of +fops. All release, releasedir and forget are treated as fops (but they won't +come with requirement to use STACK_WIND and STACK_UNWIND). + diff --git a/doc/developer-guide/translator-development.md b/doc/developer-guide/translator-development.md new file mode 100644 index 00000000000..edadd5150dc --- /dev/null +++ b/doc/developer-guide/translator-development.md @@ -0,0 +1,666 @@ +Translator development +====================== + +Setting the Stage +----------------- + +This is the first post in a series that will explain some of the details of +writing a GlusterFS translator, using some actual code to illustrate. + +Before we begin, a word about environments. GlusterFS is over 300K lines of +code spread across a few hundred files. That's no Linux kernel or anything, but + you're still going to be navigating through a lot of code in every +code-editing session, so some kind of cross-referencing is *essential*. I use +cscope with the vim bindings, and if I couldn't do Crtl+G and such to jump +between definitions all the time my productivity would be cut in half. You may +prefer different tools, but as I go through these examples you'll need +something functionally similar to follow on. OK, on with the show. + +The first thing you need to know is that translators are not just bags of +functions and variables. They need to have a very definite internal structure +so that the translator-loading code can figure out where all the pieces are. +The way it does this is to use dlsym to look for specific names within your +shared-object file, as follow (from `xlator.c`): + +``` +if (!(xl->fops = dlsym (handle, "fops"))) { + gf_log ("xlator", GF_LOG_WARNING, "dlsym(fops) on %s", + dlerror ()); + goto out; +} + +if (!(xl->cbks = dlsym (handle, "cbks"))) { + gf_log ("xlator", GF_LOG_WARNING, "dlsym(cbks) on %s", + dlerror ()); + goto out; +} + +if (!(xl->init = dlsym (handle, "init"))) { + gf_log ("xlator", GF_LOG_WARNING, "dlsym(init) on %s", + dlerror ()); + goto out; +} + +if (!(xl->fini = dlsym (handle, "fini"))) { + gf_log ("xlator", GF_LOG_WARNING, "dlsym(fini) on %s", + dlerror ()); + goto out; +} +``` + +In this example, `xl` is a pointer to the in-memory object for the translator +we're loading. As you can see, it's looking up various symbols *by name* in the + shared object it just loaded, and storing pointers to those symbols. Some of +them (e.g. init are functions, while others e.g. fops are dispatch tables +containing pointers to many functions. Together, these make up the translator's + public interface. + +Most of this glue or boilerplate can easily be found at the bottom of one of +the source files that make up each translator. We're going to use the `rot-13` +translator just for fun, so in this case you'd look in `rot-13.c` to see this: + +``` +struct xlator_fops fops = { + .readv = rot13_readv, + .writev = rot13_writev +}; + +struct xlator_cbks cbks = { +}; + +struct volume_options options[] = { +{ .key = {"encrypt-write"}, + .type = GF_OPTION_TYPE_BOOL +}, +{ .key = {"decrypt-read"}, + .type = GF_OPTION_TYPE_BOOL +}, +{ .key = {NULL} }, +}; +``` + +The `fops` table, defined in `xlator.h`, is one of the most important pieces. +This table contains a pointer to each of the filesystem functions that your +translator might implement -- `open`, `read`, `stat`, `chmod`, and so on. There +are 82 such functions in all, but don't worry; any that you don't specify here +will be see as null and filled with defaults from `defaults.c` when your +translator is loaded. In this particular example, since `rot-13` is an +exceptionally simple translator, we only fill in two entries for `readv` and +`writev`. + +There are actually two other tables, also required to have predefined names, +that are also used to find translator functions: `cbks` (which is empty in this + snippet) and `dumpops` (which is missing entirely). The first of these specify + entry points for when inodes are forgotten or file descriptors are released. +In other words, they're destructors for objects in which your translator might + have an interest. Mostly you can ignore them, because the default behavior +handles even the simpler cases of translator-specific inode/fd context +automatically. However, if the context you attach is a complex structure +requiring complex cleanup, you'll need to supply these functions. As for +dumpops, that's just used if you want to provide functions to pretty-print +various structures in logs. I've never used it myself, though I probably +should. What's noteworthy here is that we don't even define dumpops. That's +because all of the functions that might use these dispatch functions will check + for `xl->dumpops` being `NULL` before calling through it. This is in sharp +contrast to the behavior for `fops` and `cbks1`, which *must* be present. If +they're not, translator loading will fail because these pointers are not +checked every time and if they're `NULL` then we'll segfault. That's why we +provide an empty definition for cbks; it's OK for the individual function +pointers to be NULL, but not for the whole table to be absent. + +The last piece I'll cover today is options. As you can see, this is a table of +translator-specific option names and some information about their types. +GlusterFS actually provides a pretty rich set of types (`volume_option_type_t` +in `options.`h) which includes paths, translator names, percentages, and times +in addition to the obvious integers and strings. Also, the `volume_option_t` +structure can include information about alternate names, min/max/default +values, enumerated string values, and descriptions. We don't see any of these +here, so let's take a quick look at some more complex examples from afr.c and +then come back to `rot-13`. + +``` +{ .key = {"data-self-heal-algorithm"}, + .type = GF_OPTION_TYPE_STR, + .default_value = "", + .description = "Select between \"full\", \"diff\". The " + "\"full\" algorithm copies the entire file from " + "source to sink. The \"diff\" algorithm copies to " + "sink only those blocks whose checksums don't match " + "with those of source.", + .value = { "diff", "full", "" } +}, +{ .key = {"data-self-heal-window-size"}, + .type = GF_OPTION_TYPE_INT, + .min = 1, + .max = 1024, + .default_value = "1", + .description = "Maximum number blocks per file for which " + "self-heal process would be applied simultaneously." +}, +``` + +When your translator is loaded, all of this information is used to parse the +options actually provided in the volfile, and then the result is turned into a +dictionary and stored as `xl->options`. This dictionary is then processed by +your init function, which you can see being looked up in the first code +fragment above. We're only going to look at a small part of the `rot-13`'s +init for now. + +``` +priv->decrypt_read = 1; +priv->encrypt_write = 1; + +data = dict_get (this->options, "encrypt-write"); +if (data) { + if (gf_string2boolean (data->data, &priv->encrypt_write + == -1) { + gf_log (this->name, GF_LOG_ERROR, + "encrypt-write takes only boolean options"); + return -1; + } +} +``` + +What we can see here is that we're setting some defaults in our priv structure, +then looking to see if an `encrypt-write` option was actually provided. If so, +we convert and store it. This is a pretty classic use of dict_get to fetch a +field from a dictionary, and of using one of many conversion functions in +`common-utils.c` to convert `data->data` into something we can use. + +So far we've covered the basic of how a translator gets loaded, how we find its +various parts, and how we process its options. In my next Translator 101 post, +we'll go a little deeper into other things that init and its companion fini +might do, and how some other fields in our `xlator_t` structure (commonly +referred to as this) are commonly used. + +`init`, `fini`, and private context +----------------------------------- + +In the previous Translator 101 post, we looked at some of the dispatch tables +and options processing in a translator. This time we're going to cover the rest + of the "shell" of a translator -- i.e. the other global parts not specific to +handling a particular request. + +Let's start by looking at the relationship between a translator and its shared +library. At a first approximation, this is the relationship between an object +and a class in just about any object-oriented programming language. The class +defines behaviors, but has to be instantiated as an object to have any kind of +existence. In our case the object is an `xlator_t`. Several of these might be +created within the same daemon, sharing all of the same code through init/fini +and dispatch tables, but sharing *no data*. You could implement shared data (as + static variables in your shared libraries) but that's strongly discouraged. +Every function in your shared library will get an `xlator_t` as an argument, +and should use it. This lack of class-level data is one of the points where +the analogy to common OOP systems starts to break down. Another place is the +complete lack of inheritance. Translators inherit behavior (code) from exactly +one shared library -- looked up and loaded using the `type` field in a volfile +`volume ... end-volume` block -- and that's it -- not even single inheritance, +no subclasses or superclasses, no mixins or prototypes, just the relationship +between an object and its class. With that in mind, let's turn to the init +function that we just barely touched on last time. + +``` +int32_t +init (xlator_t *this) +{ + data_t *data = NULL; + rot_13_private_t *priv = NULL; + + if (!this->children || this->children->next) { + gf_log ("rot13", GF_LOG_ERROR, + "FATAL: rot13 should have exactly one child"); + return -1; + } + + if (!this->parents) { + gf_log (this->name, GF_LOG_WARNING, + "dangling volume. check volfile "); + } + + priv = GF_CALLOC (sizeof (rot_13_private_t), 1, 0); + if (!priv) + return -1; +``` + +At the very top, we see the function signature -- we get a pointer to the +`xlator_t` object that we're initializing, and we return an `int32_t` status. +As with most functions in the translator API, this should be zero to indicate +success. In this case it's safe to return -1 for failure, but watch out: in +dispatch-table functions, the return value means the status of the *function +call* rather than the *request*. A request error should be reflected as a +callback with a non-zero `op_re`t value, but the dispatch function itself +should still return zero. In fact, the handling of a non-zero return from a +dispatch function is not all that robust (we recently had a bug report in +HekaFS related to this) so it's something you should probably avoid +altogether. This only underscores the difference between dispatch functions +and `init`/`fini` functions, where non-zero returns *are* expected and handled +logically by aborting the translator setup. We can see that down at the +bottom, where we return -1 to indicate that we couldn't allocate our +private-data area (more about that later). + +The first thing this init function does is check that the translator is being +set up in the right kind of environment. Translators are called by parents and +in turn call children. Some translators are "initial" translators that inject +requests into the system from elsewhere -- e.g. mount/fuse injecting requests +from the kernel, protocol/server injecting requests from the network. Those +translators don't need parents, but `rot-13` does and so we check for that. +Similarly, some translators are "final" translators that (from the perspective +of the current process) terminate requests instead of passing them on -- e.g. +`protocol/client` passing them to another node, `storage/posix` passing them to +a local filesystem. Other translators "multiplex" between multiple children -- + passing each parent request on to one (`cluster/dht`), some +(`cluster/stripe`), or all (`cluster/afr`) of those children. `rot-13` fits +into none of those categories either, so it checks that it has *exactly one* +child. It might be more convenient or robust if translator shared libraries +had standard variables describing these requirements, to be checked in a +consistent way by the translator-loading infrastructure itself instead of by +each separate init function, but this is the way translators work today. + +The last thing we see in this fragment is allocating our private data area. +This can literally be anything we want; the infrastructure just provides the +priv pointer as a convenience but takes no responsibility for how it's used. In + this case we're using `GF_CALLOC` to allocate our own `rot_13_private_t` +structure. This gets us all the benefits of GlusterFS's memory-leak detection +infrastructure, but the way we're calling it is not quite ideal. For one thing, + the first two arguments -- from `calloc(3)` -- are kind of reversed. For +another, notice how the last argument is zero. That can actually be an +enumerated value, to tell the GlusterFS allocator *what* type we're +allocating. This can be very useful information for memory profiling and leak +detection, so it's recommended that you follow the example of any +x`xx-mem-types.h` file elsewhere in the source tree instead of just passing +zero here (even though that works). + +To finish our tour of standard initialization/termination, let's look at the +end of `init` and the beginning of `fini`: + +``` + this->private = priv; + gf_log ("rot13", GF_LOG_DEBUG, "rot13 xlator loaded"); + return 0; +} + +void +fini (xlator_t *this) +{ + rot_13_private_t *priv = this->private; + + if (!priv) + return; + this->private = NULL; + GF_FREE (priv); +``` + +At the end of init we're just storing our private-data pointer in the `priv` +field of our `xlator_t`, then returning zero to indicate that initialization +succeeded. As is usually the case, our fini is even simpler. All it really has +to do is `GF_FREE` our private-data pointer, which we do in a slightly +roundabout way here. Notice how we don't even have a return value here, since +there's nothing obvious and useful that the infrastructure could do if `fini` +failed. + +That's practically everything we need to know to get our translator through +loading, initialization, options processing, and termination. If we had defined + no dispatch functions, we could actually configure a daemon to use our +translator and it would work as a basic pass-through from its parent to a +single child. In the next post I'll cover how to build the translator and +configure a daemon to use it, so that we can actually step through it in a +debugger and see how it all fits together before we actually start adding +functionality. + +This Time For Real +------------------ + +In the first two parts of this series, we learned how to write a basic +translator skeleton that can get through loading, initialization, and option +processing. This time we'll cover how to build that translator, configure a +volume to use it, and run the glusterfs daemon in debug mode. + +Unfortunately, there's not much direct support for writing new translators. You +can check out a GlusterFS tree and splice in your own translator directory, but + that's a bit painful because you'll have to update multiple makefiles plus a +bunch of autoconf garbage. As part of the HekaFS project, I basically reverse +engineered the truly necessary parts of the translator-building process and +then pestered one of the Fedora glusterfs package maintainers (thanks +daMaestro!) to add a `glusterfs-devel` package with the required headers. Since + then the complexity level in the HekaFS tree has crept back up a bit, but I +still remember the simple method and still consider it the easiest way to get +started on a new translator. For the sake of those not using Fedora, I'm going +to describe a method that doesn't depend on that header package. What it does +depend on is a GlusterFS source tree, much as you might have cloned from GitHub + or the Gluster review site. This tree doesn't have to be fully built, but you +do need to run `autogen.sh` and configure in it. Then you can take the +following simple makefile and put it in a directory with your actual source. + +``` +# Change these to match your source code. +TARGET = rot-13.so +OBJECTS = rot-13.o + +# Change these to match your environment. +GLFS_SRC = /srv/glusterfs +GLFS_LIB = /usr/lib64 +HOST_OS = GF_LINUX_HOST_OS + +# You shouldn't need to change anything below here. + +CFLAGS = -fPIC -Wall -O0 -g \ + -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE \ + -D$(HOST_OS) -I$(GLFS_SRC) -I$(GLFS_SRC)/contrib/uuid \ + -I$(GLFS_SRC)/libglusterfs/src +LDFLAGS = -shared -nostartfiles -L$(GLFS_LIB) +LIBS = -lglusterfs -lpthread + +$(TARGET): $(OBJECTS) + $(CC) $(OBJECTS) $(LDFLAGS) -o $(TARGET) $(OBJECTS) $(LIBS) +``` + +Yes, it's still Linux-specific. Mea culpa. As you can see, we're sticking with +the `rot-13` example, so you can just copy the files from +`xlators/encryption/rot-13/src` in your GlusterFS tree to follow on. Type +`make` and you should be rewarded with a nice little `.so` file. + +``` +xlator_example$ ls -l rot-13.so +-rwxr-xr-x. 1 jeff jeff 40784 Nov 16 16:41 rot-13.so +``` + +Notice that we've built with optimization level zero and debugging symbols +included, which would not typically be the case for a packaged version of +GlusterFS. Let's put our version of `rot-13.so` into a slightly different file +on our system, so that it doesn't stomp on the installed version (not that +you'd ever want to use that anyway). + +``` +xlator_example# ls /usr/lib64/glusterfs/3git/xlator/encryption/ +crypt.so crypt.so.0 crypt.so.0.0.0 rot-13.so rot-13.so.0 +rot-13.so.0.0.0 +xlator_example# cp rot-13.so \ + /usr/lib64/glusterfs/3git/xlator/encryption/my-rot-13.so +``` + +These paths represent the current Gluster filesystem layout, which is likely to +be deprecated in favor of the Fedora layout; your paths may vary. At this point + we're ready to configure a volume using our new translator. To do that, I'm +going to suggest something that's strongly discouraged except during +development (the Gluster guys are going to hate me for this): write our own +volfile. Here's just about the simplest volfile you'll ever see. + +``` +volume my-posix + type storage/posix + option directory /srv/export +end-volume + +volume my-rot13 + type encryption/my-rot-13 + subvolumes my-posix +end-volume +``` + +All we have here is a basic brick using `/srv/export` for its data, and then +an instance of our translator layered on top -- no client or server is +necessary for what we're doing, and the system will automatically push a +mount/fuse translator on top if there's no server translator. To try this out, +all we need is the following command (assuming the directories involved already + exist). + +``` +xlator_example$ glusterfs --debug -f my.vol /srv/import +``` + +You should be rewarded with a whole lot of log output, including the text of +the volfile (this is very useful for debugging problems in the field). If you +go to another window on the same machine, you can see that you have a new +filesystem mounted. + +``` +~$ df /srv/import +Filesystem 1K-blocks Used Available Use% Mounted on +/srv/xlator_example/my.vol + 114506240 2706176 105983488 3% /srv/import +``` + +Just for fun, write something into a file in `/srv/import`, then look at the +corresponding file in `/srv/export` to see it all `rot-13`'ed for you. + +``` +~$ echo hello > /srv/import/a_file +~$ cat /srv/export/a_file +uryyb +``` + +There you have it -- functionality you control, implemented easily, layered on +top of local storage. Now you could start adding functionality -- real +encryption, perhaps -- and inevitably having to debug it. You could do that the + old-school way, with `gf_log` (preferred) or even plain old `printf`, or you +could run daemons under `gdb` instead. Alternatively, you could wait for the +next Translator 101 post, where we'll be doing exactly that. + +Debugging a Translator +---------------------- + +Now that we've learned what a translator looks like and how to build one, it's +time to run one and actually watch it work. The best way to do this is good +old-fashioned `gdb`, as follows (using some of the examples from last time). + +``` +xlator_example# gdb glusterfs +GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) +... +(gdb) r --debug -f my.vol /srv/import +Starting program: /usr/sbin/glusterfs --debug -f my.vol /srv/import +... +[2011-11-23 11:23:16.495516] I [fuse-bridge.c:2971:fuse_init] + 0-glusterfs-fuse: FUSE inited with protocol versions: + glusterfs 7.13 kernel 7.13 +``` + +If you get to this point, your glusterfs client process is already running. You +can go to another window to see the mountpoint, do file operations, etc. + +``` +~# df /srv/import +Filesystem 1K-blocks Used Available Use% Mounted on +/root/xlator_example/my.vol + 114506240 2643968 106045568 3% /srv/import +~# ls /srv/import +a_file +~# cat /srv/import/a_file +hello +``` + +Now let's interrupt the process and see where we are. + +``` +^C +Program received signal SIGINT, Interrupt. +0x0000003a0060b3dc in pthread_cond_wait@@GLIBC_2.3.2 () + from /lib64/libpthread.so.0 +(gdb) info threads + 5 Thread 0x7fffeffff700 (LWP 27206) 0x0000003a002dd8c7 + in readv () + from /lib64/libc.so.6 + 4 Thread 0x7ffff50e3700 (LWP 27205) 0x0000003a0060b75b + in pthread_cond_timedwait@@GLIBC_2.3.2 () + from /lib64/libpthread.so.0 + 3 Thread 0x7ffff5f02700 (LWP 27204) 0x0000003a0060b3dc + in pthread_cond_wait@@GLIBC_2.3.2 () + from /lib64/libpthread.so.0 + 2 Thread 0x7ffff6903700 (LWP 27203) 0x0000003a0060f245 + in sigwait () + from /lib64/libpthread.so.0 +* 1 Thread 0x7ffff7957700 (LWP 27196) 0x0000003a0060b3dc + in pthread_cond_wait@@GLIBC_2.3.2 () + from /lib64/libpthread.so.0 +``` + +Like any non-toy server, this one has multiple threads. What are they all +doing? Honestly, even I don't know. Thread 1 turns out to be in +`event_dispatch_epoll`, which means it's the one handling all of our network +I/O. Note that with socket multi-threading patch this will change, with one +thread in `socket_poller` per connection. Thread 2 is in `glusterfs_sigwaiter` +which means signals will be isolated to that thread. Thread 3 is in +`syncenv_task`, so it's a worker process for synchronous requests such as +those used by the rebalance and repair code. Thread 4 is in +`janitor_get_next_fd`, so it's waiting for a chance to close no-longer-needed +file descriptors on the local filesystem. (I admit I had to look that one up, +BTW.) Lastly, thread 5 is in `fuse_thread_proc`, so it's the one fetching +requests from our FUSE interface. You'll often see many more threads than +this, but it's a pretty good basic set. Now, let's set a breakpoint so we can +actually watch a request. + +``` +(gdb) b rot13_writev +Breakpoint 1 at 0x7ffff50e4f0b: file rot-13.c, line 119. +(gdb) c +Continuing. +``` + +At this point we go into our other window and do something that will involve a write. + +``` +~# echo goodbye > /srv/import/another_file +(back to the first window) +[Switching to Thread 0x7fffeffff700 (LWP 27206)] + +Breakpoint 1, rot13_writev (frame=0x7ffff6e4402c, this=0x638440, + fd=0x7ffff409802c, vector=0x7fffe8000cd8, count=1, offset=0, + iobref=0x7fffe8001070) at rot-13.c:119 +119 rot_13_private_t *priv = (rot_13_private_t *)this->private; +``` + +Remember how we built with debugging symbols enabled and no optimization? That +will be pretty important for the next few steps. As you can see, we're in +`rot13_writev`, with several parameters. + +* `frame` is our always-present frame pointer for this request. Also, + `frame->local` will point to any local data we created and attached to the + request ourselves. +* `this` is a pointer to our instance of the `rot-13` translator. You can examine + it if you like to see the name, type, options, parent/children, inode table, + and other stuff associated with it. +* `fd` is a pointer to a file-descriptor *object* (`fd_t`, not just a + file-descriptor index which is what most people use "fd" for). This in turn + points to an inode object (`inode_t`) and we can associate our own + `rot-13`-specific data with either of these. +* `vector` and `count` together describe the data buffers for this write, which + we'll get to in a moment. +* `offset` is the offset into the file at which we're writing. +* `iobref` is a buffer-reference object, which is used to track the life cycle + of buffers containing read/write data. If you look closely, you'll notice that + `vector[0].iov_base` points to the same address as `iobref->iobrefs[0].ptr`, which + should give you some idea of the inter-relationships between vector and iobref. + +OK, now what about that `vector`? We can use it to examine the data being +written, like this. + +``` +(gdb) p vector[0] +$2 = {iov_base = 0x7ffff7936000, iov_len = 8} +(gdb) x/s 0x7ffff7936000 +0x7ffff7936000: "goodbye\n" +``` + +It's not always safe to view this data as a string, because it might just as +well be binary data, but since we're generating the write this time it's safe +and convenient. With that knowledge, let's step through things a bit. + +``` +(gdb) s +120 if (priv->encrypt_write) +(gdb) +121 rot13_iovec (vector, count); +(gdb) +rot13_iovec (vector=0x7fffe8000cd8, count=1) at rot-13.c:57 +57 for (i = 0; i < count; i++) { +(gdb) +58 rot13 (vector[i].iov_base, vector[i].iov_len); +(gdb) +rot13 (buf=0x7ffff7936000 "goodbye\n", len=8) at rot-13.c:45 +45 for (i = 0; i < len; i++) { +(gdb) +46 if (buf[i] >= 'a' && buf[i] <= 'z') +(gdb) +47 buf[i] = 'a' + ((buf[i] - 'a' + 13) % 26); +``` + +Here we've stepped into `rot13_iovec`, which iterates through our vector +calling `rot13`, which in turn iterates through the characters in that chunk +doing the `rot-13` operation if/as appropriate. This is pretty straightforward +stuff, so let's skip to the next interesting bit. + +``` +(gdb) fin +Run till exit from #0 rot13 (buf=0x7ffff7936000 "goodbye\n", + len=8) at rot-13.c:47 +rot13_iovec (vector=0x7fffe8000cd8, count=1) at rot-13.c:57 +57 for (i = 0; i < count; i++) { +(gdb) fin +Run till exit from #0 rot13_iovec (vector=0x7fffe8000cd8, + count=1) at rot-13.c:57 +rot13_writev (frame=0x7ffff6e4402c, this=0x638440, + fd=0x7ffff409802c, vector=0x7fffe8000cd8, count=1, + offset=0, iobref=0x7fffe8001070) at rot-13.c:123 +123 STACK_WIND (frame, +(gdb) b 129 +Breakpoint 2 at 0x7ffff50e4f35: file rot-13.c, line 129. +(gdb) b rot13_writev_cbk +Breakpoint 3 at 0x7ffff50e4db3: file rot-13.c, line 106. +(gdb) c +``` + +So we've set breakpoints on both the callback and the statement following the +`STACK_WIND`. Which one will we hit first? + +``` +Breakpoint 3, rot13_writev_cbk (frame=0x7ffff6e4402c, + cookie=0x7ffff6e440d8, this=0x638440, op_ret=8, op_errno=0, + prebuf=0x7fffefffeca0, postbuf=0x7fffefffec30) + at rot-13.c:106 +106 STACK_UNWIND_STRICT (writev, frame, op_ret, op_errno, + prebuf, postbuf); +(gdb) bt +#0 rot13_writev_cbk (frame=0x7ffff6e4402c, + cookie=0x7ffff6e440d8, this=0x638440, op_ret=8, op_errno=0, + prebuf=0x7fffefffeca0, postbuf=0x7fffefffec30) + at rot-13.c:106 +#1 0x00007ffff52f1b37 in posix_writev (frame=0x7ffff6e440d8, + this=<value optimized out>, fd=<value optimized out>, + vector=<value optimized out>, count=1, + offset=<value optimized out>, iobref=0x7fffe8001070) + at posix.c:2217 +#2 0x00007ffff50e513e in rot13_writev (frame=0x7ffff6e4402c, + this=0x638440, fd=0x7ffff409802c, vector=0x7fffe8000cd8, + count=1, offset=0, iobref=0x7fffe8001070) at rot-13.c:123 +``` + +Surprise! We're in `rot13_writev_cbk` now, called (indirectly) while we're +still in `rot13_writev` before `STACK_WIND` returns (still at rot-13.c:123). If + you did any request cleanup here, then you need to be careful about what you +do in the remainder of `rot13_writev` because data may have been freed etc. +It's tempting to say you should just do the cleanup in `rot13_writev` after +the `STACK_WIND,` but that's not valid because it's also possible that some +other translator returned without calling `STACK_UNWIND` -- i.e. before +`rot13_writev` is called, so then it would be the one getting null-pointer +errors instead. To put it another way, the callback and the return from +`STACK_WIND` can occur in either order or even simultaneously on different +threads. Even if you were to use reference counts, you'd have to make sure to +use locking or atomic operations to avoid races, and it's not worth it. Unless +you *really* understand the possible flows of control and know what you're +doing, it's better to do cleanup in the callback and nothing after +`STACK_WIND.` + +At this point all that's left is a `STACK_UNWIND` and a return. The +`STACK_UNWIND` invokes our parent's completion callback, and in this case our +parent is FUSE so at that point the VFS layer is notified of the write being +complete. Finally, we return through several levels of normal function calls +until we come back to fuse_thread_proc, which waits for the next request. + +So that's it. For extra fun, you might want to repeat this exercise by stepping +through some other call -- stat or setxattr might be good choices -- but you'll + have to use a translator that actually implements those calls to see much +that's interesting. Then you'll pretty much know everything I knew when I +started writing my first for-real translators, and probably even a bit more. I +hope you've enjoyed this series, or at least found it useful, and if you have +any suggestions for other topics I should cover please let me know (via +comments or email, IRC or Twitter). diff --git a/doc/developer-guide/unittest.md b/doc/developer-guide/unittest.md new file mode 100644 index 00000000000..5c6c0a8a039 --- /dev/null +++ b/doc/developer-guide/unittest.md @@ -0,0 +1,228 @@ +# Unit Tests in GlusterFS + +## Overview +[Art-of-unittesting][definitionofunittest] provides a good definition for unit tests. A good unit test is: + +* Able to be fully automated +* Has full control over all the pieces running (Use mocks or stubs to achieve this isolation when needed) +* Can be run in any order if part of many other tests +* Runs in memory (no DB or File access, for example) +* Consistently returns the same result (You always run the same test, so no random numbers, for example. save those for integration or range tests) +* Runs fast +* Tests a single logical concept in the system +* Readable +* Maintainable +* Trustworthy (when you see its result, you don’t need to debug the code just to be sure) + +## cmocka +GlusterFS unit test framework is based on [cmocka][]. cmocka provides +developers with methods to isolate and test modules written in C language. It +also provides integration with Jenkins by providing JUnit XML compliant unit +test results. + +cmocka + +## Running Unit Tests +To execute the unit tests, all you need is to type `make check`. Here is a step-by-step example assuming you just cloned a GlusterFS tree: + +``` +$ ./autogen.sh +$ ./configure --enable-debug +$ make check +``` + +Sample output: + +``` +PASS: mem_pool_unittest +============================================================================ +Testsuite summary for glusterfs 3git +============================================================================ +# TOTAL: 1 +# PASS: 1 +# SKIP: 0 +# XFAIL: 0 +# FAIL: 0 +# XPASS: 0 +# ERROR: 0 +============================================================================ +``` + +In this example, `mem_pool_unittest` has multiple tests inside, but `make check` assumes that the program itself is the test, and that is why it only shows one test. Here is the output when we run `mem_pool_unittest` directly: + +``` +$ ./libglusterfs/src/mem_pool_unittest +[==========] Running 10 test(s). +[ RUN ] test_gf_mem_acct_enable_set +Expected assertion data != ((void *)0) occurred +[ OK ] test_gf_mem_acct_enable_set +[ RUN ] test_gf_mem_set_acct_info_asserts +Expected assertion xl != ((void *)0) occurred +Expected assertion size > ((4 + sizeof (size_t) + sizeof (xlator_t *) + 4 + 8) + 8) occurred +Expected assertion type <= xl->mem_acct.num_types occurred +[ OK ] test_gf_mem_set_acct_info_asserts +[ RUN ] test_gf_mem_set_acct_info_memory +[ OK ] test_gf_mem_set_acct_info_memory +[ RUN ] test_gf_calloc_default_calloc +[ OK ] test_gf_calloc_default_calloc +[ RUN ] test_gf_calloc_mem_acct_enabled +[ OK ] test_gf_calloc_mem_acct_enabled +[ RUN ] test_gf_malloc_default_malloc +[ OK ] test_gf_malloc_default_malloc +[ RUN ] test_gf_malloc_mem_acct_enabled +[ OK ] test_gf_malloc_mem_acct_enabled +[ RUN ] test_gf_realloc_default_realloc +[ OK ] test_gf_realloc_default_realloc +[ RUN ] test_gf_realloc_mem_acct_enabled +[ OK ] test_gf_realloc_mem_acct_enabled +[ RUN ] test_gf_realloc_ptr +Expected assertion ((void *)0) != ptr occurred +[ OK ] test_gf_realloc_ptr +[==========] 10 test(s) run. +[ PASSED ] 10 test(s). +[ FAILED ] 0 test(s). +[ REPORT ] Created libglusterfs_mem_pool_xunit.xml report +``` + + +## Writing Unit Tests + +### Enhancing your C functions + +#### Programming by Contract +Add the following to your C file: + +```c +#include <cmocka_pbc.h> +``` + +```c +/* + * Programming by Contract is a programming methodology + * which binds the caller and the function called to a + * contract. The contract is represented using Hoare Triple: + * {P} C {Q} + * where {P} is the precondition before executing command C, + * and {Q} is the postcondition. + * + * See also: + * http://en.wikipedia.org/wiki/Design_by_contract + * http://en.wikipedia.org/wiki/Hoare_logic + * http://dlang.org/dbc.html + */ + #ifndef CMOCKERY_PBC_H_ +#define CMOCKERY_PBC_H_ + +#if defined(UNIT_TESTING) || defined (DEBUG) + +#include <assert.h> + +/* + * Checks caller responsibility against contract + */ +#define REQUIRE(cond) assert(cond) + +/* + * Checks function reponsability against contract. + */ +#define ENSURE(cond) assert(cond) + +/* + * While REQUIRE and ENSURE apply to functions, INVARIANT + * applies to classes/structs. It ensures that intances + * of the class/struct are consistent. In other words, + * that the instance has not been corrupted. + */ +#define INVARIANT(invariant_fnc) do{ (invariant_fnc) } while (0); + +#else +#define REQUIRE(cond) do { } while (0); +#define ENSURE(cond) do { } while (0); +#define INVARIANT(invariant_fnc) do{ } while (0); + +#endif /* defined(UNIT_TESTING) || defined (DEBUG) */ +#endif /* CMOCKERY_PBC_H_ */ +``` + +##### Example +This is an _extremely_ simple example: + +```c +int divide (int n, int d) +{ + int ans; + + REQUIRE(d != 0); + + ans = n / d; + + // As code is added to this function throughout its lifetime, + // ENSURE will assert that data will be returned + // according to the contract. Again this is an + // extremely simple example. :-D + ENSURE( ans == (n / d) ); + + return ans; +} + +``` + +##### Important Note +`REQUIRE`, `ENSURE`, and `INVARIANT` are only available when `DEBUG` or `UNIT_TESTING` are set in the CFLAGS. You must pass `--enable-debug` to `./configure` to enable PBC on your non-unittest builds. + +#### Overriding functions +Cmockery2 provides its own memory allocation functions which check for buffer overrun and memory leaks. The following header file must be included **last** to be able to override any of the memory allocation functions: + +```c +#include <cmocka.h> +``` + +This file will only take effect with the `UNIT_TESTING` CFLAG is set. + +### Creating a unit test +Once you identify the C file you would like to test, first create a `unittest` directory under the directory where the C file is located. This will isolate the unittests to a different directory. + +Next, you need to edit the `Makefile.am` file in the directory where your C file is located. Initialize the +`Makefile.am` if it does not already have the following sections: + +``` +#### UNIT TESTS ##### +CLEANFILES += *.gcda *.gcno *_xunit.xml +noinst_PROGRAMS = +TESTS = +``` + +Now you can add the following for each of the unit tests that you would like to build: + +``` +### UNIT TEST xxx_unittest ### +xxx_unittest_CPPFLAGS = $(xxx_CPPFLAGS) +xxx_unittest_SOURCES = xxx.c \ + unittest/xxx_unittest.c +xxx_unittest_CFLAGS = $(UNITTEST_CFLAGS) +xxx_unittest_LDFLAGS = $(UNITTEST_LDFLAGS) +noinst_PROGRAMS += xxx_unittest +TESTS += xxx_unittest +``` + +Where `xxx` is the name of your C file. For example, look at `libglusterfs/src/Makefile.am`. + +Copy the simple unit test from the [cmocka API][cmockaapi] to `unittest/xxx_unittest.c`. If you would like to see an example of a unit test, please refer to `libglusterfs/src/unittest/mem_pool_unittest.c`. + +#### Mocking +You may see that the linker will complain about missing functions needed by the C file you would like to test. Identify the required functions, then place their stubs in a file called `unittest/xxx_mock.c`, then include this file in `Makefile.am` in `xxx_unittest_SOURCES`. This will allow you to you Cmockery2's mocking functions. + +#### Running the unit test +You can type `make` in the directory where the C file is located. Once you built it and there are no errors, you can execute the test either by directly executing the program (in our example above it is called `xxx_unittest` ), or by running `make check`. + +#### Debugging +Sometimes you may need to debug your unit test. To do that, you will have to point `gdb` to the binary which is located in the same directory as the source. For example, you can do the following from the root of the source tree to debug `mem_pool_unittest`: + +``` +$ gdb libglusterfs/src/mem_pool_unittest +``` + + +[cmocka]: https://cmocka.org +[definitionofunittest]: http://artofunittesting.com/definition-of-a-unit-test/ +[cmockapi]: https://api.cmocka.org diff --git a/doc/developer-guide/versioning.md b/doc/developer-guide/versioning.md new file mode 100644 index 00000000000..10c1511b79b --- /dev/null +++ b/doc/developer-guide/versioning.md @@ -0,0 +1,44 @@ +Versioning +========== + +### current + +The number of the current interface exported by the library. A current value +of '1', means that you are calling the interface exported by this library +interface 1. + +### revision + +The implementation number of the most recent interface exported by this library. +In this case, a revision value of `0` means that this is the first +implementation of the interface. + +If the next release of this library exports the same interface, but has a +different implementation (perhaps some bugs have been fixed), the revision +number will be higher, but current number will be the same. In that case, when +given a choice, the library with the highest revision will always be used by +the runtime loader. + +### age + +The number of previous additional interfaces supported by this library. If age +were '2', then this library can be linked into executables which were built with +a release of this library that exported the current interface number, current, +or any of the previous two interfaces. By definition age must be less than or +equal to current. At the outset, only the first ever interface is implemented, +so age can only be `0'. + +For every release of the library `-version-info` argument needs to be set +correctly depending on any interface changes you have made. + +This is quite straightforward when you understand what the three numbers mean: + +If you have changed any of the sources for this library, the revision number +must be incremented. This is a new revision of the current interface. If the +interface has changed, then current must be incremented, and revision reset +to '0'. + +This is the first revision of a new interface. If the new interface is a +superset of the previous interface (that is, if the previous interface has not +been broken by the changes in this new release), then age must be incremented. +This release is backwards compatible with the previous release. diff --git a/doc/developer-guide/write-behind.md b/doc/developer-guide/write-behind.md new file mode 100644 index 00000000000..0d78964fa20 --- /dev/null +++ b/doc/developer-guide/write-behind.md @@ -0,0 +1,56 @@ +performance/write-behind translator +=================================== + +Basic working +-------------- + +Write behind is basically a translator to lie to the application that the +write-requests are finished, even before it is actually finished. + +On a regular translator tree without write-behind, control flow is like this: + +1. application makes a `write()` system call. +2. VFS ==> FUSE ==> `/dev/fuse`. +3. fuse-bridge initiates a glusterfs `writev()` call. +4. `writev()` is `STACK_WIND()`ed up to client-protocol or storage translator. +5. client-protocol, on receiving reply from server, starts `STACK_UNWIND()` towards the fuse-bridge. + +On a translator tree with write-behind, control flow is like this: + +1. application makes a `write()` system call. +2. VFS ==> FUSE ==> `/dev/fuse`. +3. fuse-bridge initiates a glusterfs `writev()` call. +4. `writev()` is `STACK_WIND()`ed up to write-behind translator. +5. write-behind adds the write buffer to its internal queue and does a `STACK_UNWIND()` towards the fuse-bridge. + +write call is completed in application's percepective. after +`STACK_UNWIND()`ing towards the fuse-bridge, write-behind initiates a fresh +writev() call to its child translator, whose replies will be consumed by +write-behind itself. Write-behind _doesn't_ cache the write buffer, unless +`option flush-behind on` is specified in volume specification file. + +Windowing +--------- + +With respect to write-behind, each write-buffer has three flags: `stack_wound`, `write_behind` and `got_reply`. + +* `stack_wound`: if set, indicates that write-behind has initiated `STACK_WIND()` towards child translator. +* `write_behind`: if set, indicates that write-behind has done `STACK_UNWIND()` towards fuse-bridge. +* `got_reply`: if set, indicates that write-behind has received reply from child translator for a `writev()` `STACK_WIND()`. a request will be destroyed by write-behind only if this flag is set. + +Currently pending write requests = aggregate size of requests with write_behind = 1 and got_reply = 0. + +window size limits the aggregate size of currently pending write requests. once +the pending requests' size has reached the window size, write-behind blocks +writev() calls from fuse-bridge. Blocking is only from application's +perspective. Write-behind does `STACK_WIND()` to child translator +straight-away, but hold behind the `STACK_UNWIND()` towards fuse-bridge. +`STACK_UNWIND()` is done only once write-behind gets enough replies to +accommodate for currently blocked request. + +Flush behind +------------ + +If `option flush-behind on` is specified in volume specification file, then +write-behind sends aggregate write requests to child translator, instead of +regular per request `STACK_WIND()`s. |