doc: Add block device xlator details

Change-Id: I4c24a7f0a4339364a80e5d1be0fc8de6b9c189c1
BUG: 1086762
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
Reviewed-on: http://review.gluster.org/7809
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>

1 files changed, 406 insertions, 0 deletions

diff --git a/doc/features/bd-xlator.md b/doc/features/bd-xlator.md
new file mode 100644
index 00000000000..743e78b1eb0
--- /dev/null
+++ b/doc/features/bd-xlator.md
@@ -0,0 +1,406 @@
+#Block device translator
+
+Block device translator (BD xlator) is a translator added to GlusterFS  which provides block backend for GlusterFS. This replaces the existing bd_map translator in GlusterFS that provided similar but very limited functionality. GlusterFS expects the underlying brick to be formatted with a POSIX compatible file system. BD xlator changes that and allows for having bricks that are raw block devices like LVM which needn’t have any file systems on them. Hence with BD xlator, it becomes possible to build a GlusterFS volume comprising of bricks that are logical volumes (LV).
+
+##bd
+
+BD xlator maps underlying LVs to files and hence the LVs appear as files to GlusterFS clients. Though BD volume externally appears very similar to the usual Posix volume, not all operations are supported or possible for the files on a BD volume. Only those operations that make sense for a block device are supported and the exact semantics are described in subsequent sections.
+
+While Posix volume takes a file system directory as brick, BD volume needs a volume group (VG) as brick. In the usual use case of BD volume, a file created on BD volume will result in an LV being created in the brick VG. In addition to a VG, BD volume also needs a file system directory that should be specified at the volume creation time. This directory is necessary for supporting the notion of directories and directory hierarchy for the BD volume. Metadata about LVs (size, mapping info) is stored in this directory.
+
+BD xlator was mainly developed to use block devices directly as VM images when GlusterFS is used as storage for KVM virtualization. Some of the salient points of BD xlator are
+
+* Since BD supports file level snapshots and clones by leveraging the snapshot and clone capabilities of LVM, it can be used to fully off-load snapshot and cloning operations from QEMU to the storage (GlusterFS) itself.
+
+* BD understands dm-thin LVs and hence can support files that are backed by thinly provisioned LVs. This capability of BD xlator translates to having thinly provisioned raw VM images.
+
+* BD enables thin LVs from a thin pool to be used from multiple nodes that have visibility to GlusterFS BD volume. Thus thin pool can be used as a VM image repository allowing access/visibility to it from multiple nodes.
+
+* BD supports true zerofill by using BLKZEROOUT ioctl on underlying block devices. Thus BD allows SCSI WRITESAME to be used on underlying block device if the device supports it.
+
+Though BD xlator is primarily intended to be used with block devices, it does provide full Posix xlator compatibility for files that are created on BD volume but are not backed by or mapped to a block device. Such files which don’t have a block device mapping exist on the Posix directory that is specified during BD volume creation. BD xlator is available from GlusterFS-3.5 release.
+
+###Compiling BD translator
+
+BD xlator needs lvm2 development library. –enable-bd-xlator option can be used with `./configure` script to explicitly enable BD translator. The following snippet from the output of configure script shows that BD xlator is enabled for compilation.
+
+
+#####GlusterFS configure summary
+
+ …
+ Block Device xlator  : yes
+
+
+###Creating a BD volume
+
+BD supports hosting of both linear LV and thin LV within the same volume. However seperate examples are provided below. As noted above, the prerequisite for a BD volume is VG which is created from a loop device here, but it can be any other device too.
+
+
+* Creating BD volume with linear LV backend
+
+* Create a loop device
+
+
+    [root@node ~]# dd if=/dev/zero of=bd-loop count=1024 bs=1M
+
+    [root@node ~]# losetup /dev/loop0 bd-loop
+
+
+* Prepare a brick by creating a VG
+
+    [root@node ~]# pvcreate /dev/loop0
+
+    [root@node ~]# vgcreate bd-vg /dev/loop0
+
+
+* Create the BD volume
+
+* Create a POSIX directory first
+
+
+    [root@node ~]# mkdir /bd-meta
+
+It is recommended that this directory is created on an LV in the brick VG itself so that both data and metadata live together on the same device.
+
+
+* Create and mount the volume
+
+    [root@node ~]# gluster volume create bd node:/bd-meta?bd-vg force
+
+
+The general syntax for specifying the brick is `host:/posix-dir?volume-group-name` where “?” is the separator.
+
+
+
+    [root@node ~]# gluster volume start bd
+    [root@node ~]# gluster volume info bd
+    Volume Name: bd
+    Type: Distribute
+    Volume ID: cb042d2a-f435-4669-b886-55f5927a4d7f
+    Status: Started
+    Xlator 1: BD
+    Capability 1: offload_copy
+    Capability 2: offload_snapshot
+    Number of Bricks: 1
+    Transport-type: tcp
+    Bricks:
+    Brick1: node:/bd-meta
+    Brick1 VG: bd-vg
+
+
+
+    [root@node ~]# mount -t glusterfs node:/bd /mnt
+
+* Create a file that is backed by an LV
+
+    [root@node ~]# ls /mnt
+
+    [root@node ~]#
+
+Since the volume is empty now, so is the underlying VG.
+
+    [root@node ~]# lvdisplay bd-vg
+    [root@node ~]#
+
+Creating a file that is mapped to an LV is a 2 step operation. First the file should be created on the mount point and a specific extended attribute should be set to map the file to LV.
+
+    [root@node ~]# touch /mnt/lv
+    [root@node ~]# setfattr -n “user.glusterfs.bd” -v “lv” /mnt/lv
+
+Now an LV got created in the VG brick and the file /mnt/lv maps to this LV. Any read/write to this file ends up as read/write to the underlying LV.
+
+    [root@node ~]# lvdisplay bd-vg
+    — Logical volume —
+    LV Path                 /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287
+    LV Name                 6ff0f25f-2776-4d19-adfb-df1a3cab8287
+    VG Name                 bd-vg
+    LV UUID                 PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6
+    LV Write Access         read/write
+    LV Creation host, time node, 2013-11-26 16:15:45 +0530
+    LV Status               available
+    open                    0
+    LV Size                 4.00 MiB
+    Current LE              1
+    Segments                1
+    Allocation              inherit
+    Read ahead sectors      0
+    Block device            253:6
+
+The file gets created with default LV size which is 1 LE which is 4MB in this case.
+
+    [root@node ~]# ls -lh /mnt/lv
+    -rw-r–r–. 1 root root 4.0M Nov 26 16:15 /mnt/lv
+
+truncate can be used to set the required file size.
+
+    [root@node ~]# truncate /mnt/lv -s 256M
+    [root@node ~]# lvdisplay bd-vg
+    — Logical volume —
+    LV Path               /dev/bd-vg/6ff0f25f-2776-4d19-adfb-df1a3cab8287
+    LV Name               6ff0f25f-2776-4d19-adfb-df1a3cab8287
+    VG Name               bd-vg
+    LV UUID               PjMPcc-RkD5-RADz-6ixG-UYsk-oclz-vL0nv6
+    LV Write Access       read/write
+    LV Creation host, time node, 2013-11-26 16:15:45 +0530
+    LV Status             available
+    # open                   0
+    LV Size               256.00 MiB
+    Current LE             64
+    Segments                1
+    Allocation            inherit
+    Read ahead sectors     0
+    Block device          253:6
+
+
+    [root@node ~]# ls -lh /mnt/lv
+    -rw-r–r–. 1 root root 256M Nov 26 16:15 /mnt/lv
+
+    currently LV size has been set to        256
+
+The size of the file/LV can be specified during creation/mapping time itself like this:
+
+    setfattr -n “user.glusterfs.bd” -v “lv:256MB” /mnt/lv
+
+2. Creating BD volume with thin LV backend
+
+* Create a loop device
+
+
+    [root@node ~]# dd if=/dev/zero of=bd-loop-thin count=1024 bs=1M
+
+    [root@node ~]# losetup /dev/loop0 bd-loop-thin
+
+
+* Prepare a brick by creating a VG and thin pool
+
+
+    [root@node ~]# pvcreate /dev/loop0
+
+    [root@node ~]# vgcreate bd-vg-thin /dev/loop0
+
+
+* Create a thin pool
+
+
+    [root@node ~]# lvcreate –thin bd-vg-thin -L 1000M
+
+    Rounding up size to full physical extent 4.00 MiB
+    Logical volume “lvol0″ created
+
+lvdisplay shows the thin pool
+
+    [root@node ~]# lvdisplay bd-vg-thin
+    — Logical volume —
+    LV Name                      lvol0
+    VG Name                      bd-vg-thin
+    LV UUID                      HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn
+    LV Write Access              read/write
+    LV Creation host, time node, 2013-11-26 16:39:06 +0530
+    LV Pool transaction ID       0
+    LV Pool metadata             lvol0_tmeta
+    LV Pool data                 lvol0_tdata
+    LV Pool chunk size           64.00 KiB
+    LV Zero new blocks           yes
+    LV Status                    available
+    # open                       0
+    LV Size                      1000.00 MiB
+    Allocated pool data          0.00%
+    Allocated metadata           0.88%
+    Current LE                   250
+    Segments                     1
+    Allocation                     inherit
+    Read ahead sectors     auto
+    Block device                253:9
+
+* Create the BD volume
+
+* Create a POSIX directory first
+
+
+    [root@node ~]# mkdir /bd-meta-thin
+
+* Create and mount the volume
+
+    [root@node ~]# gluster volume create bd-thin node:/bd-meta-thin?bd-vg-thin force
+
+    [root@node ~]# gluster volume start bd-thin
+
+
+    [root@node ~]# gluster volume info bd-thin
+    Volume Name: bd-thin
+    Type: Distribute
+    Volume ID: 27aa7eb0-4ffa-497e-b639-7cbda0128793
+    Status: Started
+    Xlator 1: BD
+    Capability 1: thin
+    Capability 2: offload_copy
+    Capability 3: offload_snapshot
+    Number of Bricks: 1
+    Transport-type: tcp
+    Bricks:
+    Brick1: node:/bd-meta-thin
+    Brick1 VG: bd-vg-thin
+
+
+    [root@node ~]# mount -t glusterfs node:/bd-thin /mnt
+
+* Create a file that is backed by a thin LV
+
+
+    [root@node ~]# ls /mnt
+
+    [root@node ~]#
+
+Creating a file that is mapped to a thin LV is a 2 step operation. First the file should be created on the mount point and a specific extended attribute should be set to map the file to a thin LV.
+
+    [root@node ~]# touch /mnt/thin-lv
+ 
+    [root@node ~]# setfattr -n “user.glusterfs.bd” -v “thin:256MB” /mnt/thin-lv
+
+Now /mnt/thin-lv is a thin provisioned file that is backed by a thin LV and size has been set to 256.
+
+    [root@node ~]# lvdisplay bd-vg-thin
+    — Logical volume —
+    LV Name             lvol0
+    VG Name             bd-vg-thin
+    LV UUID             HVa3EM-IVMS-QG2g-oqU6-1UxC-RgqS-g8zhVn
+    LV Write Access     read/write
+    LV Creation host, time node, 2013-11-26 16:39:06 +0530
+    LV Pool transaction ID 1
+    LV Pool metadata    lvol0_tmeta
+    LV Pool data        lvol0_tdata
+    LV Pool chunk size  64.00 KiB
+    LV Zero new blocks  yes
+    LV Status           available
+    # open               0
+    LV Size             000.00 MiB
+    Allocated pool data 0.00%
+    Allocated metadata  0.98%
+    Current LE          250
+    Segments            1
+    Allocation          inherit
+    Read ahead sectors  auto
+    Block device         253:9
+
+
+
+
+    — Logical volume —
+    LV Path             dev/bd-vg-thin/081b01d1-1436-4306-9baf-41c7bf5a2c73
+    LV Name             081b01d1-1436-4306-9baf-41c7bf5a2c73
+    VG Name             bd-vg-thin
+    LV UUID             coxpTY-2UZl-9293-8H2X-eAZn-wSp6-csZIeB
+    LV Write Access     read/write
+    LV Creation host, time node, 2013-11-26 16:43:19 +0530
+    LV Pool name        lvol0
+    LV Status           available
+    # open               0
+    LV Size             256.00 MiB
+    Mapped size          0.00%
+    Current LE           64
+    Segments             1
+    Allocation          inherit
+    Read ahead sectors   auto
+    Block device                 253:10
+
+
+
+
+
+As can be seen from above, creation of a file resulted in creation of a thin LV in the brick.
+
+For more information on bd translator wrt snapshots and clones,
+refer #  http://raobharata.wordpress.com/2013/11/27/glusterfs-block-device-translator/
+
+###Improvisation on BD translator:
+
+First version of BD xlator  ( block backend) had few limitations  such as
+
+* Creation of directories not supported
+* Supports only single brick
+* Does not use extended attributes (and client gfid) like posix xlator
+* Creation of special files (symbolic links, device nodes etc) not
+  supported
+
+Basic limitation of not allowing directory creation was blocking
+oVirt/VDSM to consume BD xlator as part of Gluster domain since VDSM
+creates multi-level directories when GlusterFS is used as storage
+backend for storing VM images.
+
+To overcome these limitations a new BD xlator with following
+improvements are implemented.
+
+* New hybrid BD xlator that handles both regular files and block device
+  files
+* The volume will have both POSIX and BD bricks. Regular files are
+  created on POSIX bricks, block devices are created on the BD brick (VG)
+* BD xlator leverages exiting POSIX xlator for most POSIX calls and
+  hence sits above the POSIX xlator
+* Block device file is differentiated from regular file by an extended
+  attribute
+* The xattr 'user.glusterfs.bd' (BD_XATTR) plays a role in mapping a
+  posix file to Logical Volume (LV).
+* When a client sends a request to set BD_XATTR on a posix file, a new
+  LV is created and mapped to posix file. So every block device will
+  have a representative file in POSIX brick with 'user.glusterfs.bd'
+  (BD_XATTR) set.
+* Here after all operations on this file results in LV related
+  operations.
+
+For example, opening a file that has BD_XATTR set results in opening
+the LV block device, reading results in reading the corresponding LV
+block device.
+
+When BD xlator gets request to set BD_XATTR via setxattr call, it
+creates a LV and information about this LV is placed in the xattr of the
+posix file. xattr "user.glusterfs.bd" used to identify that posix file
+is mapped to BD.
+
+Usage:
+Server side:
+
+    [root@host1 ~]# gluster volume create bdvol host1:/storage/vg1_info?vg1 host2:/storage/vg2_info?vg2
+
+It creates a distributed gluster volume 'bdvol' with Volume Group vg1
+using posix brick /storage/vg1_info in host1 and Volume Group vg2 using
+/storage/vg2_info in host2.
+
+
+    [root@host1 ~]# gluster volume start bdvol
+
+Client side:
+
+    [root@node ~]# mount -t glusterfs host1:/bdvol /media
+    [root@node ~]# touch /media/posix
+
+It creates regular posix file 'posix' in either host1:/vg1 or host2:/vg2 brick
+
+    [root@node ~]# mkdir /media/image
+
+    [root@node ~]# touch /media/image/lv1
+
+
+It also creates regular posix file 'lv1' in either host1:/vg1 or
+host2:/vg2 brick
+
+    [root@node ~]# setfattr -n "user.glusterfs.bd" -v "lv" /media/image/lv1
+
+    [root@node ~]#
+
+
+Above setxattr results in creating a new LV in corresponding brick's VG
+and it sets 'user.glusterfs.bd' with value 'lv:<default-extent-size''
+
+
+    [root@node ~]# truncate -s5G /media/image/lv1
+
+
+It results in resizig LV 'lv1'to 5G
+
+New BD xlator code is placed in `xlators/storage/bd` directory.
+
+Also add volume-uuid to the VG so that same VG cannot be used for other
+bricks/volumes. After deleting a gluster volume, one has to manually
+remove the associated tag using vgchange <vg-name> --deltag
+`<trusted.glusterfs.volume-id:<volume-id>>`
+