Work In Progress Author - Satheesaran Sundaramoorthi **Purpose** ----------- Gluster volume can be used to store VM Disk images. This usecase is popularly known as 'Virt-Store' usecase. Earlier, gluster volume had to be fuse mounted and images are created/accessed over the fuse mount. With the introduction of GlusterFS libgfapi, QEMU supports glusterfs through libgfapi directly. This we call as *QEMU driver for glusterfs*. These document explains about the way to make use of QEMU driver for glusterfs Steps for the entire procedure could be split in to 2 views viz,the document from 1. Steps to be done on gluster volume side 2. Steps to be done on Hypervisor side **Steps to be done on gluster side** ------------------------------------ These are the steps that needs to be done on the gluster side Precisely this involves 1. Creating "Trusted Storage Pool" 2. Creating a volume 3. Tuning the volume for virt-store 4. Tuning glusterd to accept requests from QEMU 5. Tuning glusterfsd to accept requests from QEMU 6. Setting ownership on the volume 7. Starting the volume ##### Creating "Trusted Storage Pool" Install glusterfs rpms on the NODE. You can create a volume with a single node. You can also scale up the cluster, as we call as *Trusted Storage Pool*, by adding more nodes to the cluster gluster peer probe  ##### Creating a volume It is highly recommended to have replicate volume or distribute-replicate volume for virt-store usecase, as it would add high availability and fault-tolerance. Remember the plain distribute works equally well gluster volume create replica 2 .. where, is :/ Note: It is recommended to create sub-directories inside brick and that could be used to create a volume.For example, say, */home/brick1* is the mountpoint of XFS, then you can create a sub-directory inside it */home/brick1/b1* and use it while creating a volume.You can also use space available in root filesystem for bricks. Gluster cli, by default, throws warning in that case. You can override by using *force* option gluster volume create replica 2 .. force *If you are new to GlusterFS, you can take a look at [QuickStart](http://gluster.readthedocs.org/en/latest/Quick-Start-Guide/Quickstart/) guide.* ##### Tuning the volume for virt-store There are recommended settings available for virt-store. This provide good performance characteristics when enabled on the volume that was used for *virt-store* Refer to [Virt-store-usecase\#Tunables](Virt-store-usecase#Tunables "wikilink") for recommended tunables and for applying them on the volume, [Virt-store-usecase\#Applying\_the\_Tunables\_on\_the\_volume](Virt-store-usecase#Applying_the_Tunables_on_the_volume "wikilink") ##### Tuning glusterd to accept requests from QEMU glusterd receives the request only from the applications that run with port number less than 1024 and it blocks otherwise. QEMU uses port number greater than 1024 and to make glusterd accept requests from QEMU, edit the glusterd vol file, */etc/glusterfs/glusterd.vol* and add the following, option rpc-auth-allow-insecure on Note: If you have installed glusterfs from source, you can find glusterd vol file at */usr/local/etc/glusterfs/glusterd.vol* Restart glusterd after adding that option to glusterd vol file service glusterd restart ##### Tuning glusterfsd to accept requests from QEMU Enable the option *allow-insecure* on the particular volume gluster volume set  server.allow-insecure on **IMPORTANT :** As of now(april 2,2014)there is a bug, as *allow-insecure* is not dynamically set on a volume.You need to restart the volume for the change to take effect ##### Setting ownership on the volume Set the ownership of qemu:qemu on to the volume gluster volume set  storage.owner-uid 107 gluster volume set  storage.owner-gid 107 **IMPORTANT :** The UID and GID can differ per Linux distribution, or even installation. The UID/GID should be the one fomr the *qemu* or 'kvm'' user, you can get the IDs with these commands: id qemu getent group kvm ##### Starting the volume Start the volume gluster volume start  **Steps to be done on Hypervisor Side** --------------------------------------- Hypervisor is just the machine which spawns the Virtual Machines. This machines should be necessarily the baremetal with more memory and computing power. The following steps needs to be done on hypervisor, 1. Install qemu-kvm 2. Install libvirt 3. Create a VM Image 4. Add ownership to the Image file 5. Create libvirt XML to define Virtual Machine 6. Define the VM 7. Start the VM 8. Verification ##### Install qemu-kvm ##### Install libvirt ##### Create a VM Image Images can be created using *qemu-img* utility qemu-img create -f gluster://// - format - This can be raw or qcow2 - server - One of the gluster Node's IP or FQDN - vol-name - gluster volume name - image - Image File name - size - Size of the image Here is sample, qemu-img create -f qcow2 gluster://host.sample.com/vol1/vm1.img 10G ##### Add ownership to the Image file NFS or FUSE mount the glusterfs volume and change the ownership of the image file to qemu:qemu mount -t nfs -o vers=3 :/  Change the ownership of the image file that was earlier created using *qemu-img* utility chown qemu:qemu / ##### Create libvirt XML to define Virtual Machine *virt-install* is python wrapper which is mostly used to create VM using set of params. *virt-install* doesn't support any network filesystem [ ] Create a libvirt xml - See to that the disk section is formatted in such a way, qemu driver for glusterfs is being used. This can be seen in the following example xml description
##### Define the VM from XML Define the VM from the XML file that was created earlier virsh define  Verify that the VM is created successfully virsh list --all ##### Start the VM Start the VM virsh start  ##### Verification You can verify the disk image file that is being used by VM virsh domblklist  The above should show the volume name and image name. Here is the example, [root@test ~]# virsh domblklist vm-test2 Target Source ------------------------------------------------ vda distrepvol/test.img hdc -