summaryrefslogtreecommitdiffstats
path: root/done/GlusterFS 3.5/libgfapi with qemu libvirt.md
blob: 2309016d5a2926a9f14a949d101836517b6df666 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
        Work In Progress
        Author - Satheesaran Sundaramoorthi
        <sasundar@redhat.com>

**Purpose**
-----------

Gluster volume can be used to store VM Disk images. This usecase is
popularly known as 'Virt-Store' usecase. Earlier, gluster volume had to
be fuse mounted and images are created/accessed over the fuse mount.

With the introduction of GlusterFS libgfapi, QEMU supports glusterfs
through libgfapi directly. This we call as *QEMU driver for glusterfs*.
These document explains about the way to make use of QEMU driver for
glusterfs

Steps for the entire procedure could be split in to 2 views viz,the
document from

1.  Steps to be done on gluster volume side
2.  Steps to be done on Hypervisor side

**Steps to be done on gluster side**
------------------------------------

These are the steps that needs to be done on the gluster side Precisely
this involves

1.  Creating "Trusted Storage Pool"
2.  Creating a volume
3.  Tuning the volume for virt-store
4.  Tuning glusterd to accept requests from QEMU
5.  Tuning glusterfsd to accept requests from QEMU
6.  Setting ownership on the volume
7.  Starting the volume

##### Creating "Trusted Storage Pool"

Install glusterfs rpms on the NODE. You can create a volume with a
single node. You can also scale up the cluster, as we call as *Trusted
Storage Pool*, by adding more nodes to the cluster

        gluster peer probe <hostname>

##### Creating a volume

It is highly recommended to have replicate volume or
distribute-replicate volume for virt-store usecase, as it would add high
availability and fault-tolerance. Remember the plain distribute works
equally well

        gluster volume create replica 2 <brick1> .. <brickN>

where, <brick1> is <hostname>:/<path-of-dir> Note: It is recommended to
create sub-directories inside brick and that could be used to create a
volume.For example, say, */home/brick1* is the mountpoint of XFS, then
you can create a sub-directory inside it */home/brick1/b1* and use it
while creating a volume.You can also use space available in root
filesystem for bricks. Gluster cli, by default, throws warning in that
case. You can override by using *force* option

        gluster volume create replica 2 <brick1> .. <brickN> force

*If you are new to GlusterFS, you can take a look at
[QuickStart](http://gluster.readthedocs.org/en/latest/Quick-Start-Guide/Quickstart/) guide.*

##### Tuning the volume for virt-store

There are recommended settings available for virt-store. This provide
good performance characteristics when enabled on the volume that was
used for *virt-store*

Refer to
[Virt-store-usecase\#Tunables](Virt-store-usecase#Tunables "wikilink")
for recommended tunables and for applying them on the volume,
[Virt-store-usecase\#Applying\_the\_Tunables\_on\_the\_volume](Virt-store-usecase#Applying_the_Tunables_on_the_volume "wikilink")

##### Tuning glusterd to accept requests from QEMU

glusterd receives the request only from the applications that run with
port number less than 1024 and it blocks otherwise. QEMU uses port
number greater than 1024 and to make glusterd accept requests from QEMU,
edit the glusterd vol file, */etc/glusterfs/glusterd.vol* and add the
following,

        option rpc-auth-allow-insecure on

Note: If you have installed glusterfs from source, you can find glusterd
vol file at */usr/local/etc/glusterfs/glusterd.vol*

Restart glusterd after adding that option to glusterd vol file

        service glusterd restart

##### Tuning glusterfsd to accept requests from QEMU

Enable the option *allow-insecure* on the particular volume

        gluster volume set <volname> server.allow-insecure on

**IMPORTANT :** As of now(april 2,2014)there is a bug, as
*allow-insecure* is not dynamically set on a volume.You need to restart
the volume for the change to take effect

##### Setting ownership on the volume

Set the ownership of qemu:qemu on to the volume

        gluster volume set <vol-name> storage.owner-uid 107
        gluster volume set <vol-name> storage.owner-gid 107

**IMPORTANT :** The UID and GID can differ per Linux distribution, or
even installation. The UID/GID should be the one fomr the *qemu* or
'kvm'' user, you can get the IDs with these commands:

        id qemu
        getent group kvm

##### Starting the volume

Start the volume

        gluster volume start <vol-name>

**Steps to be done on Hypervisor Side**
---------------------------------------

Hypervisor is just the machine which spawns the Virtual Machines. This
machines should be necessarily the baremetal with more memory and
computing power. The following steps needs to be done on hypervisor,

1.  Install qemu-kvm
2.  Install libvirt
3.  Create a VM Image
4.  Add ownership to the Image file
5.  Create libvirt XML to define Virtual Machine
6.  Define the VM
7.  Start the VM
8.  Verification

##### Install qemu-kvm

##### Install libvirt

##### Create a VM Image

Images can be created using *qemu-img* utility

        qemu-img create -f <format> gluster://<server>/<vol-name>/ <image> <size>

-  format - This can be raw or qcow2
-  server - One of the gluster Node's IP or FQDN
-  vol-name - gluster volume name
-  image - Image File name
-  size - Size of the image

Here is sample,

        qemu-img create -f qcow2 gluster://host.sample.com/vol1/vm1.img 10G

##### Add ownership to the Image file

NFS or FUSE mount the glusterfs volume and change the ownership of the
image file to qemu:qemu

        mount -t nfs -o vers=3 <gluster-server>:/<vol-name> <mount-point>

Change the ownership of the image file that was earlier created using
*qemu-img* utility

        chown qemu:qemu <mount-point>/<image-file>

##### Create libvirt XML to define Virtual Machine

*virt-install* is python wrapper which is mostly used to create VM using
set of params. *virt-install* doesn't support any network filesystem [
<https://bugzilla.redhat.com/show_bug.cgi?id=1017308> ]

Create a libvirt xml - <http://libvirt.org/formatdomain.html> See to
that the disk section is formatted in such a way, qemu driver for
glusterfs is being used. This can be seen in the following example xml
description

        <disk type='network' device='disk'>
           <driver name='qemu' type='raw' cache='none'/>
           <source protocol='gluster' name='distrepvol/vm3.img'>
                <host name='10.70.37.106' port='24007'/>
            </source>
           <target dev='vda' bus='virtio'/>
           <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
        </disk>

##### Define the VM from XML

Define the VM from the XML file that was created earlier

        virsh define <xml-file-description>

Verify that the VM is created successfully

        virsh list --all

##### Start the VM

Start the VM

        virsh start <VM>

##### Verification

You can verify the disk image file that is being used by VM

        virsh domblklist <VM-Domain-Name/ID>

The above should show the volume name and image name. Here is the
example,

        [root@test ~]# virsh domblklist vm-test2
        Target     Source
        ------------------------------------------------
        vda        distrepvol/test.img
        hdc        -