1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
|
<?xml version='1.0' encoding='UTF-8'?>
<!-- This document was created with Syntext Serna Free. --><!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
<!ENTITY % BOOK_ENTITIES SYSTEM "Administration_Guide.ent">
%BOOK_ENTITIES;
]>
<chapter id="chap-Administration_Guide-Managing_Volumes">
<title>Managing GlusterFS Volumes</title>
<para>This section describes how to perform common GlusterFS management operations, including the following: </para>
<itemizedlist>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Tuning"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Expanding"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Shrinking"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Migrating"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Rebalancing"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Stop"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Delete"/></para>
</listitem>
<listitem>
<para><xref linkend="sect-Administration_Guide-Managing_Volumes-Self_heal"/></para>
</listitem>
</itemizedlist>
<section id="sect-Administration_Guide-Managing_Volumes-Tuning">
<title>Tuning Volume Options</title>
<para>You can tune volume options, as needed, while the cluster is online and available. </para>
<para><note>
<para>Red Hat recommends you to set server.allow-insecure option to ON if there are too many bricks in each volume or if there are too many services which have already utilized all the privileged ports in the system. Turning this option ON allows ports to accept/reject messages from insecure ports. So, use this option only if your deployment requires it. </para>
</note></para>
<para>To tune volume options </para>
<itemizedlist>
<listitem>
<para>Tune volume options using the following command:</para>
<para><command># gluster volume set <replaceable>VOLNAME OPTION PARAMETER</replaceable></command></para>
<para>For example, to specify the performance cache size for test-volume:</para>
<para><programlisting># gluster volume set test-volume performance.cache-size 256MB
Set volume successful</programlisting></para>
<para>The following table lists the Volume options along with its description and default value: </para>
<para><note>
<para>The default options given here are subject to modification at any given time and may not be the same for all versions.</para>
</note></para>
<informaltable frame="all">
<tgroup cols="4">
<colspec colname="c1"/>
<colspec colname="c2"/>
<colspec colname="c3"/>
<colspec colname="c4"/>
<thead>
<row>
<entry>Option</entry>
<entry>Description</entry>
<entry>Default Value</entry>
<entry>Available Options</entry>
</row>
</thead>
<tbody>
<row>
<entry>auth.allow</entry>
<entry>IP addresses of the clients which should be allowed to access the volume. </entry>
<entry>* (allow all)</entry>
<entry>Valid IP address which includes wild card patterns including *, such as 192.168.1.*</entry>
</row>
<row>
<entry>auth.reject</entry>
<entry>IP addresses of the clients which should be denied to access the volume. </entry>
<entry>NONE (reject none) </entry>
<entry>Valid IP address which includes wild card patterns including *, such as 192.168.2.*</entry>
</row>
<row>
<entry>client.grace-timeout</entry>
<entry>Specifies the duration for the lock state to be maintained on the client after a network disconnection.</entry>
<entry>10 </entry>
<entry>10 - 1800 secs</entry>
</row>
<row>
<entry>cluster.self-heal-window-size</entry>
<entry>Specifies the maximum number of blocks per file on which self-heal would happen simultaneously. </entry>
<entry>16 </entry>
<entry>0 - 1025 blocks</entry>
</row>
<row>
<entry>cluster.data-self-heal-algorithm</entry>
<entry>Specifies the type of self-heal. If you set the option as "full", the entire file is copied from source to destinations. If the option is set to "diff" the file blocks that are not in sync are copied to destinations. Reset uses a heuristic model. If the file does not exist on one of the subvolumes, or a zero-byte file exists (created by entry self-heal) the entire content has to be copied anyway, so there is no benefit from using the "diff" algorithm. If the file size is about the same as page size, the entire file can be read and written with a few operations, which will be faster than "diff" which has to read checksums and then read and write. </entry>
<entry>reset</entry>
<entry>full | diff | reset</entry>
</row>
<row>
<entry>cluster.min-free-disk</entry>
<entry>Specifies the percentage of disk space that must be kept free. Might be useful for non-uniform bricks. </entry>
<entry>10%</entry>
<entry>Percentage of required minimum free disk space</entry>
</row>
<row>
<entry>cluster.stripe-block-size</entry>
<entry>Specifies the size of the stripe unit that will be read from or written to. </entry>
<entry>128 KB (for all files)</entry>
<entry>size in bytes</entry>
</row>
<row>
<entry>cluster.self-heal-daemon</entry>
<entry>Allows you to turn-off proactive self-heal on replicated volumes.</entry>
<entry>on</entry>
<entry>On | Off</entry>
</row>
<row>
<entry>cluster.ensure-durability</entry>
<entry>This option makes sure the data/metadata is durable across abrupt shutdown of the brick. </entry>
<entry>on</entry>
<entry>On | Off</entry>
</row>
<row>
<entry>diagnostics.brick-log-level</entry>
<entry>Changes the log-level of the bricks. </entry>
<entry>INFO </entry>
<entry>DEBUG|WARNING|ERROR|CRITICAL|NONE|TRACE</entry>
</row>
<row>
<entry>diagnostics.client-log-level</entry>
<entry>Changes the log-level of the clients. </entry>
<entry>INFO </entry>
<entry>DEBUG|WARNING|ERROR|CRITICAL|NONE|TRACE</entry>
</row>
<row>
<entry>diagnostics.latency-measurement</entry>
<entry>Statistics related to the latency of each operation would be tracked. </entry>
<entry>off </entry>
<entry>On | Off</entry>
</row>
<row>
<entry>diagnostics.dump-fd-stats</entry>
<entry>Statistics related to file-operations would be tracked.</entry>
<entry>off </entry>
<entry>On | Off</entry>
</row>
<row>
<entry>feature.read-only</entry>
<entry>Enables you to mount the entire volume as read-only for all the clients (including NFS clients) accessing it.</entry>
<entry>off</entry>
<entry>On | Off</entry>
</row>
<row>
<entry>features.lock-heal</entry>
<entry>Enables self-healing of locks when the network disconnects.</entry>
<entry>on</entry>
<entry>On | Off</entry>
</row>
<row>
<entry>features.quota-timeout</entry>
<entry>For performance reasons, quota caches the directory sizes on client. You can set timeout indicating the maximum duration of directory sizes in cache, from the time they are populated, during which they are considered valid. </entry>
<entry>0</entry>
<entry>0 - 3600 secs</entry>
</row>
<row>
<entry>geo-replication.indexing</entry>
<entry>Use this option to automatically sync the changes in the filesystem from Master to Slave.</entry>
<entry>off </entry>
<entry>On | Off</entry>
</row>
<row>
<entry>network.frame-timeout</entry>
<entry>The time frame after which the operation has to be declared as dead, if the server does not respond for a particular operation. </entry>
<entry>1800 (30 mins) </entry>
<entry>1800 secs</entry>
</row>
<row>
<entry>network.ping-timeout</entry>
<entry>The time duration for which the client waits to check if the server is responsive. When a ping timeout happens, there is a network disconnect between the client and server. All resources held by server on behalf of the client get cleaned up. When a reconnection happens, all resources will need to be re-acquired before the client can resume its operations on the server. Additionally, the locks will be acquired and the lock tables updated. <para>This reconnect is a very expensive operation and should be avoided.
</para></entry>
<entry>42 Secs</entry>
<entry>42 Secs</entry>
</row>
<row>
<entry>nfs.enable-ino32</entry>
<entry>For 32-bit nfs clients or applications that do not support 64-bit inode numbers or large files, use this option from the CLI to make Gluster NFS return 32-bit inode numbers instead of 64-bit inode numbers. Applications that will benefit are those that were either: <para>* Built 32-bit and run on 32-bit machines.</para><para>* Built 32-bit on 64-bit systems.</para><para>* Built 64-bit but use a library built 32-bit, especially relevant for python and perl scripts.</para><para>Either of the conditions above can lead to application on Linux NFS clients failing with "Invalid argument" or "Value too large for defined data type" errors.</para></entry>
<entry>off</entry>
<entry>On | Off</entry>
</row>
<row>
<entry>nfs.volume-access </entry>
<entry>Set the access type for the specified sub-volume. </entry>
<entry>read-write </entry>
<entry>read-write|read-only </entry>
</row>
<row>
<entry>nfs.trusted-write </entry>
<entry>If there is an UNSTABLE write from the client, STABLE flag will be returned to force the client to not send a COMMIT request. <para>In some environments, combined with a replicated GlusterFS setup, this option can improve write performance. This flag allows users to trust Gluster replication logic to sync data to the disks and recover when required. COMMIT requests if received will be handled in a default manner by fsyncing. STABLE writes are still handled in a sync manner.</para></entry>
<entry> off </entry>
<entry>On | Off </entry>
</row>
<row>
<entry>nfs.trusted-sync</entry>
<entry> All writes and COMMIT requests are treated as async. This implies that no write requests are guaranteed to be on server disks when the write reply is received at the NFS client. Trusted sync includes trusted-write behavior. </entry>
<entry>off </entry>
<entry>On | Off </entry>
</row>
<row>
<entry>nfs.export-dir </entry>
<entry>By default, all sub-volumes of NFS are exported as individual exports. Now, this option allows you to export only the specified subdirectory or subdirectories in the volume. This option can also be used in conjunction with nfs3.export-volumes option to restrict exports only to the subdirectories specified through this option. You must provide an absolute path.</entry>
<entry>Enabled for all sub directories.</entry>
<entry>Enable | Disable </entry>
</row>
<row>
<entry>nfs.export-volumes </entry>
<entry>Enable/Disable exporting entire volumes, instead if used in conjunction with nfs3.export-dir, can allow setting up only subdirectories as exports. </entry>
<entry>on</entry>
<entry> On | Off </entry>
</row>
<row>
<entry>nfs.rpc-auth-unix </entry>
<entry>Enable/Disable the AUTH_UNIX authentication type. This option is enabled by default for better interoperability. However, you can disable it if required.</entry>
<entry>on </entry>
<entry> On | Off </entry>
</row>
<row>
<entry>nfs.rpc-auth-null </entry>
<entry>Enable/Disable the AUTH_NULL authentication type. It is not recommended to change the default value for this option. </entry>
<entry>on </entry>
<entry> On | Off </entry>
</row>
<row>
<entry>nfs.rpc-auth-allow<IP- Addresses> </entry>
<entry>Allow a comma separated list of addresses and/or hostnames to connect to the server. By default, all clients are disallowed. This allows you to define a general rule for all exported volumes.</entry>
<entry>Reject All </entry>
<entry>IP address or Host name </entry>
</row>
<row>
<entry>nfs.rpc-auth-reject IP- Addresses </entry>
<entry>Reject a comma separated list of addresses and/or hostnames from connecting to the server. By default, all connections are disallowed. This allows you to define a general rule for all exported volumes.</entry>
<entry>Reject All </entry>
<entry>IP address or Host name </entry>
</row>
<row>
<entry>nfs.ports-insecure </entry>
<entry>Allow client connections from unprivileged ports. By default only privileged ports are allowed. This is a global setting in case insecure ports are to be enabled for all exports using a single option. </entry>
<entry>off</entry>
<entry> On | Off </entry>
</row>
<row>
<entry>nfs.addr-namelookup </entry>
<entry>Turn-off name lookup for incoming client connections using this option. In some setups, the name server can take too long to reply to DNS queries resulting in timeouts of mount requests. Use this option to turn off name lookups during address authentication. Note, turning this off will prevent you from using hostnames in rpc-auth.addr.* filters. </entry>
<entry>on </entry>
<entry> On | Off </entry>
</row>
<row>
<entry>nfs.register-with- portmap </entry>
<entry>For systems that need to run multiple NFS servers, you need to prevent more than one from registering with portmap service. Use this option to turn off portmap registration for Gluster NFS. </entry>
<entry>on </entry>
<entry> On | Off </entry>
</row>
<row>
<entry>nfs.port <PORT- NUMBER> </entry>
<entry>Use this option on systems that need Gluster NFS to be associated with a non-default port number. </entry>
<entry>38465- 38467 </entry>
<entry/>
</row>
<row>
<entry>nfs.disable</entry>
<entry>Turn-off volume being exported by NFS</entry>
<entry> off </entry>
<entry>On | Off </entry>
</row>
<row>
<entry>performance.write-behind-window-size </entry>
<entry>Size of the per-file write-behind buffer.</entry>
<entry>1 MB </entry>
<entry>Write-behind cache size </entry>
</row>
<row>
<entry>performance.io-thread-count </entry>
<entry>The number of threads in IO threads translator. </entry>
<entry>16</entry>
<entry>0 - 65 </entry>
</row>
<row>
<entry>performance.flush-behind </entry>
<entry>If this option is set ON, instructs write-behind translator to perform flush in background, by returning success (or any errors, if any of previous writes were failed) to application even before flush is sent to backend filesystem. </entry>
<entry>On </entry>
<entry>On | Off </entry>
</row>
<row>
<entry>performance.cache-max-file-size </entry>
<entry>Sets the maximum file size cached by the io-cache translator. Can use the normal size descriptors of KB, MB, GB,TB or PB (for example, 6GB). Maximum size uint64. </entry>
<entry>2 ^ 64 -1 bytes </entry>
<entry>size in bytes </entry>
</row>
<row>
<entry>performance.cache-min-file-size </entry>
<entry> Sets the minimum file size cached by the io-cache translator. Values same as "max" above.</entry>
<entry>0B</entry>
<entry>size in bytes </entry>
</row>
<row>
<entry>performance.cache-refresh-timeout </entry>
<entry>The cached data for a file will be retained till 'cache-refresh-timeout' seconds, after which data re-validation is performed. </entry>
<entry>1 sec </entry>
<entry>0 - 61 </entry>
</row>
<row>
<entry>performance.cache-size </entry>
<entry>Size of the read cache.</entry>
<entry> 32 MB </entry>
<entry>size in bytes </entry>
</row>
<row>
<entry>server.allow-insecure </entry>
<entry>Allow client connections from unprivileged ports. By default only privileged ports are allowed. This is a global setting in case insecure ports are to be enabled for all exports using a single option. </entry>
<entry>on </entry>
<entry>On | Off </entry>
</row>
<row>
<entry>server.grace-timeout</entry>
<entry>Specifies the duration for the lock state to be maintained on the server after a network disconnection.</entry>
<entry>10</entry>
<entry>10 - 1800 secs</entry>
</row>
<row>
<entry>server.statedump-path </entry>
<entry>Location of the state dump file. </entry>
<entry>/tmp directory of the brick </entry>
<entry>New directory path</entry>
</row>
</tbody>
</tgroup>
</informaltable>
<para>You can view the changed volume options using the<command> # gluster volume info <replaceable>VOLNAME</replaceable></command> command. For more information, see <xref linkend="sect-Administration_Guide-Managing_Volumes-Delete"/>.</para>
</listitem>
</itemizedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Expanding">
<title>Expanding Volumes</title>
<para>You can expand volumes, as needed, while the cluster is online and available. For example, you might want to add a brick to a distributed volume, thereby increasing the distribution and adding to the capacity of the GlusterFS volume. </para>
<para>Similarly, you might want to add a group of bricks to a distributed replicated volume, increasing the capacity of the GlusterFS volume. </para>
<para><note>
<para>When expanding distributed replicated and distributed striped volumes, you need to add a number of bricks that is a multiple of the replica or stripe count. For example, to expand a distributed replicated volume with a replica count of 2, you need to add bricks in multiples of 2 (such as 4, 6, 8, etc.). </para>
</note></para>
<para><emphasis role="bold">To expand a volume</emphasis> </para>
<orderedlist>
<listitem>
<para>On the first server in the cluster, probe the server to which you want to add the new brick using the following command:</para>
<para><command># gluster peer probe <replaceable>HOSTNAME</replaceable></command></para>
<para>For example:</para>
<para><programlisting># gluster peer probe server4
Probe successful</programlisting></para>
</listitem>
<listitem>
<para>Add the brick using the following command: </para>
<para><command># gluster volume add-brick <replaceable>VOLNAME NEW-BRICK</replaceable></command></para>
<para>For example:</para>
<para><programlisting># gluster volume add-brick test-volume server4:/exp4
Add Brick successful</programlisting></para>
</listitem>
<listitem>
<para>Check the volume information using the following command: </para>
<para><command># gluster volume info </command></para>
<para>The command displays information similar to the following:</para>
<para><programlisting>Volume Name: test-volume
Type: Distribute
Status: Started
Number of Bricks: 4
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server3:/exp3
Brick4: server4:/exp4</programlisting></para>
</listitem>
<listitem>
<para>Rebalance the volume to ensure that all files are distributed to the new brick.</para>
<para>You can use the rebalance command as described in <xref linkend="sect-Administration_Guide-Managing_Volumes-Rebalancing"/>.</para>
</listitem>
</orderedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Shrinking">
<title>Shrinking Volumes</title>
<para>You can shrink volumes, as needed, while the cluster is online and available. For example, you might need to remove a brick that has become inaccessible in a distributed volume due to hardware or network failure. </para>
<para><note>
<para>Data residing on the brick that you are removing will no longer be accessible at the Gluster mount point. Note however that only the configuration information is removed - you can continue to access the data directly from the brick, as necessary. </para>
</note></para>
<para>When shrinking distributed replicated and distributed striped volumes, you need to remove a number of bricks that is a multiple of the replica or stripe count. For example, to shrink a distributed striped volume with a stripe count of 2, you need to remove bricks in multiples of 2 (such as 4, 6, 8, etc.). In addition, the bricks you are trying to remove must be from the same sub-volume (the same replica or stripe set). </para>
<para><emphasis role="bold">To shrink a volume</emphasis> </para>
<orderedlist>
<listitem>
<para>Remove the brick using the following command:</para>
<para><command># gluster volume remove-brick <varname>VOLNAME</varname><replaceable> BRICK</replaceable></command> <command>start</command></para>
<para>For example, to remove server2:/exp2:</para>
<para><programlisting># gluster volume remove-brick test-volume server2:/exp2
Removing brick(s) can result in data loss. Do you want to Continue? (y/n)</programlisting></para>
</listitem>
<listitem>
<para>Enter "y" to confirm the operation. The command displays the following message indicating that the remove brick operation is successfully started: </para>
<para><programlisting>Remove Brick successful </programlisting></para>
</listitem>
<listitem>
<para>(Optional) View the status of the remove brick operation using the following command:</para>
<para><command># gluster volume remove-brick <varname>VOLNAME</varname><replaceable> BRICK</replaceable></command><command> status</command></para>
<para>For example, to view the status of remove brick operation on server2:/exp2 brick:</para>
<para><screen># gluster volume remove-brick test-volume server2:/exp2 status
Node Rebalanced-files size scanned status
--------- ---------------- ---- ------- -----------
617c923e-6450-4065-8e33-865e28d9428f 34 340 162 in progress</screen></para>
</listitem>
<listitem>
<para>Check the volume information using the following command: </para>
<para><command># gluster volume info </command></para>
<para>The command displays information similar to the following:</para>
<para><programlisting># gluster volume info
Volume Name: test-volume
Type: Distribute
Status: Started
Number of Bricks: 3
Bricks:
Brick1: server1:/exp1
Brick3: server3:/exp3
Brick4: server4:/exp4</programlisting></para>
</listitem>
<listitem>
<para>Rebalance the volume to ensure that all files are distributed to the new brick.</para>
<para>You can use the rebalance command as described in <xref linkend="sect-Administration_Guide-Managing_Volumes-Rebalancing"/>.</para>
</listitem>
</orderedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Migrating">
<title>Migrating Volumes</title>
<para>You can migrate the data from one brick to another, as needed, while the cluster is online and available. </para>
<para><emphasis role="bold">To migrate a volume</emphasis> </para>
<orderedlist>
<listitem>
<para>Make sure the new brick, server5 in this example, is successfully added to the cluster.</para>
<para>For more information, see <xref linkend="sect-Administration_Guide-Storage_Pools-Adding_Servers"/>.</para>
</listitem>
<listitem>
<para>Migrate the data from one brick to another using the following command:</para>
<para><command> # gluster volume replace-brick <code>VOLNAME</code><code> BRICK</code><code>NEW-BRICK</code> start</command></para>
<para>For example, to migrate the data in server3:/exp3 to server5:/exp5 in test-volume:</para>
<para><programlisting># gluster volume replace-brick test-volume server3:/exp3 server5:exp5 start
Replace brick start operation successful</programlisting></para>
<para><note>
<para>You need to have the FUSE package installed on the server on which you are running the replace-brick command for the command to work.</para>
</note></para>
</listitem>
<listitem>
<para>To pause the migration operation, if needed, use the following command: </para>
<para><command># gluster volume replace-brick <varname>VOLNAME BRICK NEW-BRICK </varname> pause </command></para>
<para>For example, to pause the data migration from server3:/exp3 to server5:/exp5 in test-volume:</para>
<para><programlisting># gluster volume replace-brick test-volume server3:/exp3 server5:exp5 pause
Replace brick pause operation successful</programlisting></para>
</listitem>
<listitem>
<para>To abort the migration operation, if needed, use the following command: </para>
<para><command> # gluster volume replace-brick <varname>VOLNAME BRICK NEW-BRICK </varname>abort </command></para>
<para>For example, to abort the data migration from server3:/exp3 to server5:/exp5 in test-volume:</para>
<para><programlisting># gluster volume replace-brick test-volume server3:/exp3 server5:exp5 abort
Replace brick abort operation successful</programlisting></para>
</listitem>
<listitem>
<para>Check the status of the migration operation using the following command: </para>
<para><command> # gluster volume replace-brick <varname>VOLNAME BRICK NEW-BRICK </varname>status </command></para>
<para>For example, to check the data migration status from server3:/exp3 to server5:/exp5 in test-volume:</para>
<para><programlisting># gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 status
Current File = /usr/src/linux-headers-2.6.31-14/block/Makefile
Number of files migrated = 10567
Migration complete</programlisting></para>
<para>The status command shows the current file being migrated along with the current total number of files migrated. After completion of migration, it displays Migration complete.</para>
</listitem>
<listitem>
<para>Commit the migration of data from one brick to another using the following command: </para>
<para><command> # gluster volume replace-brick <varname>VOLNAME BRICK NEW-BRICK </varname>commit </command></para>
<para>For example, to commit the data migration from server3:/exp3 to server5:/exp5 in test-volume:</para>
<para><programlisting># gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 commit
replace-brick commit successful</programlisting></para>
</listitem>
<listitem>
<para>Verify the migration of brick by viewing the volume info using the following command: </para>
<para><command># gluster volume info <code>VOLNAME</code></command></para>
<para>For example, to check the volume information of new brick server5:/exp5 in test-volume:</para>
<para><programlisting># gluster volume info test-volume
Volume Name: testvolume
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server4:/exp4
Brick4: server5:/exp5
The new volume details are displayed.
</programlisting></para>
<para>The new volume details are displayed.</para>
<para>In the above example, previously, there were bricks; 1,2,3, and 4 and now brick 3 is replaced by brick 5.</para>
</listitem>
</orderedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Rebalancing">
<title>Rebalancing Volumes</title>
<para>After expanding or shrinking a volume (using the add-brick and remove-brick commands respectively), you need to rebalance the data among the servers. New directories created after expanding or shrinking of the volume will be evenly distributed automatically. For all the existing directories, the distribution can be fixed by rebalancing the layout and/or data. </para>
<para>This section describes how to rebalance GlusterFS volumes in your storage environment, using the following common scenarios: </para>
<itemizedlist>
<listitem>
<para>Fix Layout - Fixes the layout changes so that the files can actually go to newly added nodes. For more information, see <xref linkend="sect-Administration_Guide-Managing_Volumes-Rebalancing-Fix_Layout"/>. </para>
</listitem>
<listitem>
<para>Fix Layout and Migrate Data - Rebalances volume by fixing the layout changes and migrating the existing data. For more information, see <xref linkend="sect-Administration_Guide-Managing_Volumes-Rebalancing-Fix_Migrate"/>.</para>
</listitem>
</itemizedlist>
<section id="sect-Administration_Guide-Managing_Volumes-Rebalancing-Fix_Layout">
<title>Rebalancing Volume to Fix Layout Changes</title>
<para>Fixing the layout is necessary because the layout structure is static for a given directory. In a scenario where new bricks have been added to the existing volume, newly created files in existing directories will still be distributed only among the old bricks. The <command># gluster volume rebalance<varname> VOLNAME</varname> fix-layout start </command>command will fix the layout information so that the files can also go to newly added nodes. When this command is issued, all the file stat information which is already cached will get revalidated. </para>
<para>A fix-layout rebalance will only fix the layout changes and does not migrate data. If you want to migrate the existing data, use<command># gluster volume rebalance <varname>VOLNAME</varname> start </command> command to rebalance data among the servers. </para>
<para><emphasis role="bold">To rebalance a volume to fix layout changes</emphasis></para>
<itemizedlist>
<listitem>
<para>Start the rebalance operation on any one of the server using the following command:</para>
<para><command># gluster volume rebalance<varname> VOLNAME</varname> fix-layout start</command></para>
<para>For example:</para>
<para><programlisting># gluster volume rebalance test-volume fix-layout start
Starting rebalance on volume test-volume has been successful</programlisting></para>
</listitem>
</itemizedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Rebalancing-Fix_Migrate">
<title>Rebalancing Volume to Fix Layout and Migrate Data</title>
<para>After expanding or shrinking a volume (using the add-brick and remove-brick commands respectively), you need to rebalance the data among the servers. </para>
<para><emphasis role="bold">To rebalance a volume to fix layout and migrate the existing data</emphasis></para>
<itemizedlist>
<listitem>
<para>Start the rebalance operation on any one of the server using the following command:</para>
<para><command># gluster volume rebalance<varname> VOLNAME</varname> start</command></para>
<para>For example:</para>
<para><programlisting># gluster volume rebalance test-volume start
Starting rebalancing on volume test-volume has been successful</programlisting></para>
</listitem>
<listitem>
<para>Start the migration operation forcefully on any one of the server using the following command:</para>
<para><command># gluster volume rebalance<varname> VOLNAME</varname> start force</command></para>
<para>For example:</para>
<para><programlisting># gluster volume rebalance test-volume start force
Starting rebalancing on volume test-volume has been successful</programlisting></para>
</listitem>
</itemizedlist>
</section>
<section>
<title>Displaying Status of Rebalance Operation</title>
<para>You can display the status information about rebalance volume operation, as needed. </para>
<para><emphasis role="bold">To view status of rebalance volume</emphasis></para>
<itemizedlist>
<listitem>
<para>Check the status of the rebalance operation, using the following command:</para>
<para><command># gluster volume rebalance <replaceable>VOLNAME</replaceable> status</command></para>
<para>For example:</para>
<para><screen># gluster volume rebalance test-volume status
Node Rebalanced-files size scanned status
--------- ---------------- ---- ------- -----------
617c923e-6450-4065-8e33-865e28d9428f 416 1463 312 in progress</screen></para>
<para>The time to complete the rebalance operation depends on the number of files on the volume along with the corresponding file sizes. Continue checking the rebalance status, verifying that the number of files rebalanced or total files scanned keeps increasing.</para>
<para>For example, running the status command again might display a result similar to the following:</para>
<para><screen># gluster volume rebalance test-volume status
Node Rebalanced-files size scanned status
--------- ---------------- ---- ------- -----------
617c923e-6450-4065-8e33-865e28d9428f 498 1783 378 in progress</screen></para>
<para>The rebalance status displays the following when the rebalance is complete:</para>
<para><screen># gluster volume rebalance test-volume status
Node Rebalanced-files size scanned status
--------- ---------------- ---- ------- -----------
617c923e-6450-4065-8e33-865e28d9428f 502 1873 334 completed</screen></para>
</listitem>
</itemizedlist>
</section>
<section>
<title>Stopping Rebalance Operation</title>
<para>You can stop the rebalance operation, as needed.</para>
<para><emphasis role="bold">To stop rebalance</emphasis></para>
<itemizedlist>
<listitem>
<para>Stop the rebalance operation using the following command:</para>
<para><command># gluster volume rebalance <replaceable>VOLNAME</replaceable> stop</command></para>
<para>For example:</para>
<para><screen># gluster volume rebalance test-volume stop
Node Rebalanced-files size scanned status
--------- ---------------- ---- ------- -----------
617c923e-6450-4065-8e33-865e28d9428f 59 590 244 stopped
Stopped rebalance process on volume test-volume </screen></para>
</listitem>
</itemizedlist>
</section>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Stop">
<title>Stopping Volumes</title>
<para>To stop a volume</para>
<orderedlist>
<listitem>
<para>Stop the volume using the following command:
</para>
<para><command># gluster volume stop <varname>VOLNAME </varname></command></para>
<para>For example, to stop test-volume:</para>
<para><programlisting># gluster volume stop test-volume
Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
</programlisting></para>
</listitem>
<listitem>
<para>Enter <userinput>y</userinput> to confirm the operation. The output of the command displays the following:
</para>
<programlisting>Stopping volume test-volume has been successful</programlisting>
</listitem>
</orderedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Delete">
<title>Deleting Volumes</title>
<para>To delete a volume </para>
<orderedlist>
<listitem>
<para>Delete the volume using the following command:</para>
<para><command># gluster volume delete <varname>VOLNAME</varname></command></para>
<para>For example, to delete test-volume:</para>
<para><programlisting># gluster volume delete test-volume
Deleting volume will erase all information about the volume. Do you want to continue? (y/n)</programlisting></para>
</listitem>
<listitem>
<para>Enter <userinput role="bold">y</userinput> to confirm the operation. The command displays the following:</para>
<para><programlisting>Deleting volume test-volume has been successful</programlisting></para>
</listitem>
</orderedlist>
</section>
<section id="sect-Administration_Guide-Managing_Volumes-Self_heal">
<title>Triggering Self-Heal on Replicate</title>
<para>In replicate module, previously you had to manually trigger a self-heal when a brick goes offline and comes back online, to bring all the replicas in sync. Now the pro-active self-heal daemon runs in the background, diagnoses issues and automatically initiates self-healing every 10 minutes on the files which requires<emphasis role="italic"> healing</emphasis>. </para>
<para>You can view the list of files that need <emphasis role="italic">healing</emphasis>, the list of files which are currently/previously <emphasis role="italic">healed</emphasis>, list of files which are in split-brain state, and you can manually trigger self-heal on the entire volume or only on the files which need <emphasis role="italic">healing</emphasis>.</para>
<itemizedlist>
<listitem>
<para>Trigger self-heal only on the files which requires <emphasis role="italic">healing</emphasis>:</para>
<para><command># gluster volume heal <replaceable>VOLNAME</replaceable></command></para>
<para>For example, to trigger self-heal on files which requires <emphasis role="italic">healing</emphasis> of test-volume:</para>
<para><screen># gluster volume heal test-volume
Heal operation on volume test-volume has been successful</screen></para>
</listitem>
<listitem>
<para>Trigger self-heal on all the files of a volume:</para>
<para><command># gluster volume heal <replaceable>VOLNAME</replaceable></command> <command>full</command></para>
<para>For example, to trigger self-heal on all the files of of test-volume:</para>
<para><screen># gluster volume heal test-volume full
Heal operation on volume test-volume has been successful</screen></para>
</listitem>
<listitem>
<para>View the list of files that needs <emphasis role="italic">healing</emphasis>:</para>
<para><command># gluster volume heal <replaceable>VOLNAME</replaceable></command> <command>info</command></para>
<para>For example, to view the list of files on test-volume that needs <emphasis role="italic">healing</emphasis>:</para>
<para><screen># gluster volume heal test-volume info
Brick <emphasis role="italic">server1</emphasis>:/gfs/test-volume_0
Number of entries: 0
Brick <emphasis role="italic">server2</emphasis>:/gfs/test-volume_1
Number of entries: 101
/95.txt
/32.txt
/66.txt
/35.txt
/18.txt
/26.txt
/47.txt
/55.txt
/85.txt
...</screen></para>
</listitem>
<listitem>
<para>View the list of files that are self-healed:</para>
<para><command># gluster volume heal <replaceable>VOLNAME</replaceable></command> <command>info healed</command> </para>
<para>For example, to view the list of files on test-volume that are self-healed:</para>
<para><screen># gluster volume heal test-volume info healed
Brick <emphasis role="italic">server1</emphasis>:/gfs/test-volume_0
Number of entries: 0
Brick <emphasis role="italic">server2</emphasis>:/gfs/test-volume_1
Number of entries: 69
/99.txt
/93.txt
/76.txt
/11.txt
/27.txt
/64.txt
/80.txt
/19.txt
/41.txt
/29.txt
/37.txt
/46.txt
...</screen></para>
</listitem>
<listitem>
<para>View the list of files of a particular volume on which the self-heal failed:</para>
<para><command># gluster volume heal <replaceable>VOLNAME</replaceable></command> <command>info failed</command> </para>
<para>For example, to view the list of files of test-volume that are not self-healed:</para>
<para><screen># gluster volume heal test-volume info failed
Brick <emphasis role="italic">server1</emphasis>:/gfs/test-volume_0
Number of entries: 0
Brick server2:/gfs/test-volume_3
Number of entries: 72
/90.txt
/95.txt
/77.txt
/71.txt
/87.txt
/24.txt
...</screen></para>
</listitem>
<listitem>
<para>View the list of files of a particular volume which are in split-brain state:</para>
<para><command># gluster volume heal <replaceable>VOLNAME</replaceable></command> <command>info split-brain</command> </para>
<para>For example, to view the list of files of test-volume which are in split-brain state:</para>
<para><screen># gluster volume heal test-volume info split-brain
Brick server1:/gfs/test-volume_2
Number of entries: 12
/83.txt
/28.txt
/69.txt
...
Brick <emphasis role="italic">server2</emphasis>:/gfs/test-volume_2
Number of entries: 12
/83.txt
/28.txt
/69.txt
...</screen></para>
</listitem>
</itemizedlist>
</section>
</chapter>
|
