summaryrefslogtreecommitdiffstats
path: root/done/GlusterFS 3.5/AFR CLI enhancements.md
blob: 88f49804ef853b039427ef978cb58e924e8fcb12 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
Feature
-------

AFR CLI enhancements

SUMMARY
-------

Presently the AFR reporting via CLI has lots of problems in the
representation of logs because of which they may not be able to use the
data effectively. This feature is to correct these problems and provide
a coherent mechanism to present heal status,information and the logs
associated.

Owners
------

Venkatesh Somayajulu  
Raghavan

Current status
--------------

There are many bugs related to this which indicates the current status
and why these requirements are required.

​1) 924062 - gluster volume heal info shows only gfids in some cases and
sometimes names. This is very confusing for the end user.

​2) 852294 - gluster volume heal info hangs/crashes when there is a
large number of entries to be healed.

​3) 883698 - when self heal daemon is turned off, heal info does not
show any output. But healing can happen because of lookups from IO path.
Hence list of entries to be healed still needs to be shown.

​4) 921025 - directories are not reported when list of split brain
entries needs to be displayed.

​5) 981185 - when self heal daemon process is offline, volume heal info
gives error as "staging failure"

​6) 952084 - We need a command to resolve files in split brain state.

​7) 986309 - We need to report source information for files which got
healed during a self heal session.

​8) 986317 - Sometimes list of files to get healed also includes files
to which IO s being done since the entries for these files could be in
the xattrop directory. This could be confusing for the user.

There is a master bug 926044 that sums up most of the above problems. It
does give the QA perspective of the current representation out of the
present reporting infrastructure.

Detailed Description
--------------------

​1) One common thread among all the above complaints is that the
information presented to the user is <B>FUD</B> because of the following
reasons:

(a) Split brain itself is a scary scenario especially with VMs.  
(b) The data that we present to the users cannot be used in a stable
    manner for them to get to the list of these files. <I>For ex:</I> we
    need to give mechanisms by which he can automate the resolution out
    of split brain.  
(c) The logs that are generated are all the more scarier since we
    see repetition of some error lines running into hundreds of lines.
    Our mailing lists are filled with such emails from end users.  

Any data is useless unless it is associated with an event. For self
heal, the event that leads to self heal is the loss of connectivity to a
brick from a client. So all healing info and especially split brain
should be associated with such events.

The following is hence the proposed mechanism:

(a) Every loss of a brick from client's perspective is logged and
    available via some ID. The information provides the time from when
    the brick went down to when it came up. Also it should also report
    the number of IO transactions(modifies) that hapenned during this
    event.  
(b) The list of these events are available via some CLI command. The
    actual command needs to be detailed as part of this feature.  
(c) All volume info commands regarding list of files to be healed,
    files healed and split brain files should be associated with this
    event(s).  

​2) Provide a mechanism to show statistics at a volume and replica group
level. It should show the number of files to be healed and number of
split brain files at both the volume and replica group level.

​3) Provide a mechanism to show per volume list of files to be
healed/files healed/split brain in the following info:

This should have the following information:

(a) File name  
(b) Bricks location  
(c) Event association (brick going down)  
(d) Source  
(v) Sink

​4) Self heal crawl statistics - Introduce new CLI commands for showing
more information on self heal crawl per volume.

(a) Display why a self heal crawl ran (timeouts, brick coming up)  
(b) Start time and end time  
(c) Number of files it attempted to heal  
(d) Location of the self heal daemon

​5) Scale the logging infrastructure to handle huge number of file list
that needs to be displayed as part of the logging.

(a) Right now the system crashes or hangs in case of a high number
    of files.  
(b) It causes CLI timeouts arbitrarily. The latencies involved in
    the logging have to be studied (profiled) and mechanisms to
    circumvent them have to be introduced.  
(c) All files are displayed on the output. Have a better way of
    representing them.

Options are:

(a) Maybe write to a glusterd log file or have a seperate directory
    for afr heal logs.  
(b) Have a status kind of command. This will display the current
    status of the log building and maybe have batched way of
    representing when there is a huge list.

​6) We should provide mechanism where the user can heal split brain by
some pre-established policies:

(a) Let the system figure out the latest files (assuming all nodes
    are in time sync) and choose the copies that have the latest time.  
(b) Choose one particular brick as the source for split brain and
    heal all split brains from this brick.  
(c) Just remove the split brain information from changelog. We leave
    the exercise to the user to repair split brain where in he would
    rewrite to the split brained files. (right now the user is forced to
    remove xattrs manually for this step).

Benefits to GlusterFS
--------------------

Makes the end user more aware of healing status and provides statistics.

Scope
-----

6.1. Nature of proposed change

Modification to AFR and CLI and glusterd code

6.2. Implications on manageability

New CLI commands to be added. Existing commands to be improved.

6.3. Implications on presentation layer

N/A

6.4. Implications on persistence layer

N/A

6.5. Implications on 'GlusterFS' backend

N/A

6.6. Modification to GlusterFS metadata

N/A

6.7. Implications on 'glusterd'

Changes for healing specific commands will be introduced.

How To Test
-----------

See documentation session

User Experience
---------------

*Changes in CLI, effect on User experience...*

Documentation
-------------

<http://review.gluster.org/#/c/7792/1/doc/features/afr-statistics.md>

Status
------

Patches :

<http://review.gluster.org/6044> <http://review.gluster.org/4790>

Status:

Merged