summaryrefslogtreecommitdiffstats
path: root/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md
diff options
context:
space:
mode:
Diffstat (limited to 'doc/admin-guide/en-US/markdown/admin_monitoring_workload.md')
-rw-r--r--doc/admin-guide/en-US/markdown/admin_monitoring_workload.md931
1 files changed, 931 insertions, 0 deletions
diff --git a/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md b/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md
new file mode 100644
index 000000000..0312bd048
--- /dev/null
+++ b/doc/admin-guide/en-US/markdown/admin_monitoring_workload.md
@@ -0,0 +1,931 @@
+Monitoring your GlusterFS Workload
+==================================
+
+You can monitor the GlusterFS volumes on different parameters.
+Monitoring volumes helps in capacity planning and performance tuning
+tasks of the GlusterFS volume. Using these information, you can identify
+and troubleshoot issues.
+
+You can use Volume Top and Profile commands to view the performance and
+identify bottlenecks/hotspots of each brick of a volume. This helps
+system administrators to get vital performance information whenever
+performance needs to be probed.
+
+You can also perform statedump of the brick processes and nfs server
+process of a volume, and also view volume status and volume information.
+
+Running GlusterFS Volume Profile Command
+========================================
+
+GlusterFS Volume Profile command provides an interface to get the
+per-brick I/O information for each File Operation (FOP) of a volume. The
+per brick information helps in identifying bottlenecks in the storage
+system.
+
+This section describes how to run GlusterFS Volume Profile command by
+performing the following operations:
+
+- ?
+
+- ?
+
+- ?
+
+Start Profiling
+---------------
+
+You must start the Profiling to view the File Operation information for
+each brick.
+
+**To start profiling:**
+
+- Start profiling using the following command:
+
+`# gluster volume profile start `
+
+For example, to start profiling on test-volume:
+
+ # gluster volume profile test-volume start
+ Profiling started on test-volume
+
+When profiling on the volume is started, the following additional
+options are displayed in the Volume Info:
+
+ diagnostics.count-fop-hits: on
+
+ diagnostics.latency-measurement: on
+
+Displaying the I/0 Information
+------------------------------
+
+You can view the I/O information of each brick.
+
+To display I/O information:
+
+- Display the I/O information using the following command:
+
+`# gluster volume profile info`
+
+For example, to see the I/O information on test-volume:
+
+ # gluster volume profile test-volume info
+ Brick: Test:/export/2
+ Cumulative Stats:
+
+ Block 1b+ 32b+ 64b+
+ Size:
+ Read: 0 0 0
+ Write: 908 28 8
+
+ Block 128b+ 256b+ 512b+
+ Size:
+ Read: 0 6 4
+ Write: 5 23 16
+
+ Block 1024b+ 2048b+ 4096b+
+ Size:
+ Read: 0 52 17
+ Write: 15 120 846
+
+ Block 8192b+ 16384b+ 32768b+
+ Size:
+ Read: 52 8 34
+ Write: 234 134 286
+
+ Block 65536b+ 131072b+
+ Size:
+ Read: 118 622
+ Write: 1341 594
+
+
+ %-latency Avg- Min- Max- calls Fop
+ latency Latency Latency
+ ___________________________________________________________
+ 4.82 1132.28 21.00 800970.00 4575 WRITE
+ 5.70 156.47 9.00 665085.00 39163 READDIRP
+ 11.35 315.02 9.00 1433947.00 38698 LOOKUP
+ 11.88 1729.34 21.00 2569638.00 7382 FXATTROP
+ 47.35 104235.02 2485.00 7789367.00 488 FSYNC
+
+ ------------------
+
+ ------------------
+
+ Duration : 335
+
+ BytesRead : 94505058
+
+ BytesWritten : 195571980
+
+Stop Profiling
+--------------
+
+You can stop profiling the volume, if you do not need profiling
+information anymore.
+
+**To stop profiling**
+
+- Stop profiling using the following command:
+
+ `# gluster volume profile stop`
+
+ For example, to stop profiling on test-volume:
+
+ `# gluster volume profile stop`
+
+ `Profiling stopped on test-volume`
+
+Running GlusterFS Volume TOP Command
+====================================
+
+GlusterFS Volume Top command allows you to view the glusterfs bricks’
+performance metrics like read, write, file open calls, file read calls,
+file write calls, directory open calls, and directory real calls. The
+top command displays up to 100 results.
+
+This section describes how to run and view the results for the following
+GlusterFS Top commands:
+
+- ?
+
+- ?
+
+- ?
+
+- ?
+
+- ?
+
+- ?
+
+- ?
+
+Viewing Open fd Count and Maximum fd Count
+------------------------------------------
+
+You can view both current open fd count (list of files that are
+currently the most opened and the count) on the brick and the maximum
+open fd count (count of files that are the currently open and the count
+of maximum number of files opened at any given point of time, since the
+servers are up and running). If the brick name is not specified, then
+open fd metrics of all the bricks belonging to the volume will be
+displayed.
+
+**To view open fd count and maximum fd count:**
+
+- View open fd count and maximum fd count using the following command:
+
+ `# gluster volume top open [brick ] [list-cnt ]`
+
+ For example, to view open fd count and maximum fd count on brick
+ server:/export of test-volume and list top 10 open calls:
+
+ `# gluster volume top open brick list-cnt `
+
+ `Brick: server:/export/dir1 `
+
+ `Current open fd's: 34 Max open fd's: 209 `
+
+ ==========Open file stats========
+
+ open file name
+ call count
+
+ 2 /clients/client0/~dmtmp/PARADOX/
+ COURSES.DB
+
+ 11 /clients/client0/~dmtmp/PARADOX/
+ ENROLL.DB
+
+ 11 /clients/client0/~dmtmp/PARADOX/
+ STUDENTS.DB
+
+ 10 /clients/client0/~dmtmp/PWRPNT/
+ TIPS.PPT
+
+ 10 /clients/client0/~dmtmp/PWRPNT/
+ PCBENCHM.PPT
+
+ 9 /clients/client7/~dmtmp/PARADOX/
+ STUDENTS.DB
+
+ 9 /clients/client1/~dmtmp/PARADOX/
+ STUDENTS.DB
+
+ 9 /clients/client2/~dmtmp/PARADOX/
+ STUDENTS.DB
+
+ 9 /clients/client0/~dmtmp/PARADOX/
+ STUDENTS.DB
+
+ 9 /clients/client8/~dmtmp/PARADOX/
+ STUDENTS.DB
+
+Viewing Highest File Read Calls
+-------------------------------
+
+You can view highest read calls on each brick. If brick name is not
+specified, then by default, list of 100 files will be displayed.
+
+**To view highest file Read calls:**
+
+- View highest file Read calls using the following command:
+
+ `# gluster volume top read [brick ] [list-cnt ] `
+
+ For example, to view highest Read calls on brick server:/export of
+ test-volume:
+
+ `# gluster volume top read brick list-cnt `
+
+ `Brick:` server:/export/dir1
+
+ ==========Read file stats========
+
+ read filename
+ call count
+
+ 116 /clients/client0/~dmtmp/SEED/LARGE.FIL
+
+ 64 /clients/client0/~dmtmp/SEED/MEDIUM.FIL
+
+ 54 /clients/client2/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client6/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client5/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client0/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client3/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client4/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client9/~dmtmp/SEED/LARGE.FIL
+
+ 54 /clients/client8/~dmtmp/SEED/LARGE.FIL
+
+Viewing Highest File Write Calls
+--------------------------------
+
+You can view list of files which has highest file write calls on each
+brick. If brick name is not specified, then by default, list of 100
+files will be displayed.
+
+**To view highest file Write calls:**
+
+- View highest file Write calls using the following command:
+
+ `# gluster volume top write [brick ] [list-cnt ] `
+
+ For example, to view highest Write calls on brick server:/export of
+ test-volume:
+
+ `# gluster volume top write brick list-cnt `
+
+ `Brick: server:/export/dir1 `
+
+ ==========Write file stats========
+ write call count filename
+
+ 83 /clients/client0/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client7/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client1/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client2/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client0/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client8/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client5/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client4/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client6/~dmtmp/SEED/LARGE.FIL
+
+ 59 /clients/client3/~dmtmp/SEED/LARGE.FIL
+
+Viewing Highest Open Calls on Directories
+-----------------------------------------
+
+You can view list of files which has highest open calls on directories
+of each brick. If brick name is not specified, then the metrics of all
+the bricks belonging to that volume will be displayed.
+
+To view list of open calls on each directory
+
+- View list of open calls on each directory using the following
+ command:
+
+ `# gluster volume top opendir [brick ] [list-cnt ] `
+
+ For example, to view open calls on brick server:/export/ of
+ test-volume:
+
+ `# gluster volume top opendir brick list-cnt `
+
+ `Brick: server:/export/dir1 `
+
+ ==========Directory open stats========
+
+ Opendir count directory name
+
+ 1001 /clients/client0/~dmtmp
+
+ 454 /clients/client8/~dmtmp
+
+ 454 /clients/client2/~dmtmp
+
+ 454 /clients/client6/~dmtmp
+
+ 454 /clients/client5/~dmtmp
+
+ 454 /clients/client9/~dmtmp
+
+ 443 /clients/client0/~dmtmp/PARADOX
+
+ 408 /clients/client1/~dmtmp
+
+ 408 /clients/client7/~dmtmp
+
+ 402 /clients/client4/~dmtmp
+
+Viewing Highest Read Calls on Directory
+---------------------------------------
+
+You can view list of files which has highest directory read calls on
+each brick. If brick name is not specified, then the metrics of all the
+bricks belonging to that volume will be displayed.
+
+**To view list of highest directory read calls on each brick**
+
+- View list of highest directory read calls on each brick using the
+ following command:
+
+ `# gluster volume top readdir [brick ] [list-cnt ] `
+
+ For example, to view highest directory read calls on brick
+ server:/export of test-volume:
+
+ `# gluster volume top readdir brick list-cnt `
+
+ `Brick: `
+
+ ==========Directory readdirp stats========
+
+ readdirp count directory name
+
+ 1996 /clients/client0/~dmtmp
+
+ 1083 /clients/client0/~dmtmp/PARADOX
+
+ 904 /clients/client8/~dmtmp
+
+ 904 /clients/client2/~dmtmp
+
+ 904 /clients/client6/~dmtmp
+
+ 904 /clients/client5/~dmtmp
+
+ 904 /clients/client9/~dmtmp
+
+ 812 /clients/client1/~dmtmp
+
+ 812 /clients/client7/~dmtmp
+
+ 800 /clients/client4/~dmtmp
+
+Viewing List of Read Performance on each Brick
+----------------------------------------------
+
+You can view the read throughput of files on each brick. If brick name
+is not specified, then the metrics of all the bricks belonging to that
+volume will be displayed. The output will be the read throughput.
+
+ ==========Read throughput file stats========
+
+ read filename Time
+ through
+ put(MBp
+ s)
+
+ 2570.00 /clients/client0/~dmtmp/PWRPNT/ -2011-01-31
+ TRIDOTS.POT 15:38:36.894610
+ 2570.00 /clients/client0/~dmtmp/PWRPNT/ -2011-01-31
+ PCBENCHM.PPT 15:38:39.815310
+ 2383.00 /clients/client2/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:52:53.631499
+
+ 2340.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:38:36.926198
+
+ 2299.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ LARGE.FIL 15:38:36.930445
+
+ 2259.00 /clients/client0/~dmtmp/PARADOX/ -2011-01-31
+ COURSES.X04 15:38:40.549919
+
+ 2221.00 /clients/client0/~dmtmp/PARADOX/ -2011-01-31
+ STUDENTS.VAL 15:52:53.298766
+
+ 2221.00 /clients/client3/~dmtmp/SEED/ -2011-01-31
+ COURSES.DB 15:39:11.776780
+
+ 2184.00 /clients/client3/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:39:10.251764
+
+ 2184.00 /clients/client5/~dmtmp/WORD/ -2011-01-31
+ BASEMACH.DOC 15:39:09.336572
+
+This command will initiate a dd for the specified count and block size
+and measures the corresponding throughput.
+
+**To view list of read performance on each brick**
+
+- View list of read performance on each brick using the following
+ command:
+
+ `# gluster volume top read-perf [bs count ] [brick ] [list-cnt ]`
+
+ For example, to view read performance on brick server:/export/ of
+ test-volume, 256 block size of count 1, and list count 10:
+
+ `# gluster volume top read-perf bs 256 count 1 brick list-cnt `
+
+ `Brick: server:/export/dir1 256 bytes (256 B) copied, Throughput: 4.1 MB/s `
+
+ ==========Read throughput file stats========
+
+ read filename Time
+ through
+ put(MBp
+ s)
+
+ 2912.00 /clients/client0/~dmtmp/PWRPNT/ -2011-01-31
+ TRIDOTS.POT 15:38:36.896486
+
+ 2570.00 /clients/client0/~dmtmp/PWRPNT/ -2011-01-31
+ PCBENCHM.PPT 15:38:39.815310
+
+ 2383.00 /clients/client2/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:52:53.631499
+
+ 2340.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:38:36.926198
+
+ 2299.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ LARGE.FIL 15:38:36.930445
+
+ 2259.00 /clients/client0/~dmtmp/PARADOX/ -2011-01-31
+ COURSES.X04 15:38:40.549919
+
+ 2221.00 /clients/client9/~dmtmp/PARADOX/ -2011-01-31
+ STUDENTS.VAL 15:52:53.298766
+
+ 2221.00 /clients/client8/~dmtmp/PARADOX/ -2011-01-31
+ COURSES.DB 15:39:11.776780
+
+ 2184.00 /clients/client3/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:39:10.251764
+
+ 2184.00 /clients/client5/~dmtmp/WORD/ -2011-01-31
+ BASEMACH.DOC 15:39:09.336572
+
+
+Viewing List of Write Performance on each Brick
+-----------------------------------------------
+
+You can view list of write throughput of files on each brick. If brick
+name is not specified, then the metrics of all the bricks belonging to
+that volume will be displayed. The output will be the write throughput.
+
+This command will initiate a dd for the specified count and block size
+and measures the corresponding throughput. To view list of write
+performance on each brick:
+
+- View list of write performance on each brick using the following
+ command:
+
+ `# gluster volume top write-perf [bs count ] [brick ] [list-cnt ] `
+
+ For example, to view write performance on brick server:/export/ of
+ test-volume, 256 block size of count 1, and list count 10:
+
+ `# gluster volume top write-perf bs 256 count 1 brick list-cnt `
+
+ `Brick`: server:/export/dir1
+
+ `256 bytes (256 B) copied, Throughput: 2.8 MB/s `
+
+ ==========Write throughput file stats========
+
+ write filename Time
+ throughput
+ (MBps)
+
+ 1170.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ SMALL.FIL 15:39:09.171494
+
+ 1008.00 /clients/client6/~dmtmp/SEED/ -2011-01-31
+ LARGE.FIL 15:39:09.73189
+
+ 949.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:38:36.927426
+
+ 936.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ LARGE.FIL 15:38:36.933177
+ 897.00 /clients/client5/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:39:09.33628
+
+ 897.00 /clients/client6/~dmtmp/SEED/ -2011-01-31
+ MEDIUM.FIL 15:39:09.27713
+
+ 885.00 /clients/client0/~dmtmp/SEED/ -2011-01-31
+ SMALL.FIL 15:38:36.924271
+
+ 528.00 /clients/client5/~dmtmp/SEED/ -2011-01-31
+ LARGE.FIL 15:39:09.81893
+
+ 516.00 /clients/client6/~dmtmp/ACCESS/ -2011-01-31
+ FASTENER.MDB 15:39:01.797317
+
+Displaying Volume Information
+=============================
+
+You can display information about a specific volume, or all volumes, as
+needed.
+
+**To display volume information**
+
+- Display information about a specific volume using the following
+ command:
+
+ `# gluster volume info ``VOLNAME`
+
+ For example, to display information about test-volume:
+
+ # gluster volume info test-volume
+ Volume Name: test-volume
+ Type: Distribute
+ Status: Created
+ Number of Bricks: 4
+ Bricks:
+ Brick1: server1:/exp1
+ Brick2: server2:/exp2
+ Brick3: server3:/exp3
+ Brick4: server4:/exp4
+
+- Display information about all volumes using the following command:
+
+ `# gluster volume info all`
+
+ # gluster volume info all
+
+ Volume Name: test-volume
+ Type: Distribute
+ Status: Created
+ Number of Bricks: 4
+ Bricks:
+ Brick1: server1:/exp1
+ Brick2: server2:/exp2
+ Brick3: server3:/exp3
+ Brick4: server4:/exp4
+
+ Volume Name: mirror
+ Type: Distributed-Replicate
+ Status: Started
+ Number of Bricks: 2 X 2 = 4
+ Bricks:
+ Brick1: server1:/brick1
+ Brick2: server2:/brick2
+ Brick3: server3:/brick3
+ Brick4: server4:/brick4
+
+ Volume Name: Vol
+ Type: Distribute
+ Status: Started
+ Number of Bricks: 1
+ Bricks:
+ Brick: server:/brick6
+
+Performing Statedump on a Volume
+================================
+
+Statedump is a mechanism through which you can get details of all
+internal variables and state of the glusterfs process at the time of
+issuing the command.You can perform statedumps of the brick processes
+and nfs server process of a volume using the statedump command. The
+following options can be used to determine what information is to be
+dumped:
+
+- **mem** - Dumps the memory usage and memory pool details of the
+ bricks.
+
+- **iobuf** - Dumps iobuf details of the bricks.
+
+- **priv** - Dumps private information of loaded translators.
+
+- **callpool** - Dumps the pending calls of the volume.
+
+- **fd** - Dumps the open fd tables of the volume.
+
+- **inode** - Dumps the inode tables of the volume.
+
+**To display volume statedump**
+
+- Display statedump of a volume or NFS server using the following
+ command:
+
+ `# gluster volume statedump [nfs] [all|mem|iobuf|callpool|priv|fd|inode]`
+
+ For example, to display statedump of test-volume:
+
+ # gluster volume statedump test-volume
+ Volume statedump successful
+
+ The statedump files are created on the brick servers in the` /tmp`
+ directory or in the directory set using `server.statedump-path`
+ volume option. The naming convention of the dump file is
+ `<brick-path>.<brick-pid>.dump`.
+
+- By defult, the output of the statedump is stored at
+ ` /tmp/<brickname.PID.dump>` file on that particular server. Change
+ the directory of the statedump file using the following command:
+
+ `# gluster volume set server.statedump-path `
+
+ For example, to change the location of the statedump file of
+ test-volume:
+
+ # gluster volume set test-volume server.statedump-path /usr/local/var/log/glusterfs/dumps/
+ Set volume successful
+
+ You can view the changed path of the statedump file using the
+ following command:
+
+ `# gluster volume info `
+
+Displaying Volume Status
+========================
+
+You can display the status information about a specific volume, brick or
+all volumes, as needed. Status information can be used to understand the
+current status of the brick, nfs processes, and overall file system.
+Status information can also be used to monitor and debug the volume
+information. You can view status of the volume along with the following
+details:
+
+- **detail** - Displays additional information about the bricks.
+
+- **clients** - Displays the list of clients connected to the volume.
+
+- **mem** - Displays the memory usage and memory pool details of the
+ bricks.
+
+- **inode** - Displays the inode tables of the volume.
+
+- **fd** - Displays the open fd (file descriptors) tables of the
+ volume.
+
+- **callpool** - Displays the pending calls of the volume.
+
+**To display volume status**
+
+- Display information about a specific volume using the following
+ command:
+
+ `# gluster volume status [all| []] [detail|clients|mem|inode|fd|callpool]`
+
+ For example, to display information about test-volume:
+
+ # gluster volume status test-volume
+ STATUS OF VOLUME: test-volume
+ BRICK PORT ONLINE PID
+ --------------------------------------------------------
+ arch:/export/1 24009 Y 22445
+ --------------------------------------------------------
+ arch:/export/2 24010 Y 22450
+
+- Display information about all volumes using the following command:
+
+ `# gluster volume status all`
+
+ # gluster volume status all
+ STATUS OF VOLUME: volume-test
+ BRICK PORT ONLINE PID
+ --------------------------------------------------------
+ arch:/export/4 24010 Y 22455
+
+ STATUS OF VOLUME: test-volume
+ BRICK PORT ONLINE PID
+ --------------------------------------------------------
+ arch:/export/1 24009 Y 22445
+ --------------------------------------------------------
+ arch:/export/2 24010 Y 22450
+
+- Display additional information about the bricks using the following
+ command:
+
+ `# gluster volume status detail`
+
+ For example, to display additional information about the bricks of
+ test-volume:
+
+ # gluster volume status test-volume details
+ STATUS OF VOLUME: test-volume
+ -------------------------------------------
+ Brick : arch:/export/1
+ Port : 24009
+ Online : Y
+ Pid : 16977
+ File System : rootfs
+ Device : rootfs
+ Mount Options : rw
+ Disk Space Free : 13.8GB
+ Total Disk Space : 46.5GB
+ Inode Size : N/A
+ Inode Count : N/A
+ Free Inodes : N/A
+
+ Number of Bricks: 1
+ Bricks:
+ Brick: server:/brick6
+
+- Display the list of clients accessing the volumes using the
+ following command:
+
+ `# gluster volume status clients`
+
+ For example, to display the list of clients connected to
+ test-volume:
+
+ # gluster volume status test-volume clients
+ Brick : arch:/export/1
+ Clients connected : 2
+ Hostname Bytes Read BytesWritten
+ -------- --------- ------------
+ 127.0.0.1:1013 776 676
+ 127.0.0.1:1012 50440 51200
+
+- Display the memory usage and memory pool details of the bricks using
+ the following command:
+
+ `# gluster volume status mem`
+
+ For example, to display the memory usage and memory pool details of
+ the bricks of test-volume:
+
+ Memory status for volume : test-volume
+ ----------------------------------------------
+ Brick : arch:/export/1
+ Mallinfo
+ --------
+ Arena : 434176
+ Ordblks : 2
+ Smblks : 0
+ Hblks : 12
+ Hblkhd : 40861696
+ Usmblks : 0
+ Fsmblks : 0
+ Uordblks : 332416
+ Fordblks : 101760
+ Keepcost : 100400
+
+ Mempool Stats
+ -------------
+ Name HotCount ColdCount PaddedSizeof AllocCount MaxAlloc
+ ---- -------- --------- ------------ ---------- --------
+ test-volume-server:fd_t 0 16384 92 57 5
+ test-volume-server:dentry_t 59 965 84 59 59
+ test-volume-server:inode_t 60 964 148 60 60
+ test-volume-server:rpcsvc_request_t 0 525 6372 351 2
+ glusterfs:struct saved_frame 0 4096 124 2 2
+ glusterfs:struct rpc_req 0 4096 2236 2 2
+ glusterfs:rpcsvc_request_t 1 524 6372 2 1
+ glusterfs:call_stub_t 0 1024 1220 288 1
+ glusterfs:call_stack_t 0 8192 2084 290 2
+ glusterfs:call_frame_t 0 16384 172 1728 6
+
+- Display the inode tables of the volume using the following command:
+
+ `# gluster volume status inode`
+
+ For example, to display the inode tables of the test-volume:
+
+ # gluster volume status test-volume inode
+ inode tables for volume test-volume
+ ----------------------------------------------
+ Brick : arch:/export/1
+ Active inodes:
+ GFID Lookups Ref IA type
+ ---- ------- --- -------
+ 6f3fe173-e07a-4209-abb6-484091d75499 1 9 2
+ 370d35d7-657e-44dc-bac4-d6dd800ec3d3 1 1 2
+
+ LRU inodes:
+ GFID Lookups Ref IA type
+ ---- ------- --- -------
+ 80f98abe-cdcf-4c1d-b917-ae564cf55763 1 0 1
+ 3a58973d-d549-4ea6-9977-9aa218f233de 1 0 1
+ 2ce0197d-87a9-451b-9094-9baa38121155 1 0 2
+
+- Display the open fd tables of the volume using the following
+ command:
+
+ `# gluster volume status fd`
+
+ For example, to display the open fd tables of the test-volume:
+
+ # gluster volume status test-volume fd
+
+ FD tables for volume test-volume
+ ----------------------------------------------
+ Brick : arch:/export/1
+ Connection 1:
+ RefCount = 0 MaxFDs = 128 FirstFree = 4
+ FD Entry PID RefCount Flags
+ -------- --- -------- -----
+ 0 26311 1 2
+ 1 26310 3 2
+ 2 26310 1 2
+ 3 26311 3 2
+
+ Connection 2:
+ RefCount = 0 MaxFDs = 128 FirstFree = 0
+ No open fds
+
+ Connection 3:
+ RefCount = 0 MaxFDs = 128 FirstFree = 0
+ No open fds
+
+- Display the pending calls of the volume using the following command:
+
+ `# gluster volume status callpool`
+
+ Each call has a call stack containing call frames.
+
+ For example, to display the pending calls of test-volume:
+
+ # gluster volume status test-volume
+
+ Pending calls for volume test-volume
+ ----------------------------------------------
+ Brick : arch:/export/1
+ Pending calls: 2
+ Call Stack1
+ UID : 0
+ GID : 0
+ PID : 26338
+ Unique : 192138
+ Frames : 7
+ Frame 1
+ Ref Count = 1
+ Translator = test-volume-server
+ Completed = No
+ Frame 2
+ Ref Count = 0
+ Translator = test-volume-posix
+ Completed = No
+ Parent = test-volume-access-control
+ Wind From = default_fsync
+ Wind To = FIRST_CHILD(this)->fops->fsync
+ Frame 3
+ Ref Count = 1
+ Translator = test-volume-access-control
+ Completed = No
+ Parent = repl-locks
+ Wind From = default_fsync
+ Wind To = FIRST_CHILD(this)->fops->fsync
+ Frame 4
+ Ref Count = 1
+ Translator = test-volume-locks
+ Completed = No
+ Parent = test-volume-io-threads
+ Wind From = iot_fsync_wrapper
+ Wind To = FIRST_CHILD (this)->fops->fsync
+ Frame 5
+ Ref Count = 1
+ Translator = test-volume-io-threads
+ Completed = No
+ Parent = test-volume-marker
+ Wind From = default_fsync
+ Wind To = FIRST_CHILD(this)->fops->fsync
+ Frame 6
+ Ref Count = 1
+ Translator = test-volume-marker
+ Completed = No
+ Parent = /export/1
+ Wind From = io_stats_fsync
+ Wind To = FIRST_CHILD(this)->fops->fsync
+ Frame 7
+ Ref Count = 1
+ Translator = /export/1
+ Completed = No
+ Parent = test-volume-server
+ Wind From = server_fsync_resume
+ Wind To = bound_xl->fops->fsync
+
+