Summary:
========
Currently we don't have a memory and cpu leak
testing framework which blocks automation development
of all testcases around memory and CPU leaks.
To solve this problem we need the libraries for:
1. Logging memory and CPU utilization
2. Checking for OOM killer on gluster processes
3. Checking for memory leaks
4. Checking for cpu usage spikes
5. Wrapper functions in base class to make development easy
6. Compute statistics of usage
Detailed description:
=====================
We have already added script to log CPU and memory
usage through patch [1]. In this patch we would be
using patch [1] and building logic to process the
CSV files generated by the script which would
help us to achieve the following:
1. Checking if there are memory leaks or CPU spikes
2. Computing statistics of memory and CPU usage
Function sets added:
~~~~~~~~~~~~~~~~~~~~
Set 1 - Functions to perfrom logging using script
-------------------------------------------------
Public functions:
1. check_upload_memory_and_cpu_logger_script()
2. log_memory_and_cpu_usage_on_servers()
3. log_memory_and_cpu_usage_on_clients()
4. log_memory_and_cpu_usage_on_cluster()
5. wait_for_logging_processes_to_stop()
6. kill_all_logging_processes()
Private functions to support public functions:
1. _start_logging_processes()
2. _process_wait_flag_append()
Set 2 - Functions to check for OOM killers
------------------------------------------
Public functions:
1. check_for_oom_killers_on_servers()
2. check_for_oom_killers_on_clients()
Private functions to support public functions:
1. _check_for_oom_killers()
Set 3 - Functions to check for memory leaks
-------------------------------------------
Public functions:
1. check_for_memory_leaks_in_glusterd()
2. check_for_memory_leaks_in_glusterfs()
3. check_for_memory_leaks_in_glusterfsd()
4. check_for_memory_leaks_in_glusterfs_fuse()
Private functions to support public functions:
1. _perform_three_point_check_for_memory_leak()
Set 4 - Functions to check for cpu usage spikes
-----------------------------------------------
Public functions:
1. check_for_cpu_usage_spikes_on_glusterd()
2. check_for_cpu_usage_spikes_on_glusterfs()
3. check_for_cpu_usage_spikes_on_glusterfsd()
4. check_for_cpu_usage_spikes_on_glusterfs_fuse()
Private functions to support public functions:
1. _check_for_cpu_usage_spikes()
Set 7 - Functions to calculate stats
------------------------------------
Public functions:
1. compute_data_usage_stats_on_servers()
2. compute_data_usage_stats_on_clients()
Private functions to support public functions:
1. _get_min_max_mean_median()
2. _compute_min_max_mean_median()
Set 6 - Wrapper functions added to base class
---------------------------------------------
1. start_memory_and_cpu_usage_logging()
2. compute_and_print_usage_stats()
3. check_for_memory_leaks_and_oom_kills_on_servers()
4. check_for_memory_leaks_and_oom_kills_on_clients()
5. check_for_cpu_usage_spikes_on_servers()
6. check_for_cpu_spikes_on_clients()
Set 7 - Other generic functions
-------------------------------
Public functions:
1. create_dataframe_from_csv()
Third party libraries added to glusto-tests through
patch:
1. Numpy(It is being used in file_dir_ops.py but it's
installed on clients and not on management node.)
2. Pandas
3. Statistics
How do I use it in my testcase?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For example if we take the testcase to Check the server side
memory leak with fuse mount, we would need to perfrom the
below steps:
1. Create Disperse volumes
2. Fuse mount on the client
3. Start creating files in 1000's in parallel with
3. Create directories in 1000's in parallel
4. Do linux untar and wait for completion
5. Watch the memory usage on the server side and check for the OOM killers.
Here steps 1-4 would be as usual, for step five what we need
to do would be after step 2 we would need to start logging memory usage
with the below code:
```
proc_dict = cls.start_memory_and_cpu_usage_logging()
assertIsNotNone(proc_dict, <Error message>)
```
Once step 4 is complete we would need to wait for the logging process
to stop with the below code:
```
ret = wait_for_logging_processes_to_stop(proc_dict, cluster=True)
assertTrue(ret, <Error message>)
```
And lastly to check for memory leaks and OOM killers we would need use
the below code:
```
ret = cls.check_for_memory_leaks_and_oom_kills_on_servers()
assertTrue(ret, 'Memory leaks or OOM killer found')
```
NOTE:
Interval and count of function start_memory_and_cpu_usage_logging()
and gain of check_for_memory_leaks_and_oom_kills_on_servers() would
need tweaking on a case by case scenario.
Links:
======
[1] https://review.gluster.org/#/c/glusto-tests/+/24659/
Change-Id: Ib617fae102b8280723e54d0a38f77791558f5658
Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>