cluster/tier: Adding compaction option for metadata databases

Problem: As metadata in the database fills up, querying the database
take a long time. As a result, tier migration slows down.  To
counteract this, we added a way to enable the compaction methods of
the underlying database. The goal is to reduce the size of the
underlying file by eliminating database fragmentation.

NOTE: There is currently a bug where sometimes a brick will
attempt to activate compaction. This happens even compaction is already
turned on.

The cause is narrowed down to the compact_mode_switch flipping its value.

Changes: libglusterfs/src/gfdb - Added a gfdb function to compact the
underlying database, compact_db() This is a no-op if the database has
no such option.

- Added a compaction function for SQLite3 that does the following

1) Changes the auto_vacuum pragma of the database
2) Compacts the database according to the type of compaction requested

- Compaction type can be changed by changing the macro
  GF_SQL_COMPACT_DEF to one of the 4 compaction types in
  gfdb_sqlite3.h

  It is currently set to GF_SQL_COMPACT_INCR, or incremental
  vacuuming.

xlators/cluster/dht/src - Added the following command-line option to
enable SQLite3 compaction.

gluster volume set <vol-name> tier-compact <off|on>

- Added the following command-line option to change the frequency the
  hot and cold tier are ordered to compact.

gluster volume set <vol-name> tier-hot-compact-frequency <int>
gluster volume set <vol-name> tier-cold-compact-frequency <int>

- tier daemon periodically sends the (new)
  GFDB_IPC_CTR_SET_COMPACT_PRAGMA IPC to the CTR xlator. The IPC
  triggers compaction of the database.

  The inputs are both gf_boolean_t.

  IPC Input:

  compact_active: Is compaction currently on for the db.
  compact_mode_switched: Did we flip the compaction switch recently?

  IPC Output:

  0 if the compaction succeeds.
  Non-zero otherwise.

xlators/features/changetimerecorder/src/ - When the CTR gets the
compaction IPC, it launches a thread that will perform the
compaction. The IPC ends after the thread is launched. To avoid extra
allocations, the parameters are passed using static variables.

Change-Id: I5e1433becb9eeff2afe8dcb4a5798977bf5ba0dd
Signed-off-by: Diogenes Nunez <dnunez@redhat.com>
Reviewed-on: http://review.gluster.org/15031
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>

1 files changed, 40 insertions, 0 deletions

diff --git a/libglusterfs/src/gfdb/gfdb_data_store.c b/libglusterfs/src/gfdb/gfdb_data_store.c
index 9c042f9e82e..cb567503fa3 100644
--- a/libglusterfs/src/gfdb/gfdb_data_store.c
+++ b/libglusterfs/src/gfdb/gfdb_data_store.c
@@ -433,6 +433,43 @@ delete_record (gfdb_conn_node_t *_conn_node,
         return ret;
 }
 
+/*Libgfdb API Function: Compact the database.
+ *
+ * Arguments:
+ *      _conn_node                      :  GFDB Connection node
+ *      _compact_active                 :  Is compaction currently on?
+ *      _compact_mode_switched          :  Was the compaction switch flipped?
+ * Returns : if successful return 0 or
+ *          -ve value in case of failure*/
+int
+compact_db (gfdb_conn_node_t *_conn_node, gf_boolean_t _compact_active,
+            gf_boolean_t _compact_mode_switched)
+{
+        int ret                                 = 0;
+        gfdb_db_operations_t *db_operations_t   = NULL;
+        void *gf_db_connection                  = NULL;
+
+        CHECK_CONN_NODE(_conn_node);
+
+        db_operations_t = &_conn_node->gfdb_connection.gfdb_db_operations;
+        gf_db_connection = _conn_node->gfdb_connection.gf_db_connection;
+
+        if (db_operations_t->compact_db_op) {
+
+                ret = db_operations_t->compact_db_op (gf_db_connection,
+                                                      _compact_active,
+                                                      _compact_mode_switched);
+                if (ret) {
+                        gf_msg (GFDB_DATA_STORE, GF_LOG_ERROR, 0,
+                                LG_MSG_COMPACT_FAILED, "Compaction operation "
+                                "failed");
+                }
+
+        }
+
+        return ret;
+}
+
 
 
 
@@ -835,5 +872,8 @@ void get_gfdb_methods (gfdb_methods_t *methods)
         /* Link info related functions */
         methods->gfdb_link_info_new = gfdb_link_info_new;
         methods->gfdb_link_info_free = gfdb_link_info_free;
+
+        /* Compaction related functions */
+        methods->compact_db = compact_db;
 }