glusterfs.git/libglusterfs/src/mem-pool.h, branch v3.7.9

core: use reference counting for mem_acct structures

2015-05-09T21:27:36+00:00

When freeing memory, our memory-accounting code expects to be able to
dereference from the (previously) allocated block to its owning
translator.  However, as we have already found once in option
validation and twice in logging, that translator might itself have
been freed and the dereference attempt causes on of our daemons to
crash with SIGSEGV.  This patch attempts to fix that as follows:

 * We no longer embed a struct mem_acct directly in a struct xlator,
   but instead allocate it separately.

 * Allocated memory blocks now contain a pointer to the mem_acct
   instead of the xlator.

 * The mem_acct structure contains a reference count, manipulated in
   both the normal and translator allocate/free code using atomic
   increments and decrements.

 * Because it's now a separate structure, we can defer freeing the
   mem_acct until its reference count reaches zero (either way).

 * Some unit tests were disabled, because they embedded their own
   copies of the implementation for what they were supposedly testing.
   Life's too short to spend time fixing tests that seem designed to
   impede progress by requiring a certain implementation as well as
   behavior.

Change-Id: Id929b11387927136f78626901729296b6c0d0fd7
BUG: 1219026
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/10417
Tested-by: Gluster Build System 
Reviewed-by: Krishnan Parthasarathi 
Reviewed-by: Niels de Vos 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-on: http://review.gluster.org/10723
Tested-by: NetBSD Build System

mem-pool: invalidate memory on GF_FREE to aid debugging

2015-03-30T12:37:07+00:00

Debugging where memory gets free'd with help from overwriting the memory
before it is free'd with some structures (repeatedly). The struct
mem_invalid starts with a magic value (0xdeadc0de), followed by a
pointer to the xlator, the mem-type. the size of the GF_?ALLOC()
requested area and the baseaddr pointer to what GF_?ALLOC() returned.

With these details, and the 'struct mem_header' that is placed when
calling GF_?ALLOC(), it is possible to identify overruns and possible
use-after-free. A memory dump (core) or running with a debugger is
needed to read the surrounding memory of corrupt structures.

This additional memory invalidation/poisoning needs to be enabled by
passing --enable-debug to ./configure.

Change-Id: I9f5f37dc4b5b59142adefc90897d32e89be67b82
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/10019
Reviewed-by: Venky Shankar 
Tested-by: Gluster Build System 
Reviewed-by: Emmanuel Dreyfus 
Reviewed-by: Kaleb KEITHLEY

libgfapi: revamp glfs_new function

2015-03-30T12:31:34+00:00

Current glfs_new() function is not flexible enough to error out
and destroy the struct members or objects initialized just before the
error path/condition. This make the structs or objects to continue or
left out with partially recorded data in fs and ctx structs and cause
crashes/issues later in the code path. This patch avoid the issue.

Change-Id: Ie4514b82b24723a46681cc7832a08870afc0cb28
BUG: 1202492 
Signed-off-by: Humble Devassy Chirammal 
Reviewed-on: http://review.gluster.org/9903
Tested-by: Gluster Build System 
Reviewed-by: Poornima G 
Reviewed-by: soumya k 
Reviewed-by: Kaleb KEITHLEY

testing: Switch to cmocka the successor of cmockery2

2015-03-05T21:51:37+00:00

This uses https://cmocka.org/ as the unit testing framework.

With this change, unit testing is made optional as well. We assume there
is no cmocka available while building. cmocka will be enabled by default
later on. For now, to build with cmocka run:

    $ ./configure --enable-cmocka

This change is based on the work of Andreas (replacing cmockery2 with
cmocka) and Kaleb (make cmockery2 an optional build dependency).

The only modifications I made, are additional #defines in unittest.h for
making sure the unit tests function as expected.

Change-Id: Iea4cbcdaf09996b49ffcf3680c76731459cb197e
BUG: 1067059
Merged-change: http://review.gluster.org/9762/
Signed-off-by: Andreas Schneider 
Signed-off-by: Kaleb S. KEITHLEY 
Signed-off-by: Niels de Vos 
Change-Id: Ia2e955481c102d5dce17695a9205395a6030e985
Reviewed-on: http://review.gluster.org/9738
Tested-by: Gluster Build System

mem-accounting: store type string and print in meta

2014-05-06T00:27:51+00:00

Change-Id: Ied1de473c72967110b54f033768f7ac60bf66edc
BUG: 1089216
Signed-off-by: Anand Avati 
Reviewed-on: http://review.gluster.org/7541
Tested-by: Gluster Build System 
Reviewed-by: Harshavardhana

log: enhance gluster log format with message ID and standardize errno reporting

2014-03-28T11:53:37+00:00

	Currently there are quite a slew of logs in Gluster that do not
	lend themselves to trivial analysis by various tools that help
	collect	and monitor logs, due to the textual nature of the logs.

	This FEAT is to make this better by giving logs message IDs so
	that the tools do not have to do complex log parsing to break
	it down to problem areas and suggest troubleshooting options.

	With this patch, a new set of logging APIs are introduced that
	take additionally a message ID and an error number, so as to
	print the message ID and the descriptive string for the error.

	New APIs:
		- gf_msg, gf_msg_debug/trace, gf_msg_nomem, gf_msg_callingfn

	These APIs follow the functionality of the previous gf_log*
	counterparts, and hence are 1:1 replacements, with the delta
	that, gf_msg, gf_msg_callingfn take additional parameters as
	specified above.

	Defining the log messages:
	Each invocation of gf_msg/gf_msg_callingfn, should provide an ID
	and an errnum (if available). Towards this, a common message id
	file is provided, which contains defines to various messages and
	their respective strings. As other messages are changed to the
	new infrastructure APIs, it is intended that this file is edited
	to add these messages as well.

	Framework enhanced:
	The logging framework is also enhanced to be able to support
	different logging backends in the future. Hence new configuration
	options for logging framework and logging formats are introduced.

	Backward compatibility:
	Currently the framework supports logging in the traditional
	format, with the inclusion of an error string based on the errnum
	passed in. Hence the shift to these new APIs would retain the log
	file names, locations, and format with the exception of an
	additional error string where applicable.

	Testing done:
	Tested the new APIs with different messages in normal code paths
	Tested with configurations set to gluster logs (syslog pending)
	Tested nomem variants, inducing the message in normal code paths
	Tested ident generation for normal code paths (other paths
		pending)
	Tested with sample gfapi program for gfapi messages
	Test code is stripped from the commit

	Pending work (not to be addressed in this patch (future)):
	- Logging framework should be configurable
	- Logging format should be configurable
	- Once all messages move to the new APIs deprecate/delete older
	  APIs to prevent misuse/abuse using the same
	- Repeated log messages should be suppressed (as a configurable
	  option)
	- Logging framework assumes that only one init is possible, but
	  there is no protection around the same (in existing code)
	- gf_log_fini is not invoked anywhere and does very little
	  cleanup (in existing code)
	- DOxygen comments to message id headers for each message

Change-Id: Ia043fda99a1c6cf7817517ef9e279bfcf35dcc24
BUG: 1075611
Signed-off-by: ShyamsundarR 
Reviewed-on: http://review.gluster.org/6547
Reviewed-by: Krutika Dhananjay 
Tested-by: Gluster Build System 
Reviewed-by: Raghavendra G 
Reviewed-by: Pranith Kumar Karampuri 
Reviewed-by: Vijay Bellur

build: GlusterFS Unit Test Framework

2014-03-06T12:10:46+00:00

This patch will allow for developers to create unit tests for
their code.  Documentation has been added to the patch and
is available here:

doc/hacker-guide/en-US/markdown/unittest.md

Also, unit tests are run when RPM is created.

BUG: 1067059
Change-Id: I95cf8bb0354d4ca4ed4476a0f2385436a17d2369
Signed-off-by: Vijay Bellur 
Signed-off-by: Luis Pabon 
Reviewed-on: http://review.gluster.org/7145
Tested-by: Gluster Build System 
Reviewed-by: Rajesh Joseph 
Reviewed-by: Justin Clift 
Tested-by: Justin Clift

stripe: remove unused param, handle mem alloc failure

2013-08-28T23:58:43+00:00

Change-Id: I9c27b1edab111031ca8eea9cc49480ea01e39089
BUG: 1002207
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/5716
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

libglusterfs/mem-pool: implement gf_memdup

2012-11-28T06:44:19+00:00

Change-Id: I1d7a1e7418f424fe08c041d562098b947c01580f
BUG: 808400
Signed-off-by: Raghavendra 
Reviewed-on: http://review.gluster.org/4124
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

libglusterfs: implement gf_strndup

2012-11-21T09:41:12+00:00

Change-Id: Ifb7bf22e8cf4ad1faccf7999c36919693912093f
BUG: 808400
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/4135
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur