glusterfs.git/xlators/mgmt/glusterd/src/Makefile.am, branch v3.5qa2

bd: posix/multi-brick support to BD xlator

2013-11-13T19:38:42+00:00

Current BD xlator (block backend) has a few limitations such as
* Creation of directories not supported
* Supports only single brick
* Does not use extended attributes (and client gfid) like posix xlator
* Creation of special files (symbolic links, device nodes etc) not
  supported

Basic limitation of not allowing directory creation is blocking
oVirt/VDSM to consume BD xlator as part of Gluster domain since VDSM
creates multi-level directories when GlusterFS is used as storage
backend for storing VM images.

To overcome these limitations a new BD xlator with following
improvements is suggested.

* New hybrid BD xlator that handles both regular files and block device
  files
* The volume will have both POSIX and BD bricks. Regular files are
  created on POSIX bricks, block devices are created on the BD brick (VG)
* BD xlator leverages exiting POSIX xlator for most POSIX calls and
  hence sits above the POSIX xlator
* Block device file is differentiated from regular file by an extended
  attribute
* The xattr 'user.glusterfs.bd' (BD_XATTR) plays a role in mapping a
  posix file to Logical Volume (LV).
* When a client sends a request to set BD_XATTR on a posix file, a new
  LV is created and mapped to posix file. So every block device will
  have a representative file in POSIX brick with 'user.glusterfs.bd'
  (BD_XATTR) set.
* Here after all operations on this file results in LV related
  operations.

For example opening a file that has BD_XATTR set results in opening
the LV block device, reading results in reading the corresponding LV
block device.

When BD xlator gets request to set BD_XATTR via setxattr call, it
creates a LV and information about this LV is placed in the xattr of the
posix file. xattr "user.glusterfs.bd" used to identify that posix file
is mapped to BD.

Usage:
Server side:
[root@host1 ~]# gluster volume create bdvol host1:/storage/vg1_info?vg1 host2:/storage/vg2_info?vg2
It creates a distributed gluster volume 'bdvol' with Volume Group vg1
using posix brick /storage/vg1_info in host1 and Volume Group vg2 using
/storage/vg2_info in host2.

[root@host1 ~]# gluster volume start bdvol

Client side:
[root@node ~]# mount -t glusterfs host1:/bdvol /media
[root@node ~]# touch /media/posix
It creates regular posix file 'posix' in either host1:/vg1 or host2:/vg2 brick
[root@node ~]# mkdir /media/image
[root@node ~]# touch /media/image/lv1
It also creates regular posix file 'lv1' in either host1:/vg1 or
host2:/vg2 brick
[root@node ~]# setfattr -n "user.glusterfs.bd" -v "lv" /media/image/lv1
[root@node ~]#
Above setxattr results in creating a new LV in corresponding brick's VG
and it sets 'user.glusterfs.bd' with value 'lv: --deltag
>

Changes from previous version V5:
* Removed support for delayed deleting of LVs

Changes from previous version V4:
* Consolidated the patches
* Removed usage of BD_XATTR_SIZE and consolidated it in BD_XATTR.

Changes from previous version V3:
* Added support in FUSE to support full/linked clone
* Added support to merge snapshots and provide information about origin
* bd_map xlator removed
* iatt structure used in inode_ctx. iatt is cached and updated during
fsync/flush
* aio support
* Type and capabilities of volume are exported through getxattr

Changes from version 2:
* Used inode_context for caching BD size and to check if loc/fd is BD or
  not.
* Added GlusterFS server offloaded copy and snapshot through setfattr
  FOP. As part of this libgfapi is modified.
* BD xlator supports stripe
* During unlinking if a LV file is already opened, its added to delete
  list and bd_del_thread tries to delete from this list when a last
  reference to that file is closed.

Changes from previous version:
* gfid is used as name of LV
* ? is used to specify VG name for creating BD volume in volume
  create, add-brick. gluster volume create volname host:/path?vg
* open-behind issue is fixed
* A replicate brick can be added dynamically and LVs from source brick
  are replicated to destination brick
* A distribute brick can be added dynamically and rebalance operation
  distributes existing LVs/files to the new brick
* Thin provisioning support added.
* bd_map xlator support retained
* setfattr -n user.glusterfs.bd -v "lv" creates a regular LV and
  setfattr -n user.glusterfs.bd -v "thin" creates thin LV
* Capability and backend information added to gluster volume info (and
--xml) so
  that management tools can exploit BD xlator.
* tracing support for bd xlator added

TODO:
* Add support to display snapshots for a given LV
* Display posix filename for list-origin instead of gfid

Change-Id: I00d32dfbab3b7c806e0841515c86c3aa519332f2
BUG: 1028672
Signed-off-by: M. Mohan Kumar 
Reviewed-on: http://review.gluster.org/4809
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

bd_map: Remove bd_map xlator

2013-11-13T19:38:28+00:00

Remove bd_map xlator and CLI related changes.

Change-Id: If7086205df1907127c1a1fa4ba603f1c48421d09
BUG: 1028672
Signed-off-by: M. Mohan Kumar 
Reviewed-on: http://review.gluster.org/5747
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

config: better (i.e. more portable) test for libxml2

2013-03-25T21:50:25+00:00

Over the weekend I tried to build on MacOS X¹ and ran into the following
issues:
1) The recent change to autogen.sh to test for pkg-config falls down.
2) After removing the pkg-config test in autogen.sh, w/o pkg-config the
   PKG_CHECK_MODULES macro invocation in configure[.ac] falls down. N.B.
   Solaris users run into this too, even through there's a (broken)
   pkg-config package that can be installed.
3) There are other problems in the code related to fuse that are beyond the
   scope of this.

It seems that pkg-config is only a requirement for the definition of the
PKG_CHECK_MODULES macro used to detect libxml2. Since this seems to be
inherently unportable — at least to MacOS X and Solaris — I'd like to:
A) Change the use of the PKG_CHECK_MODULES macro to the more portable
   AM_PATH_XML2 macro provided by the libxml2 package in
   /usr/.../share/aclocal/libxml.m4
2) Revisit the decision to add the check for pkg-config in autogen.sh in
   BZ 921817.

For now this is just an rfc. If people are agreeable I'll reenter this
change against BZ 921817.

¹Mountain Lion 10.8.3, XCode 4.6.1

Change-Id: I237b1ed8919088345b8fd943423b2a6ad289981b
BUG: 921817
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: http://review.gluster.org/4720
Reviewed-by: Justin Clift 
Tested-by: Justin Clift 
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

glusterd: Moved the volume entry table to a separate file.

2013-02-15T22:50:35+00:00

Change-Id: I893f41bd505fc0e02aec1e71f7a6209759b24a89
BUG: 903478
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/4517
Tested-by: Amar Tumballi 
Reviewed-by: Amar Tumballi 
Reviewed-by: Kaushal M 
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

Use proper libtool option -avoid-version instead of bogus -avoidversion

2013-02-07T23:12:56+00:00

Change-Id: I1c9541058c7d07786539a3266ca125a6a15287d8
BUG: 859835
Signed-off-by: Anand Avati 
Original-author: Kacper Kowalik (Xarthisius) 
Signed-off-by: Kacper Kowalik (Xarthisius) 
Reviewed-on: http://review.gluster.org/3967
Tested-by: Gluster Build System

BD Backend: Volume creation support

2012-11-29T17:39:25+00:00

A new parameter type is added to volume create command. To use BD xlator
one has to specify following argument in addition to normal volume
create

device vg brick:

for example,
$ gluster volume create lv_volume device vg host:/vg1

Changes from previous version

* New type 'backend' added to volinfo structure to differentiate between
  posix and bd xlator
* Most of the volume related commands are updated to handle BD xlator,
  like add-brick, heal-brick etc refuse to work when volume is BD xlator
  type
* Only one VG (ie brick) can be specified for BD xlator during volume
  creation
* volume info shows VG info if its of type BD xlator

BUG: 805138
Change-Id: I0ff90aca04840c71f364fabb0ab43ce33f9278ce
Signed-off-by: M. Mohan Kumar 
Reviewed-on: http://review.gluster.org/3717
Reviewed-by: Vijay Bellur 
Tested-by: Vijay Bellur

build: libraries must be in LDADD/LIBADD

2012-10-03T19:32:14+00:00

Libraries must not be listed in LDFLAGS, because _order is important_.
Details see
http://stackoverflow.com/questions/4241683/linker-flags-in-wrong-place
http://www.gentoo.org/proj/en/qa/asneeded.xml#doc_chap2_sect3

Change-Id: I0479456d6702748c555031bb20641ce430732ec7
BUG: 862082
Original-author: Jan Engelhardt 
Signed-off-by: Jan Engelhardt 
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/4030
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

build: split CPPFLAGS from CFLAGS

2012-10-03T19:26:45+00:00

Automake provides a separate variable for preprocessor flags
(*_CPPFLAGS). They are already uses in a few places, so make it
consistent and use it everywhere. Note that cflags obtained from
pkg-config often are cppflags, which is why LIBXML2_CFLAGS moves with
into AM_CPPFLAGS, for example.

Change-Id: I15feed1d18b2ca497371271c4b5876d5ec6289dd
BUG: 862082
Original-author: Jan Engelhardt 
Signed-off-by: Jan Engelhardt 
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/4029
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

build: move -L arguments out of CFLAGS

2012-10-03T19:25:34+00:00

"-L" is a linker flag, and as such should be in LDFLAGS, not CFLAGS, to
have guaranteed effect.

Change-Id: I014db3f48823d4923b45e0695b62b124eb1a1506
BUG: 862082
Original-author: Jan Engelhardt 
Signed-off-by: Jan Engelhardt 
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/4028
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

build: remove useless explicit -fPIC -shared from

2012-10-03T19:24:59+00:00

 CFLAGS

libtool will automatically add "-fPIC" to the compiler command line as
needed, so there is no need to specify it separately.

"-shared" is normally a linker flag and has an odd effect when used with
libtool --mode=compile, namely that it inhibits production of static
objects. For that however, using AC_DISABLE_STATIC is a lot simpler.

Change-Id: Ic4cba0fad18ffd985cf07f8d6951a976ae59a48f
BUG: 862082
Original-author: Jan Engelhardt 
Signed-off-by: Jan Engelhardt 
Signed-off-by: Jeff Darcy 
Reviewed-on: http://review.gluster.org/4027
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati