glusterfs.git/libglusterfs/src/Makefile.am, branch v3.10.7

cluster/tier: handle fast demotions

2016-10-19T19:51:48+00:00

Demote files on priority if hi-watermark has been breached and continue
to demote until the watermark drops below hi-watermark.

Monitor watermark more frequently.
Trigger demotion as soon as hi-watermark is breached.
Add cluster.tier-emergency-demote-query-limit option to limit number
of files returned from the database query for every iteration of
tier_migrate_using_query_file(). If watermark hasn't dropped below
hi-watermark during the first iteration, the next iteration will be
triggered approximately 1 second after tier_demote() returns to the
main tiering loop.
Update changetimerecorder xlator to handle query for emergency demote
mode.

Add tier-ctr-interface.h:
Move tier and ctr interface specific macros and struct definition from
libglusterfs/src/gfdb/gfdb_data_store.h to new header
libglusterfs/src/tier-ctr-interface.h

Change-Id: If56af78c6c81d37529b9b6e65ae606ba5c99a811
BUG: 1366648
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15158
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Dan Lambright

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

compound fops: Some fixes to compound fops framework

2016-08-31T06:50:40+00:00

Change-Id: I808fd5f9f002a35bff94d310c5d61a781e49570b
BUG: 1360169
Signed-off-by: Anuradha Talur 
Reviewed-on: http://review.gluster.org/15010
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

glusterd/cli: cli to get local state representation from glusterd

2016-08-26T15:23:37+00:00

Currently there is no existing CLI that can be used to get the
local state representation of the cluster as maintained in glusterd
in a readable as well as parseable format.

The CLI added has the following usage:

 # gluster get-state [daemon] [odir ] [file ]

This would dump data points that reflect the local state
representation of the cluster as maintained in glusterd (no other
daemons are supported as of now) to a file inside the specified
output directory. The default output directory and filename is
/var/run/gluster and glusterd_state_ respectively. The
option for specifying the daemon name leaves room to add support for
other daemons in the future. Following are the data points captured
as of now to represent the state from the local glusterd pov:

 * Peer:
    - Primary hostname
    - uuid
    - state
    - connection status
    - List of hostnames

 * Volumes:
    - name, id, transport type, status
    - counts: bricks, snap, subvol, stripe, arbiter, disperse,
 redundancy
    - snapd status
    - quorum status
    - tiering related information
    - rebalance status
    - replace bricks status
    - snapshots

 * Bricks:
    - Path, hostname (for all bricks these info will be shown)
    - port, rdma port, status, mount options, filesystem type and
signed in status for bricks running locally.

 * Services:
    - name, online status for initialised services

 * Others:
    - Base port, last allocated port
    - op-version
    - MYUUID

Change-Id: I4a45cc5407ab92d8afdbbd2098ece851f7e3d618
BUG: 1353156
Signed-off-by: Samikshan Bairagya 
Reviewed-on: http://review.gluster.org/14873
Reviewed-by: Avra Sengupta 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee

build: fix eventtypes.h generation

2016-08-26T12:38:00+00:00

make 'eventtypes.py' as dependent i.e. must build eventtypes.h

Else the consequence will be like:
error: 'EVENT_CLIENT_GRACE_TIMER_START' undeclared (first use in this function)

Change-Id: I5fe2491d8d1e0c430b307026695d25475250ae79
BUG: 1370406
Signed-off-by: Prasanna Kumar Kalever 
Reviewed-on: http://review.gluster.org/15327
Tested-by: Prasanna Kumar Kalever 
Smoke: Gluster Build System 
Reviewed-by: Aravinda VK 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

eventsapi: Fix disable-events issue

2016-08-24T08:23:06+00:00

Events related sources are not loaded in libglusterfs when
configure is run with --disable-events option. Due to this
every call of gf_event should be guarded with USE_EVENTS macro.

To prevent this, USE_EVENTS macro was included in events.c
itself(Patch #15054)

Instead of disabling building entire directory "events", selectively
disabled the code. So that constants and empty function gf_event is
exposed. Code will not fail even if gf_event is called when events is
disabled.

BUG: 1368042
Change-Id: Ia6abfe9c1e46a7640c4d8ff5ccf0e9c30c87f928
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15198
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

eventsapi: Auto generate header files during make

2016-08-02T05:49:59+00:00

$SRC/libglusterfs/src/eventtypes.h and $SRC/events/src/eventtypes.py are
generated by running `python $SRC/events/eventskeygen.py`

Header files generation step is added to make file itself, Now All new
events should be added to only to $SRC/events/eventskeygen.py file.

BUG: 1361094
Change-Id: I384961ef2978ca2d0be37f288b39ac0d834bdf06
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15035
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kotresh HR 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee

features/ctr: Check for NULL local

2016-07-25T20:14:09+00:00

This is a defensive fix to prevent a crash reported
during a rename operation. This is not reproducible
under normal circumstances.

This patch also moves ctr-messages.h to the src dir
of the changetimerecorder xlator.

Change-Id: I46eb926d67bf4c19387c8b26e354c635a5fb284c
BUG: 1358196
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/14964
NetBSD-regression: NetBSD Build System 
Tested-by: Zhou Zhengping 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Vijay Bellur

eventsapi: Gluster Eventing Feature implementation

2016-07-18T20:52:20+00:00

[Depends on http://review.gluster.org/14627]

Design is available in `glusterfs-specs`, A change from the design
is support of webhook instead of Websockets as discussed in the design

http://review.gluster.org/13115

Since Websocket support depends on REST APIs, I will add Websocket support
once REST APIs patch gets merged

Usage:
Run following command to start/stop Eventsapi server in all Peers,
which will collect the notifications from any Gluster daemon and emits
to configured client.

    gluster-eventsapi start|stop|restart|reload

Status of running services can be checked using,

    gluster-eventsapi status

Events listener is a HTTP(S) server which listens to events emited by
the Gluster. Create a HTTP Server to listen on POST and register that
URL using,

    gluster-eventsapi webhook-add  [--bearer-token ]

For example, if HTTP Server running in `http://192.168.122.188:9000`
then add that URL using,

    gluster-eventsapi webhook-add http://192.168.122.188:9000

If it expects a Token then specify it using `--bearer-token` or `-t`

We can also test Webhook if all peer nodes can send message or not
using,

    gluster-eventsapi webhook-test  [--bearer-token ]

Configurations can be viewed/updated using,

    gluster-eventsapi config-get [--name]
    gluster-eventsapi config-set  
    gluster-eventsapi config-reset 

If any one peer node was down during config-set/reset or webhook
modifications, Run sync command from good node when a peer node comes
back. Automatic update is not yet implemented.

    gluster-eventsapi sync

Basic Events Client(HTTP Server) is included with the code, Start
running the client with required port and start listening to the
events.

    /usr/share/glusterfs/scripts/eventsdash.py --port 8080

Default port is 9000, if no port is specified, once it started running
then configure gluster-eventsapi to send events to that client.

Eventsapi Client can be outside of the Cluster, it can be run event on
Windows. But only requirement is the client URL should be accessible
by all peer nodes.(Or ngrok(https://ngrok.com) like tools can be used)

Events implemented with this patch,
- Volume Create
- Volume Start
- Volume Stop
- Volume Delete
- Peer Attach
- Peer Detach

It is easy to add/support more events, since it touches Gluster cmd
code and to avoid merge conflicts I will add support for more events
once this patch merges.

BUG: 1334044
Change-Id: I316827ac9dd1443454df7deffe4f54835f7f6a08
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/14248
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy

features/bitrot: Move throttling code to libglusterfs

2016-07-18T12:03:17+00:00

Since throttling is a separate feature by itself,
move throttling code to libglusterfs.

Change-Id: If9b99885ceb46e5b1865a4af18b2a2caecf59972
BUG: 1352019
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/14846
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Ravishankar N 
Reviewed-by: Jeff Darcy