glusterfs.git/xlators/features, branch v3.5.0qa3

cli, glusterd: More quota fixes ...

2013-12-03T09:48:31+00:00

... which may be grouped under the following categories:

1. Fix incorrect cli exit status for 'quota list' cmd
2. Print appropriate error message on quota parse errors in cli

        Authored by: Anuradha Talur 

3. glusterd: Improve quota validation during stage-op
4. Fix peer probe issues resulting from quota conf checksum mismatches
5. Enhancements to CLI output in the event of quota command failures

        Authored by: Kaushal Madappa 

7. Move aux mount location from /tmp to /var/run/gluster

        Authored by: Krishnan Parthasarathi 

8. Fix performance issues in quota limit-usage

        Authored by: Krutika Dhananjay 

Note: Some functions that were used in earlier version of quota,
      that aren't called anymore have been removed.

Change-Id: I963d4145f3ecdfe30c61bfa8920baccb33d2d4bd
BUG: 969461
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/6386
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

cluster/dht: instruct marker whenever it shouldn't do accounting

2013-11-26T18:27:21+00:00

This is needed for two reasons:

* since dht-linkfiles are internal, they shouldn't be accounted.
* hardlink handling in marker is broken. link/unlink of hardlinks
  present in same directory can break marker accounting. Hence, if src
  and dst are in same directory in case of rename, dht - if it breaks
  rename into link/unlink operations - should instruct marker to not to
  do accounting.

Change-Id: I9c9f7384569f75a2792f6450ee7a5279bf751ae7
BUG: 1022995
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/6203
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

features/marker-quota: exclude dht-linkfiles from being accounted.

2013-11-26T18:27:03+00:00

Change-Id: I3239f5e8477664dcc04434e4d455ae447493a7ac
BUG: 1022995
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/6153
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

features/quota: make writes short when the entire write cannot fit into available space.

2013-11-26T18:26:26+00:00

This patch aims to prevent creation of infinite zero byte sized files
due to amount of storage available before exceeding quota limit
being less than write sizes. Imagine x bytes of storage is available
before we exceed quota limit and quota enforcer is receiving writes of
size y and (y > x). In this scenario, if we run a shell script like:

   # for i in $(seq 1 10); do dd if=/dev/zero of=$i bs=y count=1; done

Then, we would end up with 10 zero byte sized files, because we allow
only complete writes and all writes will fail because of lack of space.
However, creates succeed since a create itself will consume zero
bytes. In this pattern of creates and writes, size of volume would
never grow and x bytes of space will always be available and we can
end up with an infinite number of zero byte sized files.

Change-Id: Ice148d6a2207883e41759f7b0be73abaa3198b41
BUG: 1012216
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/6035
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

features/quota: Improvements to quota

2013-11-26T18:24:02+00:00

* Two stages of quota enforcement is done:

  Soft and hard quota Upon reaching soft quota limit on the directory
  it logs/alerts in the quota daemon log (ie DEFAULT_LOG_DIR/quotad.log)
  and no more writes allowed after hard
  quota limit. After reaching the soft-limit the daemon alerts the
  user/admin repeatively for every 'alert-time', which is
  configurable.

* Quota enforcer is moved to server-side.

  It  takes care of enforcing quota. Since enforcer doesn't have the
  cluster view, it relies on another service called
  quota-aggregator. Aggregator, on query can return the size of a
  directory based on the cluster view.

  Enforcer is always loaded in the server graph and is by passed if
  the feature is not enabled.

  Options specific to enforcer:

  server-quota - Specifies whether the feature is on/off. It is used
  to by pass the quota if turned off.

  deem-statfs - If set to on, it takes quota limits into consideration
  while estimating fs size. (df command). The algorithm followed is,
  i.   Adjust statvfs based on limit configured on root.
  ii.  If limit is set on the inode passed, use size/limits on that inode to
       populate statvfs. Otherwise, use size/limits configured on root.
  iii. Upon statvfs, update the ctx->size on the inode.
  iv.  Don't let DHT aggregate, instead take the maximum of the usages from the
       subvols of the DHT, since each of it contains the complete information.

  Enforcer also makes use of gfid-to-path conversion functionality to
  work correctly when a client like nfs predominently relies on
  nameless lookups.

* Quota Aggregator acts as a thin client to provide cluster view

  Its a lightweight *gluster client* process with no mount point,
  started upon enabling quota or restarting the volume. This is a
  single process run on each brick, which can answer queries on all
  volumes in the cluster. Its volfile stored in
  GLUSTERD_DEFAULT_WORKING_DIR/quotad/quotad.vol.

Credits:
Raghavendra Bhat        
Varun Shastry           
Shishir Gowda           
Kruthika Dhananjay      
Brian Foster            
Krishnan Parthasarathi  

Change-Id: Id1cb25b414951da34c665a55f77385d482e0f9de
BUG: 969461
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/5952
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

features/marker: quota friendly changes

2013-11-26T18:23:02+00:00

* handles renames on dht linkfiles correctly

* nameless lookup friendly changes. uses gfid-to-path conversion
  functionality from storage/posix to build ancestry till root.

* log message cleanup.

* build inode contexts in readdirp

* Accounting still not correct with hardlinks.

Credits:
========
Vijay Bellur 
Raghavendra Bhat 

Change-Id: I415b6fbbc9691f5a38d9fd3c5d083a61e578bb81
BUG: 969461
Signed-off-by: Raghavendra G 
Reviewed-on: http://review.gluster.org/5953
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

NetBSD missing backtrace(3) portability fix

2013-11-19T21:30:39+00:00

Implement backtrace(3) and backtrace_symbols(3) which do not exist in NetBSD
While there, remove duplicate #include 

BUG: 764655
Change-Id: Iccd695765906e085c3f8fcb670506d4fea68fa39
Signed-off-by: Emmanuel Dreyfus 
Reviewed-on: http://review.gluster.org/6285
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati

zerofill: Change the type of len argument of glfs_zerofill() to off_t

2013-11-15T07:29:48+00:00

glfs_zerofill() can be potentially called to zero-out entire file and
hence allow for bigger value of length parameter.

Change-Id: I75f1d11af298915049a3f3a7cb3890a2d72fca63
BUG: 1028673
Signed-off-by: Bharata B Rao 
Reviewed-on: http://review.gluster.org/6266
Tested-by: Gluster Build System 
Reviewed-by: M. Mohan Kumar 
Tested-by: M. Mohan Kumar 
Reviewed-by: Anand Avati

features/compress: Compression/DeCompression translator

2013-11-12T03:35:01+00:00

* When a writev call occurs, the client compresses the data before
  sending it to server. On the server, compressed data is decompressed.
  Similarly, when a readv call occurs, the server compresses the data
  before sending it to client. On the client, the compressed data is
  decompressed. Thus the amount of data sent over the wire is minimized.

* Compression/Decompression is done using Zlib library.

* During normal operation, this is the format of data sent over wire :
   + trailer(8)
  The trailer contains the CRC32 checksum and length of original
  uncompressed data. This is used for validation.

HOW TO USE
----------
Turning on compression xlator:
gluster volume set  compress on

Configurable options:
gluster volume set  compress.compression-level 8
gluster volume set  compress.min-size 50

Change-Id: Ib7a66b6f1f70fe002b7c513588cdf75c69370805
BUG: 923540
Original-author : Venky Shankar 
Signed-off-by: Venky Shankar 
Signed-off-by: Prashanth Pai 
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/3251
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/qemu-block: simplify coroutine model to use single synctask, ucontext

2013-11-11T07:45:46+00:00

The current coroutine model, mapping synctasks 1-1 with qemu internal
Coroutines, has some unresolved raciness issues. This problem usually
manifests as lifecycle mismatches between top-level (gluster created)
synctasks and the subsequently created internal coroutines from that
context. Qemu's internal queueing (and locking) can cause situations
where the top-level synctask is destroyed before the internal scheduler
has released references to memory, leading to use after free crashes
and asserts.

Simplify the coroutine model to use a single synctask as a coroutine
processor and rely on the existing native ucontext coroutine
implementation. The syncenv thread is donated to qemu and ensures a
single top-level coroutine is processed at a time. Qemu now has
complete control over coroutine scheduling.

BUG: 986775
Change-Id: I38223479a608d80353128e390f243933fc946fd6
Signed-off-by: Brian Foster 
Reviewed-on: http://review.gluster.org/6110
Tested-by: Gluster Build System 
Reviewed-by: Anand Avati