glusterfs.git/libglusterfs/src/generator.py, branch v4.1.0alpha

core/build/various: python3 compat, prepare for python2 -> python3

2018-04-12T11:04:27+00:00

Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines

Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.

This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.

In a follow-up patch python2 will eventually be changed to python3.

Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.

updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY

experimental/cloudsync: Download xlator for archival feature

2018-04-10T01:09:29+00:00

spec-files:
https://review.gluster.org/#/c/18854/

Overview:
* Cloudsync maintains three file states in it's inode-ctx i.e
  1 - LOCAL,
  2 - REMOTE,
  3 - DOWNLOADING.

* A data modifying fop is allowed only if the state is LOCAL.
  If the state is REMOTE or DOWNLOADING, client will download
  or wait for the download to finish initiated by other client.

* Multiple download and upload from different clients are synchronized
  by inodelk.

* In POSIX a state check is done (part of different commit)before
  allowing the fop to continue. If the state is remote/downloading the
  fop is unwound with EREMOTE. The client will then download the file
  and continue with the fop again.

* Basic Algo for fop (let's say write fop):
  - If LOCAL -> resume fop
  - If REMOTE ->
	- INODELK
	- STAT (this gets state and heal the state if needed)
	- DOWNLOAD
	- resume fop

Note:
* Developers will need to write plugins for download, based on the
remote store they choose. In phase-1, support will be added for
one remote store per volume. In future, more options for multiple
remote stores will be explored.

TODOs:
 - Implement stat/lookup/readdirp to return size info from xattr
 - Make plugins configurable
 - Implement unlink fop
 - Add metrics collection
 - Add sharding support

Design Contributions:
Aravinda V K 
Amar Tumballi 
Ram Ankireddypalle 
Susant Palai 

updates: #387
Change-Id: Iddf711ee7ab4e946ae3e472ff62791a7b85e6d4b
Signed-off-by: Susant Palai

gfapi: return pre/post attributes from glfs_ftruncate

2018-02-12T21:34:46+00:00

Updates: #389
Change-Id: I8faea0828921fb17f05f7321c3cb01747373f21e
Signed-off-by: Kinglong Mee

gfapi: return pre/post attributes from glfs_fsync/fdatasync

2018-02-12T21:34:46+00:00

Updates: #389
Change-Id: I4153df72d5eeecefa7579170899db4c340128bea
Signed-off-by: Kinglong Mee

gfapi: return pre/post attributes from glfs_pread/pwrite

2018-02-12T21:34:46+00:00

As nfs-ganesha, a wcc data contains pre/post attributes is return
in read/write rpc reply. nfs-ganesha get those attributes by
two getattr between the real read/write right now.

But, gluster has return pre/post attributes from glusterfsd,
those attributes are skipped in syncop/gfapi, if gfapi return them,
the upper user (nfs-ganesha) can use them directly without any
duplicate getattr.

Updates: #389
Change-Id: I7b643ae4241cfe2aeb17063de00192d81674024a
Signed-off-by: Kinglong Mee

rio/everywhere: add icreate/namelink fop

2017-12-05T21:23:57+00:00

icreate creates inode, while namelink links the basename to it's
parent gfid.

For now mkdir is the primary user of these fops. Better distribution is
acheived by creating the inode on ,(say) mds1 and linking the basename to it's
parent gfid on mds2. The inode serves readdirp, stat etc.

More details about the fops are present at:
https://review.gluster.org/#/c/13395/3/design/DHT2/DHT2_Icreate_Namelink_Notes.md

This backport of three patches from experimental branch.
1- https://review.gluster.org/#/c/18085/
2- https://review.gluster.org/#/c/18086/
3- https://review.gluster.org/#/c/18094/

Updates gluster/glusterfs#243
Change-Id: I1bd3d5a441a3cfab1acfeb52f15c6c867d362592
Signed-off-by: Susant Palai

libglusterfs: Add put fop

2017-12-05T14:21:01+00:00

Problem: It had been a longtime request to implement put fop
in gluster. put fop in gluster may not have the exact sementics
of HTTP PUT, but can be easily extended to do so. The subsequent
patches, will contain more semantics on the put fop and its
guarentees.

Why compound fop framework is not used for put?
Compound fop framework currently doesn't allow compounding of
entry fop and inode fops, i.e. fops on multiple inodes cannot be
combined in compound fop.

Updates #353
Change-Id: Idb7891b3e056d46d570bb7e31bad1b6a28656ada
Signed-off-by: Poornima G

jbr: Sending rollback from failed fop to fdl

2016-11-08T19:25:25+00:00

In case of a failed fop, the failure is detected
by the leader in the jbr-server in two places. First
during a quorum check of +ve responses when it
receives responses from all the followers. At this
point if the fop hasn't been successfully journaled
at a quorum of followers (as in there is no merit in
trying the fop in the leader as the quorum will never
be met), then we fail the fop.

Also if this quorum is met, then the fop is tried on
the leader, and after the leader completes the fop
a quorum check similar to the previous one is done
again, this time including the leaders outcome. If
quorum is not met, then we fail the fop.

In both these cases, when the fop fails we send a -ve
ack to the client. With this patch, now we will also
send a rollback through a GF_FOP_IPC to all the followers(and
also to the leader in the second case of failure). This
rollback will contain the index and term number of the
fop which failed. This will be recorded in the respective
journals of the bricks and will be used to rollback the
fop on that brick later.

A subsequent write, and it's respective rollback would
look something like the following in the journal.

The trusted.jbr.term and trusted.jbr.index present in the
dict of both the logs, relate them, and the presence of
"rollback-fop" in the dict of IPC indicates that it is a
rollback fop, and the value 13(stands for GF_FOP_WRITE)
indicates what kind of rollback operation it is.

=== GF_FOP_WRITE
fd = 
vector = <158 bytes>
offset = 0 (0x0)
flags = 32769 (0x8001)
xdata = dict {
 trusted.jbr.term = 0 <2 bytes>
 trusted.jbr.index = 4 <2 bytes>
}
=== GF_FOP_IPC
xdata = dict {
 trusted.jbr.term = 0 <2 bytes>
 trusted.jbr.index = 4 <2 bytes>
 rollback-fop = 13 <3 bytes>
}

Change-Id: I70b6a143d20697153d58e2f719e34ecd1ed160a5
BUG: 1349385
Signed-off-by: Avra Sengupta 
Reviewed-on: http://review.gluster.org/14783
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Jeff Darcy 
Smoke: Gluster Build System

core: add setactivelk () fop

2016-05-02T01:04:42+00:00

Change-Id: Ic2ba77a1fdd27801a6e579e04e6c0dd93cd7127b
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/14011
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

core: add getactivelk () fop

2016-05-02T01:04:21+00:00

Change-Id: Ifd0ff278dcf43da064021f5c25e5dcd34347fcde
BUG: 1326085
Signed-off-by: Susant Palai 
Reviewed-on: http://review.gluster.org/13970
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System