summaryrefslogtreecommitdiffstats
path: root/xlators/features/changelog
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'upstream'HEADmasterJeff Darcy2014-04-287-9/+344
|\ | | | | | | | | | | | | | | | | | | | | | | Conflicts: rpc/xdr/src/glusterfs3-xdr.c rpc/xdr/src/glusterfs3-xdr.h xlators/features/changelog/src/Makefile.am xlators/features/changelog/src/changelog-helpers.h xlators/features/changelog/src/changelog.c xlators/mgmt/glusterd/src/glusterd-sm.c Change-Id: I9972a5e6184503477eb77a8b56c50a4db4eec3e2
| * features/libgfchangelog: APIs to process history changelogs.Kotresh H R2014-04-284-2/+336
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Create directories in following fashion for history API's usage when consumer is registered with libgfchangelog shared library through gf_changelog_register. scratch_dir/.history scratch_dir/.history/.current scratch_dir/.history/.processed scratch_dir/.history/.processing 2. Added new file 'gf-history-changelog.c' and following APIs are provided for consumers to process history changelogs. 1. gf_history_changelog_scan: Move processed history changelog file from .processing to .processed 2. gf_history_changelog_next_change: Return the next history changelog file entry. Zero means all history chanelogs are consumed. 3. gf_history_changelog_done: Scan .processing directory and generate a list of change entries. 4. gf_history_changelog_start_fresh: For a set of changelogs, start from the begining. NOTE: Though this patch provides above funcationalities. It is considered functionally full with the patch (http://review.gluster.org/#/c/6930/). Change-Id: I200780c7278e0a6c008910d93faad5858a4b3e76 Original-author: Kotresh H R <khiremat@redhat.com> Signed-off-by: Kotresh H R <khiremat@redhat.com> Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6998 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
| * feature/changelog: Change default rollover and fsync-interval timeKotresh H R2014-04-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | This will change the following default configurables. 1. rollover-time: from 60 to 15. 2. fsync-interval: from 0 to 5. Change-Id: I9c8db01376967c4f19547ec87f54833f8b139d29 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7545 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
| * build: MacOSX Porting fixesHarshavardhana2014-04-242-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git@forge.gluster.org:~schafdog/glusterfs-core/osx-glusterfs Working functionality on MacOSX - GlusterD (management daemon) - GlusterCLI (management cli) - GlusterFS FUSE (using OSXFUSE) - GlusterNFS (without NLM - issues with rpc.statd) Change-Id: I20193d3f8904388e47344e523b3787dbeab044ac BUG: 1089172 Signed-off-by: Harshavardhana <harsha@harshavardhana.net> Signed-off-by: Dennis Schafroth <dennis@schafroth.com> Tested-by: Harshavardhana <harsha@harshavardhana.net> Tested-by: Dennis Schafroth <dennis@schafroth.com> Reviewed-on: http://review.gluster.org/7503 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* | Merge branch 'upstream'Jeff Darcy2014-03-241-1/+1
|\|
| * build: do not create versioned <xlator>.so filesNiels de Vos2014-03-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There has been a misspelled option in the Makefile.am files. The option is called -avoid-version, and not -avoidversion. It is not trivial to provide a test-case for this. One way would be to check generated RPMs with a command like this (output should be empty): $ rpm -qlp *.rpm | grep -E '/xlator/.+.so.0' Change-Id: I2a6cc557eada4d098b73af5a254f8c75707543da BUG: 1078365 Signed-off-by: Niels de Vos <ndevos@redhat.com> Reviewed-on: http://review.gluster.org/7299 Reviewed-by: Lalatendu Mohanty <lmohanty@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* | features/changelog: NSR Journal changesVenky Shankar2014-03-124-117/+443
| | | | | | | | | | | | | | | | | | | | | | | | | | * Sequential updates to journal * Journal update in the FOP path * NSR specific format changes TBD --- * POST-OP record Change-Id: I5b21b7624ccb095295a0c69abf00866e0d6cd818 Signed-off-by: Venky Shankar <vshankar@redhat.com>
* | features/changelog: fix possible missed UNLOCK()Venky Shankar2014-02-131-5/+11
| | | | | | | | | | Change-Id: I6d53c8fcbfdc1f80cd6ef9fb6bb6c7b32541116b Signed-off-by: Venky Shankar <vshankar@redhat.com>
* | Merge remote-tracking branch 'upstream/master'Jeff Darcy2014-02-114-4/+5
|\| | | | | | | | | | | | | Conflicts: xlators/mgmt/glusterd/src/Makefile.am Change-Id: Ida5ec4aecc358cb2268bdfdb1a8c9bab750f9575
| * changelog: When connection is unsuccessful, close socket too.Raghavendra Talur2014-02-071-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Bug fix for Coverity CID: 1124791 Change-Id: I0362d45123ebc250290f3a5231f7fb113fa41212 BUG: 789278 Signed-off-by: Raghavendra Talur <rtalur@redhat.com> Reviewed-on: http://review.gluster.org/6900 Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com> Reviewed-by: Santosh Pradhan <spradhan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
| * changelog: Restrict length of unix socket files to UNIX_PATH_MAX.Raghavendra Talur2014-02-054-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | Bug fix for Coverity-CID 1124847. Change-Id: I410ef8e06cbc491b1f72535298fae5e9bc77220d BUG: 789278 Signed-off-by: Raghavendra Talur <rtalur@redhat.com> Reviewed-on: http://review.gluster.org/6870 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* | Example of how to add extra info to changelog records.Jeff Darcy2014-02-111-1/+9
| | | | | | | | | | | | | | | | This is a bit hacky so far, but this is the direction we need to go for all fops that take extra data like modes, uids, times, or keys. Change-Id: I02968611bc46beda108b4caf65cd6a8928679e16 Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
* | features/changelog: remove dead code and fix compile warningsVenky Shankar2014-01-311-116/+1
| | | | | | | | | | Change-Id: I869121a31c057a73534e59db91a7d2392037763a Signed-off-by: Venky Shankar <vshankar@redhat.com>
* | NSR reconciliation: Adding index for each opRaghavan P2014-01-231-2/+5
| | | | | | | | | | | | | | We need to have the leader generate an index number for each changelog entry. Change-Id: If451dc00a21dbc481771b713ac75fd0af6887014 Signed-off-by: Raghavan P <rpichai@redhat.com>
* | Merge branch 'upstream'Jeff Darcy2014-01-211-1/+1
|\| | | | | | | | | | | | | | | Conflicts: api/src/glfs-fops.c api/src/glfs-handleops.c Change-Id: I6811674cc4ec4be6fa6e4cdebb4bc428194bebd8
| * build: Start using library versioning for various librariesHarshavardhana2014-01-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | According to libtool three individual numbers stand for CURRENT:REVISION:AGE, or C:R:A for short. The libtool script typically tacks these three numbers onto the end of the name of the .so file it creates. The formula for calculating the file numbers on Linux and Solaris is /path/to/library/<library_name>.(C - A).(A).(R) As you release new versions of your library, you will update the library's C:R:A. Although the rules for changing these version numbers can quickly become confusing, a few simple tips should help keep you on track. The libtool documentation goes into greater depth. In essence, every time you make a change to the library and release it, the C:R:A should change. A new library should start with 0:0:0. Each time you change the public interface (i.e., your installed header files), you should increment the CURRENT number. This is called your interface number. The main use of this interface number is to tag successive revisions of your API. The AGE number is how many consecutive versions of the API the current implementation supports. Thus if the CURRENT library API is the sixth published version of the interface and it is also binary compatible with the fourth and fifth versions (i.e., the last two), the C:R:A might be 6:0:2. When you break binary compatibility, you need to set AGE to 0 and of course increment CURRENT. The REVISION marks a change in the source code of the library that doesn't affect the interface-for example, a minor bug fix. Anytime you increment CURRENT, you should set REVISION back to 0. Change-Id: Id72e74c1642c804fea6f93ec109135c7c16f1810 BUG: 862082 Signed-off-by: Harshavardhana <harsha@harshavardhana.net> Reviewed-on: http://review.gluster.org/5645 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Niels de Vos <ndevos@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
| * features/changelog: more changelog fixes.Ajeet Jha2013-12-128-38/+212
| | | | | | | | | | | | | | | | | | | | | | | | | | | | -> log additional records. -> include FOP number for metadata. -> prevent crash if inode is not found in a fop. Change-Id: I9edd4b71819ebd68c6a2b4150ae279c471d129da BUG: 1036536 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6403 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Amar Tumballi <amarts@gmail.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* | features/changelog: couple of performance improvementsVenky Shankar2014-01-166-10/+34
| | | | | | | | | | | | | | | | | | Changes being: * Changes journal preallocation size to 512 MB * Lock-less updation of journal Change-Id: I7c2b7ee224b3d338e4fdd35cc078ea2a13251292 Signed-off-by: Venky Shankar <vshankar@redhat.com>
* | Restore NSR-specific changelog patches.Jeff Darcy2014-01-1413-664/+585
| | | | | | | | | | Change-Id: I40fb3a024086dddbdaba3d9014eeda92dc432f48 Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
* | Temporarily revert NSR-specific changelog patches.Jeff Darcy2014-01-1413-585/+664
| | | | | | | | | | | | | | | | | | This is necessary so that patches from upstream can merge cleanly. Otherwise there are some nasty conflicts, and resolving them by hand gets even uglier than this approach. Change-Id: I4235f8ba0ad63563c2e7dec1a1e8eeb636657574 Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
* | Roll-up patch for NSR so far.Jeff Darcy2013-12-1116-468/+2550
|/ | | | | | | | Previous history: https://forge.gluster.org/~jdarcy/glusterfs-core/glusterfs-nsr Change-Id: I2b56328788753c6a74d9589815f2dd705ac9ce6a Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
* features/changelog : Improvement in changelog "encoding-change".ajha2013-09-294-17/+26
| | | | | | | | | | | | | | | | | change in encoding method of changelog was critical section for "fop dispatch thread", "roll-over thread" and "reconfigure dispatch thread". In this patch the "encoding-method" is changed by the reconfigure dispatch thread lazily during handle_change, which solves the concurrency among the racing threads. BUG: 1002940 Change-Id: I78c3e8887efa46d0fcc60755cdf4243031cfa3eb Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/5844 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
* core: block unused signals in created threadsAnand Avati2013-09-252-8/+8
| | | | | | | | | | | | | | | Block all signal except those which are set for explicit handling in glusterfs_signals_setup(). Since thread spawning code in libglusterfs and xlators can get called from application threads when used through libgfapi, it is necessary to do this blocking. Change-Id: Ia320f80521a83d2edcda50b9ad414583a0175281 BUG: 1011662 Signed-off-by: Anand Avati <avati@redhat.com> Reviewed-on: http://review.gluster.org/5995 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* gsyncd / geo-rep: Introduce basic crawl instrumentationVenky Shankar2013-09-041-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | This patch extends the persistent instrumentation work done by Aravinda (@avishwa), by introducing a handfull of instrumentation variables for crawl. These variables are "pulled up" by glusterd in the event of a geo-replication status cli command and looks something like below: "Uptime=00:21:10;FilesSyned=2982;FilesPending=0;BytesPending=0;DeletesPending=0;" "FilesPending", "BytesPending" and "DeletesPending" are short-lived variables that are non-zero when a changelog is being processes (ie. when an active sync in ongoing). After a successfull changelog process "FilesPending" is summed up into "FilesSynced". The three short-lived variabled are then reset to zero and the data is persisted Additionally this patch also reverts some of the changes made for BZ #986929 (those were not needed). Change-Id: I948f1a0884ca71bc5e5bcfdc017d16c8c54fc30b BUG: 990420 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/5441 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* glusterd: Try to start all bricks on 'start force'Kaushal M2013-08-181-4/+6
| | | | | | | | | | | | | | | | | | | A volume would fail to start if any one of the bricks fails staging or fails to start, even with the 'force' option. With this patch, when the 'force' option is given for a volume start, glusterd will continue and start other bricks even if one fails staging or starting. Also did a small fix in changelog, to prevent it crashing when it fails to init. Change-Id: I7efbd9ab13d12d69b0335ae54143fa17586f8f98 BUG: 994375 Signed-off-by: Kaushal M <kaushal@redhat.com> Reviewed-on: http://review.gluster.org/5510 Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* log: set ident to openlogBala.FA2013-08-131-1/+2
| | | | | | | | | | | | | | | | | | | | at syslog side, log message is identified by its properties like programname, pid, etc. brick/mount processes need to be identified uniquely as they are different process of gluterfsd/glusterfs. At rsyslog side, log separated by programname/app-name with pid works but bit hard to identify them in long run which process is for what brick/mount. This patch fixes by setting identity string at openlog() which sets programname/app-name as similar to old style log file prefixed by gluster, glusterd, glusterfs or glusterfsd Change-Id: Ia05068943fa67ae1663aaded1444cf84ea648db8 BUG: 928648 Signed-off-by: Bala.FA <barumuga@redhat.com> Reviewed-on: http://review.gluster.org/5541 Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* features/changelog: fixes when enabling changelogVenky Shankar2013-07-252-29/+41
| | | | | | | | | | | | | Other enhancements being: * ignore fops made by rebalance * ignore internally triggered fops BUG: 987734 Change-Id: I7dd164ae3c209fdb8ec43a27e67b8846f937c93b Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/5380 Reviewed-by: Amar Tumballi <amarts@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* features/changelog: changelog translatorAvra Sengupta2013-07-2223-0/+4970
This is the initial version of the Changelog Translator. What is it ----------- Goal is to capture changes performed on a GlusterFS volume. The translator needs to be loaded on the server (bricks) and captures changes in a plain text file inside a configured directory path (controlled by "changelog-dir", should be somewhere in <export>/.glusterfs/changelog by default). Changes are classified into 3 types: - Data: : TYPE-I - Metadata : TYPE-II - Entry : TYPE-III Changelog file is rolled over after a certain time interval (defauls to 60 seconds) after which a changelog is started. The thing to be noted here is that for a time interval (time slice) multiple changes for an inode are recorded only once (ie. say for 100+ writes on an inode that happens within the time slice has only a single corresponding entry in the changelog file). That way we do not bloat up the changelog and also save lots of writes. Changelog Format ----------------- TYPE-I and TYPE-II changes have the gfid on the entity on which the operation happened. TYPE-III being a entry op requires the parent gfid and the basename. Changelog format has been kept to a minimal and it's upto the consumers to do the heavy loading of figuring out deletes, renames etc.. A single changelog file records all three types of changes, with each change starting with an identifier ("D": DATA, "M": METADATA and "E": ENTRY). Option is provided for the encoding type (See TUNABLES). Consumers ---------- The only consumer as of today would be geo-replication, although backup utilities, self-heal, bit-rot detection could be possible consumers in the future. CLI ---- By default, change-logging is disabled (the translator is present in the server graph but does nothing). When enabled (via cli) each brick starts to log the changes. There are a set of tunable that can be used to change the translators behaviour: - enable/disable changelog (disabled by default) gluster volume set <volume> changelog {on|off} - set the logging directory (<brick>/.glusterfs/changelogs is the default) gluster volume set <volume> changelog-dir /path/to/dir - select encoding type (binary (default) or ascii) gluster volume set <volume> encoding {binary|ascii} - change the rollover time for the logs (60 secs by default) gluster volume set <volume> rollover-time <secs> - when secs > 0, changelog file is not open()'d with O_SYNC flag - and fsync is trigerred periodically every <secs> seconds. gluster volume set <volume> fsync-interval <secs> features/changelog: changelog consumer library (libgfchangelog) A shared library is provided for the consumer of the changelogs for easy acess via APIs. Application can link against this library and request for changelog updates. Conversion of binary logs to human-readable ascii format is also taken care by the library which keeps a copy of the changelog in application provided working directory. Change-Id: I75575fb7f1c53d2bec3dba1a329ea7bb3c628497 BUG: 847839 Original Author: Venky Shankar <vshankar@redhat.com> Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/5127 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>