diff options
| author | Humble Devassy Chirammal <hchiramm@redhat.com> | 2015-09-24 14:53:52 +0530 |
|---|---|---|
| committer | Humble Devassy Chirammal <humble.devassy@gmail.com> | 2015-10-10 05:49:01 -0700 |
| commit | a4f982be9b21323038704069a56fb2448369d6a0 (patch) | |
| tree | 1daf99ef973b95b004938bb0e76b544907180b84 /doc | |
| parent | bad9539437ca1d69e470159277bbb6b5675cbae3 (diff) | |
Porting developer guide to source code repo from glusterdocs project
Change-Id: Ib8d9c668ebb05863918e6ec2b89908f206626f38
BUG: 1206539
Signed-off-by: Humble Devassy Chirammal <hchiramm@redhat.com>
Reviewed-on: http://review.gluster.org/12227
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Tested-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Tested-by: Raghavendra Talur <rtalur@redhat.com>
Diffstat (limited to 'doc')
30 files changed, 3013 insertions, 12 deletions
diff --git a/doc/developer-guide/Backport-Guidelines.md b/doc/developer-guide/Backport-Guidelines.md new file mode 100644 index 00000000000..af48dc00f03 --- /dev/null +++ b/doc/developer-guide/Backport-Guidelines.md @@ -0,0 +1,41 @@ +Bugs often get fixed in master before release branches. When a bug is +fixed in the master branch, it might be desirable or necessary in a +stable branch. To put the fix in stable branch we need to backport the +fix to stable branch. + +Anyone in the community can suggest a backport. If you are interested to +suggest a backport, please check the [Backport +Wishlist](./Backport Wishlist.md). + +This page describes the steps needed to backport simple changes. Changes +that do not apply cleanly will need some manual modifications and using +`git cherry-pick` may not always be the easiest solution. + +1. Git clone the GlusterFS code + + git clone ssh://username@review.gluster.org/glusterfs + +2. Create and checkout a new branch for your work, based on the branch + for the backport version + + git checkout -t -b bug-123456/release-3.5 origin/release-3.5 + +3. Cherry pick the change from master. + + $ git cherry-pick -x a0b1c2d3e4f5 + - verify that the change has been merged in the master branch. + +4. Update/correct the commit message. + + $ git commit -s --amend --date="$(date)" +[This is one example](https://github.com/gluster/glusterfs/commit/40407afb529f6e5fa2f79e9778c2f527122d75eb) of the commit message that has a good description for a backport. Notice the indention of the patch-metadata like BUG, Change-ID and Reviewed-on tags. There is also the original commit-id that was cherry picked from the master branch. + -make sure to quote the review tags + -update the BUG reference, point to the BUG that is used for this +particular release-branch + -add a Signed-off-by tag + +5. Run `./rfc.sh` to post the backport for review. + + ./rfc.sh +After submitting patch(es), make sure to move the bug to the *POST* +status. diff --git a/doc/developer-guide/Backport-Wishlist.md b/doc/developer-guide/Backport-Wishlist.md new file mode 100644 index 00000000000..f191820c803 --- /dev/null +++ b/doc/developer-guide/Backport-Wishlist.md @@ -0,0 +1,193 @@ +Bugs often get fixed in master before release branches. + +When a bug is fixed in the master branch it might be desirable or +necessary to backport the fix to a stable branch. + +This page is intended to help organize support (and prioritization) for +backporting bug fixes of importance to the community. + +### GlusterFs 3.6 + +Requested Backports for 3.6.0 +----------------------------- + +The tracker bug for 3.6.0 : +<https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.6.0> + +Please add 'glusterfs-3.6.0' in the 'Blocks' field of bugs to propose +inclusion in GlusterFS 3.6.0. + +### GlusterFs 3.5 + +Requested Backports for 3.5.3 +----------------------------- + +Current [list of bugs planned for +inclusion](https://bugzilla.redhat.com/showdependencytree.cgi?hide_resolved=0&id=glusterfs-3.5.3). + +- File a new bug for backporting a patch to 3.5.3: + [<https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&blocked=glusterfs-3.5.3&version=3.5.2&short_desc=backport%20request%20for%20>... + new glusterfs-3.5.3 backport request] + +### GlusterFs 3.4 + +Requested Backports for 3.4.6 +----------------------------- + +The tracker bug for 3.4.6 : +<https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.4.6> + +Please add 'glusterfs-3.4.6' in the 'Blocks' field of bugs to propose +inclusion in GlusterFS 3.4.6. + +<https://bugzilla.redhat.com:443/show_bug.cgi?id=1116150> +<https://bugzilla.redhat.com:443/show_bug.cgi?id=1117851> + +Requested Backports for 3.4.4 +----------------------------- + +<https://bugzilla.redhat.com/show_bug.cgi?id=859581> - "self-heal +process can sometimes create directories instead of symlinks for the +root gfid file in .glusterfs" + +<https://bugzilla.redhat.com/show_bug.cgi?id=1041109> - "structure needs +cleaning" message appear when accessing files. + +<https://bugzilla.redhat.com/show_bug.cgi?id=1073023> - glusterfs mount +crash after remove brick, detach peer and termination + +Requested Backports for 3.4.3 +----------------------------- + +<https://bugzilla.redhat.com/show_bug.cgi?id=859581> - "self-heal +process can sometimes create directories instead of symlinks for the +root gfid file in .glusterfs" + +<https://bugzilla.redhat.com/show_bug.cgi?id=1041109> - "structure needs +cleaning" message appear when accessing files. + +<https://bugzilla.redhat.com/show_bug.cgi?id=977492> - large NFS writes +to Gluster slow down then stop + +<https://bugzilla.redhat.com/show_bug.cgi?id=1073023> - glusterfs mount +crash after remove brick, detach peer and termination + +Requested Backports for 3.3.3 +----------------------------- + +[Enable fusermount by default, make nightly autobuilding +work](https://bugzilla.redhat.com/1058666) + +Requested Backports for 3.4.2 +----------------------------- + +Please enter bugzilla ID or patch URL here: + +1) Until RDMA handling is improved, we should output a warning when +using RDMA volumes - +<https://bugzilla.redhat.com/show_bug.cgi?id=1017176> + +2) Unable to shrink volumes without dataloss - +<https://bugzilla.redhat.com/show_bug.cgi?id=1024369> + +3) cluster/dht: Allow non-local clients to function with nufa volumes. +- <http://review.gluster.org/5414> + +Requested Backports for 3.4.1 +----------------------------- + +Please enter bugzilla ID or patch URL here. + +<https://bugzilla.redhat.com/show_bug.cgi?id=812230> - "quota context +not set in inode" + +<https://bugzilla.redhat.com/show_bug.cgi?id=893778> - "NFS crash bug" + +A note for whoever reviews this list: These are the fixes for issues +that have caused actual service disruption in our production +installation and thus are absolutely required for us (-- Lubomir +Rintel): + +<https://bugzilla.redhat.com/show_bug.cgi?id=994392> - "Setting ACL +entries fails with glusterfs-3.4.0" + +<https://bugzilla.redhat.com/show_bug.cgi?id=991622> - "fd leaks +observed while running dbench with "open-behind" volume option set to +"on" on a replicate volume" + +These are issues that we've stumbled upon during the git log review and +that seemed scary enough for us to cherry-pick them to avoid risk, +despite not being actually hit. Hope that helps deciding whether it's +worthwhile cherry-picking them (-- Lubomir Rintel): + +<https://bugzilla.redhat.com/show_bug.cgi?id=961691> "CLI crash upon +executing "gluster peer status" command" + +<https://bugzilla.redhat.com/show_bug.cgi?id=965995> "quick-read and +open-behind xlator: Make options (volume\_options ) structure NULL +terminated." + +<https://bugzilla.redhat.com/show_bug.cgi?id=958691> "nfs-root-squash: +rename creates a file on a file residing inside a sticky bit set +directory" + +<https://bugzilla.redhat.com/show_bug.cgi?id=982919> "DHT : files are +stored on directory which doesn't have hash range(hash layout)" + +<https://bugzilla.redhat.com/show_bug.cgi?id=976189> "statedump crashes +in ioc\_inode\_dump" + +<https://bugzilla.redhat.com/show_bug.cgi?id=982174> "cli crashes when +setting diagnostics.client-log-level is set to trace" + +<https://bugzilla.redhat.com/show_bug.cgi?id=989579> "glusterfsd crashes +on smallfile benchmark" + +<http://review.gluster.org/5821>, "tests: call 'cleanup' at the end of +each test", <https://bugzilla.redhat.com/show_bug.cgi?id=1004756>, +backport of 983975 + +<http://review.gluster.org/5822>, "glusterfs-api.pc.in contains an +rpath", <https://bugzilla.redhat.com/show_bug.cgi?id=1004751>, backport +of 1002220 + +<http://review.gluster.org/5824> "glusterd.service (systemd), ensure +glusterd starts before any local gluster mounts", +<https://bugzilla.redhat.com/show_bug.cgi?id=1004796>, backport of +1004795 + +<https://bugzilla.redhat.com/show_bug.cgi?id=819130> meta, check that +glusterfs.spec.in has all relevant updates + +<https://bugzilla.redhat.com/show_bug.cgi?id=1012400> - Glusterd would +not store all the volumes when a global options were set leading to peer +rejection + +Requested Backports +------------------- + +- Please backport [gfapi: Closed the logfile fd and initialize to NULL + in glfs\_fini](http://review.gluster.org/#/c/6552) into release-3.5 + - Done +- Please backport [cluster/dht: Make sure loc has + gfid](http://review.gluster.org/5178) into release-3.4 +- Please backport [Bug 887098](http://goo.gl/QjeMP) into release-3.3 + (FyreFoX) - Done +- Please backport [Bug 856341](http://goo.gl/9cGAC) into release-3.2 + and release-3.3 (the-me o/b/o Debian) - Done for release-3.3 +- Please backport [Bug 895656](http://goo.gl/ZNs3J) into release-3.2 + and release-3.3 (semiosis, x4rlos) - Done for release-3.3 +- Please backport [Bug 918437](http://goo.gl/1QRyw) into release-3.3 + (tjstansell) - Done +- Please backport into [Bug + 884597](https://bugzilla.redhat.com/show_bug.cgi?id=884597) + release-3.3 (nocko) - Done + +Unaddressed bugs +---------------- + +- [Bug 838784](https://bugzilla.redhat.com/show_bug.cgi?id=838784) +- [Bug 893778](https://bugzilla.redhat.com/show_bug.cgi?id=893778) +- [Bug 913699](https://bugzilla.redhat.com/show_bug.cgi?id=913699); + possibly related to [Bug + 884597](https://bugzilla.redhat.com/show_bug.cgi?id=884597)
\ No newline at end of file diff --git a/doc/developer-guide/Bug-Reporting-Guidelines.md b/doc/developer-guide/Bug-Reporting-Guidelines.md new file mode 100644 index 00000000000..d03878adebd --- /dev/null +++ b/doc/developer-guide/Bug-Reporting-Guidelines.md @@ -0,0 +1,128 @@ +Before filing a bug +------------------- + +If you are finding any issues, these preliminary checks as useful: + +- Is SELinux enabled? (you can use `getenforce` to check) +- Are iptables rules blocking any data traffic? (`iptables -L` can + help check) +- Are all the nodes reachable from each other? [ Network problem ] +- Please search Bugzilla to see if the bug has already been reported + - Choose GlusterFS as the "product", and then type something + relevant in the "words" box. If you are seeing a crash or abort, + searching for part of the abort message might be effective. If + you are feeling adventurous you can select the "Advanced search" + tab; this gives a lot more control but isn't much better for + finding existing bugs. + - If a bug has been already filed for a particular release and you + found the bug in another release, + - please clone the existing bug for the release, you found the + issue. + - If the existing bug is against mainline and you found the + issue for a release, then the cloned bug *depends on* should + be set to the BZ for mainline bug. + +Anyone can search in Bugzilla, you don't need an account. Searching +requires some effort, but helps avoid duplicates, and you may find that +your problem has already been solved. + +Reporting A Bug +--------------- + +- You should have a Bugzilla account +- Here is the link to file a bug: + [Bugzilla](https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS) +- The template for filing a bug can be found [ + *here*](./Bug reporting template.md) + +*Note: Please go through all below sections to understand what +information we need to put in a bug. So it will help the developer to +root cause and fix it* + +### Required Information + +You should gather the information below before creating the bug report. + +#### Package Information + +- Location from which the packages are used +- Package Info - version of glusterfs package installed + +#### Cluster Information + +- Number of nodes in the cluster +- Hostnames and IPs of the gluster Node [if it is not a security + issue] + - Hostname / IP will help developers in understanding & + correlating with the logs +- Output of `gluster peer status` +- Node IP, from which the "x" operation is done + - "x" here means any operation that causes the issue + +#### Volume Information + +- Number of volumes +- Volume Names +- Volume on which the particular issue is seen [ if applicable ] +- Type of volumes +- Volume options if available +- Output of `gluster volume info` +- Output of `gluster volume status` +- Get the statedump of the volume with the problem + +` $ gluster volume statedump `<vol-name> + +This dumps statedump per brick process in `/var/run/gluster` + +*NOTE: Collect statedumps from one gluster Node in a directory.* + +Repeat it in all Nodes containing the bricks of the volume. All the so +collected directories could be archived,compressed and attached to bug + +#### Brick Information + +- xfs options when brick partition was done + - This could be obtained with this command : + +` $ xfs_info /dev/mapper/vg1-brick` + +- Extended attributes on the bricks + - This could be obtained with this command: + +` $ getfattr -d -m. -ehex /rhs/brick1/b1` + +#### Client Information + +- OS Type ( Windows, RHEL ) +- OS Version : In case of Linux distro get the following : + +` $ uname -r`\ +` $ cat /etc/issue` + +- Fuse or NFS Mount point on the client with output of mount commands +- Output of `df -Th` command + +#### Tool Information + +- If any tools are used for testing, provide the info/version about it +- if any IO is simulated using a script, provide the script + +#### Logs Information + +- You can check logs for check for issues/warnings/errors. + - Self-heal logs + - Rebalance logs + - Glusterd logs + - Brick logs + - NFS logs (if applicable) + - Samba logs (if applicable) + - Client mount log +- Add the entire logs as attachment, if its very large to paste as a + comment + +#### SOS report for CentOS/Fedora + +- Get the sosreport from the involved gluster Node and Client [ in + case of CentOS /Fedora ] +- Add a meaningful name/IP to the sosreport, by renaming/adding + hostname/ip to the sosreport name diff --git a/doc/developer-guide/Bug-Triage.md b/doc/developer-guide/Bug-Triage.md new file mode 100644 index 00000000000..bcd475e81fb --- /dev/null +++ b/doc/developer-guide/Bug-Triage.md @@ -0,0 +1,400 @@ +Bug Triage Guidelines +===================== + +- Triaging of bugs is an important task; when done correctly, it can + reduce the time between reporting a bug and the availability of a + fix enormously. + +- Triager should focus on new bugs, and try to define the problem + easily understandable and as accurate as possible. The goal of the + triagers is to reduce the time that developers need to solve the bug + report. + +- A triager is like an assistant that helps with the information + gathering and possibly the debugging of a new bug report. Because a + triager helps preparing a bug before a developer gets involved, it + can be a very nice role for new community members that are + interested in technical aspects of the software. + +- Triagers will stumble upon many different kind of issues, ranging + from reports about spelling mistakes, or unclear log messages to + memory leaks causing crashes or performance issues in environments + with several hundred storage servers. + +Nobody expects that triagers can prepare all bug reports. Therefore most +developers will be able to assist the triagers, answer questions and +suggest approaches to debug and data to gather. Over time, triagers get +more experienced and will rely less on developers. + +**Bug triage can be summarised as below points:** + +- Is there enough information in the bug description? +- Is it a duplicate bug? +- Is it assigned to correct component of GlusterFS? +- Are the Bugzilla fields correct? +- Is the bug summary is correct? +- Assigning bugs or Adding people to the "CC" list +- Fix the Severity And Priority. +- Todo, If the bug present in multiple GlusterFS versions. +- Add appropriate Keywords to bug. + +The detailed discussion about the above points are below. + +Weekly meeting about Bug Triaging +--------------------------------- + +We try to meet every week in \#gluster-meeting on Freenode. The meeting +date and time for the next meeting is normally updated in the +[agenda](https://public.pad.fsfe.org/p/gluster-bug-triage). + +Getting Started: Find reports to triage +--------------------------------------- + +There are many different techniques and approaches to find reports to +triage. One easy way is to use these pre-defined Bugzilla reports (a +report is completely structured in the URL and can manually be +modified): + +- New **bugs** that do not have the 'Triaged' keyword [Bugzilla + link](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&f1=keywords&keywords=Triaged%2CFutureFeature&keywords_type=nowords&list_id=3014117&o1=nowords&product=GlusterFS&query_format=advanced&v1=Triaged) +- New **features** that do not have the 'Triaged' keyword (identified + by FutureFeature keyword, probably of interest only to project + leaders) [Bugzilla + link](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&f1=keywords&f2=keywords&list_id=3014699&o1=nowords&o2=allwords&product=GlusterFS&query_format=advanced&v1=Triaged&v2=FutureFeature) +- New glusterd bugs: [Bugzilla + link](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&product=GlusterFS&f1=keywords&o1=nowords&v1=Triaged&component=glusterd) +- New Replication(afr) bugs: [Bugzilla + link](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&component=replicate&f1=keywords&list_id=2816133&o1=nowords&product=GlusterFS&query_format=advanced&v1=Triaged) +- New distribute(DHT) bugs: [Bugzilla + links](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&component=distribute&f1=keywords&list_id=2816148&o1=nowords&product=GlusterFS&query_format=advanced&v1=Triaged) + +- New bugs against version 3.6: + [<https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&product=GlusterFS&f1=keywords&f2=version&o1=nowords&o2=regexp&v1=Triaged&v2>=\^3.6 + Bugzilla link] +- New bugs against version 3.5: + [<https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&product=GlusterFS&f1=keywords&f2=version&o1=nowords&o2=regexp&v1=Triaged&v2>=\^3.5 + Bugzilla link] +- New bugs against version 3.4: + [<https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&product=GlusterFS&f1=keywords&f2=version&o1=nowords&o2=regexp&v1=Triaged&v2>=\^3.4 + Bugzilla link] + +- [<https://bugzilla.redhat.com/page.cgi?id=browse.html&product=GlusterFS&product_version>=&bug\_status=all&tab=recents + bugzilla tracker] (can include already Triaged bugs) + +- [Untriaged NetBSD + bugs](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&keywords=Triaged&keywords_type=nowords&op_sys=NetBSD&product=GlusterFS) +- [Untriaged FreeBSD + bugs](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&keywords=Triaged&keywords_type=nowords&op_sys=FreeBSD&product=GlusterFS) +- [Untriaged Mac OS + bugs](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&keywords=Triaged&keywords_type=nowords&op_sys=Mac%20OS&product=GlusterFS) + +In addition to manually checking Bugzilla for bugs to triage, it is also +possible to receive emails when new +bugs are filed or existing bugs get updated. + +If at any point you feel like you do not know what to do with a certain +report, please first ask [irc or mailing +lists](http://www.gluster.org/community/index.html) before changing +something. + +Is there enough information? +---------------------------- + +To make a report useful, the same rules apply as for +[bug reporting guidelines](./Bug Reporting Guidelines.md). + +It's hard to generalize what makes a good report. For "average" +reporters is definitely often helpful to have good steps to reproduce, +GlusterFS software version , and information about the test/production +environment, Linux/GNU distribution. + +If the reporter is a developer, steps to reproduce can sometimes be +omitted as context is obvious. *However, this can create a problem for +contributors that need to find their way, hence it is strongly advised +to list the steps to reproduce an issue.* + +Other tips: + +- There should be only one issue per report. Try not to mix related or + similar looking bugs per report. + +- It should be possible to call the described problem fixed at some + point. "Improve the documentation" or "It runs slow" could never be + called fixed, while "Documentation should cover the topic Embedding" + or "The page at <http://en.wikipedia.org/wiki/Example> should load + in less than five seconds" would have a criterion. A good summary of + the bug will also help others in finding existing bugs and prevent + filing of duplicates. + +- If the bug is a graphical problem, you may want to ask for a + screenshot to attach to the bug report. Make sure to ask that the + screenshot should not contain any confidential information. + +Is it a duplicate? +------------------ + +Some reports in Bugzilla have already been reported before so you can +[search for an already existing +report](https://bugzilla.redhat.com/query.cgi?format=advanced). We do +not recommend to spend too much time on it; if a bug is filed twice, +someone else will mark it as a duplicate later. If the bug is a +duplicate, mark it as a duplicate in the resolution box below the +comment field by setting the **CLOSED DUPLICATE** status, and shortly +explain your action in a comment for the reporter. When marking a bug as +a duplicate, it is required to reference the original bug. + +If you think that you have found a duplicate but you are not totally +sure, just add a comment like "This bug looks related to bug XXXXX" (and +replace XXXXX by the bug number) so somebody else can take a look and +help judging. + +You can also take a look at +https://bugzilla.redhat.com/page.cgi?id=browse.html&product=GlusterFS&product_version>=&bug\_status=all&tab=duplicates's +list of existing duplicates + +Is it assigned to correct component of GlusterFS? +------------------------------------------------- + +Make sure the bug is assigned on right component. Below are the list of +GlusterFs components in bugzilla. + +- access control - Access control translator +- BDB - Berkeley DB backend storage +- booster - LD\_PRELOAD'able access client +- build - Compiler, package management and platform specific warnings + and errors +- cli -gluster command line +- core - Core features of the filesystem +- distribute - Distribute translator (previously DHT) +- errorgen - Error Gen Translator +- fuse -mount/fuse translator and patched fuse library +- georeplication - Gluster Geo-Replication +- glusterd - Management daemon +- HDFS - Hadoop application support over GlusterFS +- ib-verbs - Infiniband verbs transport +- io-cache - IO buffer caching translator +- io-threads - IO threads performance translator +- libglusterfsclient- API interface to access glusterfs volumes + programatically +- locks - POSIX and internal locks +- logging - Centralized logging, log messages, log rotation etc +- nfs- NFS component in GlusterFS +- nufa- Non-Uniform Filesystem Scheduler Translator +- object-storage - Object Storage +- porting - Porting GlusterFS to different operating systems and + platforms +- posix - POSIX (API) based backend storage +- protocol -Client and Server protocol translators +- quick-read- Quick Read Translator +- quota - Volume & Directory quota translator +- rdma- RDMA transport +- read-ahead - Read ahead (file) performance translator +- replicate- Replication translator (previously AFR) +- rpc - RPC Layer +- scripts - Build scripts, mount scripts, etc. +- stat-prefetch - Stat prefetch translator +- stripe - Striping (RAID-0) cluster translator +- trace- Trace translator +- transport - Socket (IPv4, IPv6, unix, ib-sdp) and generic transport + code +- unclassified - Unclassified - to be reclassified as other components +- unify - Unify translator and schedulers +- write-behind- Write behind performance translator +- libgfapi - APIs for GlusterFS +- tests- GlusterFS Test Framework +- gluster-hadoop - Hadoop support on GlusterFS +- gluster-hadoop-install - Automated Gluster volume configuration for + Hadoop Environments +- gluster-smb - gluster smb +- puppet-gluster - A puppet module for GlusterFS + +Tips for searching: + +- As it is often hard for reporters to find the right place (product + and component) where to file a report, also search for duplicates + outside same product and component of the bug report you are + triaging. +- Use common words and try several times with different combinations, + as there could be several ways to describe the same problem. If you + choose the proper and common words, and you try several times with + different combinations of those, you ensure to have matching + results. +- Drop the ending of a verb (e.g. search for "delet" so you get + reports for both "delete" and "deleting"), and also try similar + words (e.g. search both for "delet" and "remov"). +- Search using the date range delimiter: Most of the bug reports are + recent, so you can try to increase the search speed using date + delimiters by going to "Search by Change History" on the [search + page](https://bugzilla.redhat.com/query.cgi?format=advanced). + Example: search from "2011-01-01" or "-730d" (to cover the last two + years) to "Now". + +Are the fields correct? +----------------------- + +### Summary + +Sometimes the summary does not summarize the bug itself well. You may +want to update the bug summary to make the report distinguishable. A +good title may contain: + +- A brief explanation of the root cause (if it was found) +- Some of the symptoms people are experiencing + +### Adding people to the "CC" or changing the "Assigned to" field + +Normally, developers and potential assignees of an area are already +CC'ed by default, but sometimes reports describe general issues or are +filed against common bugzilla products. Only if you know developers who +work in the area covered by the bug report, and if you know that these +developers accept getting CCed or assigned to certain reports, you can +add that person to the CC field or even assign the bug report to +her/him. + +To get an idea who works in which area, check To know component owners , +you can check the "MAINTAINERS" file in root of glusterfs code directory +or querying changes in [Gerrit](http://review.gluster.org) (see +[Simplified dev workflow](./Simplified Development Workflow.md)) + +### Severity And Priority + +Please see below for information on the available values and their +meanings. + +#### Severity + +This field is a pull-down of the external weighting of the bug report's +importance and can have the following values: + + Severity |Definition + -------------|------------------------------------------------------------------------------------------------------------------------------------------------------------- + urgent |catastrophic issues which severely impact the mission-critical operations of an organization. This may mean that the operational servers, development systems or customer applications are down or not functioning and no procedural workaround exists. + high |high-impact issues in which the customer's operation is disrupted, but there is some capacity to produce + medium |partial non-critical functionality loss, or issues which impair some operations but allow the customer to perform their critical tasks. This may be a minor issue with limited loss or no loss of functionality and limited impact to the customer's functionality + low |general usage questions, recommendations for product enhancement, or development work + unspecified |importance not specified + +#### Priority + +This field is a pull-down of the internal weighting of the bug report's +importance and can have the following values: + + Priority |Definition + -------------|------------------------ + urgent |extremely important + high |very important + medium |average importance + low |not very important + unspecified |importance not specified + + +### Bugs present in multiple Versions + +During triaging you might come across a particular bug which is present +across multiple version of GlusterFS. Here are the course of actions: + +- We should have separate bugs for each release (We should + clone bugs if required) +- Bugs in released versions should be depended on bug for mainline + (master branch) if the bug is applicable for mainline. + - This will make sure that the fix would get merged in master + branch first then the fix can get ported to other stable + releases. + +*Note: When a bug depends on other bugs, that means the bug cannot be +fixed unless other bugs are fixed (depends on), or this bug stops other +bugs being fixed (blocks)* + +Here are some examples: + +- A bug is raised for GlusterFS 3.5 and the same issue is present in + mainline (master branch) and GlusterFS 3.6 + - Clone the original bug for mainline. + - Clone another for 3.6. + - And have the GlusterFS 3.6 bug and GlusterFS 3.5 bug 'depend on' + the 'mainline' bug + +- A bug is already present for mainline, and the same issue is seen in + GlusterFS 3.5. + - Clone the original bug for GlusterFS 3.5. + - And have the cloned bug (for 3.5) 'depend on' the 'mainline' + bug. + +### Keywords + +Many predefined searches for Bugzilla include keywords. One example are +the searches for the triaging. If the bug is 'NEW' and 'Triaged' is no +set, you (as a triager) can pick it and use this page to triage it. When +the bug is 'NEW' and 'Triaged' is in the list of keyword, the bug is +ready to be picked up by a developer. + +**Triaged** +: Once you are done with triage add the **Triaged** keyword to the + bug, so that others will know the triaged state of the bug. The + predefined search at the top of this page will then not list the + Triaged bug anymore. Instead, the bug should have moved to [this + list](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&keywords=Triaged&product=GlusterFS). + +**EasyFix** +: By adding the **EasyFix** keyword, the bug gets added to the [list + of bugs that should be simple to fix](./Easy Fix Bugs.md). + Adding this keyword is encouraged for simple and well defined bugs + or feature enhancements. + +**Patch** +: When a patch for the problem has been attached or included inline, + add the **Patch** keyword so that it is clear that some preparation + for the development has been done already. If course, it would have + been nicer if the patch was sent to Gerrit for review, but not + everyone is ready to pass the Gerrit hurdle when they report a bug. + +You can also add the **Patch** keyword when a bug has been fixed in + mainline and the patch(es) has been identified. Add a link to the + Gerrit change(s) so that backporting to a stable release is made + simpler. + +**Documentation** +: Add the **Documentation** keyword when a bug has been reported for + the documentation. This helps editors and writers in finding the + bugs that they can resolve. + +**Tracking** +: This keyword is used for bugs which are used to track other bugs for + a particular release. For example [3.6 tracker + bug](https://bugzilla.redhat.com/showdependencytree.cgi?maxdepth=2&hide_resolved=1&id=glusterfs-3.6.0) + +**FutureFeature** +: This keyword is used for bugs which are used to request for a + feature enhancement ( RFE - Requested Feature Enhancement) for + future releases of GlusterFS. If you open a bug by requesting a + feature which you would like to see in next versions of GlusterFS + please report with this keyword. + +Add yourself to the CC list +--------------------------- + +By adding yourself to the CC list of bug reports that you change, you +will receive followup emails with all comments and changes by anybody on +that individual report. This helps learning what further investigations +others make. You can change the settings in Bugzilla on which actions +you want to receive mail. + +Bugs For Group Triage +--------------------- + +If you come across a bug/ bugs or If you think any bug should to go +thorough the bug triage group, please set NEEDINFO for bugs@gluster.org +on the bug. + +Resolving bug reports +--------------------- + +See the [Bug report life cycle](./Bug report Life Cycle.md) for +the meaning of the bug status and resolutions. + +Example of Triaged Bugs +----------------------- + +This Bugzilla +[filter](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&keywords=Triaged&keywords_type=anywords&list_id=2739593&product=GlusterFS&query_format=advanced) +will list NEW, Triaged Bugs diff --git a/doc/developer-guide/Bug-report-Life-Cycle.md b/doc/developer-guide/Bug-report-Life-Cycle.md new file mode 100644 index 00000000000..3749bd6272a --- /dev/null +++ b/doc/developer-guide/Bug-report-Life-Cycle.md @@ -0,0 +1,57 @@ +This page describes the life of a bug report. + +- When a bug is first reported, it is given the **NEW** status. +- Once a developer has started, or is planning to work on a bug, the + status **ASSIGNED** is set. The "Assigned to" field should mention a + specific developer. +- If an initial + [patch](https://en.wikipedia.org/wiki/Patch_(computing)) for a bug + has been put into the [Gerrit code review + tool](http://review.gluster.org), the status **POST** should be set + manually. The status **POST** should only be used when all patches + for a specific bug have been posted for review. +- After a review of the patch, and passing any automated regression + tests, the patch will get merged by one of the maintainers. When the + patch has been merged into the git repository, a comment is added to + the bug. Only when all needed patches have been merged, the assigned + engineer will need to change the status to **MODIFIED**. +- Once a package is available with fix for the bug, the status should + be moved to **ON\_QA**. + - The **Fixed in version** field should get the name/release of + the package that contains the fix. Packages for multiple + distributions will mostly get available within a few days after + the *make dist* tarball was created. + - This tells the bug reporter that a package is available with fix + for the bug and that they should test the package. + - The release maintainer need to do this change to bug status, + scripts are available (ask *ndevos*). +- The status **VERIFIED** is set if a QA tester or the reporter + confirmed the fix after fix is merged and new build with the fix + resolves the issue. +- In case the version does not fix the reported bug, the status should + be moved back to **ASSIGNED** with a clear note on what exactly + failed. +- When a report has been solved it is given **CLOSED** status. This + can mean: + - **CLOSED/CURRENTRELEASE** when a code change that fixes the + reported problem has been merged in + [Gerrit](http://review.gluster.org). + - **CLOSED/WONTFIX** when the reported problem or suggestion is + valid, but any fix of the reported problem or implementation of + the suggestion would be barred from approval by the project's + Developers/Maintainers (or product managers, if existing). + - **CLOSED/WORKSFORME** when the problem can not be reproduced, + when missing information has not been provided, or when an + acceptable workaround exists to achieve a similar outcome as + requested. + - **CLOSED/CANTFIX** when the problem is not a bug, or when it is + a change that is outside the power of GlusterFS development. For + example, bugs proposing changes to third-party software can not + be fixed in the GlusterFS project itself. + - **CLOSED/DUPLICATE** when the problem has been reported before, + no matter if the previous report has been already resolved or + not. + +If a bug report was marked as *CLOSED* or *VERIFIED* and it turns out +that this was incorrect, the bug can be changed to the status *ASSIGNED* +or *NEW*.
\ No newline at end of file diff --git a/doc/developer-guide/Bug-reporting-template.md b/doc/developer-guide/Bug-reporting-template.md new file mode 100644 index 00000000000..f8d1157d447 --- /dev/null +++ b/doc/developer-guide/Bug-reporting-template.md @@ -0,0 +1,55 @@ +Template for bug description +---------------------------- +This template should be in-line to the [Bug reporting guidelines](./Bug Reporting Guidelines.md). +The template is replacement for the default description template present in [Bugzilla](https://bugzilla.redhat.com) + + work in progress + +------------------------------------------------------------------------ + +Description of problem: + +Version of GlusterFS package installed: + +Location from which the packages are used: + +GlusterFS Cluster Information: + +- Number of volumes +- Volume Names +- Volume on which the particular issue is seen [ if applicable ] +- Type of volumes +- Volume options if available +- Output of `gluster volume info` +- Output of `gluster volume status` +- Get the statedump of the volume with the problem + +` $ gluster volume statedump `<vol-name> + +- Client Information + - OS Type: + - Mount type: + - OS Version: + +How reproducible: + +Steps to Reproduce: + +- 1. +- 2. +- 3. + +Actual results: + +Expected results: + +Logs Information: + +- Provide possible issues, warnings, errors as a comment to the bug + - Look for issues/warnings/errors in self-heal logs, rebalance logs, glusterd logs, brick logs, mount logs/nfs logs/smb logs + - Add the entire logs as attachment, if it is very large to paste as a comment + +Additional info: + + [Bug\_reporting\_guidelines]: Bug_reporting_guidelines "wikilink" + [Bugzilla]: https://bugzilla.redhat.com diff --git a/doc/developer-guide/Building GlusterFS.md b/doc/developer-guide/Building GlusterFS.md new file mode 100644 index 00000000000..160216921ad --- /dev/null +++ b/doc/developer-guide/Building GlusterFS.md @@ -0,0 +1,148 @@ +This page describes how to build and install GlusterFS. + +Build Requirements +------------------ + +The following packages are required for building GlusterFS, + +- GNU Autotools + - Automake + - Autoconf + - Libtool +- lex (generally flex) +- GNU Bison +- OpenSSL +- libxml2 +- Python 2.x +- libaio +- libibverbs +- librdmacm +- readline +- lvm2 +- glib2 +- liburcu +- cmocka +- libacl +- sqlite + +### Fedora + +The following yum command installs all the build requirements for +Fedora, + + # yum install automake autoconf libtool flex bison openssl-devel libxml2-devel python-devel libaio-devel libibverbs-devel librdmacm-devel readline-devel lvm2-devel glib2-devel userspace-rcu-devel libcmocka-devel libacl-devel sqlite-devel + +### Ubuntu + +The following apt-get command will install all the build requirements on +Ubuntu, + + $ sudo apt-get install make automake autoconf libtool flex bison pkg-config libssl-dev libxml2-dev python-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev + +Building from Source +-------------------- + +This section describes how to build GlusterFS from source. It is assumed +you have a copy of the GlusterFS source (either from a released tarball +or a git clone). All the commands below are to be run with the source +directory as the working directory. + +### Configuring for building + +Run the below commands once for configuring and setting up the build +process. + +Run autogen to generate the configure script. + + $ ./autogen.sh + +Once autogen completes successfully a configure script is generated. Run +the configure script to generate the makefiles. + + $ ./configure + +If the above build requirements have been installed, running the +configure script should give the below configure summary, + + GlusterFS configure summary + =========================== + FUSE client : yes + Infiniband verbs : yes + epoll IO multiplex : yes + argp-standalone : no + fusermount : yes + readline : yes + georeplication : yes + Linux-AIO : yes + Enable Debug : no + systemtap : no + Block Device xlator : yes + glupy : yes + Use syslog : yes + XML output : yes + QEMU Block formats : yes + Encryption xlator : yes + +During development it is good to enable a debug build. To do this run +configure with a '--enable-debug' flag. + + $ ./configure --enable-debug + +Further configuration flags can be found by running configure with a +'--help' flag, + + $ ./configure --help + +### Building + +Once configured, GlusterFS can be built with a simple make command. + + $ make + +To speed up the build process on a multicore machine, add a '-jN' flag, +where N is the number of parallel jobs. + +### Installing + +Run 'make install' to install GlusterFS. By default, GlusterFS will be +installed into '/usr/local' prefix. To change the install prefix, give +the appropriate option to configure. If installing into the default +prefix, you might need to use 'sudo' or 'su -c' to install. + + $ sudo make install + +### Running GlusterFS + +GlusterFS can be only run as root, so the following commands will need +to be run as root. If you've installed into the default '/usr/local' +prefix, add '/usr/local/sbin' and '/usr/local/bin' to your PATH before +running the below commands. + +A source install will generally not install any init scripts. So you +will need to start glusterd manually. To manually start glusterd just +run, + + # glusterd + +This will start glusterd and fork it into the background as a daemon +process. You now run 'gluster' commands and make use of GlusterFS. + +Building packages +----------------- + +### Building RPMs + +Building RPMs is really simple. On a RPM based system, for eg. Fedora, +get the source and do the configuration steps as shown in the 'Building +from Source' section. After the configuration step, run the following +steps to build RPMs, + + $ cd extras/LinuxRPM + $ make glusterrpms + +This will create rpms from the source in 'extras/LinuxRPM'. *(Note: You +will need to install the rpmbuild requirements including rpmbuild and +mock)* + +A more detailed description for building RPMs can be found at +[CompilingRPMS](./Compiling RPMS.md). diff --git a/doc/developer-guide/Compiling-RPMS.md b/doc/developer-guide/Compiling-RPMS.md new file mode 100644 index 00000000000..b1bd39b26f8 --- /dev/null +++ b/doc/developer-guide/Compiling-RPMS.md @@ -0,0 +1,178 @@ +How to compile GlusterFS RPMs from git source, for RHEL/CentOS, and Fedora +-------------------------------------------------------------------------- + +Creating rpm's of GlusterFS from git source is fairly easy, once you +know the steps. + +RPMS can be compiled on at least the following OS's: + +- Red Hat Enterprise Linux 5, 6 (& 7 when available) +- CentOS 5, 6 (& 7 when available) +- Fedora 16-20 + +Specific instructions for compiling are below. If you're using: + +- Fedora 16-20 - Follow the Fedora steps, then do all of the Common + steps. +- CentOS 5.x - Follow the CentOS 5.x steps, then do all of the Common + steps +- CentOS 6.x - Follow the CentOS 6.x steps, then do all of the Common + steps. +- RHEL 6.x - Follow the RHEL 6.x steps, then do all of the Common + steps. + +Note - these instructions have been explicitly tested on all of CentOS +5.10, RHEL 6.4, CentOS 6.4+, and Fedora 16-20. Other releases of +RHEL/CentOS and Fedora may work too, but haven't been tested. Please +update this page appropriately if you do so. :) + +### Preparation steps for Fedora 16-20 (only) + +1. Install gcc, the python development headers, and python setuptools: + + $ sudo yum -y install gcc python-devel python-setuptools + +2. If you're compiling GlusterFS version 3.4, then install +python-swiftclient. Other GlusterFS versions don't need it: + + $ sudo easy_install simplejson python-swiftclient + +Now follow through the **Common Steps** part below. + +### Preparation steps for CentOS 5.x (only) + +You'll need EPEL installed first and some CentOS specific packages. The +commands below will get that done for you. After that, follow through +the "Common steps" section. + +1. Install EPEL first: + + $ curl -OL http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm + $ sudo yum -y install epel-release-5-4.noarch.rpm --nogpgcheck + +2. Install the packages required only on CentOS 5.x: + + $ sudo yum -y install buildsys-macros gcc ncurses-devel python-ctypes python-sphinx10 \ + redhat-rpm-config + +Now follow through the **Common Steps** part below. + +### Preparation steps for CentOS 6.x (only) + +You'll need EPEL installed first and some CentOS specific packages. The +commands below will get that done for you. After that, follow through +the "Common steps" section. + +1. Install EPEL first: + + $ sudo yum -y install http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm + +2. Install the packages required only on CentOS: + + $ sudo yum -y install python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config + +Now follow through the **Common Steps** part below. + +### Preparation steps for RHEL 6.x (only) + +You'll need EPEL installed first and some RHEL specific packages. The 2 +commands below will get that done for you. After that, follow through +the "Common steps" section. + +1. Install EPEL first: + + $ sudo yum -y install http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm + +2. Install the packages required only on RHEL: + + $ sudo yum -y --enablerepo=rhel-6-server-optional-rpms install python-webob1.0 \ + python-paste-deploy1.5 python-sphinx10 redhat-rpm-config + +Now follow through the **Common Steps** part below. + +### Common Steps + +These steps are for both Fedora and RHEL/CentOS. At the end you'll have +the complete set of GlusterFS RPMs for your platform, ready to be +installed. + +**NOTES for step 1 below:** + +- If you're on RHEL/CentOS 5.x and get a message about lvm2-devel not + being available, it's ok. You can ignore it. :) +- If you're on RHEL/CentOS 6.x and get any messages about + python-eventlet, python-netifaces, python-sphinx and/or pyxattr not + being available, it's ok. You can ignore them. :) + +1. Install the needed packages + + $ sudo yum -y --disablerepo=rhs* --enablerepo=*optional-rpms install git autoconf \ + automake bison cmockery2-devel dos2unix flex fuse-devel glib2-devel libaio-devel \ + libattr-devel libibverbs-devel librdmacm-devel libtool libxml2-devel lvm2-devel make \ + openssl-devel pkgconfig pyliblzma python-devel python-eventlet python-netifaces \ + python-paste-deploy python-simplejson python-sphinx python-webob pyxattr readline-devel \ + rpm-build systemtap-sdt-devel tar libcmocka-devel + +2. Clone the GlusterFS git repository + + $ git clone git://git.gluster.org/glusterfs + $ cd glusterfs + +3. Choose which branch to compile + +If you want to compile the latest development code, you can skip this +step and go on to the next one. + +If instead you want to compile the code for a specific release of +GlusterFS (such as v3.4), get the list of release names here: + + $ git branch -a | grep release + remotes/origin/release-2.0 + remotes/origin/release-3.0 + remotes/origin/release-3.1 + remotes/origin/release-3.2 + remotes/origin/release-3.3 + remotes/origin/release-3.4 + remotes/origin/release-3.5 + +Then switch to the correct release using the git "checkout" command, and +the name of the release after the "remotes/origin/" bit from the list +above: + + $ git checkout release-3.4 + +**NOTE -** The CentOS 5.x instructions have only been tested for the +master branch in GlusterFS git. It is unknown (yet) if they work for +branches older then release-3.5. + +4. Configure and compile GlusterFS + +Now you're ready to compile Gluster: + + $ ./autogen.sh + $ ./configure --enable-fusermount + $ make dist + +5. Create the GlusterFS RPMs + + $ cd extras/LinuxRPM + $ make glusterrpms + +That should complete with no errors, leaving you with a directory +containing the RPMs. + + $ ls -l *rpm + -rw-rw-r-- 1 jc jc 3966111 Mar 2 12:15 glusterfs-3git-1.el5.centos.src.rpm + -rw-rw-r-- 1 jc jc 1548890 Mar 2 12:17 glusterfs-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 66680 Mar 2 12:17 glusterfs-api-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 20399 Mar 2 12:17 glusterfs-api-devel-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 123806 Mar 2 12:17 glusterfs-cli-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 7850357 Mar 2 12:17 glusterfs-debuginfo-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 112677 Mar 2 12:17 glusterfs-devel-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 100410 Mar 2 12:17 glusterfs-fuse-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 187221 Mar 2 12:17 glusterfs-geo-replication-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 299171 Mar 2 12:17 glusterfs-libs-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 44943 Mar 2 12:17 glusterfs-rdma-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 123065 Mar 2 12:17 glusterfs-regression-tests-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 16224 Mar 2 12:17 glusterfs-resource-agents-3git-1.el5.centos.x86_64.rpm + -rw-rw-r-- 1 jc jc 654043 Mar 2 12:17 glusterfs-server-3git-1.el5.centos.x86_64.rpm
\ No newline at end of file diff --git a/doc/developer-guide/Developers-Index.md b/doc/developer-guide/Developers-Index.md new file mode 100644 index 00000000000..a523a4681c9 --- /dev/null +++ b/doc/developer-guide/Developers-Index.md @@ -0,0 +1,127 @@ +Developers +========== + +### From GlusterDocumentation + +Contributing to the Gluster community +------------------------------------- + +Are you itching to send in patches and participate as a developer in the +Gluster community? Here are a number of starting points for getting +involved. We don't require a signed contributor license agreement or +copyright assignment, but we do require a "signed-off-by" line on each +code check-in. + +- [Simplified Developer Workflow](./Simplified Development Workflow.md) + - A simpler and faster intro to developing with GlusterFS, than the + doc below. +- [Developer Workflow](./Development Workflow.md) - this tells + you about our patch requirements, tools we use, and more. Required + reading if you want to contribute code. +- [License + Change](http://www.gluster.org/2012/05/glusterfs-license-change/) - + we recently changed the client library code to a dual license under + the GPL v2 and the LGPL v3 or later +- [GlusterFS Coding Standards](./coding-standard.md) + +Compiling Gluster +----------------- + +- [Compiling RPMS](./Compiling RPMS.md) - Step by step + instructions for compiling Gluster RPMS +- [Building GlusterFS](./Building GlusterFS.md) - How to compile + Gluster from source code. Including instructions for Ubuntu. + +Developing +---------- + +- [Projects](./Projects.md) - Ideas for projects you could + create +- [Language Bindings](./Language Bindings.md) - Connect to + GlusterFS using various language bindings +- [EasyFix\_Bugs](./Easy Fix Bugs.md) - Easy to fix bugs of + GlusterFS. One of the best place to start contributing to GlusterFS. +- [Fixing issues reported by tools for static code + analysis](./Fixing issues reported by tools for static code analysis.md) + - This is a good starting point for developers to fix bugs in + GlusterFS project. +- [Backport Wishlist](./Backport Wishlist.md) - Problems fixed + in the master branch might need to get fixed in stable release + branches too. + The [Backport Guidelines](./Backport Guidelines.md) describe the steps that + branches too. + +Adding File operations +---------------------- + +- [Steps to be followed when adding a new FOP to GlusterFS ](./adding-fops.md) + +Automatic File Replication +-------------------------- + +- [Cluster/afr translator](./afr.md) +- [History of Locking in AFR](./afr-locks-evolution.md) +- [Self heal Daemon](./afr-self-heal-daemon.md) + +Data Structures +--------------- + +- [inode data structure](./datastructure-inode.md) +- [iobuf data structure](./datastructure-iobuf.md) +- [mem-pool data structure](./datastructure-mem-pool.md) + +Find the gfapi symbol versions [here](./gfapi-symbol-versions.md) + +Daemon Management Framework +--------------------------- + +- [How to introduce new daemons using daemon management framework](./daemon-management-framework.md) + +Translators +----------- + +- [Block Device Tanslator](./bd-xlator.md) +- [Performance/write-Behind Translator](./write-behind.md) +- [Translator Development](./translator-development.md) +- [Storage/posix Translator](./posix.md) +- [Compression translator](./network_compression.md) + +Testing/Debugging +----------------- + +- [Unit Tests in GlusterFS](./unittest.md) +- [Using the Gluster Test + Framework](./Using Gluster Test Framework.md) - Step by + step instructions for running the Gluster Test Framework +- [Our Jenkins Infrastructure](./Jenkins Infrastructure.md) - A + braindump of the Jenkins infrastructure we have in place for + automated testing +- [Manual steps for setting up a Jenkins slave VM in + Rackspace](./Jenkins Manual Setup.md) - Steps for setting up a slave + VM in Rackspace +- [Coredump Analysis](./coredump-analysis.md) - Steps to analize coredumps generated by regression machines. + +Bug Handling +------------ + +- [Bug reporting guidelines](./Bug Reporting Guidelines.md) - + Guideline for reporting a bug in GlusterFS +- [Bug triage guidelines](./Bug Triage.md) - Guideline on how to + triage bugs for GlusterFS +- [Bug report life cycle in + Bugzilla](./Bug report Life Cycle.md) - Information about bug + life cycle + +Patch Acceptance +---------------- + +- The [Guidelines For + Maintainers](./Guidelines For Maintainers.md) explains when + maintainers can merge patches. + +Release Process +--------------- + +- [Versioning](./versioning.md) +- [GlusterFS Release Process](./GlusterFS Release process.md) - + Our release process / checklist diff --git a/doc/developer-guide/Development-Workflow.md b/doc/developer-guide/Development-Workflow.md new file mode 100644 index 00000000000..d36b932c0a8 --- /dev/null +++ b/doc/developer-guide/Development-Workflow.md @@ -0,0 +1,457 @@ +Development work flow of Gluster +================================ + +This document provides a detailed overview of the development model +followed by the GlusterFS project. + +For a simpler overview visit +[Simplified develoment workflow](./Simplified Development Workflow.md). + +Basics +------ + +The GlusterFS development model largely revolves around the features and +functionality provided by Git version control system, Gerrit code review +system and Jenkins continuous integration system. It is a primer for a +contributor to the project. + +### Git + +Git is a extremely flexible, distributed version control system. +GlusterFS' main git repository is at <http://git.gluster.org> and public +mirrors are at GlusterForge +(https://forge.gluster.org/glusterfs-core/glusterfs) and at GitHub +(https://github.com/gluster/glusterfs). The development repo is hosted +inside Gerrit and every code merge is instantly replicated to the public +mirrors. + +A good introduction to Git can be found at +<http://www-cs-students.stanford.edu/~blynn/gitmagic/>. + +### Gerrit + +Gerrit is an excellent code review system which is developed with a git +based workflow in mind. The GlusterFS project code review system is +hosted at [review.gluster.org](http://review.gluster.org). Gerrit works +on "Change"s. A change is a set of modifications to various files in +your repository to accomplish a task. It is essentially one large git +commit with all the necessary changes which can be both built and +tested. + +Gerrit usage is described later in 'Review Process' section. + +### Jenkins + +Jenkins is a Continuous Integration build system. Jenkins is hosted at +<http://build.gluster.org>. Jenkins is configured to work with Gerrit by +setting up hooks. Every "Change" which is pushed to Gerrit is +automatically picked up by Jenkins, built and smoke tested. Output of +all builds and tests can be viewed at +<http://build.gluster.org/job/smoke/>. Jenkins is also setup with a +'regression' job which is designed to execute test scripts provided as +part of the code change. + +Preparatory Setup +----------------- + +Here is a list of initial one-time steps before you can start hacking on +code. + +### Register + +Sign up for an account at <http://review.gluster.org> by clicking +'Register' on the right-hand top. You can use your gmail login as the +openID identity. + +### Preferred email + +On first login, add your git/work email to your identity. You will have +to click on the URL which is sent to your email and set up a proper Full +Name. Make sure you set your git/work email as your preferred email. +This should be the email address from which all your code commits are +associated. + +### Set Username + +Select yourself a username. + +### Watch glusterfs + +In Gerrit settings, watch the 'glusterfs' project. Tick on all the three +(New Changes, All Comments, Submitted Changes) types of notifications. + +### Email filters + +Set up a filter rule in your mail client to tag or classify mails with +the header + + List-Id: <gerrit-glusterfs.review.gluster.org> + +as mails originating from the review system. + +### SSH keys + +Provide your SSH public key into Gerrit so that you can successfully +access the development git repo as well as push changes for +review/merge. + +### Clone a working tree + +Get yourself a working tree by cloning the development repository from +Gerrit + + sh$ git clone ssh://[username)@]git.gluster.org/glusterfs.git glusterfs + +Branching policy +---------------- + +This section describes both, the branching policies on the public repo +as well as the suggested best-practice for local branching + +### Master/release branches + +In glusterfs.git, the master branch is the forward development branch. +This is where new features come in first. In fact this is where almost +every change (commit) comes in first. The master branch is always kept +in a buildable state and smoke tests pass. + +Release trains (3.1.z, 3.2.z, 3.2.z) each have a branch originating from +master. Code freeze of each new release train is marked by the creation +of the release-3.y branch. At this point no new features are added to +the release-3.y branch. All fixes and commits first get into master. +From there, only bug fixes get backported to the relevant release +branches. From the release-3.y branch, actual release code snapshots +(e.g. glusterfs-3.2.2 etc.) are tagged (git annotated tag with 'git tag +-a') shipped as a tarball. + +### Personal per-task branches + +As a best practice, it is recommended you perform all code changes for a +task in a local branch in your working tree. The local branch should be +created from the upstream branch to which you intend to submit the +change. If you are submitting changes to master branch, first create a +local task branch like this - + + sh$ git checkout master + sh$ git branch bug-XYZ && git checkout bug-XYZ + ... <hack, commit> + +If you are backporting a fix to a release branch, or making a new change +to a release branch, your commands would be slightly different. If you +are checking out a release branch in your local working tree for the +first time, make sure to set it up as a remote tracking branch like this +- + + sh$ git checkout -b release-3.2 origin/release-3.2 + +The above step is not necessary to be repeated. In the future if you +want to work to the release branch - + + sh$ git checkout release-3.2 + sh$ git branch bug-XYZ-release-3.2 && git checkout bug-XYZ-release-3.2 + ... <cherry-pick, hack, commit> + +Building +-------- + +### Environment Setup + +**For details about the required packages for the build environment +refer : [Building GlusterFS](./Building GlusterFS.md)** + +Ubuntu: + +To setup the build environment on an Ubuntu system, type the following +command to install the required packages: + + sudo apt-get -y install python-pyxattr libreadline-dev systemtap-sdt-dev + tar python-pastedeploy python-simplejson python-sphinx python-webob libssl-dev + pkg-config python-dev python-eventlet python-netifaces libaio-dev libibverbs-dev + libtool libxml2-dev liblvm2-dev make autoconf automake bison dos2unix flex libfuse-dev + +CentOS/RHEL/Fedora: + +On Fedora systems, install the required packages by following the +instructions in [CompilingRPMS](./Compiling RPMS.md). + +### Creating build environment + +Once the required packages are installed for your appropiate system, +generate the build configuration: + + sh$ ./autogen.sh + sh$ ./configure --enable-fusermount + +### Build and install + +#### GlusterFS + +Ubuntu: + +Type the following to build and install GlusterFS on the system: + + sh$ make + sh$ make install + +CentOS/RHEL/Fedora: + +In an rpm based system, there are two methods to build GlusterFS. One is +to use the method describe above for *Ubuntu*. The other is to build and +install RPMS as described in [CompilingRPMS](./Compiling RPMS.md). + +#### GlusterFS UFO/SWIFT + +To build and run Gluster UFO you can do the following: + +1. Build, create, and install the RPMS as described in + [CompilingRPMS](./Compiling RPMS.md). +2. Configure UFO/SWIFT as described in [Howto Using UFO SWIFT - A quick + and dirty setup + guide](http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide) + +Commit policy +------------- + +For a Gerrit based work flow, each commit should be an independent, +buildable and testable change. Typically you would have a local branch +per task, and most of the times that branch will have one commit. + +If you have a second task at hand which depends on the changes of the +first one, then technically you can have it as a separate commit on top +of the first commit. But it is important that the first commit should be +a testable change by itself (if not, it is an indication that the two +commits are essentially part of a single change). Gerrit accommodates +these situations by marking Change 1 as a "dependency" of Change 2 +(there is a 'Dependencies' tab in the Change page in Gerrit) +automatically when you push the changes for review from the same local +branch. + +You will need to sign-off your commit (git commit -s) before sending the +patch for review. By signing off your patch, you agree to the terms +listed under "Developer's Certificate of Origin" section in the +CONTRIBUTING file available in the repository root. + +Provide a meaningful commit message. Your commit message should be in +the following format + +- A short one line subject describing what the patch accomplishes +- An empty line following the subject +- Situation necessitating the patch +- Description of the code changes +- Reason for doing it this way (compared to others) +- Description of test cases + +### Test cases + +Part of the workflow is to aggregate and execute pre-commit test cases +which accompany patches, cumulatively for every new patch. This +guarantees that tests which are working till the present are not broken +with the new patch. Every change submitted to Gerrit much include test +cases in + + tests/group/script.t + +as part of the patch. This is so that code changes and accompanying test +cases are reviewed together. All new commits now come under the +following categories w.r.t test cases: + +#### New 'group' directory and/or 'script.t' + +This is typically when code is adding a new module and/or feature + +#### Extend/Modify old test cases in existing scripts + +This is typically when present behavior (default values etc.) of code is +changed + +#### No test cases + +This is typically when code change is trivial (e.g. fixing typos in +output strings, code comments) + +#### Only test case and no code change + +This is typically when we are adding test cases to old code (already +existing before this regression test policy was enforced) + +More details on how to work with test case scripts can be found in + +tests/README + +Review process +-------------- + +### rfc.sh + +After doing the local commit, it is time to submit the code for review. +There is a script available inside glusterfs.git called rfc.sh. You can +submit your changes for review by simply executing + + sh$ ./rfc.sh + +This script does the following: + +- The first time it is executed, it downloads a git hook from + <http://review.gluster.org/tools/hooks/commit-msg> and sets it up + locally to generate a Change-Id: tag in your commit message (if it + was not already generated.) +- Rebase your commit against the latest upstream HEAD. This rebase + also causes your commits to undergo massaging from the just + downloaded commit-msg hook. +- Prompt for a Bug Id for each commit (if it was not already provded) + and include it as a "BUG:" tag in the commit log. You can just hit + <enter> at this prompt if your submission is purely for review + purposes. +- Push the changes to review.gluster.org for review. If you had + provided a bug id, it assigns the topic of the change as "bug-XYZ". + If not it sets the topic as "rfc". + +On a successful push, you will see a URL pointing to the change in +review.gluster.org + +Auto verification +----------------- + +The integration between Jenkins and Gerrit triggers an event in Jenkins +on every push of changes, to pick up the change and run build and smoke +test on it. + +If the build and smoke tests execute successfuly, Jenkins marks the +change as '+0 Verified'. If they fail, '-1 Verified' is marked on the +change. This means passing the automated smoke test is a necessary +condition but not sufficient. + +It is important to note that Jenkins verification is only a generic +verification of high level tests. More concentrated testing effort for +the patch is necessary with manual verification. + +If auto verification fails, it is a good reason to skip code review till +a fixed change is pushed later. You can click on the build URL +automatically put as a comment to inspect the reason for auto +verification failure. In the Jenkins job page, you can click on the +'Console Output' link to see the exact point of failure. + +Reviewing / Commenting +---------------------- + +Code review with Gerrit is relatively easy compared to other available +tools. Each change is presented as multiple files and each file can be +reviewed in Side-by-Side mode. While reviewing it is possible to comment +on each line by double-clicking on it and writing in your comments in +the text box. Such in-line comments are saved as drafts, till you +finally publish them as a Review from the 'Change page'. + +There are many small and handy features in Gerrit, like 'starring' +changes you are interested to follow, setting the amount of context to +view in the side-by-side view page etc. + +Incorporate, Amend, rfc.sh, Reverify +------------------------------------ + +Code review comments are notified via email. After incorporating the +changes in code, you can mark each of the inline comment as 'done' +(optional). After all the changes to your local files, amend the +previous commit with these changes with - + + sh$ git commit -a --amend + +Push the amended commit by executing rfc.sh. If your previous push was +an "rfc" push (i.e, without a Bug Id) you will be prompted for a Bug Id +again. You can re-push an rfc change without any other code change too +by giving a Bug Id. + +On the new push, Jenkins will re-verify the new change (independent of +what the verification result was for the previous push). + +It is the Change-Id line in the commit log (which does not change) that +associates the new push as an update for the old push (even though they +had different commit ids) under the same Change. In the side-by-side +view page, it is possible to set knobs in the 'Patch History' tab to +view changes between patches as well. This is handy to inspect how +review comments were incorporated. + +If further changes are found necessary, comments can be made on the new +patch as well, and the same cycle repeats. + +If no further changes are necessary, the reviewer can mark the patch as +reviewed with a certain score depending on the depth of review and +confidence (+1 or +2). A -1 review indicates non-agreement for the +change to get merged upstream. + +Regression tests and test cases +------------------------------- + +All code changes which are not trivial (typo fixes, code comment +changes) must be accompanied with either a new test case script or +extend/modify an existing test case script. It is important to review +the test case in conjunction with the code change to analyse whether the +code change is actually verified by the test case. + +Regression tests (i.e, execution of all test cases accumulated with +every commit) is not automatically triggered as the test cases can be +extensive and is quite expensive to execute for every change submission +in the review/resubmit cycle. Instead it is triggered by the +maintainers, after code review. Passing the regression test is a +necessary condition for merge along with code review points. + +Submission Qualifiers +--------------------- + +For a change to get merged, there are two qualifiers which are enforced +by the Gerrit system. They are - A change should have at least one '+2 +Reviewed', and a change should have at least one '+1 Verified' +(regression test). The project maintainer will merge the changes once a +patch meets these qualifiers. + +Submission Disqualifiers +------------------------ + +There are three types of "negative votes". + +-1 Verified + +-1 Code-Review ("I would prefer that you didn't submit this") + +-2 Code-Review ("Do not submit") + +The implication and scope of each of the three are different. They +behave differently as changes are resubmitted as new patchsets. + +### -1 Verified + +Anybody voting -1 Verified will prevent \*that patchset only\* from +getting merged. The flag is automatically cleared on the next patchset +post. The intention is that this vote is based on the result of some +kind of testing. A voter is expected to explain the test case which +failed. Jenkins jobs (smoke, regression, ufounit) use this field for +voting -1/0/+1. When voting -1, Jenkins posts the link to the URL which +has the console output of the failed job. + +### -1 Code-Review ("I would prefer that you didn't submit this") + +This is an advisory vote based on the content of the patch. Typically +issues in source code (both design and implementation), source code +comments, log messages, license headers etc. found by human inspection. +The reviewer explains the specific issues by commenting against the most +relevant lines of source code in the patch. On a resubmission, -1 votes +are cleared automatically. It is the responsibility of the maintainers +to honor -1 Code-Review votes from reviewers (by not merging the +patches), and inspecting that -1 comments on previous submissions are +addressed in the new patchset. Generally this is the recommended +"negative" vote. + +### -2 Code-Review ("Do not submit") + +This is a stronger vote which actually prevents Gerrit from merging the +patch. The -2 vote persists even after resubmission and continues to +prevent the patch from getting merged, until the voter revokes the -2 +vote (and then is further subjected to Submission Qualifiers). Typically +one would vote -2 if they are \*against the goal\* of what the patch is +trying to achieve (and not an issue with the patch, which can change on +resubmission). A reviewer would also vote -2 on a patch even if there is +agreement with the goal, but the issue in the code is of such a critical +nature that the reviewer personally wants to inspect the next patchset +and only then revoke the vote after finding the new patch satisfactory. +This prevents the merge of the patch in the mean time. Every registered +user has the right to exercise the -2 Code review vote, and cannot be +overridden by the maintainers. diff --git a/doc/developer-guide/Easy-Fix-Bugs.md b/doc/developer-guide/Easy-Fix-Bugs.md new file mode 100644 index 00000000000..9ba36213a73 --- /dev/null +++ b/doc/developer-guide/Easy-Fix-Bugs.md @@ -0,0 +1,35 @@ +Fixing easy bugs is an excellent method to start contributing patches to +Gluster. + +- Bugs which are marked with EasyFix flag can be found from below + BugZilla query. + - [Bugzilla Query For EasyFix + Bugs](https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&keywords=EasyFix&list_id=2626252&product=GlusterFS) + - [RSS-feed for EasyFix Gluster Bugs](http://goo.gl/OpQwlv) +- To fix EasyFix bugs, + - When you pick an EasyFix you want to work on, assign it to + yourself and move it to ASSIGNED + - Check + [Bug report life cycle](./Bug report Life Cycle.md) and + follow it. + - Check Developers page for details about development workflow, + GlusterFS design documents etc. + +Sometimes an *Easy Fix* bug has a patch attached. In those cases, +the *Patch* keyword has been added to the bug. These bugs can be +used by new contributors that would like to verify their workflow. [Bug +1099645](https://bugzilla.redhat.com/1099645) is one example of those. + +### Guidelines for new comers + +- While trying to write a patch, do not hesitate to ask questions. +- If something in the documentation is unclear, we do need to know so + that we can improve it. +- There are no stupid questions, and it's more stupid to not ask + questions that others can easily answer. Always assume that if you + have a question, someone else would like to hear the answer too. + +[Reach out](http://gluster.org/community/index.html) to the developers +in \#gluster or \#gluster-dev on Freenode IRC, or on one of the mailing +lists, try to keep the discussions public so that anyone can learn from +it. diff --git a/doc/developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md b/doc/developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md new file mode 100644 index 00000000000..5a3dceb7d0e --- /dev/null +++ b/doc/developer-guide/Fixing-issues-reported-by-tools-for-static-code-analysis.md @@ -0,0 +1,66 @@ +Static Code Analysis Tools +-------------------------- + +Bug fixes for issues reported by *Static Code Analysis Tools* should +follow [Development Work Flow](./Development Workflow.md) + +### Coverity + +GlusterFS is part of [Coverity's](https://scan.coverity.com/) scan +program. + +- To see Coverity issues you have to be a member of the GlusterFS + project in Coverity scan website. +- Here is the link to [Coverity scan + website](https://scan.coverity.com/projects/987) +- Go to above link and subscribe to GlusterFS project (as + contributor). It will send a request to Admin for including you in + the Project. +- Once admins for the GlusterFS Coverity scan approve your request, + you will be able to see the defects raised by Coverity. +- [BZ 789278](https://bugzilla.redhat.com/show_bug.cgi?id=789278) + should be used as a umbrella bug for Coverity issues in master + branch unless you are trying to fix a specific bug in Bugzilla. + - While sending patches for fixing Coverity issues please use the + same bug number. + - For 3.6 branch the Coverity tracking bug is + [1122834](https://bugzilla.redhat.com/show_bug.cgi?id=1122834) +- When you decide to work on some issue, please assign it to your name + in the same Coverity website. So that we don't step on each others + work. +- When marking a bug intentional in Coverity scan website, please put + an explanation for the same. So that it will help others to + understand the reasoning behind it. + +*If you have more questions please send it to +[gluster-devel](http://www.gluster.org/interact/mailinglists) mailing +list* + +### CPP Check + +Cppcheck is available in Fedora and EL's EPEL repo + +- Install Cppcheck + + yum install cppcheck + +- Clone GlusterFS code + + git clone https://github.com/gluster/glusterfs) glusterfs + +- Run Cpp check + + cppcheck glusterfs/ 2>cppcheck.log + +- [BZ 1091677](https://bugzilla.redhat.com/show_bug.cgi?id=1091677) + should be used for submitting patches to master branch for Cppcheck + reported issues. + +### Daily Runs + +We now have daily runs of various static source code analysis tools on +the glusterfs sources. There are daily analyses of the master, +release-3.6, and release-3.5 branches. + +Results are posted at +<http://download.gluster.org/pub/gluster/glusterfs/static-analysis/> diff --git a/doc/developer-guide/GlusterFS-Release-process.md b/doc/developer-guide/GlusterFS-Release-process.md new file mode 100644 index 00000000000..504b012def7 --- /dev/null +++ b/doc/developer-guide/GlusterFS-Release-process.md @@ -0,0 +1,73 @@ +Release Process for GlusterFS +============================= + +Create tarball +-------------- + +1. Add the release-notes to the docs/release-notes/ directory in the + sources +2. after merging the release-notes, create a tag like v3.6.2 +3. push the tag to git.gluster.org +4. create the tarball with the [release job in + Jenkins](http://build.gluster.org/job/release/) + +Notify packagers +---------------- + +Notify the packagers that we need packages created. Provide the link to the +source tarball from the Jenkins release job to the [packagers +mailinglist](mailto:packaging@gluster.org). A list of the people involved in +the package maintenance for the different distributions is in the `MAINTAINERS` +file in the sources. + +Create a new Tracker Bug for the next release +--------------------------------------------- + +The tracker bugs are used as guidance for blocker bugs and should get created when a release is made. To create one + +- file a [new bug in Bugzilla](https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS) +- base the contents on previous tracker bugs, like the one for [glusterfs-3.5.5](https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.5.5) +- set the '''Alias''' (it is a text-field) of the bug to 'glusterfs-a.b.c' where a.b.c is the next minor version +- save the new bug +- you should now be able to use the 'glusterfs-a.b.c' to access the bug, use the alias to replace the BZ# in URLs, or '''blocks''' fields +- bugs that were not fixed in this release, but were added to the tracker should be moved to the new tracker + + +Create Release Announcement +--------------------------- + +Create the Release Announcement (this is often done while people are +making the packages). The contents of the release announcement can be +based on the release notes, or should at least have a pointer to them. + +Examples: + +- [blog](http://blog.gluster.org/2014/11/glusterfs-3-5-3beta2-is-now-available-for-testing/) +- [release + notes](https://github.com/gluster/glusterfs/blob/v3.5.3/doc/release-notes/3.5.3.md) + +Send Release Announcement +------------------------- + +Once the Fedora/EL RPMs are ready (and any others that are ready by +then), send the release announcement: + +- Gluster Mailing lists + - gluster-announce, gluster-devel, gluster-users +- Gluster Blog +- Gluster Twitter account +- Gluster Facebook page +- Gluster LinkedIn group - Justin has access +- Gluster G+ + +Close Bugs +---------- + +Close the bugs that have all their patches included in the release. +Leave a note in the bug report with a pointer to the release +announcement. + +Other things to consider +------------------------ + +- Translations? - Are there strings needing translation? diff --git a/doc/developer-guide/Guidelines-For-Maintainers.md b/doc/developer-guide/Guidelines-For-Maintainers.md new file mode 100644 index 00000000000..71612cfe8c7 --- /dev/null +++ b/doc/developer-guide/Guidelines-For-Maintainers.md @@ -0,0 +1,70 @@ +### Guidelines For Maintainers + +GlusterFS has maintainers, sub-maintainers and release maintainers to +manage the project's codebase. Sub-maintainers are the owners for +specific areas/components of the source tree. Maintainers operate across +all components in the source tree.Release maintainers are the owners for +various release branches (release-x.y) present in the GlusterFS +repository. + +In the guidelines below, release maintainers and sub-maintainers are +also implied when there is a reference to maintainers unless it is +explicitly called out. + +### Guidelines that Maintainers are expected to adhere to + +1. Ensure qualitative and timely management of patches sent for review. + +2. For merging patches into the repository, it is expected of +maintainers to: + + a> Merge patches of owned components only. + b> Seek approvals from all maintainers before merging a patchset spanning multiple components. + c> Ensure that regression tests pass for all patches before merging. + d> Ensure that regression tests accompany all patch submissions. + e> Ensure that documentation is updated for a noticeable change in user perceivable behavior or design. + f> Encourage code unit tests from patch submitters to improve the overall quality of the codebase. + g> Not merge patches written by themselves until there is a +2 Code Review vote by other reviewers. + +3. The responsibility of merging a patch into a release branch in +normal circumstances will be that of the release maintainer's. Only in +exceptional situations, maintainers & sub-maintainers will merge patches +into a release branch. + +4. Release maintainers will ensure approval from appropriate +maintainers before merging a patch into a release branch. + +5. Maintainers have a responsibility to the community, it is expected +of maintainers to: + + a> Facilitate the community in all aspects. + b> Be very active and visible in the community. + c> Be objective and consider the larger interests of the community ahead of individual interests. + d> Be receptive to user feedback. + e> Address concerns & issues affecting users. + f> Lead by example. + +### Queries on Guidelines + +Any questions or comments regarding these guidelines can be routed to +gluster-devel at gluster dot org. + +### Patches in Gerrit + +Gerrit can be used to list patches that need reviews and/or can get +merged. Some queries have been prepared for this, edit the search box in +Gerrit to make your own variation: + +- [3.5 open reviewed/verified (non + rfc)](http://review.gluster.org/#/q/project:glusterfs+branch:release-3.5+status:open+%28label:Code-Review%253D%252B1+OR+label:Code-Review%253D%252B2+OR+label:Verified%253D%252B1%29+NOT+topic:rfc+NOT+label:Code-Review%253D-2,n,z) +- [All open 3.5 patches (non + rfc)](http://review.gluster.org/#/q/project:glusterfs+branch:release-3.5+status:open+NOT+topic:rfc,n,z) +- [Open NFS (master + branch)](http://review.gluster.org/#/q/project:glusterfs+branch:master+status:open+message:nfs,n,z) + +An other option can be used in combination with the Gerrit queries, and +has support for filename/directory matching (the queries above do not). +Go to the [settings](http://review.gluster.org/#/settings/projects) in +your Gerrit profile, and enter filters like these: + + diff --git a/doc/developer-guide/Jenkins-Infrastructure.md b/doc/developer-guide/Jenkins-Infrastructure.md new file mode 100644 index 00000000000..cb5d63ecfb7 --- /dev/null +++ b/doc/developer-guide/Jenkins-Infrastructure.md @@ -0,0 +1,127 @@ +We're using Gerrit and [Jenkins](http://jenkins-ci.org) at the moment. +Our Gerrit instance: + +http://review.gluster.org + +It's hosted on an ancient VM (badly needs upgrading) in some hosting +place called iWeb. We're wanting to migrate this to a Rackspace VM in +the very near future. + +Our main Jenkins instance: + +http://build.gluster.org + +That's also a pretty-out-of-date version of Jenkins, on an badly +outdated VM. That one's in Rackspace at least. We intend on migrating to +a new VM (and new Jenkins) in the not-too-far-future. No ETA yet. ;) + +As well as those two main pieces, we have a bunch of VM's in Rackspace +with various OS's on them: + +http://build.gluster.org/computer/ + +In that list we have: + +- bulk\*.cloud.gluster.org\ + + - Temporary VM's used for running bulk regression tests on, for + analysing our spurious regression failure problem + - Setup and maintained by Justin Clift + +- freebsd0.cloud.gluster.org\ + + - FreeBSD 10.0 VM in Rackspace. Used for automatic smoke testing + on FreeBSD of all proposed patches (uses a Gerrit trigger). + +- g4s-rackspace-\* (apart from gfs-rackspace-f20-1), and + tiny-rackspace-f20-1\ + + - Various VM's in Rackspace with Fedora and EL6 on them, setup by + Luis Pabon. From their description in Jenkins, they're nodes for + "open-stack swift executing functional test suite against + Gluster-for-Swift". + +- gfs-rackspace-f20-1\ + + - A VM in Rackspace for automatically building RPMs on. Setup + + maintained by Luis Pabon. + +- netbsd0.cloud.gluster.org\ + + - NetBSD 6.1.4 VM in Rackspace. Used for automatic smoke testing + on NetBSD 6.x of all proposed patches (uses a Gerrit trigger). + - Setup and maintained by Manu Dreyfus + +- netbsd7.cloud.gluster.org\ + + - NetBSD 7 (beta) VM in Rackspace. Used for automatic smoke + testing on NetBSD 7 of all proposed patches (uses a Gerrit + trigger). + - Setup and maintained by Manu Dreyfus + +- nbslave7\*.cloud.gluster.org\ + + - NetBSD 7 slaves VMs for running our regression tests on + - Setup and maintained by Manu Dreyfus + +- slave20.cloud.gluster.org - slave49.cloud.gluster.org\ + + - CentOS 6.5 VM's in Rackspace. Used for automatic regression + testing of all proposed patches (uses a Gerrit trigger). + - Setup and maintained by Michael Scherer + +Work is being done on the GlusterFS regression tests so they'll function +on FreeBSD and NetBSD (instead of just Linux). When that's complete, +we'll automatically run full regression testing on FreeBSD and NetBSD +for all proposed patches too. + +Non Jenkins VMs +--------------- + +**backups.cloud.gluster.org** + + Server holding our nightly backups. Setup and maintained by Michael + Scherer. + +**bareos-dev.cloud.gluster.org, bareos-data.cloud.gluster.org** + + Shared VMs to debug Bareos and libgfapi integration. Maintained by + Niels de Vos. + +**bugs.cloud.gluster.org** + + Hosting + [gluster-bugs-webui](https://github.com/gluster/gluster-bugs-webui) + for bug triage/checking. Maintained by Niels de Vos. + +**docs.cloud.gluster.org** + + Documentation server, running readTheDocs - managed by Soumya Deb. + +**download.gluster.org** + + Our primary download server - holds the Gluster binaries we + generate, which people can download. + +**gluster-sonar** + + Hosts our Gluster + [SonarQube](http://sonar.peircean.com/dashboard/index/com.peircean.glusterfs:glusterfs-java-filesystem) + instance. Setup and maintained by Louis Zuckerman. + +**salt-master.gluster.org** + + Our Configuration Mgmt master VM. Maintained by Michael Scherer. + +**munin.gluster.org** + + Munin master. Maintained by Michael Scherer. + +**webbuilder.gluster.org** + + Our builder for the website. Maintained by Michael Scherer. + +**www.gluster.org aka supercolony.gluster.org** + + The main website server. Maintained by Michael Scherer, Justin + Clift, Others ( add your name ) diff --git a/doc/developer-guide/Jenkins-Manual-Setup.md b/doc/developer-guide/Jenkins-Manual-Setup.md new file mode 100644 index 00000000000..3622c7265a0 --- /dev/null +++ b/doc/developer-guide/Jenkins-Manual-Setup.md @@ -0,0 +1,146 @@ +Setting up Jenkins slaves on Rackspace for GlusterFS regression testing +======================================================================= + +This is for RHEL/CentOS 6.x. The below commands should be run as root. + +### Install additional required packages + + yum -y install cmockery2-devel dbench libacl-devel mock nfs-utils yajl perl-Test-Harness salt-minion + +### Enable yum-cron for automatic rpm updates + + chkconfig yum-cron on + +### Add the mock user + + useradd -g mock mock + +### Disable eth1 + +Because GlusterFS can fail if more than 1 ethernet interface + + sed -i 's/ONBOOT=yes/ONBOOT=no/' /etc/sysconfig/network-scripts/ifcfg-eth1 + +### Disable IPv6 + +As per <https://access.redhat.com/site/node/8709> + + sed -i 's/IPV6INIT=yes/IPV6INIT=no/' /etc/sysconfig/network-scripts/ifcfg-eth0 + echo 'options ipv6 disable=1' > /etc/modprobe.d/ipv6.conf + chkconfig ip6tables off + sed -i 's/NETWORKING_IPV6=yes/NETWORKING_IPV6=no/' /etc/sysconfig/network + echo ' ' >> /etc/sysctl.conf + echo '# ipv6 support in the kernel, set to 0 by default' >> /etc/sysctl.conf + echo 'net.ipv6.conf.all.disable_ipv6 = 1' >> /etc/sysctl.conf + echo 'net.ipv6.conf.default.disable_ipv6 = 1' >> /etc/sysctl.conf + sed -i 's/v inet6/- inet6/' /etc/netconfig + +### Update hostname + + vi /etc/sysconfig/network + vi /etc/hosts + +### Remove IPv6 and eth1 interface from /etc/hosts + + sed -i 's/^10\./#10\./' /etc/hosts + sed -i 's/^2001/#2001/' /etc/hosts + +### Install ntp + + yum -y install ntp + chkconfig ntpdate on + service ntpdate start + +### Install OpenJDK, needed for Jenkins slaves + + yum -y install java-1.7.0-openjdk + +### Create the Jenkins user + + useradd -G wheel jenkins + chmod 755 /home/jenkins + +### Set the Jenkins password + + passwd jenkins + +### Copy the Jenkins SSH key from build.gluster.org + + mkdir /home/jenkins/.ssh + chmod 700 /home/jenkins/.ssh + cp `<somewhere>` /home/jenkins/.ssh/id_rsa + chown -R jenkins:jenkins /home/jenkins/.ssh + chmod 600 /home/jenkins/.ssh/id_rsa + +### Generate the SSH known hosts file for jenkins user + + su - jenkins + mkdir ~/foo + cd ~/foo + git clone `[`ssh://build@review.gluster.org/glusterfs.git`](ssh://build@review.gluster.org/glusterfs.git) + (this will ask if the new host fingerprint should be added. Choose yes) + cd .. + rm -rf ~/foo + exit + +### Install git from RPMForge + + yum -y install http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm + yum -y --enablerepo=rpmforge-extras update git + +### Install the GlusterFS patch acceptance tests + + git clone git://forge.gluster.org/gluster-patch-acceptance-tests/gluster-patch-acceptance-tests.git /opt/qa + +### Add the loopback mount point to /etc/fstab + +For the 1GB Rackspace VM's use this: + + echo '/backingstore /d xfs loop 0 2' >> /etc/fstab + mount /d + +For the 2GB and above Rackspace VM's use this: + + echo '/dev/xvde /d xfs defaults 0 2' >> /etc/fstab + mount /d + +### Create the directories needed for the regression testing + + JDIRS="/var/log/glusterfs /var/lib/glusterd /var/run/gluster /d /d/archived_builds /d/backends /d/build /d/logs /home/jenkins/root" + mkdir -p $JDIRS + chown jenkins:jenkins $JDIRS + chmod 755 $JDIRS + ln -s /d/build /build + +### Create the directories where regression logs are archived + + ADIRS="/archives/archived_builds /archives/logs" + mkdir -p $ADIRS + chown jenkins:jenkins $ADIRS + chmod 755 $ADIRS + +### Install Nginx + +For making logs available over http + + yum -y install http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm + yum -y install nginx + lokkit -s http + +### Copy the Nginx config file into place + + cp -f /opt/qa/nginx/default.conf /etc/nginx/conf.d/default.conf + +### Enable wheel group for sudo + + sed -i 's/# %wheel\tALL=(ALL)\tNOPASSWD/%wheel\tALL=(ALL)\tNOPASSWD/' /etc/sudoers + +### Reboot (for networking changes to take effect) + + reboot + +### Add forward and reverse DNS entries for the slave into Rackspace DNS + +Rackspace recently added [API calls for its Cloud +DNS](https://developer.rackspace.com/docs/cloud-dns/getting-started/?lang=python) +service, so we should be able to fully automate this part as well now.
\ No newline at end of file diff --git a/doc/developer-guide/Language-Bindings.md b/doc/developer-guide/Language-Bindings.md new file mode 100644 index 00000000000..89ef6df3d78 --- /dev/null +++ b/doc/developer-guide/Language-Bindings.md @@ -0,0 +1,39 @@ +GlusterFS 3.4 introduced the libgfapi client API for C programs. This +page lists bindings to the libgfapi C library from other languages. + +Go +-- + +- [gogfapi](https://forge.gluster.org/gogfapi) - Go language bindings + for libgfapi, aiming to provide an api consistent with the default + Go file apis. + +Java +---- + +- [libgfapi-jni](https://github.com/semiosis/libgfapi-jni/) - Low + level JNI binding for libgfapi +- [glusterfs-java-filesystem](https://github.com/semiosis/glusterfs-java-filesystem) + - High level NIO.2 FileSystem Provider implementation for the Java + platform +- [libgfapi-java-io](https://github.com/gluster/libgfapi-java-io) - + Java bindings for libgfapi, similar to java.io + +Python +------ + +- [libgfapi-python](https://github.com/gluster/libgfapi-python) - + Libgfapi bindings for Python + +Ruby +---- + +- [libgfapi-ruby](https://github.com/spajus/libgfapi-ruby) - Libgfapi + bindings for Ruby using FFI + +Rust +---- + +- [gfapi-sys](https://github.com/cholcombe973/Gfapi-sys) - Libgfapi + bindings for Rust using FFI + diff --git a/doc/developer-guide/Projects.md b/doc/developer-guide/Projects.md new file mode 100644 index 00000000000..5c41fef9daf --- /dev/null +++ b/doc/developer-guide/Projects.md @@ -0,0 +1,99 @@ +This page contains a list of project ideas which will be suitable for +students (for GSOC, internship etc.) + +Projects with mentors +--------------------- + +### gfsck - A GlusterFS filesystem check + +- A tool to check filesystem integrity and repairing +- I'm currently working on it +- Owner: Xavier Hernandez (Datalab) + +### Sub-directory mount support for native GlusterFS mounts + +Allow clients to directly mount directories inside a GlusterFS volume, +like how NFS clients can mount directories inside an NFS export. + +Mentor: Kaushal <kshlmster at gmail dot com> + +### GlusterD services high availablity + +GlusterD should restart the processes it manages, bricks, nfs server, +self-heal daemon and quota daemon, whenever it detects they have died. + +Mentor : Atin Mukherjee <atin.mukherjee83@gmail.com> + +### Language bindings for libgfapi + +- API/library for accessing gluster volumes + +### oVirt gui for stats + +Have pretty graphs and tables in ovirt for the GlusterFS top and profile +commands. + +### Monitoring integrations - munin others + +The more monitoring support we have for GlusterFS the better. + +### More compression algorithms for compression xlator + +The on-wire compression translator should be extended to support more +compression algorithms. Ideally it should be pluggable. + +### Cinder GlusterFS backup driver + +Write a driver for cinder, a part of openstack, to allow backup onto +GlusterFS volumes + +### rsockets - sockets for rdma transport + +Coding for RDMA using the familiar socket api should lead to a more +robust rdma transport + +### Data import tool + +Create a tool which will allow importing already existing data in the +brick directories into the gluster volume. This is most likely going to +be a special rebalance process. + +### Rebalance improvements + +Improve rebalance performance. + +### Meta translator + +The meta xlator provides a /proc like interface to GlusterFS xlators. +This could be improved upon and the meta xlator could be made a standard +part of the volume graph. + +### Geo-replication using rest-api + +Might be suitable for geo replication over WAN. + +### Quota using underlying FS' quota + +GlusterFS quota is currently maintained completely in GlusterFSs +namespace using xattrs. We could make use of the quota capabilities of +the underlying fs (XFS) for better performance. + +### Snapshot pluggability + +Snapshot should be able to make use of snapshot support provided by +btrfs for example. + +### Compression at rest + +Lessons learnt while implementing encryption at rest can be used with +the compression at rest. + +### File-level deduplication + +GlusterFS works on files. So why not have dedup at the level files as +well. + +### Composition xlator for small files + +Merge small files into a designated large file using our own custom +semantics. This can improve our small file performance.
\ No newline at end of file diff --git a/doc/developer-guide/Simplified-Development-Workflow.md b/doc/developer-guide/Simplified-Development-Workflow.md new file mode 100644 index 00000000000..c95e3ba4f67 --- /dev/null +++ b/doc/developer-guide/Simplified-Development-Workflow.md @@ -0,0 +1,238 @@ +Simplified development workflow for GlusterFS +============================================= + +This page gives a simplified model of the development workflow used by +the GlusterFS project. This will give the steps required to get a patch +accepted into the GlusterFS source. + +Visit [Development Work Flow](./Development Workflow.md) a more +detailed description of the workflow. + +Initial preperation +------------------- + +The GlusterFS development workflow revolves around +[Git](http://git.gluster.org/?p=glusterfs.git;a=summary), +[Gerrit](http://review.gluster.org) and +[Jenkins](http://build.gluster.org). + +Using these tools requires some initial preparation. + +### Dev system setup + +You should install and setup Git on your development system. Use your +distribution specific package manger to install git. After installation +configure git. At the minimum, set a git user email. To set the email +do, + + $ git config --global user.name "Name" + $ git config --global user.email <email address> + +You should also generate an ssh key pair if you haven't already done it. +To generate a key pair do, + + $ ssh-keygen + +and follow the instructions. + +Next, install the build requirements for GlusterFS. Refer +[Building GlusterFS - Build Requirements](./Building GlusterFS.md#Build Requirements) +for the actual requirements. + +### Gerrit setup + +To contribute to GlusterFS, you should first register on +[gerrit](http://review.gluster.org). + +After registration, you will need to select a username, set a preferred +email and upload the ssh public key in gerrit. You can do this from the +gerrit settings page. Make sure that you set the preferred email to the +email you configured for git. + +### Get the source + +Git clone the GlusterFS source using + + <ssh://><username>@review.gluster.org/glusterfs.git + +(replace <username> with your gerrit username). + + $ git clone (ssh://)<username> @review.gluster.org/glusterfs.git + +This will clone the GlusterFS source into a subdirectory named glusterfs +with the master branch checked out. + +It is essential that you use this link to clone, or else you will not be +able to submit patches to gerrit for review. + +Actual development +------------------ + +The commands in this section are to be run inside the glusterfs source +directory. + +### Create a development branch + +It is recommended to use separate local development branches for each +change you want to contribute to GlusterFS. To create a development +branch, first checkout the upstream branch you want to work on and +update it. More details on the upstream branching model for GlusterFS +can be found at + +[Development Work Flow - Branching\_policy](./Development Workflow.md#branching-policy). +For example if you want to develop on the master branch, + + $ git checkout master + $ git pull + +Now, create a new branch from master and switch to the new branch. It is +recommended to have descriptive branch names. Do, + + $ git branch <descriptive-branch-name> + $ git checkout <descriptive-branch-name> + +or, + + $ git checkout -b <descriptive-branch-name> + +to do both in one command. + +### Hack + +Once you've switched to the development branch, you can perform the +actual code changes. [Build](./Building GlusterFS) and test to +see if your changes work. + +#### Tests + +Unless your changes are very minor and trivial, you should also add a +test for your change. Tests are used to ensure that the changes you did +are not broken inadvertently. More details on tests can be found at + +[Development Workflow - Test cases](./Development Workflow.md#test-cases) +and +[Development Workflow - Regression tests and test cases](./Development Workflow.md#regression-tests-and-test-cases) + +### Regression test + +Once your change is working, run the regression test suite to make sure +you haven't broken anything. The regression test suite requires a +working GlusterFS installation and needs to be run as root. To run the +regression test suite, do + + # make install + # ./run-tests.sh + +### Commit your changes + +If you haven't broken anything, you can now commit your changes. First +identify the files that you modified/added/deleted using git-status and +stage these files. + + $ git status + $ git add <list of modified files> + +Now, commit these changes using + + $ git commit -s + +Provide a meaningful commit message. The commit message policy is +described at + +[Development Work Flow - Commit policy](./Development Workflow.md#commit-policy). + +It is essential that you commit with the '-s' option, which will +sign-off the commit with your configured email, as gerrit is configured +to reject patches which are not signed-off. + +### Submit for review + +To submit your change for review, run the rfc.sh script, + + $ ./rfc.sh + +The script will ask you to enter a bugzilla bug id. Every change +submitted to GlusterFS needs a bugzilla entry to be accepted. If you do +not already have a bug id, file a new bug at [Red Hat +Bugzilla](https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS). +If the patch is submitted for review, the rfc.sh script will return the +gerrit url for the review request. + +More details on the rfc.sh script are available at +[Development Work Flow - rfc.sh](./Development Workflow.md#rfc.sh). + +Review process +-------------- + +Your change will now be reviewed by the GlusterFS maintainers and +component owners on [gerrit](http://review.gluster.org). You can follow +and take part in the review process on the change at the review url. The +review process involves several steps. + +To know component owners , you can check the "MAINTAINERS" file in root +of glusterfs code directory + +### Automated verification + +Every change submitted to gerrit triggers an initial automated +verification on [jenkins](http://build.gluster.org). The automated +verification ensures that your change doesn't break the build and has an +associated bug-id. + +More details can be found at + +[Development Work Flow - Auto verification](./Development Workflow.md#auto-verification). + +### Formal review + +Once the auto verification is successful, the component owners will +perform a formal review. If they are okay with your change, they will +give a positive review. If not they will give a negative review and add +comments on the reasons. + +More information regarding the review qualifiers and disqualifiers is +available at + +[Development Work Flow - Submission Qualifiers](./Development Workflow.md#submission-qualifiers) +and +[Development Work Flow - Submission Disqualifiers](./Development Workflow.md#submission-disqualifiers). + +If your change gets a negative review, you will need to address the +comments and resubmit your change. + +#### Resubmission + +Switch to your development branch and make new changes to address the +review comments. Build and test to see if the new changes are working. + +Stage your changes and commit your new changes using, + + $ git commit --amend + +'--amend' is required to ensure that you update your original commit and +not create a new commit. + +Now you can resubmit the updated commit for review using the rfc.sh +script. + +The formal review process could take a long time. To increase chances +for a speedy review, you can add the component owners as reviewers on +the gerrit review page. This will ensure they notice the change. The +list of component owners can be found in the MAINTAINERS file present in +the GlusterFS source + +### Verification + +After a component owner has given a positive review, a maintainer will +run the regression test suite on your change to verify that your change +works and hasn't broken anything. This verification is done with the +help of jenkins. + +If the verification fails, you will need to make necessary changes and +resubmit an updated commit for review. + +### Acceptance + +After successful verification, a maintainer will merge/cherry-pick (as +necessary) your change into the upstream GlusterFS source. Your change +will now be available in the upstream git repo for everyone to use.
\ No newline at end of file diff --git a/doc/developer-guide/Using-Gluster-Test-Framework.md b/doc/developer-guide/Using-Gluster-Test-Framework.md new file mode 100644 index 00000000000..5256e973fbc --- /dev/null +++ b/doc/developer-guide/Using-Gluster-Test-Framework.md @@ -0,0 +1,270 @@ +Description +----------- + +The Gluster Test Framework, is a suite of scripts used for regression +testing of Gluster. + +It runs well on RHEL and CentOS (possibly Fedora too, presently being +tested), and is automatically run against every patch submitted to +Gluster [for review](http://review.gluster.org). + +The Gluster Test Framework is part of the main Gluster code base, living +under the "tests" subdirectory: + + http://git.gluster.org/?p=glusterfs.git;a=summary + +WARNING +------- + +Running the Gluster Test Framework deletes “/var/lib/glusterd/\*”. + +**DO NOT run it on a server with any data.** + +Preparation steps for Ubuntu 14.04 LTS +-------------------------------------- + +1. \# apt-get install dbench git libacl1-dev mock nfs-common +nfs-kernel-server libtest-harness-perl libyajl-dev xfsprogs psmisc attr +acl lvm2 rpm + +2. \# apt-get install python-webob python-paste python-sphinx + +3. \# apt-get install autoconf automake bison dos2unix flex libfuse-dev +libaio-dev libibverbs-dev librdmacm-dev libtool libxml2-dev +libxml2-utils liblvm2-dev make libssl-dev pkg-config libpython-dev +python-eventlet python-netifaces python-simplejson python-pyxattr +libreadline-dev systemtap-sdt-dev tar + +4) Install cmockery2 from github (https://github.com/lpabon/cmockery2) +and compile and make install as in Readme + +5) + + sudo groupadd mock + sudo useradd -g mock mock + +6) mkdir /var/run/gluster + +**Note**: redhat-rpm-config package is not found in ubuntu + +Preparation steps for CentOS 7 (only) +------------------------------------- + +1. Install EPEL: + + $ sudo yum install -y http://epel.mirror.net.in/epel/7/x86_64/e/epel-release-7-1.noarch.rpm + +2. Install the CentOS 7.x dependencies: + + $ sudo yum install -y --enablerepo=epel cmockery2-devel dbench git libacl-devel mock nfs-utils perl-Test-Harness yajl xfsprogs psmisc + + $ sudo yum install -y --enablerepo=epel python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config + +==\> Despite below missing packages it worked for me + + No package python-webob1.0 available. + No package python-paste-deploy1.5 available. + No package python-sphinx10 available. + + $ sudo yum install -y --enablerepo=epel autoconf automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \ + librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \ + python-devel python-eventlet python-netifaces python-paste-deploy \ + python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \ + systemtap-sdt-devel tar + +3. Create the mock user + + $ sudo useradd -g mock mock + +Preparation steps for CentOS 6.3+ (only) +---------------------------------------- + +1. Install EPEL: + + $ sudo yum install -y http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm + +2. Install the CentOS 6.x dependencies: + + $ sudo yum install -y --enablerepo=epel cmockery2-devel dbench git libacl-devel mock nfs-utils perl-Test-Harness yajl xfsprogs + $ sudo yum install -y --enablerepo=epel python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config + $ sudo yum install -y --enablerepo=epel autoconf automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \ + librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \ + python-devel python-eventlet python-netifaces python-paste-deploy \ + python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \ + systemtap-sdt-devel tar + +3. Create the mock user + + $ sudo useradd -g mock mock + +Preparation steps for RHEL 6.3+ (only) +-------------------------------------- + +1. Ensure you have the "Scalable Filesystem Support" group installed + +This provides the xfsprogs package, which is required by the test +framework. + +2. Install EPEL: + + $ sudo yum install -y http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm + +3. Install the CentOS 6.x dependencies: + + $ sudo yum install -y --enablerepo=epel cmockery2-devel dbench git libacl-devel mock nfs-utils yajl perl-Test-Harness + $ sudo yum install -y --enablerepo=rhel-6-server-optional-rpms python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config + $ sudo yum install -y --disablerepo=rhs* --enablerepo=*optional-rpms autoconf \ + automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \ + librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \ + python-devel python-eventlet python-netifaces python-paste-deploy \ + python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \ + systemtap-sdt-devel tar + +4. Create the mock user + + $ sudo useradd -g mock mock + +Preparation steps for Fedora 16-19 (only) +----------------------------------------- + +**Still in development** + +1. Install the Fedora dependencies: + + $ sudo yum install -y attr cmockery2-devel dbench git mock nfs-utils perl-Test-Harness psmisc xfsprogs + $ sudo yum install -y python-webob1.0 python-paste-deploy1.5 python-sphinx10 redhat-rpm-config + $ sudo yum install -y autoconf automake bison dos2unix flex fuse-devel libaio-devel libibverbs-devel \ + librdmacm-devel libtool libxml2-devel lvm2-devel make openssl-devel pkgconfig \ + python-devel python-eventlet python-netifaces python-paste-deploy \ + python-simplejson python-sphinx python-webob pyxattr readline-devel rpm-build \ + systemtap-sdt-devel tar + +3. Create the mock user + + $ sudo useradd -g mock mock + +Common steps +------------ + +1. Ensure DNS for your server is working + +The Gluster Test Framework fails miserably if the full domain name for +your server doesn't resolve back to itself. + +If you don't have a working DNS infrastructure in place, adding an entry +for your server to its /etc/hosts file will work. + +2. Install the version of Gluster you are testing + +Either install an existing set of rpms: + + $ sudo yum install [your gluster rpms here] + +Or compile your own ones (fairly easy): + + http://www.gluster.org/community/documentation/index.php/CompilingRPMS + +3. Clone the GlusterFS git repository + + $ git clone git://git.gluster.org/glusterfs + $ cd glusterfs + +Ensure mock can access the directory +------------------------------------ + +Some tests run as the user "mock". If the mock user can't access the +tests subdirectory directory, these tests fail. (rpm.t is one such test) + +This is a known gotcha when the git repo is cloned to your home +directory. Home directories generally don't have world readable +permissions. You can fix this by adjusting your home directory +permissions, or placing the git repo somewhere else (with access for the +mock user). + +Running the tests +----------------- + +The tests need to run as root, so they can mount volumes and manage +gluster processes as needed. + +It's also best to run them directly as the root user, instead of through +sudo. Strange things sporadicly happen (for me) when using the full test +framework through sudo, that haven't happened (yet) when running +directly as root. Hangs in dbench particularly, which are part of at +least one test. + + # ./run-tests.sh + +The test framework takes just over 45 minutes to run in a VM here (4 +cpu's assigned, 8GB ram, SSD storage). It may take significantly more or +less time for you, depending on the hardware and software you're using. + +Showing debug information +------------------------- + +To display verbose information while the tests are running, set the +DEBUG environment variable to 1 prior to running the tests. + + # DEBUG=1 ./run-tests.sh + +Log files +--------- + +Verbose output from the rpm.t test goes into "rpmbuild-mock.log", +located in the same directory the test is run from. + +Reporting bugs +-------------- + +If you hit a bug when running the test framework, **please** create a +bug report for it on Bugzilla so it gets fixed: + + https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=tests + +Creating your own tests +----------------------- + +The test scripts are written in bash, with their filenames ending in .t +instead of .sh. + +When creating your own test scripts, create them in an appropriate +subdirectory under "tests" (eg "bugs" or "features") and use descriptive +names like "bug-XXXXXXX-checking-feature-X.t" + +Also include the "include.rc" file, which defines the test types and +host/brick/volume defaults: + + . $(dirname $0)/../include.rc + +There are 5 test types available at present, but feel free to add more +if you need something that doesn't yet exist. The test types are +explained in more detail below. + +Also essential is the "cleanup" command, which removes any existing +Gluster configuration (**without backing it up**), and also kills any +running gluster processes. + +There is a basic test template you can copy, named bug-000000.t in the +bugs subdirectory: + + $ cp bugs/bug-000000.t somedir/descriptive-name.t + +### TEST + +- Example of usage in basic/volume.t + +### TEST\_IN\_LOOP + +- Example of usage in basic/rpm.t + +### EXPECT + +- Example of usage in basic/volume.t + +### EXPECT\_WITHIN + +- Example of usage in basic/volume-status.t + +### EXPECT\_KEYWORD + +- Defined in include.rc, but seems to be unused?
\ No newline at end of file diff --git a/doc/developer-guide/afr/afr-locks-evolution.md b/doc/developer-guide/afr-locks-evolution.md index 7d2a136d871..7d2a136d871 100644 --- a/doc/developer-guide/afr/afr-locks-evolution.md +++ b/doc/developer-guide/afr-locks-evolution.md diff --git a/doc/developer-guide/afr/self-heal-daemon.md b/doc/developer-guide/afr-self-heal-daemon.md index d5e081f5f49..b85ddd1c856 100644 --- a/doc/developer-guide/afr/self-heal-daemon.md +++ b/doc/developer-guide/afr-self-heal-daemon.md @@ -13,7 +13,8 @@ replicate xlators (subvolumes) of *only* the bricks present in that particular n The shd does two types of self-heal crawls: Index heal and Full heal. For both these types of crawls, the basic idea is the same: For each file encountered while crawling, perform metadata, data and entry heals under appropriate locks. * An overview of how each of these heals is performed is detailed in the 'Self-healing' section of *doc/features/afr-v1.md* -* The different file locks which the shd takes for each of these heals is detailed in *doc/developer-guide/afr/afr-locks-evolution.md* +* The different file locks which the shd takes for each of these heals is detailed in *doc/developer +-guide/afr-locks-evolution.md* Metadata heal refers to healing extended attributes, mode and permissions of a file or directory. Data heal refers to healing the file contents. @@ -32,10 +33,10 @@ each entry in index heal, a check is made if a full heal is queued. If it is, th In index heal, each shd reads the entries present inside .glusterfs/indices/xattrop/ folder and triggers heal on each entry with appropriate locks. The .glusterfs/indices/xattrop/ directory contains a base entry of the name "xattrop-<virtual-gfid-string>". All other entries are hardlinks to the base entry. The -*names* of the hardlinks are the gfid strings of the files that may need heal. +*names* of the hardlinks are the gfid strings of the files that may need heal. When a client (mount) performs an operation on the file, the index xlator present in each brick process adds the hardlinks in the pre-op phase of the FOP's transaction -and removes it in post-op phase if the operation is successful. Thus if an entry is present inside the .glusterfs/indices/xattrop directory when there is no I/O +and removes it in post-op phase if the operation is successful. Thus if an entry is present inside the .glusterfs/indices/xattrop/ directory when there is no I/O happening on the file, it means the file needs healing (or atleast an examination if the brick crashed after the post-op completed but just before the removal of the hardlink). ####Index heal steps: @@ -85,6 +86,7 @@ In shd process of *highest UUID node per replica* { /* Recurse*/ again opendir+readdir (directory) followed by self_heal_entry() of each entry. } + } } </code></pre> diff --git a/doc/developer-guide/afr/afr.md b/doc/developer-guide/afr.md index 566573a4e26..566573a4e26 100644 --- a/doc/developer-guide/afr/afr.md +++ b/doc/developer-guide/afr.md diff --git a/doc/developer-guide/coredump-analysis.md b/doc/developer-guide/coredump-analysis.md new file mode 100644 index 00000000000..16fa9165fd0 --- /dev/null +++ b/doc/developer-guide/coredump-analysis.md @@ -0,0 +1,55 @@ +This document explains how to analyze core-dumps obtained from regression +machines, with examples. +1) Download the core-tarball and extract it. +2) 'cd' into directory where the tarball is extracted. +~~~ +[root@atalur Downloads]# pwd +/home/atalur/Downloads +[root@atalur Downloads]# ls +build build-install-20150625_05_42_39.tar.bz2 lib64 usr +~~~ +3) Determine the core file you need to examine. There can be more than one core file. +You can list them from './build/install/cores' directory. +~~~ +[root@atalur Downloads]# ls build/install/cores/ +core.9341 liblist.txt liblist.txt.tmp +~~~ +In case you are unsure which binary generated the core-file, executing 'file' command on it will help. +~~~ +[root@atalur Downloads]# file ./build/install/cores/core.9341 +./build/install/cores/core.9341: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy' +~~~ +As seen, the core file was generated by glusterfsd binary, and path to it is provided (/build/install/sbin/glusterfsd). +4) Now, run the following command on the core: +~~~ +gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.xxx' <target, say ./build/install/sbin/glusterd> +In this case, +gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/core.9341' ./build/install/sbin/glusterfsd +~~~ +5) You can cross check if all shared libraries are available and loaded by using 'info sharedlibrary' command from +inside gdb. +6) Once verified, usual gdb commands based on requirement can be used to debug the core. +'bt' or 'backtrace' from gdb of core used in examples: +~~~ +Core was generated by `/build/install/sbin/glusterfsd -s slave26.cloud.gluster.org --volfile-id patchy'. +Program terminated with signal SIGABRT, Aborted. +#0 0x00007f512a54e625 in raise () from ./lib64/libc.so.6 +(gdb) bt +#0 0x00007f512a54e625 in raise () from ./lib64/libc.so.6 +#1 0x00007f512a54fe05 in abort () from ./lib64/libc.so.6 +#2 0x00007f512a54774e in __assert_fail_base () from ./lib64/libc.so.6 +#3 0x00007f512a547810 in __assert_fail () from ./lib64/libc.so.6 +#4 0x00007f512b9fc434 in __gf_free (free_ptr=0x7f50f4000e50) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:304 +#5 0x00007f512b9b6657 in loc_wipe (loc=0x7f510c20d1a0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/xlator.c:685 +#6 0x00007f511cb8201d in mq_start_quota_txn_v2 (this=0x7f5118019b60, loc=0x7f510c20d2b8, ctx=0x7f50f4000bf0, contri=0x7f50f4000d60) + at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:2921 +#7 0x00007f511cb82c55 in mq_initiate_quota_task (opaque=0x7f510c20d2b0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3199 +#8 0x00007f511cb81820 in mq_synctask (this=0x7f5118019b60, task=0x7f511cb829fa <mq_initiate_quota_task>, spawn=_gf_false, loc=0x7f510c20d430, dict=0x0, buf=0x0, contri=0) + at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:2789 +#9 0x00007f511cb82f82 in mq_initiate_quota_blocking_txn (this=0x7f5118019b60, loc=0x7f510c20d430) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3230 +#10 0x00007f511cb82844 in mq_reduce_parent_size_task (opaque=0x7f510c000df0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/features/marker/src/marker-quota.c:3117 +#11 0x00007f512ba0f9dc in synctask_wrap (old_task=0x7f510c0053e0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:370 +#12 0x00007f512a55f8f0 in ?? () from ./lib64/libc.so.6 +#13 0x0000000000000000 in ?? () +(gdb) +~~~ diff --git a/doc/developer-guide/daemon-management-framework.md b/doc/developer-guide/daemon-management-framework.md index cf29caa95ce..592192e665d 100644 --- a/doc/developer-guide/daemon-management-framework.md +++ b/doc/developer-guide/daemon-management-framework.md @@ -25,12 +25,9 @@ Data members & functions of different management objects - connection object - process object - online status - - Methods - -- manager, start, stop which can be abstracted as a common - methods or specific to service requirements - -- init API is invoked on demand of the service and currently integrated - into manager. - -- build method is to initialize the method pointers + - Methods - manager, start, stop which can be abstracted as a common methods + or specific to service requirements + - init API can be invoked using the service management object The above structures defines the skeleton of the daemon management framework. Introduction of new daemons in GlusterFS needs to inherit these properties. Any diff --git a/doc/developer-guide/data-structures/inode.md b/doc/developer-guide/datastructure-inode.md index a340ab9ca8e..a340ab9ca8e 100644 --- a/doc/developer-guide/data-structures/inode.md +++ b/doc/developer-guide/datastructure-inode.md diff --git a/doc/developer-guide/data-structures/iobuf.md b/doc/developer-guide/datastructure-iobuf.md index 5f521f1485f..5f521f1485f 100644 --- a/doc/developer-guide/data-structures/iobuf.md +++ b/doc/developer-guide/datastructure-iobuf.md diff --git a/doc/developer-guide/data-structures/mem-pool.md b/doc/developer-guide/datastructure-mem-pool.md index c71aa2a8ddd..c71aa2a8ddd 100644 --- a/doc/developer-guide/data-structures/mem-pool.md +++ b/doc/developer-guide/datastructure-mem-pool.md diff --git a/doc/developer-guide/gfapi-symbol-versions/gfapi-symbol-versions.md b/doc/developer-guide/gfapi-symbol-versions.md index c7a3ac9380e..e4f4fe9f052 100644 --- a/doc/developer-guide/gfapi-symbol-versions/gfapi-symbol-versions.md +++ b/doc/developer-guide/gfapi-symbol-versions.md @@ -29,7 +29,7 @@ file remains libfoo.so.0 forever. Legacy APIs may or may not have an associated symbol version. New APIs may or may not have an associated symbol version either. In general symbol versions are reserved for APIs that have changed. Either the function's signature has changed, i.e. the -return type or the number of paramaters, and/or the parameter types have +return time or the number of paramaters, and/or the parameter types have changed. Another reason for using symbol versions on an API is when the behaviour or functionality of the API changes dramatically. As with a library that doesn't use versioned symbols, old and new applications diff --git a/doc/developer-guide/translator-development.md b/doc/developer-guide/translator-development.md index 9153c874d0f..3bf7e153354 100644 --- a/doc/developer-guide/translator-development.md +++ b/doc/developer-guide/translator-development.md @@ -51,7 +51,7 @@ if (!(xl->fini = dlsym (handle, "fini"))) { In this example, `xl` is a pointer to the in-memory object for the translator we're loading. As you can see, it's looking up various symbols *by name* in the shared object it just loaded, and storing pointers to those symbols. Some of -them (e.g. init are functions, while others e.g. fops are dispatch tables +them (e.g. init) are functions, while others (e.g. fops) are dispatch tables containing pointers to many functions. Together, these make up the translator's public interface. @@ -102,7 +102,7 @@ various structures in logs. I've never used it myself, though I probably should. What's noteworthy here is that we don't even define dumpops. That's because all of the functions that might use these dispatch functions will check for `xl->dumpops` being `NULL` before calling through it. This is in sharp -contrast to the behavior for `fops` and `cbks1`, which *must* be present. If +contrast to the behavior for `fops` and `cbks`, which *must* be present. If they're not, translator loading will fail because these pointers are not checked every time and if they're `NULL` then we'll segfault. That's why we provide an empty definition for cbks; it's OK for the individual function |