summaryrefslogtreecommitdiffstats
path: root/geo-replication
Commit message (Collapse)AuthorAgeFilesLines
* feature/geo-rep: Fix to retain pause state of gsyncd on restartKotresh H R2014-06-202-14/+18
| | | | | | | | | | | | | | On soft reboot, geo-rep monitor is writing 'faulty' into status file. It should not do it if previous state is paused as glusterd depend on the state file on node restart. Change-Id: Idd45abf13350b087371935f1b4f6e1a346433d27 BUG: 1101410 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/8097 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* feature/geo-rep: Fix for changelog agent becoming zombie.Kotresh H R2014-06-171-4/+12
| | | | | | | | | | | | | | | | | | Monitor process spawns changelog agent and is not wait on it, hence becoming zombie. When worker is dies/killed, it respawns both worker and corresponding agent leaving the earlier changelog agent in zombie state. This patch addresses this issue by waiting on agent process in montor process. Change-Id: I571b7d6487133848edca67e7446f1caa70ae01c9 BUG: 1103643 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7956 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep/gverify: Never use ping to check for host reachabilityHarshavardhana2014-06-121-1/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On many linux distributions with iptables enabled, ICMP traffic is usually dropped even when port 22 is open for SSH service So practically `ping` is an unreliable command ~~~ root@rhs1:/var/log/glusterfs # gluster volume geo-replication geo-test \ 17.16.10.1::geo-test-slave create push-pem force 172.16.10.1 not reachable. geo-replication command failed ~~~ ~~~ root@rhs1:/var/log/glusterfs # ping 172.16.10.1 PING rhs2.sjc.redhat.com (172.16.10.1) 56(84) bytes of data. From rhs2.sjc.redhat.com (172.16.10.1) icmp_seq=1 Destination Host Prohibited ... ... ~~~ ~~~ root@rhs2:/var/log/glusterfs # service iptables status | grep 22 4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 root@rhs2:/var/log/glusterfs # service iptables status | grep icmp-host-prohibited 25 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited ~~~ Change-Id: I33206ca071aa5d755c0762f7c486da222ec3c7db BUG: 1105337 Signed-off-by: Harshavardhana <harsha@harshavardhana.net> Reviewed-on: http://review.gluster.org/7997 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Making replica failover check interval configurableAravinda VK2014-06-112-1/+3
| | | | | | | | | | | | | | | | | | Replica failover check interval is hardcoded to 60 sec by default. Now this option is made configurable and defaulted to 1 sec. To change the default value gluster volume geo-replication <MASTERVOL> \ <SLAVEHOST>::<SLAVEVOL> config replica_failover_interval 15 Change-Id: Iada1b80d510452dcfedebd8a21bebd62394b0597 BUG: 1066410 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/8003 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: entry_ops errors handlingAravinda VK2014-06-112-8/+4
| | | | | | | | | | | | | | | Xattr.lsetxattr_l call will not raise OSError which errno_wrap can handle, so we need to use Xattr.lsetxattr to get proper OSError and errno_wrap ignores or retries accordingly. Change-Id: Ie0a777152ddbaf9ed80c977e4704974fec997bea BUG: 1105083 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7972 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: FSH recommended log locationsVenky Shankar2014-06-104-8/+31
| | | | | | | | | | | | | | | | Upgrading "working_dir" on the fly is a bit unclean yet (though it works) as currently config upgrade does not support "old" values to be expanded by using configuration variables. Change-Id: I44ed65c281f2e0ce3b6b467addc5c1c88ac674e7 BUG: 1077516 Signed-off-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Kotresh H R <khiremat@redhat.com> Signed-off-by: Aravinda VK <avishwan@redhat.com> Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/7375 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* gsyncd / geo-rep: Xsync crawl metadata synchronizationVenky Shankar2014-06-101-0/+2
| | | | | | | | | | | | | Added "metadata" record for directory and file creations during the intial crawl. Change-Id: I811ae26e0144cadf7249cb64541ec354ab83fe66 BUG: 1106604 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/8018 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* feature/geo-rep: Fix to retain pause state of gsyncd on restart.Kotresh H R2014-06-053-2/+16
| | | | | | | | | | | | | | | A new gsyncd options '--pause-on-start' is introduced. When node reboots, if the status is paused, gsyncd is started with this option. After gsyncd spawns worker and agent, worker will send SIGSTOP to negative pid of monitor to enter pause mode. Change-Id: I5aad82c9a9fc8c243f384940b77d25e26e520d6d BUG: 1101410 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7885 Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* glusterd/geo-rep: Not setting the user name as the user group by defaultAvra Sengupta2014-06-052-4/+3
| | | | | | | | | | | | | Also removing multiple mounts in gverify.sh Change-Id: I1567a9f711222c5f571a12f7c4ab49cef32d60c9 BUG: 1101948 Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/7911 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd : Use --remote-host option during cli invocationVenky Shankar2014-05-251-0/+1
| | | | | | | | | | | | Required for unprivileged geo-replication sessions to execute glusterd commands (which are however restricted). Change-Id: Ib83b81defa061717f4465ffa665450d0f5d3d20d BUG: 1077452 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/7833 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* gsyncd / geo-rep: Mountbroker cli to use INET socketsVenky Shankar2014-05-251-1/+1
| | | | | | | | | | | | | | | | | | | | unprivileged geo-replication session runs the slave gsyncd process as unprivileged, thereby executing gluster cli commands as an unprivileged user. By default, cli to glusterd uses unix domain sockets, thereby restricting cli command execution by non root users. This patch introduces '--remote-host' cli option to force cli to use INET socket. For this to work, the following needs to be added in glusterd volfile option rpc-auth-allow-insecure on Change-Id: I84b1711281bbcbde156200f80ebdb065afb55488 BUG: 1077452 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/7820 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* gsyncd / geo-rep: fix cli query for volinfo fetchVenky Shankar2014-05-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | With an unprivileged geo-replication session, monitor was using user@slave for --remote-host option for gluster cli, thereby failing to sucessfully connect to the slave glusterd. This patch fixes the issue by selecting the hostname/IP from the speicified slave endpoint url. - For privileged geo-replication sessions, this patch has no effect as the slave endpoint url is just the hostname/IP. Change-Id: I88f66c406a8d9a34db7fc626965f949075e3ceac BUG: 1077452 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/7818 Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* glusterd/geo-rep: Creating .ssh dir with right ownershipAvra Sengupta2014-05-252-6/+19
| | | | | | | | | | | | | | Also adding -P option to the usage of df for portability in gverify.sh Change-Id: I0be19d26ea63769a934c6ccbfc04ef80768ebc9a BUG: 1099041 Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/7812 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* glusterd/geo-rep: Use getent passwd instead of $HOMEAvra Sengupta2014-05-202-3/+3
| | | | | | | | | | | | | | | | $HOME might not be set in the env variables, as is the case when these scripts are executed using the runner framework. Hence using getent passwd instead of $HOME Change-Id: I99f6bcd788d727be534b3040600d66c8dbb7ee92 BUG: 1099041 Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/7803 Reviewed-by: Atin Mukherjee <amukherj@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd/geo-rep: Fix remote vol info fetching for non-rootKotresh H R2014-05-151-1/+1
| | | | | | | | | | Signed-off-by: Kotresh H R <khiremat@redhat.com> Change-Id: If1d2cab3fcfe2391105551e54f0b9729a7c204e4 BUG: 1077452 Reviewed-on: http://review.gluster.org/7767 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* glusterd/geo-rep: Allow gverify.sh and S56glusterd-geo-rep-create-post.shAvra Sengupta2014-05-144-39/+96
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | to operate for non-root privileged slave volume Mounting the slave-volume on local node, to perform disk checks in order to allow gverify.sh to operate for non-root privileged slave volume Allowing the hook script S56glusterd-geo-rep-create-post.sh to operate for non-root privileged slave volume Modified peer_add_secret_pub.in to accept username as argument and add the pem keys to the users's_home_dir/.ssh/authorized_keys Wrote set_geo_rep_pem_keys.sh which accepts username as argument and copies the pem keys from the user's home directory to $GLUSTERD_WORKING_DIR/geo-replication/ and then copies the keys to other nodes in the cluster and add them to the respective authorized keys. The script takes as argument the user name and assumes that the user will be present in all the nodes in the cluster. It is not needed for root. To summarize: For a privileged slave user, execute the following on master node as super user: gluster system:: execute gsec_create gluster volume geo-replication <master_vol> [root@]<slave_ip>::<slave_vol> create push_pem For a non-privileged slave user execute the following on master node as super user: gluster system:: execute gsec_create gluster volume geo-replication <master_vol> <slave_user>@<slave_ip>::<slave_vol> create push_pem then on the slave node execute the following as super user: /usr/local/libexec/glusterfs/set_geo_rep_pem_keys.sh <slave_user> BUG: 1077452 Change-Id: I88020968aa5b13a2c2ab86b1d6661b60071f6f5e Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/7744 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: Partial support for Non-root geo-replication.Venky Shankar2014-05-142-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | This patch enables geo-replication to be run as an unprivileged user. As of now, this is just the partial support, but is very close to achieve full functionality. Current limitation * Geo-replication executed Gluster CLI commands on the slave via SSH. On a non-root setup, Gluster CLI would run as an unprivileged user, failing to execute the command. As a workaround (for testing), setuid(2) Gluster CLI executable or use the glusterd option to accept commands by unprivileged CLI process. The nature of cli commands are "system::" commands (for key management) and remote volume info fetching. Remote volume info fetching has been modified to use --remote-host gluster cli option rather than ssh and remote cli execution. Change-Id: Ica89e2ba9b7f48fd6e1c876c477d7822dc693617 BUG: 1077452 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/7658 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: Changelog History API changesAravinda VK2014-05-132-10/+17
| | | | | | | | | | | | | | | Additional argument added to API gf_history_changelog, actual_end - The end time till where changelogs are available. Added sort to history_get_changes API output. BUG: 1091961 Change-Id: Id043409882a83cd0a7b9adc3d34d5147d17e532e Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7747 Reviewed-by: ajeet jha <ajha@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Pause and Resume feature for geo-replicationAravinda VK2014-05-097-71/+162
| | | | | | | | | | | | | | | | Changelog consumption/processing now happens in seperate process group than monitor. When monitor process group gets SIGSTOP all worker process, ssh, rsync will be paused except the changelog processing. When it gets SIGCONT it resumes its operation. Changelog agent runs as RepceServer, geo-rep worker communicates with changelog agent using RepceClient. Change-Id: I35c333e4d8b13d03a7808aed601960eef23cfa04 BUG: 1093602 Signed-off-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7322
* geo-rep: Changelog History consumption more fixesAravinda VK2014-05-093-12/+14
| | | | | | | | | | | Number of parallel threads to process changelog history is made configurable via sync_jobs Change-Id: Idcd8e655d9df540cfa48648b9e98af941f95e9d0 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7660 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* geo-rep: Loading libgfchangelog.so only while running geo-repAravinda VK2014-05-093-5/+6
| | | | | | | | | | | | | | | | | In source install, libgfchangelog is installed in /usr/local/lib When glusterd runs /usr/local/libexec/glusterfs/python/gsyncd --version it fails to find library without LD_LIBRARY_PATH. This patch avoids loading library when it is run from glusterd during start. BUG: 1096026 Change-Id: I59912227ac27ff4877d947a7c8f1fe2e8c5be06e Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7713 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: Initialize default memory accountingVenky Shankar2014-05-061-4/+11
| | | | | | | | | | | | | | | | commit 7fba3a8 enables memory accounting by default. Since geo-replication binary (gsyncd) does not declare a memory accounting function, call like GF_{CALLOC,MALLOC} result int asserts. This patch initializes default memory accounting. Change-Id: I9ad1de8bca6745a9899a006a863a7cfeef73d52f BUG: 1094708 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/7683 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Consume Changelog History APIAravinda VK2014-04-304-31/+198
| | | | | | | | | | | | | | | | | | | | | | | | Every time when geo-rep restarts it first does FS crawl using XCrawl and then switches to Changelog Mode. This is because changelog only had live API, that is we can get changes only after registering. Now this(http://review.gluster.org/#/c/6930/) patch introduces History API for changelogs. If history is available then geo-rep will use it instead of FS Crawl. History API returns TS till what time history is available for given start and end time. If TS < endtime then switch to FS Crawl. (History => FS Crawl => Live Changelog) If TS >= endtime, then switch directly to Changelog mode (History => Live Changelog) Change-Id: I4922f62b9f899c40643bd35720e0c81c36b2f255 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/6938 Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: fix the code bug introduced due to flake8 refactoringAravinda VK2014-04-081-6/+9
| | | | | | | | | | | Sorry for the bug, which got introduced due to code refactoring. http://review.gluster.org/#/c/7311/ Change-Id: Ide519ca114aa8a7d7624d7af99945c857f069069 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7417 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: code pep8/flake8 fixesAravinda VK2014-04-0712-447/+912
| | | | | | | | | | | | | | | | | | | | | | | | | | | pep8 is a style guide for python. http://legacy.python.org/dev/peps/pep-0008/ pep8 can be installed using, `pip install pep8` Usage: `pep8 <python file>`, For example, `pep8 master.py` will display all the coding standard errors. flake8 is used to identify unused imports and other issues in code. pip install flake8 cd $GLUSTER_REPO/geo-replication/ flake8 syncdaemon Updated license headers to each source file. Change-Id: I01c7d0a6091d21bfa48720e9fb5624b77fa3db4a Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7311 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Prashanth Pai <ppai@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: Fix ValueError - signal only works in main threadAravinda VK2014-03-201-8/+7
| | | | | | | | | | | | | | | | | | | | When a worker process not confirmed within 60 seconds of start then monitor thread was terminated instead of stopping and restarting the worker thread. Before terminate monitor thread tries to add a signal handler for SIGTERM to cleanup the stuff before terminate. Signal handling will not work inside thread, so ValueError was raised. This patch will not terminate monitor thread, instead only kills and restarts the worker. Change-Id: I14df26c0cc3097af29293c81536c13b86075e28f BUG: 1078068 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/7294 Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: quick-fix for remote xtime set failedKotresh H R2014-03-071-1/+1
| | | | | | | | | | | | | | | | Remote xtime is required for failover/failback, this patch is quick fix to avoid the OSError. Code is masked out, this need to be resolved when failover/failback is worked on. Change-Id: If339d88a2ccd8ef18a3b3c015df765c93dcb020c BUG: 1073844 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7206 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Config file upgradeAravinda VK2014-02-102-1/+55
| | | | | | | | | | | | | | | | | | | When old config file is used with new geo-rep, config item like 'georep_session_working_dir' was missing in old config file. With this patch geo-rep sets the default value for new items. Following config options supported: - georep_session_working_dir - gluster_params - ssh_command_tar BUG: 1036539 Change-Id: I389c62e749f3b567f9ecf96d4b41367ef962c025 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/6934 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* gverify: Correcting variable namesAvra Sengupta2014-02-071-2/+2
| | | | | | | | | | Change-Id: I851a50e0ebb7554b821fb591375bb6ebd6240ade BUG: 1036539 Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/6939 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-replication: gverify should use disk_size and available_sizeHarshavardhana2014-02-061-14/+27
| | | | | | | | | | | | | | | "commit ef4bb42382a5c9c3ca0907136669a659c2a20311" was incomplete since reduction in sizes doesn't really have any affect on the problem when disks sizes are similar. Actual fix is to verify disk_size and then verify available_size Change-Id: Idb57f606df229c0bdaeda8852c74e84d29109ffc BUG: 1020154 Signed-off-by: Harshavardhana <harsha@harshavardhana.net> Reviewed-on: http://review.gluster.org/6844 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: invoke changelog process() on non-empty change listVenky Shankar2014-02-061-1/+2
| | | | | | | | | Change-Id: Ida4890abdc90d683a4a83127a1573bbb3829ea23 BUG: 1036539 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/6793 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* gsyncd / geo-rep: ignore DHTs sticky bit file during crawlVenky Shankar2014-02-062-0/+25
| | | | | | | | | | Change-Id: Ide927759c6a3d5301475eac9f6e785aa901d426e BUG: 1036539 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/6792 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* gsyncd / geo-rep: "patch" up missing stimeVenky Shankar2014-02-062-0/+5
| | | | | | | | | | | | | In cases (mostly upgrade) of unavailability of "stime" key and availability of "xtime" (slave's xtime), introduce "stime" key on the fly by setting it to the value to "xtime". Change-Id: Iaa424662d838154c8abc2cf00830c7f9d6be45ac BUG: 1036539 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/6791 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* gsyncd / geo-rep: cleanup the "tar" processVenky Shankar2014-02-061-0/+4
| | | | | | | | | | | | A missing cleanup for the "tar" process (when tar+ssh is used as the sync engine). Change-Id: Ib9599b43e7ec606c70b7c5598793417142be3c0b BUG: 1036539 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/6794 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: get new config value after config value resetAravinda VK2014-01-231-0/+1
| | | | | | | | | | | | | | | | | | | | When config.read is called it preserves the previous values from the the previously opened config file. For example import ConfigParser config = ConfigParser.RawConfigParser() config.read("defaults.conf") config.read("preferences.conf") When change in config file is identified it will open new instance of config to avoid getting old config values. Change-Id: Iec677e61ebd2c59c95aea94481f569d78bd913e4 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/6747 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: Fix no error when slave size less than master sizeAravinda VK2014-01-231-2/+2
| | | | | | | | | | | | | | | | in df command output, default block-size is 1K, but BUFFER_SIZE is in bytes, so verify will succeed even if slave size is less than master size. Added -B1 to df command(that is --block-size=1) to get all values in bytes. BUG: 1056518 Change-Id: I2384d2a6f2505d967e0538b95fbf111c15b26f30 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/6746 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* geo-rep: optimizing update stime after directory synchronizationAravinda VK2014-01-221-6/+14
| | | | | | | | | | | | | | | | | | | | | Since xsync crawl generates new changelog when number of entries reaches 8K or when directory is reached. If a directory has number of files less than 8K then respective changelog file will have less entries. Since xsync generated changelog files processed one after the other, so syncjobs are underutilized. hence low bandwidth utilization. With this patch, changelog will be generated for 8K entries only, but stime will be accumulated. Multiple dirs stime will be updated together since the generated changelog will have entries accross the dirs. Change-Id: Ib0b40962a070f855f47f887d0840e412fb7928e1 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/6744 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: geo-replication fixesAjeet Jha2013-12-126-269/+769
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | -> "threaded" hybrid crawl. -> Enabling metatadata synchronization. -> Handling EINVAL/ESTALE gracefully while syncing metadata. -> Improvments to changelog crawl code. -> Initial crawl changelog generation format. -> No gsyncd restart when checkpoint updated. -> Fix symlink handling in hybrid crawl. -> Slave's xtime key is 'stime'. -> tar+ssh as data synchronization. -> Instead of 'raise', just log in warning level for xtime missing cases. -> Fix for JSON object load failure -> Get new config value after config value reset. -> Skip already processed changelogs. -> Saving status of each individual worker thread. -> GFID fetch on slave for purges. -> Add tar ssh keys and config options. -> Fix nlink count when using backend. -> Include "data" operation for hardlink. -> Use changelog time prefix as slave's time. -> Process changelogs in parallel. Change-Id: I09fcbb2e2e418149a6d8435abd2ac6b2f015bb06 BUG: 1036539 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6404 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* geo-rep: Increasing the buffer_size for master-salve size comparision.Ajeet Jha2013-10-241-1/+2
| | | | | | | | | | | | The buffer size is increased to 100MB, considering the space required by slave filesystem bookkeeping. Change-Id: Ib296f0e03b4d7a2de6d5ff0ae9ab09a5261e3e7c BUG: 1020154 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6102 Reviewed-by: Avra Sengupta <asengupt@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: logrotate: Logrotate handlingAravinda VK2013-10-032-15/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In existing georep logrotate was implemented by handling SIGSTOP and SIGCONT, gsyncd was failing to start again after SIGSTOP. New approach uses WatchedFileHandler in logging, which tracks the log file changes or logrotate. Reopens the log file if logrotate is triggered or if same log file is updated from other process. As per python doc: http://docs.python.org/2/library/logging.handlers.html: The WatchedFileHandler class, located in the logging.handlers module, is a FileHandler which watches the file it is logging to. If the file changes, it is closed and reopened using the file name. A file change can happen because of usage of programs such as newsyslog and logrotate which perform log file rotation. This handler, intended for use under Unix/Linux, watches the file to see if it has changed since the last emit. (A file is deemed to have changed if its device or inode have changed.) If the file has changed, the old file stream is closed, and the file opened to get a new stream. Change-Id: I30f65eb1e9778b12943d6e43b60a50344a7885c6 BUG: 1012776 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/5968 Reviewed-by: Amar Tumballi <amarts@redhat.com> Reviewed-by: Harshavardhana <harsha@harshavardhana.net> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* geo-rep: pack the value of 'readlink()' while crawling symlinksAmar Tumballi2013-09-201-1/+4
| | | | | | | | | | | without having 'readlink()' value, symlink fails on slave mounts Change-Id: Ib84be8db77c033029ba8ba454fd629a496fc3fe1 Signed-off-by: Amar Tumballi <amarts@redhat.com> BUG: 847839 Reviewed-on: http://review.gluster.org/5950 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-replication: treat MKNOD also as regular file createAmar Tumballi2013-09-201-1/+1
| | | | | | | | | | Change-Id: Iec04f642282b554a4d1b5f5c8cdc099fd001b3f4 Original-Author: Venky Shankar <vshankar@redhat.com> Signed-off-by: Amar Tumballi <amarts@redhat.com> BUG: 847839 Reviewed-on: http://review.gluster.org/5949 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: create hardlink as hardlinks in xsync crawlAmar Tumballi2013-09-201-4/+9
| | | | | | | | | BUG: 847839 Change-Id: Ieaa754f15611392c09afcc7190b9ff9da39d7fce Signed-off-by: Amar Tumballi <amarts@redhat.com> Reviewed-on: http://review.gluster.org/5934 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* geo-rep: retry in case of ENOENT errors in entry creationsAmar Tumballi2013-09-172-6/+13
| | | | | | | | | | Change-Id: I8961633a7371c941a3feee44c949d5c934eca998 Original-Author: Venky Shankar <vshankar@redhat.com> Signed-off-by: Amar Tumballi <amarts@redhat.com> BUG: 847839 Reviewed-on: http://review.gluster.org/5933 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* gsyncd / geo-rep: use the correct virtual xattr to collect 'gfid'Venky Shankar2013-09-051-1/+1
| | | | | | | | | Change-Id: Ifea32ad1b2695f1668428fae6b263014bf320b70 BUG: 996379 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/5589 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Amar Tumballi <amarts@redhat.com>
* gsyncd / geo-rep: distributify slaveVenky Shankar2013-09-041-14/+1
| | | | | | | | | | | | | | | | | commit fbb8fd92 introduced slave distributification but had some problems (monitor would crash upon gsyncd start). This patch fixes the issue and makes code more pythonic ;) Change-Id: I2cbf5669d81966046a4aeeb4a6ad11a947aa8f09 BUG: 1003807 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Tested-by: Amar Tumballi <amarts@redhat.com> Reviewed-on: http://review.gluster.org/5761 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Avra Sengupta <asengupt@redhat.com> Tested-by: Avra Sengupta <asengupt@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* gsyncd / geo-rep: Fix "regular file" overloading link()Venky Shankar2013-09-041-0/+1
| | | | | | | | | | | | | | | ... missing entry2pb() call before going ahead with create. Change-Id: I48de4df3f0ea6c789c9eeb10cd332ab461f3c868 BUG: 1003805 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Tested-by: Amar Tumballi <amarts@redhat.com> Reviewed-on: http://review.gluster.org/5760 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Avra Sengupta <asengupt@redhat.com> Tested-by: Avra Sengupta <asengupt@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* geo-replication: fix the logic of choosing the remote node to syncVenky Shankar2013-09-041-2/+11
| | | | | | | | | | | | Change-Id: Ie15636357d89e94b6bfad0e168b1fcad53508c47 BUG: 1003807 Signed-off-by: Amar Tumballi <amarts@redhat.com> Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/5759 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Avra Sengupta <asengupt@redhat.com> Tested-by: Avra Sengupta <asengupt@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* gsyncd / geo-rep: maximum number of retries per changelogVenky Shankar2013-09-041-0/+11
| | | | | | | | | | | | | | | | | | With entry retries MAX_RETRIES is the threashold for the number of times we retry a changelog. After hitting this limit, the changelog is marked as processed after warning for incompleteness in the log. Change-Id: I777b71b4cdaa63335d9c936f2d64e5c0264a0f26 BUG: 1003803 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Tested-by: Amar Tumballi <amarts@redhat.com> Reviewed-on: http://review.gluster.org/5758 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Avra Sengupta <asengupt@redhat.com> Tested-by: Avra Sengupta <asengupt@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* gsyncd / geo-rep: fix hardlink creation on slaveVenky Shankar2013-09-041-2/+3
| | | | | | | | | | | | | Change-Id: I20fbd518bf519cbf2362b97aeb8be7c3b105087a BUG: 1003805 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Tested-by: Amar Tumballi <amarts@redhat.com> Reviewed-on: http://review.gluster.org/5757 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Avra Sengupta <asengupt@redhat.com> Tested-by: Avra Sengupta <asengupt@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
generated by cgit (git 2.34.1) at 2024-11-14 17:21:43 +0000