summaryrefslogtreecommitdiffstats
path: root/cli/src/cli-cmd-volume.c
Commit message (Collapse)AuthorAgeFilesLines
* Added checks for fix-layout & migrate-data usage.Rahul2011-03-181-2/+15
| | | | | | | | Signed-off-by: Rahul <rahul@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 2547 (rebalance command usage and help mis-match) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2547
* TOP: cli, rpc/xdr related changesshishir gowda2011-03-161-0/+49
| | | | | | | | Signed-off-by: shishir gowda <shishirng@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 2516 (Implement gluster volume top command) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2516
* cli: QUOTA cli related changes.Mohammed Junaid Ahmed2011-03-161-0/+47
| | | | | | | | Signed-off-by: Junaid <junaid@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 2473 (Support for volume and directory level quota) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2473
* cli: gluster profile CLIPranith K2011-03-101-0/+48
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1965 (need a cmd to get io-stat details) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1965
* Display correct usage of rebalance command when it is wrongly entered.v3.1.3qa2Rahul2011-03-011-0/+7
| | | | | | | | Signed-off-by: Rahul <rahul@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1922 (Volume not present wrong message displayed on command line) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1922
* gluster rebalance: give option to split the commandAmar Tumballi2011-03-011-5/+15
| | | | | | | | | | | | | | | | | the 'gluster volume rebalance <VOLNAME> start' is enhanced with two more options: * 'gluster volume rebalance <VOLNAME> fix-layout start' (for fixing layout only) * 'gluster volume rebalance <VOLNAME> migrate-data start' (for migrating data only) Also the old way of running rebalance in one shot will still work fine * 'gluster volume rebalance <VOLNAME> start' Signed-off-by: Amar Tumballi <amar@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2258 (enhance gluster volume rebalance) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2258
* Gsync : Do not expose volume gsync, when geo-replication is disabledKaushik BV2011-03-011-1/+2
| | | | | | | | Signed-off-by: Kaushik BV <kaushikbv@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1570 (geosync related changes) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1570
* glusterd: separate out cli specific programs and mgmt specific programsAmar Tumballi2011-03-011-19/+19
| | | | | | | | Signed-off-by: Amar Tumballi <amar@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2333 (make glusterd more rpc friendly) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2333
* Solaris: disable cli support rebalance and replace brickshishir gowda2011-02-221-0/+8
| | | | | | | | Signed-off-by: shishir gowda <shishirng@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2440 (Fix solaris build issues seen on latest git) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2440
* Typo fixes in log and cli messages.Sachidananda2011-02-171-2/+2
| | | | | | | | Signed-off-by: Sachidananda Urs <sac@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2321 (Typo in logs and cli messages) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2321
* gsync: cli support for gsyncd.Mohammed Junaid Ahmed2011-02-101-0/+45
| | | | | | | | Signed-off-by: Junaid <junaid@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1570 (geosync related changes) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1570
* cli: Show 'commit' option in replace-brick.Gaurav2011-01-271-1/+1
| | | | | | | | Signed-off-by: Gaurav <gaurav@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2273 (replace-brick help does not show commit option) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2273
* glusterd,cli: print single error message on failurePranith K2010-12-271-57/+154
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1922 (Volume not present wrong message displayed on command line) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1922
* cli: remove duplication of cmd helpPranith K2010-12-141-134/+24
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2089 (Documentation bug in replace-brick options) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2089
* cli: remove special behavior for help commands without readlinePranith K2010-11-141-4/+0
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 2091 (Gluster CLI exits after issuing volume help) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2091
* cli: remove volume type from add-brick and remove-brick cmdsPranith K2010-11-091-2/+2
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1983 ("type" in add-brick is completely redundant) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1983
* cli: Fix memory leaksVijay Bellur2010-10-301-0/+4
| | | | | | | | Signed-off-by: Vijay Bellur <vijay@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1198 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1198
* takes either 'y' or 'yes' and 'n' or 'no' for stop volume, delete volume and ↵Mohammed Junaid2010-10-261-10/+30
| | | | | | | | | | remove-brick Signed-off-by: Mohammed Junaid <junaid@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1942 (cli: loose parsing of confirmation answer) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1942
* Add confirmation to volume delete commandshishir gowda2010-10-111-0/+10
| | | | | | | | Signed-off-by: shishir gowda <shishirng@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 971 (dynamic volume management) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=971
* mgmt/Glusterd: new command volume reset <volname>, volume set enhancementsv3.1.0qa39Kaushik BV2010-10-031-0/+45
| | | | | | | | | | | | - Write the reconfigured options in 'info' file to make it persistant - Implementation of volume set <volname> history - Implementation of volume reset <volname> Signed-off-by: Kaushik BV <kaushikbv@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1159 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1159
* mgmt/glusterd: volume start forcePranith K2010-09-291-11/+22
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1736 (implement volume start force) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1736
* mgmt/glusterd: Misc fixes to pump / cli / glusterd wrt replace brick.v3.1.0qa30v3.1.0betaPavan Sondur2010-09-231-1/+1
| | | | | | | | | | Patches from Vijay and Shishir have been pulled in into this one big patch. Signed-off-by: Pavan Vilas Sondur <pavan@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1235 (Bug for all pump/migrate commits) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1235
* cli: Make volume rename hiddenVijay Bellur2010-09-221-2/+2
| | | | | | | | Signed-off-by: Vijay Bellur <vijay@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 971 (dynamic volume management) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=971
* cli: Avoid exposing volume set-transportVijay Bellur2010-09-221-2/+2
| | | | | | | | Signed-off-by: Vijay Bellur <vijay@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 971 (dynamic volume management) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=971
* mgmt/glusterd: replace-brick validationsPranith Kumar K2010-09-221-1/+1
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1657 (validations for replace-brick while stage op) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1657
* cli, mgmt/glusterd: volume sync commandPranith Kumar K2010-09-201-0/+50
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1310 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1310
* Glusterd: gluster volume set <volume> <option> <value>Kaushik BV2010-09-181-2/+34
| | | | | | | | Signed-off-by: Kaushik BV <kaushikbv@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1159 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1159
* cli: add a simplistic option parser, convert "mode script" to "--mode=script"Csaba Henk2010-09-161-1/+1
| | | | | | | | Signed-off-by: Csaba Henk <csaba@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1570 (geosync related changes) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1570
* cli: fixed a minor log rotate failure msgAmar Tumballi2010-09-071-1/+1
| | | | | | | | Signed-off-by: Amar Tumballi <amar@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1550 (volume log rotate failed.) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1550
* Change cli volume/peer cmd's --help option to helpshishir gowda2010-09-071-1/+1
| | | | | | | | Signed-off-by: shishir gowda <shishirng@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1529 (need man pages for 'gluster' and 'glusterd') URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1529
* cli: add script mode for glusterPranith Kumar K2010-09-071-18/+32
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1554 (add script mode for gluster) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1554
* cli: change volume create help stringPranith Kumar K2010-09-061-1/+1
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1542 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1542
* cli: prompt user before remove-brick and stop volumePranith Kumar K2010-09-061-0/+26
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1538 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1538
* cli: stripe/replica count should be greater than 1Pranith Kumar K2010-09-061-6/+0
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1534 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1534
* cli: print volume info cmd usage in case of parse errorsPranith Kumar K2010-09-061-0/+9
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1533 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1533
* Create volume adding transport type option rdmashishir gowda2010-09-061-1/+2
| | | | | | | | | | | | Adding transport type option to cli volume create command. This is optional, and defaults to TCP. The other transport supported is rdma Signed-off-by: shishir gowda <shishirng@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1507 (need to add 'transport <type>' options to create brick) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1507
* cli: Add support for GET volumeVijay Bellur2010-09-051-0/+10
| | | | | | | | Signed-off-by: Vijay Bellur <vijay@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 971 (dynamic volume management) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=971
* Hiding add-brick/remove-brick type optionshishir gowda2010-09-041-2/+2
| | | | | | | | | | | | Currently add-brick and remove-brick type option of replica/stripe along with count does not change the behaviour. Hiding the usage options. Signed-off-by: shishir gowda <shishirng@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1528 (Hide type option for add-brick and remove brick) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1528
* cli: Changes for GET_NEXT volume infoVijay Bellur2010-09-021-5/+22
| | | | | | | | Signed-off-by: Vijay Bellur <vijay@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1255 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1255
* 'gluster volume log' feature addedAmar Tumballi2010-08-311-0/+145
| | | | | | | | | * 'gluster volume log filename <VOLNAME> [BRICK] <path>' * 'gluster volume log locate <VOLNAME> [BRICK]' * 'gluster volume log rotate <VOLUME> [BRICK]' Signed-off-by: Amar Tumballi <amar@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com>
* cli, mgmt/glusterd: added volume stop <VOLNAME> force functionalityPranith Kumar K2010-08-281-9/+23
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1361 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1361
* cli: command parse validationPranith Kumar K2010-08-271-32/+51
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Vijay Bellur <vijay@dev.gluster.com> BUG: 1431 (cli: check if arguments exist before the accessing them) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1431
* cli: print usage message when parsing of cmd failsPranith Kumar K2010-08-251-10/+43
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1432 (print usage string in case of command parse errors) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1432
* Help Added to the Gluster-CLI for all sub commands (some deletions in the ↵Kaushik BV2010-08-251-0/+24
| | | | | | | | | | old patch) Signed-off-by: Kaushik BV <kaushikbv@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1156 (gluster --help should display all the options) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1156
* cli: print volume create command usage in-case of parse errorsPranith Kumar K2010-08-251-1/+8
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1263 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1263
* cli: validating if the arguments exist before accessing in volume createPranith Kumar K2010-08-251-3/+7
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1202 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1202
* Modify the help message for volume createRaghavendra Bhat2010-08-181-1/+1
| | | | | | | | Signed-off-by: Raghavendra Bhat <raghavendrabhat@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1229 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1229
* changed GNU General Public License to GNU Affero General Public LicensePranith Kumar K2010-08-171-3/+3
| | | | | | | | Signed-off-by: Pranith Kumar K <pranithk@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1388 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1388
* cli: Implement help commandVijay Bellur2010-08-171-16/+31
| | | | | | | | Signed-off-by: Vijay Bellur <vijay@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1229 () URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1229
* 'gluster volume rebalance' related fixesAmar Tumballi2010-08-111-13/+41
| | | | | | | | Signed-off-by: Amar Tumballi <amar@gluster.com> Signed-off-by: Anand V. Avati <avati@dev.gluster.com> BUG: 1307 (gluster volume defrag <VOLNAME> status) URL: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1307
generated by cgit (git 2.34.1) at 2026-01-26 11:52:41 +0000
iv class='ctx'> */
@@ -34,10 +25,10 @@ auth_null_request_init (rpcsvc_request_t *req, void *priv)
if (!req)
return -1;
- memset (req->cred.authdata, 0, RPCSVC_MAX_AUTH_BYTES);
+ memset (req->cred.authdata, 0, GF_MAX_AUTH_BYTES);
req->cred.datalen = 0;
- memset (req->verf.authdata, 0, RPCSVC_MAX_AUTH_BYTES);
+ memset (req->verf.authdata, 0, GF_MAX_AUTH_BYTES);
req->verf.datalen = 0;
return 0;
diff --git a/rpc/rpc-lib/src/auth-unix.c b/rpc/rpc-lib/src/auth-unix.c
index 30b395bd..6251d60a 100644
--- a/rpc/rpc-lib/src/auth-unix.c
+++ b/rpc/rpc-lib/src/auth-unix.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
@@ -35,7 +26,7 @@ auth_unix_request_init (rpcsvc_request_t *req, void *priv)
{
if (!req)
return -1;
- memset (req->verf.authdata, 0, RPCSVC_MAX_AUTH_BYTES);
+ memset (req->verf.authdata, 0, GF_MAX_AUTH_BYTES);
req->verf.datalen = 0;
req->verf.flavour = AUTH_NULL;
@@ -54,6 +45,7 @@ int auth_unix_authenticate (rpcsvc_request_t *req, void *priv)
ret = xdr_to_auth_unix_cred (req->cred.authdata, req->cred.datalen,
&aup, machname, req->auxgids);
if (ret == -1) {
+ gf_log ("", GF_LOG_WARNING, "failed to decode unix credentials");
ret = RPCSVC_AUTH_REJECT;
goto err;
}
diff --git a/rpc/rpc-lib/src/protocol-common.h b/rpc/rpc-lib/src/protocol-common.h
index 40283bf4..704b1540 100644
--- a/rpc/rpc-lib/src/protocol-common.h
+++ b/rpc/rpc-lib/src/protocol-common.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2007-2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _PROTOCOL_COMMON_H
@@ -64,6 +55,9 @@ enum gf_fop_procnum {
GFS3_OP_READDIRP,
GFS3_OP_RELEASE,
GFS3_OP_RELEASEDIR,
+ GFS3_OP_FREMOVEXATTR,
+ GFS3_OP_FALLOCATE,
+ GFS3_OP_DISCARD,
GFS3_OP_MAXVALUE,
} ;
@@ -72,71 +66,11 @@ enum gf_handshake_procnum {
GF_HNDSK_SETVOLUME,
GF_HNDSK_GETSPEC,
GF_HNDSK_PING,
+ GF_HNDSK_SET_LK_VER,
+ GF_HNDSK_EVENT_NOTIFY,
GF_HNDSK_MAXVALUE,
};
-enum gf_mgmt_procnum_ {
- GD_MGMT_NULL, /* 0 */
- GD_MGMT_PROBE_QUERY,
- GD_MGMT_FRIEND_ADD,
- GD_MGMT_CLUSTER_LOCK,
- GD_MGMT_CLUSTER_UNLOCK,
- GD_MGMT_STAGE_OP,
- GD_MGMT_COMMIT_OP,
- GD_MGMT_FRIEND_REMOVE,
- GD_MGMT_FRIEND_UPDATE,
- GD_MGMT_CLI_PROBE,
- GD_MGMT_CLI_DEPROBE,
- GD_MGMT_CLI_LIST_FRIENDS,
- GD_MGMT_CLI_CREATE_VOLUME,
- GD_MGMT_CLI_GET_VOLUME,
- GD_MGMT_CLI_DELETE_VOLUME,
- GD_MGMT_CLI_START_VOLUME,
- GD_MGMT_CLI_STOP_VOLUME,
- GD_MGMT_CLI_RENAME_VOLUME,
- GD_MGMT_CLI_DEFRAG_VOLUME,
- GD_MGMT_CLI_SET_VOLUME,
- GD_MGMT_CLI_ADD_BRICK,
- GD_MGMT_CLI_REMOVE_BRICK,
- GD_MGMT_CLI_REPLACE_BRICK,
- GD_MGMT_CLI_LOG_FILENAME,
- GD_MGMT_CLI_LOG_LOCATE,
- GD_MGMT_CLI_LOG_ROTATE,
- GD_MGMT_CLI_SYNC_VOLUME,
- GD_MGMT_CLI_RESET_VOLUME,
- GD_MGMT_MAXVALUE,
-};
-
-typedef enum gf_mgmt_procnum_ gf_mgmt_procnum;
-
-enum gf_cli_procnum {
- GF1_CLI_NULL = GD_MGMT_MAXVALUE+1, /* 0 */
- GF1_CLI_PROBE,
- GF1_CLI_DEPROBE,
- GF1_CLI_LIST_FRIENDS,
- GF1_CLI_CREATE_VOLUME,
- GF1_CLI_GET_VOLUME,
- GF1_CLI_GET_NEXT_VOLUME,
- GF1_CLI_DELETE_VOLUME,
- GF1_CLI_START_VOLUME,
- GF1_CLI_STOP_VOLUME,
- GF1_CLI_RENAME_VOLUME,
- GF1_CLI_DEFRAG_VOLUME,
- GF1_CLI_SET_VOLUME,
- GF1_CLI_ADD_BRICK,
- GF1_CLI_REMOVE_BRICK,
- GF1_CLI_REPLACE_BRICK,
- GF1_CLI_LOG_FILENAME,
- GF1_CLI_LOG_LOCATE,
- GF1_CLI_LOG_ROTATE,
- GF1_CLI_GETSPEC,
- GF1_CLI_PMAP_PORTBYBRICK,
- GF1_CLI_SYNC_VOLUME,
- GF1_CLI_RESET_VOLUME,
- GF1_CLI_MAXVALUE,
-};
-
-
enum gf_pmap_procnum {
GF_PMAP_NULL = 0,
GF_PMAP_PORTBYBRICK,
@@ -161,37 +95,133 @@ enum gf_probe_resp {
GF_PROBE_LOCALHOST,
GF_PROBE_FRIEND,
GF_PROBE_ANOTHER_CLUSTER,
- GF_PROBE_VOLUME_CONFLICT
+ GF_PROBE_VOLUME_CONFLICT,
+ GF_PROBE_SAME_UUID,
+ GF_PROBE_UNKNOWN_PEER,
+ GF_PROBE_ADD_FAILED,
+ GF_PROBE_QUORUM_NOT_MET
};
enum gf_deprobe_resp {
GF_DEPROBE_SUCCESS,
GF_DEPROBE_LOCALHOST,
GF_DEPROBE_NOT_FRIEND,
- GF_DEPROBE_BRICK_EXIST
+ GF_DEPROBE_BRICK_EXIST,
+ GF_DEPROBE_FRIEND_DOWN,
+ GF_DEPROBE_QUORUM_NOT_MET,
};
enum gf_cbk_procnum {
GF_CBK_NULL = 0,
GF_CBK_FETCHSPEC,
GF_CBK_INO_FLUSH,
+ GF_CBK_EVENT_NOTIFY,
GF_CBK_MAXVALUE,
};
-#define GLUSTER3_1_FOP_PROGRAM 1298437 /* Completely random */
-#define GLUSTER3_1_FOP_VERSION 310 /* 3.1.0 */
-#define GLUSTER3_1_FOP_PROCCNT GFS3_OP_MAXVALUE
+enum gluster_cli_procnum {
+ GLUSTER_CLI_NULL, /* 0 */
+ GLUSTER_CLI_PROBE,
+ GLUSTER_CLI_DEPROBE,
+ GLUSTER_CLI_LIST_FRIENDS,
+ GLUSTER_CLI_CREATE_VOLUME,
+ GLUSTER_CLI_GET_VOLUME,
+ GLUSTER_CLI_GET_NEXT_VOLUME,
+ GLUSTER_CLI_DELETE_VOLUME,
+ GLUSTER_CLI_START_VOLUME,
+ GLUSTER_CLI_STOP_VOLUME,
+ GLUSTER_CLI_RENAME_VOLUME,
+ GLUSTER_CLI_DEFRAG_VOLUME,
+ GLUSTER_CLI_SET_VOLUME,
+ GLUSTER_CLI_ADD_BRICK,
+ GLUSTER_CLI_REMOVE_BRICK,
+ GLUSTER_CLI_REPLACE_BRICK,
+ GLUSTER_CLI_LOG_ROTATE,
+ GLUSTER_CLI_GETSPEC,
+ GLUSTER_CLI_PMAP_PORTBYBRICK,
+ GLUSTER_CLI_SYNC_VOLUME,
+ GLUSTER_CLI_RESET_VOLUME,
+ GLUSTER_CLI_FSM_LOG,
+ GLUSTER_CLI_GSYNC_SET,
+ GLUSTER_CLI_PROFILE_VOLUME,
+ GLUSTER_CLI_QUOTA,
+ GLUSTER_CLI_TOP_VOLUME,
+ GLUSTER_CLI_GETWD,
+ GLUSTER_CLI_STATUS_VOLUME,
+ GLUSTER_CLI_STATUS_ALL,
+ GLUSTER_CLI_MOUNT,
+ GLUSTER_CLI_UMOUNT,
+ GLUSTER_CLI_HEAL_VOLUME,
+ GLUSTER_CLI_STATEDUMP_VOLUME,
+ GLUSTER_CLI_LIST_VOLUME,
+ GLUSTER_CLI_CLRLOCKS_VOLUME,
+ GLUSTER_CLI_UUID_RESET,
+ GLUSTER_CLI_BD_OP,
+ GLUSTER_CLI_UUID_GET,
+ GLUSTER_CLI_COPY_FILE,
+ GLUSTER_CLI_SYS_EXEC,
+ GLUSTER_CLI_MAXVALUE,
+};
-#define GLUSTERD1_MGMT_PROGRAM 1298433 /* Completely random */
-#define GLUSTERD1_MGMT_VERSION 1 /* 0.0.1 */
-#define GLUSTERD1_MGMT_PROCCNT GD_MGMT_MAXVALUE
+enum glusterd_mgmt_procnum {
+ GLUSTERD_MGMT_NULL, /* 0 */
+ GLUSTERD_MGMT_CLUSTER_LOCK,
+ GLUSTERD_MGMT_CLUSTER_UNLOCK,
+ GLUSTERD_MGMT_STAGE_OP,
+ GLUSTERD_MGMT_COMMIT_OP,
+ GLUSTERD_MGMT_MAXVALUE,
+};
-#define GLUSTER3_1_CLI_PROGRAM 1298433 /* Completely random */
-#define GLUSTER3_1_CLI_VERSION 1 /* 0.0.1 */
-#define GLUSTER3_1_CLI_PROCCNT GF1_CLI_MAXVALUE
+enum glusterd_friend_procnum {
+ GLUSTERD_FRIEND_NULL, /* 0 */
+ GLUSTERD_PROBE_QUERY,
+ GLUSTERD_FRIEND_ADD,
+ GLUSTERD_FRIEND_REMOVE,
+ GLUSTERD_FRIEND_UPDATE,
+ GLUSTERD_FRIEND_MAXVALUE,
+};
+
+enum glusterd_brick_procnum {
+ GLUSTERD_BRICK_NULL, /* 0 */
+ GLUSTERD_BRICK_TERMINATE,
+ GLUSTERD_BRICK_XLATOR_INFO,
+ GLUSTERD_BRICK_XLATOR_OP,
+ GLUSTERD_BRICK_STATUS,
+ GLUSTERD_BRICK_OP,
+ GLUSTERD_BRICK_XLATOR_DEFRAG,
+ GLUSTERD_NODE_PROFILE,
+ GLUSTERD_NODE_STATUS,
+ GLUSTERD_BRICK_BD_OP,
+ GLUSTERD_BRICK_MAXVALUE,
+};
+
+enum glusterd_mgmt_hndsk_procnum {
+ GD_MGMT_HNDSK_NULL,
+ GD_MGMT_HNDSK_VERSIONS,
+ GD_MGMT_HNDSK_VERSIONS_ACK,
+ GD_MGMT_HNDSK_MAXVALUE,
+};
+
+typedef enum {
+ GF_AFR_OP_INVALID,
+ GF_AFR_OP_HEAL_INDEX,
+ GF_AFR_OP_HEAL_FULL,
+ GF_AFR_OP_INDEX_SUMMARY,
+ GF_AFR_OP_HEALED_FILES,
+ GF_AFR_OP_HEAL_FAILED_FILES,
+ GF_AFR_OP_SPLIT_BRAIN_FILES
+} gf_xl_afr_op_t ;
+
+typedef enum {
+ GF_BD_OP_INVALID,
+ GF_BD_OP_NEW_BD,
+ GF_BD_OP_DELETE_BD,
+ GF_BD_OP_CLONE_BD,
+ GF_BD_OP_SNAPSHOT_BD,
+} gf_xl_bd_op_t ;
#define GLUSTER_HNDSK_PROGRAM 14398633 /* Completely random */
-#define GLUSTER_HNDSK_VERSION 1 /* 0.0.1 */
+#define GLUSTER_HNDSK_VERSION 2 /* 0.0.2 */
#define GLUSTER_PMAP_PROGRAM 34123456
#define GLUSTER_PMAP_VERSION 1
@@ -199,4 +229,25 @@ enum gf_cbk_procnum {
#define GLUSTER_CBK_PROGRAM 52743234 /* Completely random */
#define GLUSTER_CBK_VERSION 1 /* 0.0.1 */
+#define GLUSTER_FOP_PROGRAM 1298437 /* Completely random */
+#define GLUSTER_FOP_VERSION 330 /* 3.3.0 */
+#define GLUSTER_FOP_PROCCNT GFS3_OP_MAXVALUE
+
+/* Second version */
+#define GD_MGMT_PROGRAM 1238433 /* Completely random */
+#define GD_MGMT_VERSION 2 /* 0.0.2 */
+
+#define GD_FRIEND_PROGRAM 1238437 /* Completely random */
+#define GD_FRIEND_VERSION 2 /* 0.0.2 */
+
+#define GLUSTER_CLI_PROGRAM 1238463 /* Completely random */
+#define GLUSTER_CLI_VERSION 2 /* 0.0.2 */
+
+#define GD_BRICK_PROGRAM 4867634 /*Completely random*/
+#define GD_BRICK_VERSION 2
+
+/* OP-VERSION handshake */
+#define GD_MGMT_HNDSK_PROGRAM 1239873 /* Completely random */
+#define GD_MGMT_HNDSK_VERSION 1
+
#endif /* !_PROTOCOL_COMMON_H */
diff --git a/rpc/rpc-lib/src/rpc-clnt.c b/rpc/rpc-lib/src/rpc-clnt.c
index 61bc5263..b1d004aa 100644
--- a/rpc/rpc-lib/src/rpc-clnt.c
+++ b/rpc/rpc-lib/src/rpc-clnt.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
@@ -23,7 +14,7 @@
#include "config.h"
#endif
-#define RPC_CLNT_DEFAULT_REQUEST_COUNT 4096
+#define RPC_CLNT_DEFAULT_REQUEST_COUNT 512
#include "rpc-clnt.h"
#include "byte-order.h"
@@ -32,6 +23,7 @@
#include "protocol-common.h"
#include "mem-pool.h"
#include "xdr-rpc.h"
+#include "rpc-common-xdr.h"
void
rpc_clnt_reply_deinit (struct rpc_req *req, struct mem_pool *pool);
@@ -69,6 +61,21 @@ __saved_frames_get_timedout (struct saved_frames *frames, uint32_t timeout,
return bailout_frame;
}
+static int
+_is_lock_fop (struct saved_frame *sframe)
+{
+ int fop = 0;
+
+ if (SFRAME_GET_PROGNUM (sframe) == GLUSTER_FOP_PROGRAM &&
+ SFRAME_GET_PROGVER (sframe) == GLUSTER_FOP_VERSION)
+ fop = SFRAME_GET_PROCNUM (sframe);
+
+ return ((fop == GFS3_OP_LK) ||
+ (fop == GFS3_OP_INODELK) ||
+ (fop == GFS3_OP_FINODELK) ||
+ (fop == GFS3_OP_ENTRYLK) ||
+ (fop == GFS3_OP_FENTRYLK));
+}
struct saved_frame *
__saved_frames_put (struct saved_frames *frames, void *frame,
@@ -78,7 +85,6 @@ __saved_frames_put (struct saved_frames *frames, void *frame,
saved_frame = mem_get (rpcreq->conn->rpc_clnt->saved_frames_pool);
if (!saved_frame) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "out of memory");
goto out;
}
/* THIS should be saved and set back */
@@ -91,7 +97,11 @@ __saved_frames_put (struct saved_frames *frames, void *frame,
saved_frame->rpcreq = rpcreq;
gettimeofday (&saved_frame->saved_at, NULL);
- list_add_tail (&saved_frame->list, &frames->sf.list);
+ if (_is_lock_fop (saved_frame))
+ list_add_tail (&saved_frame->list, &frames->lk_sf.list);
+ else
+ list_add_tail (&saved_frame->list, &frames->sf.list);
+
frames->count++;
out:
@@ -103,9 +113,8 @@ void
saved_frames_delete (struct saved_frame *saved_frame,
rpc_clnt_connection_t *conn)
{
- if (!saved_frame || !conn) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("rpc-clnt", saved_frame, out);
+ GF_VALIDATE_OR_GOTO ("rpc-clnt", conn, out);
pthread_mutex_lock (&conn->lock);
{
@@ -119,7 +128,7 @@ saved_frames_delete (struct saved_frame *saved_frame,
conn->rpc_clnt->reqpool);
}
- mem_put (conn->rpc_clnt->saved_frames_pool, saved_frame);
+ mem_put (saved_frame);
out:
return;
}
@@ -135,7 +144,6 @@ call_bail (void *data)
struct saved_frame *saved_frame = NULL;
struct saved_frame *trav = NULL;
struct saved_frame *tmp = NULL;
- struct tm frame_sent_tm;
char frame_sent[256] = {0,};
struct timeval timeout = {0,};
struct iovec iov = {0,};
@@ -164,7 +172,7 @@ call_bail (void *data)
(void *) clnt);
if (conn->timer == NULL) {
- gf_log (conn->trans->name, GF_LOG_DEBUG,
+ gf_log (conn->trans->name, GF_LOG_WARNING,
"Cannot create bailout timer");
}
}
@@ -182,14 +190,14 @@ call_bail (void *data)
pthread_mutex_unlock (&conn->lock);
list_for_each_entry_safe (trav, tmp, &list, list) {
- localtime_r (&trav->saved_at.tv_sec, &frame_sent_tm);
- strftime (frame_sent, 32, "%Y-%m-%d %H:%M:%S", &frame_sent_tm);
+ gf_time_fmt (frame_sent, sizeof frame_sent,
+ trav->saved_at.tv_sec, gf_timefmt_FT);
snprintf (frame_sent + strlen (frame_sent),
256 - strlen (frame_sent),
".%"GF_PRI_SUSECONDS, trav->saved_at.tv_usec);
gf_log (conn->trans->name, GF_LOG_ERROR,
- "bailing out frame type(%s) op(%s(%d)) xid = 0x%lx "
+ "bailing out frame type(%s) op(%s(%d)) xid = 0x%x "
"sent = %s. timeout = %d",
trav->rpcreq->prog->progname,
(trav->rpcreq->prog->procnames) ?
@@ -198,12 +206,14 @@ call_bail (void *data)
trav->rpcreq->procnum, trav->rpcreq->xid, frame_sent,
conn->frame_timeout);
+ clnt = rpc_clnt_ref (clnt);
trav->rpcreq->rpc_status = -1;
trav->rpcreq->cbkfn (trav->rpcreq, &iov, 1, trav->frame);
rpc_clnt_reply_deinit (trav->rpcreq, clnt->reqpool);
+ clnt = rpc_clnt_unref (clnt);
list_del_init (&trav->list);
- mem_put (conn->rpc_clnt->saved_frames_pool, trav);
+ mem_put (trav);
}
out:
return;
@@ -250,11 +260,11 @@ saved_frames_new (void)
saved_frames = GF_CALLOC (1, sizeof (*saved_frames),
gf_common_mt_rpcclnt_savedframe_t);
if (!saved_frames) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "out of memory");
return NULL;
}
INIT_LIST_HEAD (&saved_frames->sf.list);
+ INIT_LIST_HEAD (&saved_frames->lk_sf.list);
return saved_frames;
}
@@ -276,7 +286,15 @@ __saved_frame_copy (struct saved_frames *frames, int64_t callid,
if (tmp->rpcreq->xid == callid) {
*saved_frame = *tmp;
ret = 0;
- break;
+ goto out;
+ }
+ }
+
+ list_for_each_entry (tmp, &frames->lk_sf.list, list) {
+ if (tmp->rpcreq->xid == callid) {
+ *saved_frame = *tmp;
+ ret = 0;
+ goto out;
}
}
@@ -296,10 +314,20 @@ __saved_frame_get (struct saved_frames *frames, int64_t callid)
list_del_init (&tmp->list);
frames->count--;
saved_frame = tmp;
- break;
+ goto out;
}
}
+ list_for_each_entry (tmp, &frames->lk_sf.list, list) {
+ if (tmp->rpcreq->xid == callid) {
+ list_del_init (&tmp->list);
+ frames->count--;
+ saved_frame = tmp;
+ goto out;
+ }
+ }
+
+out:
if (saved_frame) {
THIS = saved_frame->capital_this;
}
@@ -307,20 +335,20 @@ __saved_frame_get (struct saved_frames *frames, int64_t callid)
return saved_frame;
}
+
void
saved_frames_unwind (struct saved_frames *saved_frames)
{
struct saved_frame *trav = NULL;
struct saved_frame *tmp = NULL;
- struct tm *frame_sent_tm = NULL;
- char timestr[256] = {0,};
-
+ char timestr[1024] = {0,};
struct iovec iov = {0,};
+ list_splice_init (&saved_frames->lk_sf.list, &saved_frames->sf.list);
+
list_for_each_entry_safe (trav, tmp, &saved_frames->sf.list, list) {
- frame_sent_tm = localtime (&trav->saved_at.tv_sec);
- strftime (timestr, sizeof(timestr), "%Y-%m-%d %H:%M:%S",
- frame_sent_tm);
+ gf_time_fmt (timestr, sizeof timestr,
+ trav->saved_at.tv_sec, gf_timefmt_FT);
snprintf (timestr + strlen (timestr),
sizeof(timestr) - strlen (timestr),
".%"GF_PRI_SUSECONDS, trav->saved_at.tv_usec);
@@ -328,23 +356,26 @@ saved_frames_unwind (struct saved_frames *saved_frames)
if (!trav->rpcreq || !trav->rpcreq->prog)
continue;
- gf_log ("rpc-clnt", GF_LOG_ERROR,
- "forced unwinding frame type(%s) op(%s(%d)) "
- "called at %s",
- trav->rpcreq->prog->progname,
- (trav->rpcreq->prog->procnames) ?
- trav->rpcreq->prog->procnames[trav->rpcreq->procnum]
- : "--",
- trav->rpcreq->procnum, timestr);
+ gf_log_callingfn (trav->rpcreq->conn->trans->name,
+ GF_LOG_ERROR,
+ "forced unwinding frame type(%s) op(%s(%d)) "
+ "called at %s (xid=0x%x)",
+ trav->rpcreq->prog->progname,
+ ((trav->rpcreq->prog->procnames) ?
+ trav->rpcreq->prog->procnames[trav->rpcreq->procnum]
+ : "--"),
+ trav->rpcreq->procnum, timestr,
+ trav->rpcreq->xid);
saved_frames->count--;
trav->rpcreq->rpc_status = -1;
trav->rpcreq->cbkfn (trav->rpcreq, &iov, 1, trav->frame);
+
rpc_clnt_reply_deinit (trav->rpcreq,
trav->rpcreq->conn->rpc_clnt->reqpool);
list_del_init (&trav->list);
- mem_put (trav->rpcreq->conn->rpc_clnt->saved_frames_pool, trav);
+ mem_put (trav);
}
}
@@ -389,8 +420,8 @@ rpc_clnt_reconnect (void *trans_ptr)
gf_log (trans->name, GF_LOG_TRACE,
"attempting reconnect");
- ret = rpc_transport_connect (trans, conn->config.remote_port);
-
+ ret = rpc_transport_connect (trans,
+ conn->config.remote_port);
conn->reconnect =
gf_timer_call_after (clnt->ctx, tv,
rpc_clnt_reconnect,
@@ -413,7 +444,7 @@ rpc_clnt_reconnect (void *trans_ptr)
int
rpc_clnt_fill_request_info (struct rpc_clnt *clnt, rpc_request_info_t *info)
{
- struct saved_frame saved_frame = {{}, 0};
+ struct saved_frame saved_frame;
int ret = -1;
pthread_mutex_lock (&clnt->conn.lock);
@@ -424,10 +455,9 @@ rpc_clnt_fill_request_info (struct rpc_clnt *clnt, rpc_request_info_t *info)
pthread_mutex_unlock (&clnt->conn.lock);
if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_CRITICAL, "cannot lookup the saved "
- "frame corresponding to xid (%d) for msg arrived on "
- "transport %s",
- info->xid, clnt->conn.trans->name);
+ gf_log (clnt->conn.trans->name, GF_LOG_CRITICAL,
+ "cannot lookup the saved "
+ "frame corresponding to xid (%d)", info->xid);
goto out;
}
@@ -485,7 +515,7 @@ rpc_clnt_connection_cleanup (rpc_clnt_connection_t *conn)
clnt = conn->rpc_clnt;
- gf_log ("rpc-clnt", GF_LOG_DEBUG,
+ gf_log (conn->trans->name, GF_LOG_TRACE,
"cleaning up state in transport object %p", conn->trans);
pthread_mutex_lock (&conn->lock);
@@ -500,6 +530,12 @@ rpc_clnt_connection_cleanup (rpc_clnt_connection_t *conn)
}
conn->connected = 0;
+
+ if (conn->ping_timer) {
+ gf_timer_call_cancel (clnt->ctx, conn->ping_timer);
+ conn->ping_timer = NULL;
+ conn->ping_started = 0;
+ }
}
pthread_mutex_unlock (&conn->lock);
@@ -590,7 +626,7 @@ rpc_clnt_reply_deinit (struct rpc_req *req, struct mem_pool *pool)
iobref_unref (req->rsp_iobref);
}
- mem_put (pool, req);
+ mem_put (req);
out:
return;
}
@@ -613,7 +649,8 @@ rpc_clnt_reply_init (rpc_clnt_connection_t *conn, rpc_transport_pollin_t *msg,
ret = xdr_to_rpc_reply (msgbuf, msglen, &rpcmsg, &progmsg,
req->verf.authdata);
if (ret != 0) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "RPC reply decoding failed");
+ gf_log (conn->trans->name, GF_LOG_WARNING,
+ "RPC reply decoding failed");
goto out;
}
@@ -623,35 +660,13 @@ rpc_clnt_reply_init (rpc_clnt_connection_t *conn, rpc_transport_pollin_t *msg,
goto out;
}
- gf_log ("rpc-clnt", GF_LOG_TRACE, "recieved rpc message (RPC XID: 0x%lx"
+ gf_log (conn->trans->name, GF_LOG_TRACE,
+ "received rpc message (RPC XID: 0x%x"
" Program: %s, ProgVers: %d, Proc: %d) from rpc-transport (%s)",
saved_frame->rpcreq->xid,
saved_frame->rpcreq->prog->progname,
saved_frame->rpcreq->prog->progver,
saved_frame->rpcreq->procnum, conn->trans->name);
-/* TODO: */
- /* TODO: AUTH */
- /* The verifier that is sent in a reply is a string that can be used as
- * a shorthand in credentials for future transactions. We can opt not to
- * use this shorthand, preffering to use the original AUTH_UNIX method
- * for authentication (containing all the details for authentication in
- * credential itself). Hence it is not mandatory for us to be checking
- * the verifier. See Appendix A of rfc-5531 for more details.
- */
-
- /*
- * ret = rpc_authenticate (req);
- * if (ret == RPC_AUTH_REJECT) {
- * gf_log ("rpc-clnt", GF_LOG_ERROR, "Failed authentication");
- * ret = -1;
- * goto out;
- * }
- */
-
- /* If the error is not RPC_MISMATCH, we consider the call as accepted
- * since we are not handling authentication failures for now.
- */
- req->rpc_status = 0;
out:
if (ret != 0) {
@@ -679,11 +694,13 @@ rpc_clnt_handle_cbk (struct rpc_clnt *clnt, rpc_transport_pollin_t *msg)
clnt = rpc_clnt_ref (clnt);
ret = xdr_to_rpc_call (msgbuf, msglen, &rpcmsg, &progmsg, NULL,NULL);
if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "RPC call decoding failed");
+ gf_log (clnt->conn.trans->name, GF_LOG_WARNING,
+ "RPC call decoding failed");
goto out;
}
- gf_log ("rpc-clnt", GF_LOG_INFO, "recieved rpc message (XID: 0x%lx, "
+ gf_log (clnt->conn.trans->name, GF_LOG_TRACE,
+ "received rpc message (XID: 0x%lx, "
"Ver: %ld, Program: %ld, ProgVers: %ld, Proc: %ld) "
"from rpc-transport (%s)", rpc_call_xid (&rpcmsg),
rpc_call_rpcvers (&rpcmsg), rpc_call_program (&rpcmsg),
@@ -692,16 +709,23 @@ rpc_clnt_handle_cbk (struct rpc_clnt *clnt, rpc_transport_pollin_t *msg)
procnum = rpc_call_progproc (&rpcmsg);
- list_for_each_entry (program, &clnt->programs, program) {
- if ((program->prognum == rpc_call_program (&rpcmsg))
- && (program->progver == rpc_call_progver (&rpcmsg))) {
- found = 1;
- break;
+ pthread_mutex_lock (&clnt->lock);
+ {
+ list_for_each_entry (program, &clnt->programs, program) {
+ if ((program->prognum == rpc_call_program (&rpcmsg))
+ && (program->progver
+ == rpc_call_progver (&rpcmsg))) {
+ found = 1;
+ break;
+ }
}
}
+ pthread_mutex_unlock (&clnt->lock);
+
if (found && (procnum < program->numactors) &&
(program->actors[procnum].actor)) {
- program->actors[procnum].actor (&progmsg);
+ program->actors[procnum].actor (clnt, program->mydata,
+ &progmsg);
}
out:
@@ -724,23 +748,24 @@ rpc_clnt_handle_reply (struct rpc_clnt *clnt, rpc_transport_pollin_t *pollin)
xid = ntoh32 (*((uint32_t *)pollin->vector[0].iov_base));
saved_frame = lookup_frame (conn, xid);
if (saved_frame == NULL) {
- gf_log ("rpc-clnt", GF_LOG_CRITICAL, "cannot lookup the "
- "saved frame for reply with xid (%d)", xid);
+ gf_log (conn->trans->name, GF_LOG_ERROR,
+ "cannot lookup the saved frame for reply with xid (%u)",
+ xid);
goto out;
}
req = saved_frame->rpcreq;
if (req == NULL) {
- gf_log ("rpc-clnt", GF_LOG_CRITICAL,
- "saved_frame for reply with xid (%d)", xid);
+ gf_log (conn->trans->name, GF_LOG_ERROR,
+ "no request with frame for xid (%u)", xid);
goto out;
}
ret = rpc_clnt_reply_init (conn, pollin, req, saved_frame);
if (ret != 0) {
req->rpc_status = -1;
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "initialising rpc reply "
- "failed");
+ gf_log (conn->trans->name, GF_LOG_WARNING,
+ "initialising rpc reply failed");
}
req->cbkfn (req, req->rsp, req->rspcnt, saved_frame->frame);
@@ -751,7 +776,7 @@ rpc_clnt_handle_reply (struct rpc_clnt *clnt, rpc_transport_pollin_t *pollin)
out:
if (saved_frame) {
- mem_put (conn->rpc_clnt->saved_frames_pool, saved_frame);
+ mem_put (saved_frame);
}
clnt = rpc_clnt_unref (clnt);
@@ -794,17 +819,19 @@ out:
return;
}
+static void
+rpc_clnt_destroy (struct rpc_clnt *rpc);
int
rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
rpc_transport_event_t event, void *data, ...)
{
- rpc_clnt_connection_t *conn = NULL;
- struct rpc_clnt *clnt = NULL;
- int ret = -1;
- rpc_request_info_t *req_info = NULL;
- rpc_transport_pollin_t *pollin = NULL;
- struct timeval tv = {0, };
+ rpc_clnt_connection_t *conn = NULL;
+ struct rpc_clnt *clnt = NULL;
+ int ret = -1;
+ rpc_request_info_t *req_info = NULL;
+ rpc_transport_pollin_t *pollin = NULL;
+ struct timeval tv = {0, };
conn = mydata;
if (conn == NULL) {
@@ -817,11 +844,12 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
switch (event) {
case RPC_TRANSPORT_DISCONNECT:
{
- rpc_clnt_connection_cleanup (&clnt->conn);
+ rpc_clnt_connection_cleanup (conn);
pthread_mutex_lock (&conn->lock);
{
- if (conn->reconnect == NULL) {
+ if (!conn->rpc_clnt->disabled
+ && (conn->reconnect == NULL)) {
tv.tv_sec = 10;
conn->reconnect =
@@ -833,15 +861,13 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
pthread_mutex_unlock (&conn->lock);
if (clnt->notifyfn)
- ret = clnt->notifyfn (clnt, clnt->mydata, RPC_CLNT_DISCONNECT,
- NULL);
+ ret = clnt->notifyfn (clnt, clnt->mydata,
+ RPC_CLNT_DISCONNECT, NULL);
break;
}
case RPC_TRANSPORT_CLEANUP:
- /* this event should not be received on a client for, a
- * transport is only disconnected, but never destroyed.
- */
+ rpc_clnt_destroy (clnt);
ret = 0;
break;
@@ -854,6 +880,12 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
case RPC_TRANSPORT_MSG_RECEIVED:
{
+ pthread_mutex_lock (&conn->lock);
+ {
+ gettimeofday (&conn->last_received, NULL);
+ }
+ pthread_mutex_unlock (&conn->lock);
+
pollin = data;
if (pollin->is_reply)
ret = rpc_clnt_handle_reply (clnt, pollin);
@@ -879,8 +911,17 @@ rpc_clnt_notify (rpc_transport_t *trans, void *mydata,
case RPC_TRANSPORT_CONNECT:
{
+ /* Every time there is a disconnection, processes
+ should try to connect to 'glusterd' (ie, default
+ port) or whichever port given as 'option remote-port'
+ in volume file. */
+ /* Below code makes sure the (re-)configured port lasts
+ for just one successful attempt */
+ conn->config.remote_port = 0;
+
if (clnt->notifyfn)
- ret = clnt->notifyfn (clnt, clnt->mydata, RPC_CLNT_CONNECT, NULL);
+ ret = clnt->notifyfn (clnt, clnt->mydata,
+ RPC_CLNT_CONNECT, NULL);
break;
}
@@ -904,7 +945,7 @@ rpc_clnt_connection_deinit (rpc_clnt_connection_t *conn)
}
-inline int
+static inline int
rpc_clnt_connection_init (struct rpc_clnt *clnt, glusterfs_ctx_t *ctx,
dict_t *options, char *name)
{
@@ -917,7 +958,7 @@ rpc_clnt_connection_init (struct rpc_clnt *clnt, glusterfs_ctx_t *ctx,
ret = dict_get_int32 (options, "frame-timeout",
&conn->frame_timeout);
if (ret >= 0) {
- gf_log (name, GF_LOG_DEBUG,
+ gf_log (name, GF_LOG_INFO,
"setting frame-timeout to %d", conn->frame_timeout);
} else {
gf_log (name, GF_LOG_DEBUG,
@@ -927,8 +968,9 @@ rpc_clnt_connection_init (struct rpc_clnt *clnt, glusterfs_ctx_t *ctx,
conn->trans = rpc_transport_load (ctx, options, name);
if (!conn->trans) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "loading of new rpc-transport"
+ gf_log (name, GF_LOG_WARNING, "loading of new rpc-transport"
" failed");
+ ret = -1;
goto out;
}
@@ -939,7 +981,7 @@ rpc_clnt_connection_init (struct rpc_clnt *clnt, glusterfs_ctx_t *ctx,
ret = rpc_transport_register_notify (conn->trans, rpc_clnt_notify,
conn);
if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "registering notify failed");
+ gf_log (name, GF_LOG_WARNING, "registering notify failed");
rpc_clnt_connection_cleanup (conn);
conn = NULL;
goto out;
@@ -947,39 +989,37 @@ rpc_clnt_connection_init (struct rpc_clnt *clnt, glusterfs_ctx_t *ctx,
conn->saved_frames = saved_frames_new ();
if (!conn->saved_frames) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "creation of saved_frames "
+ gf_log (name, GF_LOG_WARNING, "creation of saved_frames "
"failed");
rpc_clnt_connection_cleanup (conn);
goto out;
}
- rpc_clnt_reconnect (conn->trans);
-
ret = 0;
out:
return ret;
}
-
struct rpc_clnt *
-rpc_clnt_init (struct rpc_clnt_config *config, dict_t *options,
- glusterfs_ctx_t *ctx, char *name)
+rpc_clnt_new (dict_t *options, glusterfs_ctx_t *ctx, char *name,
+ uint32_t reqpool_size)
{
int ret = -1;
struct rpc_clnt *rpc = NULL;
rpc = GF_CALLOC (1, sizeof (*rpc), gf_common_mt_rpcclnt_t);
if (!rpc) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "out of memory");
goto out;
}
pthread_mutex_init (&rpc->lock, NULL);
rpc->ctx = ctx;
- rpc->reqpool = mem_pool_new (struct rpc_req,
- RPC_CLNT_DEFAULT_REQUEST_COUNT);
+ if (!reqpool_size)
+ reqpool_size = RPC_CLNT_DEFAULT_REQUEST_COUNT;
+
+ rpc->reqpool = mem_pool_new (struct rpc_req, reqpool_size);
if (rpc->reqpool == NULL) {
pthread_mutex_destroy (&rpc->lock);
GF_FREE (rpc);
@@ -988,7 +1028,7 @@ rpc_clnt_init (struct rpc_clnt_config *config, dict_t *options,
}
rpc->saved_frames_pool = mem_pool_new (struct saved_frame,
- RPC_CLNT_DEFAULT_REQUEST_COUNT);
+ reqpool_size);
if (rpc->saved_frames_pool == NULL) {
pthread_mutex_destroy (&rpc->lock);
mem_pool_destroy (rpc->reqpool);
@@ -1009,6 +1049,8 @@ rpc_clnt_init (struct rpc_clnt_config *config, dict_t *options,
goto out;
}
+ rpc->auth_null = dict_get_str_boolean (options, "auth-null", 0);
+
rpc = rpc_clnt_ref (rpc);
INIT_LIST_HEAD (&rpc->programs);
@@ -1018,6 +1060,22 @@ out:
int
+rpc_clnt_start (struct rpc_clnt *rpc)
+{
+ struct rpc_clnt_connection *conn = NULL;
+
+ if (!rpc)
+ return -1;
+
+ conn = &rpc->conn;
+
+ rpc_clnt_reconnect (conn->trans);
+
+ return 0;
+}
+
+
+int
rpc_clnt_register_notify (struct rpc_clnt *rpc, rpc_clnt_notify_t fn,
void *mydata)
{
@@ -1028,7 +1086,7 @@ rpc_clnt_register_notify (struct rpc_clnt *rpc, rpc_clnt_notify_t fn,
}
ssize_t
-xdr_serialize_glusterfs_auth (char *dest, struct auth_glusterfs_parms *au)
+xdr_serialize_glusterfs_auth (char *dest, struct auth_glusterfs_parms_v2 *au)
{
ssize_t ret = -1;
XDR xdr;
@@ -1036,10 +1094,11 @@ xdr_serialize_glusterfs_auth (char *dest, struct auth_glusterfs_parms *au)
if ((!dest) || (!au))
return -1;
- xdrmem_create (&xdr, dest, 1024,
- XDR_ENCODE);
+ xdrmem_create (&xdr, dest, GF_MAX_AUTH_BYTES, XDR_ENCODE);
- if (!xdr_auth_glusterfs_parms (&xdr, au)) {
+ if (!xdr_auth_glusterfs_parms_v2 (&xdr, au)) {
+ gf_log (THIS->name, GF_LOG_WARNING,
+ "failed to encode auth glusterfs elements");
ret = -1;
goto ret;
}
@@ -1052,8 +1111,8 @@ ret:
int
-rpc_clnt_fill_request (int prognum, int progver, int procnum, int payload,
- uint64_t xid, struct auth_glusterfs_parms *au,
+rpc_clnt_fill_request (int prognum, int progver, int procnum,
+ uint64_t xid, struct auth_glusterfs_parms_v2 *au,
struct rpc_msg *request, char *auth_data)
{
int ret = -1;
@@ -1072,19 +1131,26 @@ rpc_clnt_fill_request (int prognum, int progver, int procnum, int payload,
request->rm_call.cb_vers = progver;
request->rm_call.cb_proc = procnum;
- /* TODO: Using AUTH_GLUSTERFS for time-being. Make it modular in
- * future so it is easy to plug-in new authentication schemes.
+ /* TODO: Using AUTH_(GLUSTERFS/NULL) in a kludgy way for time-being.
+ * Make it modular in future so it is easy to plug-in new
+ * authentication schemes.
*/
- ret = xdr_serialize_glusterfs_auth (auth_data, au);
- if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "cannot encode credentials");
- goto out;
- }
-
- request->rm_call.cb_cred.oa_flavor = AUTH_GLUSTERFS;
- request->rm_call.cb_cred.oa_base = auth_data;
- request->rm_call.cb_cred.oa_length = ret;
+ if (auth_data) {
+ ret = xdr_serialize_glusterfs_auth (auth_data, au);
+ if (ret == -1) {
+ gf_log ("rpc-clnt", GF_LOG_DEBUG,
+ "cannot encode credentials");
+ goto out;
+ }
+ request->rm_call.cb_cred.oa_flavor = AUTH_GLUSTERFS_v2;
+ request->rm_call.cb_cred.oa_base = auth_data;
+ request->rm_call.cb_cred.oa_length = ret;
+ } else {
+ request->rm_call.cb_cred.oa_flavor = AUTH_NULL;
+ request->rm_call.cb_cred.oa_base = NULL;
+ request->rm_call.cb_cred.oa_length = 0;
+ }
request->rm_call.cb_verf.oa_flavor = AUTH_NONE;
request->rm_call.cb_verf.oa_base = NULL;
request->rm_call.cb_verf.oa_length = 0;
@@ -1132,51 +1198,57 @@ out:
struct iobuf *
rpc_clnt_record_build_record (struct rpc_clnt *clnt, int prognum, int progver,
- int procnum, size_t payload, uint64_t xid,
- struct auth_glusterfs_parms *au, struct iovec *recbuf)
+ int procnum, size_t hdrsize, uint64_t xid,
+ struct auth_glusterfs_parms_v2 *au,
+ struct iovec *recbuf)
{
- struct rpc_msg request = {0, };
- struct iobuf *request_iob = NULL;
- char *record = NULL;
- struct iovec recordhdr = {0, };
- size_t pagesize = 0;
- int ret = -1;
- char auth_data[RPC_CLNT_MAX_AUTH_BYTES] = {0, };
+ struct rpc_msg request = {0, };
+ struct iobuf *request_iob = NULL;
+ char *record = NULL;
+ struct iovec recordhdr = {0, };
+ size_t pagesize = 0;
+ int ret = -1;
+ size_t xdr_size = 0;
+ char auth_data[GF_MAX_AUTH_BYTES] = {0, };
if ((!clnt) || (!recbuf) || (!au)) {
goto out;
}
+ /* Fill the rpc structure and XDR it into the buffer got above. */
+ if (clnt->auth_null)
+ ret = rpc_clnt_fill_request (prognum, progver, procnum,
+ xid, NULL, &request, NULL);
+ else
+ ret = rpc_clnt_fill_request (prognum, progver, procnum,
+ xid, au, &request, auth_data);
+
+ if (ret == -1) {
+ gf_log (clnt->conn.trans->name, GF_LOG_WARNING,
+ "cannot build a rpc-request xid (%"PRIu64")", xid);
+ goto out;
+ }
+
+ xdr_size = xdr_sizeof ((xdrproc_t)xdr_callmsg, &request);
+
/* First, try to get a pointer into the buffer which the RPC
* layer can use.
*/
- request_iob = iobuf_get (clnt->ctx->iobuf_pool);
+ request_iob = iobuf_get2 (clnt->ctx->iobuf_pool, (xdr_size + hdrsize));
if (!request_iob) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "Failed to get iobuf");
goto out;
}
- pagesize = ((struct iobuf_pool *)clnt->ctx->iobuf_pool)->page_size;
+ pagesize = iobuf_pagesize (request_iob);
record = iobuf_ptr (request_iob); /* Now we have it. */
- /* Fill the rpc structure and XDR it into the buffer got above. */
- ret = rpc_clnt_fill_request (prognum, progver, procnum, payload, xid,
- au, &request, auth_data);
- if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "cannot build a rpc-request "
- "xid (%"PRIu64")", xid);
- goto out;
- }
-
recordhdr = rpc_clnt_record_build_header (record, pagesize, &request,
- payload);
-
- //GF_FREE (request.rm_call.cb_cred.oa_base);
+ hdrsize);
if (!recordhdr.iov_base) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "Failed to build record "
- " header");
+ gf_log (clnt->conn.trans->name, GF_LOG_ERROR,
+ "Failed to build record header");
iobuf_unref (request_iob);
request_iob = NULL;
recbuf->iov_base = NULL;
@@ -1193,43 +1265,50 @@ out:
struct iobuf *
rpc_clnt_record (struct rpc_clnt *clnt, call_frame_t *call_frame,
- rpc_clnt_prog_t *prog,int procnum, size_t payload_len,
+ rpc_clnt_prog_t *prog, int procnum, size_t hdrlen,
struct iovec *rpchdr, uint64_t callid)
{
- struct auth_glusterfs_parms au = {0, };
- struct iobuf *request_iob = NULL;
+ struct auth_glusterfs_parms_v2 au = {0, };
+ struct iobuf *request_iob = NULL;
+ char owner[4] = {0,};
if (!prog || !rpchdr || !call_frame) {
goto out;
}
- au.pid = call_frame->root->pid;
- au.uid = call_frame->root->uid;
- au.gid = call_frame->root->gid;
- au.ngrps = call_frame->root->ngrps;
- au.lk_owner = call_frame->root->lk_owner;
- if (!au.lk_owner)
- au.lk_owner = au.pid;
+ au.pid = call_frame->root->pid;
+ au.uid = call_frame->root->uid;
+ au.gid = call_frame->root->gid;
+ au.groups.groups_len = call_frame->root->ngrps;
+ au.lk_owner.lk_owner_len = call_frame->root->lk_owner.len;
- gf_log ("", GF_LOG_TRACE, "Auth Info: pid: %u, uid: %d"
- ", gid: %d, owner: %"PRId64,
- au.pid, au.uid, au.gid, au.lk_owner);
+ if (au.groups.groups_len)
+ au.groups.groups_val = call_frame->root->groups;
- memcpy (au.groups, call_frame->root->groups, 16);
+ if (call_frame->root->lk_owner.len)
+ au.lk_owner.lk_owner_val = call_frame->root->lk_owner.data;
+ else {
+ owner[0] = (char)(au.pid & 0xff);
+ owner[1] = (char)((au.pid >> 8) & 0xff);
+ owner[2] = (char)((au.pid >> 16) & 0xff);
+ owner[3] = (char)((au.pid >> 24) & 0xff);
- //rpc_transport_get_myname (clnt->conn.trans, myname, UNIX_PATH_MAX);
- //au.aup_machname = myname;
+ au.lk_owner.lk_owner_val = owner;
+ au.lk_owner.lk_owner_len = 4;
+ }
+
+ gf_log (clnt->conn.trans->name, GF_LOG_TRACE, "Auth Info: pid: %u, uid: %d"
+ ", gid: %d, owner: %s", au.pid, au.uid, au.gid,
+ lkowner_utoa (&call_frame->root->lk_owner));
- /* Assuming the client program would like to speak to the same versioned
- * program on server.
- */
request_iob = rpc_clnt_record_build_record (clnt, prog->prognum,
prog->progver,
- procnum, payload_len,
+ procnum, hdrlen,
callid, &au,
rpchdr);
if (!request_iob) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "cannot build rpc-record");
+ gf_log (clnt->conn.trans->name, GF_LOG_WARNING,
+ "cannot build rpc-record");
goto out;
}
@@ -1239,9 +1318,11 @@ out:
int
rpcclnt_cbk_program_register (struct rpc_clnt *clnt,
- rpcclnt_cb_program_t *program)
+ rpcclnt_cb_program_t *program, void *mydata)
{
- int ret = -1;
+ int ret = -1;
+ char already_registered = 0;
+ rpcclnt_cb_program_t *tmp = NULL;
if (!clnt)
goto out;
@@ -1249,18 +1330,52 @@ rpcclnt_cbk_program_register (struct rpc_clnt *clnt,
if (program->actors == NULL)
goto out;
- INIT_LIST_HEAD (&program->program);
+ pthread_mutex_lock (&clnt->lock);
+ {
+ list_for_each_entry (tmp, &clnt->programs, program) {
+ if ((program->prognum == tmp->prognum)
+ && (program->progver == tmp->progver)) {
+ already_registered = 1;
+ break;
+ }
+ }
+ }
+ pthread_mutex_unlock (&clnt->lock);
+
+ if (already_registered) {
+ gf_log_callingfn (clnt->conn.trans->name, GF_LOG_DEBUG,
+ "already registered");
+ ret = 0;
+ goto out;
+ }
+
+ tmp = GF_CALLOC (1, sizeof (*tmp),
+ gf_common_mt_rpcclnt_cb_program_t);
+ if (tmp == NULL) {
+ goto out;
+ }
+
+ memcpy (tmp, program, sizeof (*tmp));
+ INIT_LIST_HEAD (&tmp->program);
+
+ tmp->mydata = mydata;
- list_add_tail (&program->program, &clnt->programs);
+ pthread_mutex_lock (&clnt->lock);
+ {
+ list_add_tail (&tmp->program, &clnt->programs);
+ }
+ pthread_mutex_unlock (&clnt->lock);
ret = 0;
- gf_log ("rpc-clnt", GF_LOG_DEBUG, "New program registered: %s, Num: %d,"
- " Ver: %d", program->progname, program->prognum,
+ gf_log (clnt->conn.trans->name, GF_LOG_DEBUG,
+ "New program registered: %s, Num: %d, Ver: %d",
+ program->progname, program->prognum,
program->progver);
out:
if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "Program registration failed:"
+ gf_log (clnt->conn.trans->name, GF_LOG_ERROR,
+ "Program registration failed:"
" %s, Num: %d, Ver: %d", program->progname,
program->prognum, program->progver);
}
@@ -1292,9 +1407,14 @@ rpc_clnt_submit (struct rpc_clnt *rpc, rpc_clnt_prog_t *prog,
goto out;
}
+ conn = &rpc->conn;
+
+ if (conn->trans == NULL) {
+ goto out;
+ }
+
rpcreq = mem_get (rpc->reqpool);
if (rpcreq == NULL) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -1304,7 +1424,6 @@ rpc_clnt_submit (struct rpc_clnt *rpc, rpc_clnt_prog_t *prog,
if (!iobref) {
iobref = iobref_new ();
if (!iobref) {
- gf_log ("rpc-clnt", GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -1313,82 +1432,73 @@ rpc_clnt_submit (struct rpc_clnt *rpc, rpc_clnt_prog_t *prog,
callid = rpc_clnt_new_callid (rpc);
- conn = &rpc->conn;
-
rpcreq->prog = prog;
rpcreq->procnum = procnum;
rpcreq->conn = conn;
rpcreq->xid = callid;
rpcreq->cbkfn = cbkfn;
- pthread_mutex_lock (&conn->lock);
- {
- if (conn->connected == 0) {
- rpc_transport_connect (conn->trans,
- conn->config.remote_port);
- }
+ ret = -1;
- ret = -1;
-
- if (proghdr) {
- proglen += iov_length (proghdr, proghdrcount);
- }
+ if (proghdr) {
+ proglen += iov_length (proghdr, proghdrcount);
+ }
- if (progpayload) {
- proglen += iov_length (progpayload,
- progpayloadcount);
- }
+ request_iob = rpc_clnt_record (rpc, frame, prog,
+ procnum, proglen,
+ &rpchdr, callid);
+ if (!request_iob) {
+ gf_log (conn->trans->name, GF_LOG_WARNING,
+ "cannot build rpc-record");
+ goto out;
+ }
- request_iob = rpc_clnt_record (rpc, frame, prog,
- procnum, proglen,
- &rpchdr, callid);
- if (!request_iob) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG,
- "cannot build rpc-record");
- goto unlock;
- }
+ iobref_add (iobref, request_iob);
- iobref_add (iobref, request_iob);
+ req.msg.rpchdr = &rpchdr;
+ req.msg.rpchdrcount = 1;
+ req.msg.proghdr = proghdr;
+ req.msg.proghdrcount = proghdrcount;
+ req.msg.progpayload = progpayload;
+ req.msg.progpayloadcount = progpayloadcount;
+ req.msg.iobref = iobref;
- req.msg.rpchdr = &rpchdr;
- req.msg.rpchdrcount = 1;
- req.msg.proghdr = proghdr;
- req.msg.proghdrcount = proghdrcount;
- req.msg.progpayload = progpayload;
- req.msg.progpayloadcount = progpayloadcount;
- req.msg.iobref = iobref;
+ req.rsp.rsphdr = rsphdr;
+ req.rsp.rsphdr_count = rsphdr_count;
+ req.rsp.rsp_payload = rsp_payload;
+ req.rsp.rsp_payload_count = rsp_payload_count;
+ req.rsp.rsp_iobref = rsp_iobref;
+ req.rpc_req = rpcreq;
- req.rsp.rsphdr = rsphdr;
- req.rsp.rsphdr_count = rsphdr_count;
- req.rsp.rsp_payload = rsp_payload;
- req.rsp.rsp_payload_count = rsp_payload_count;
- req.rsp.rsp_iobref = rsp_iobref;
- req.rpc_req = rpcreq;
+ pthread_mutex_lock (&conn->lock);
+ {
+ if (conn->connected == 0) {
+ ret = rpc_transport_connect (conn->trans,
+ conn->config.remote_port);
+ }
ret = rpc_transport_submit_request (rpc->conn.trans,
&req);
if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_TRACE, "failed to "
- "submit rpc-request "
- "(XID: 0x%lx Program: %s, ProgVers: %d, "
+ gf_log (conn->trans->name, GF_LOG_WARNING,
+ "failed to submit rpc-request "
+ "(XID: 0x%x Program: %s, ProgVers: %d, "
"Proc: %d) to rpc-transport (%s)", rpcreq->xid,
rpcreq->prog->progname, rpcreq->prog->progver,
rpcreq->procnum, rpc->conn.trans->name);
}
if ((ret >= 0) && frame) {
- gettimeofday (&conn->last_sent, NULL);
/* Save the frame in queue */
__save_frame (rpc, frame, rpcreq);
gf_log ("rpc-clnt", GF_LOG_TRACE, "submitted request "
- "(XID: 0x%lx Program: %s, ProgVers: %d, "
+ "(XID: 0x%x Program: %s, ProgVers: %d, "
"Proc: %d) to rpc-transport (%s)", rpcreq->xid,
rpcreq->prog->progname, rpcreq->prog->progver,
rpcreq->procnum, rpc->conn.trans->name);
}
}
-unlock:
pthread_mutex_unlock (&conn->lock);
if (ret == -1) {
@@ -1398,7 +1508,9 @@ unlock:
ret = 0;
out:
- iobuf_unref (request_iob);
+ if (request_iob) {
+ iobuf_unref (request_iob);
+ }
if (new_iobref && iobref) {
iobref_unref (iobref);
@@ -1408,7 +1520,7 @@ out:
if (rpcreq) {
rpcreq->rpc_status = -1;
cbkfn (rpcreq, NULL, 0, frame);
- mem_put (rpc->reqpool, rpcreq);
+ mem_put (rpcreq);
}
}
return ret;
@@ -1428,6 +1540,36 @@ rpc_clnt_ref (struct rpc_clnt *rpc)
return rpc;
}
+
+static void
+rpc_clnt_trigger_destroy (struct rpc_clnt *rpc)
+{
+ if (!rpc)
+ return;
+
+ rpc_clnt_disable (rpc);
+ rpc_transport_unref (rpc->conn.trans);
+}
+
+static void
+rpc_clnt_destroy (struct rpc_clnt *rpc)
+{
+ if (!rpc)
+ return;
+
+ saved_frames_destroy (rpc->conn.saved_frames);
+ pthread_mutex_destroy (&rpc->lock);
+ pthread_mutex_destroy (&rpc->conn.lock);
+
+ /* mem-pool should be destroyed, otherwise,
+ it will cause huge memory leaks */
+ mem_pool_destroy (rpc->reqpool);
+ mem_pool_destroy (rpc->saved_frames_pool);
+
+ GF_FREE (rpc);
+ return;
+}
+
struct rpc_clnt *
rpc_clnt_unref (struct rpc_clnt *rpc)
{
@@ -1441,31 +1583,74 @@ rpc_clnt_unref (struct rpc_clnt *rpc)
}
pthread_mutex_unlock (&rpc->lock);
if (!count) {
- rpc_clnt_destroy (rpc);
+ rpc_clnt_trigger_destroy (rpc);
return NULL;
}
return rpc;
}
+
+char
+rpc_clnt_is_disabled (struct rpc_clnt *rpc)
+{
+
+ rpc_clnt_connection_t *conn = NULL;
+ char disabled = 0;
+
+ if (!rpc) {
+ goto out;
+ }
+
+ conn = &rpc->conn;
+
+ pthread_mutex_lock (&conn->lock);
+ {
+ disabled = rpc->disabled;
+ }
+ pthread_mutex_unlock (&conn->lock);
+
+out:
+ return disabled;
+}
+
void
-rpc_clnt_destroy (struct rpc_clnt *rpc)
+rpc_clnt_disable (struct rpc_clnt *rpc)
{
- if (!rpc)
- return;
+ rpc_clnt_connection_t *conn = NULL;
- rpc_transport_destroy (rpc->conn.trans);
- rpc_clnt_connection_cleanup (&rpc->conn);
- rpc_clnt_reconnect_cleanup (&rpc->conn);
- saved_frames_destroy (rpc->conn.saved_frames);
- pthread_mutex_destroy (&rpc->lock);
- pthread_mutex_destroy (&rpc->conn.lock);
+ if (!rpc) {
+ goto out;
+ }
- /* mem-pool should be destroyed, otherwise,
- it will cause huge memory leaks */
- mem_pool_destroy (rpc->reqpool);
- mem_pool_destroy (rpc->saved_frames_pool);
+ conn = &rpc->conn;
- GF_FREE (rpc);
+ pthread_mutex_lock (&conn->lock);
+ {
+ rpc->disabled = 1;
+
+ if (conn->timer) {
+ gf_timer_call_cancel (rpc->ctx, conn->timer);
+ conn->timer = NULL;
+ }
+
+ if (conn->reconnect) {
+ gf_timer_call_cancel (rpc->ctx, conn->reconnect);
+ conn->reconnect = NULL;
+ }
+ conn->connected = 0;
+
+ if (conn->ping_timer) {
+ gf_timer_call_cancel (rpc->ctx, conn->ping_timer);
+ conn->ping_timer = NULL;
+ conn->ping_started = 0;
+ }
+
+ }
+ pthread_mutex_unlock (&conn->lock);
+
+ rpc_transport_disconnect (rpc->conn.trans);
+
+out:
return;
}
@@ -1473,15 +1658,41 @@ rpc_clnt_destroy (struct rpc_clnt *rpc)
void
rpc_clnt_reconfig (struct rpc_clnt *rpc, struct rpc_clnt_config *config)
{
- if (config->rpc_timeout)
+ if (config->rpc_timeout) {
+ if (config->rpc_timeout != rpc->conn.config.rpc_timeout)
+ gf_log (rpc->conn.trans->name, GF_LOG_INFO,
+ "changing timeout to %d (from %d)",
+ config->rpc_timeout,
+ rpc->conn.config.rpc_timeout);
rpc->conn.config.rpc_timeout = config->rpc_timeout;
+ }
+
+ if (config->remote_port) {
+ if (config->remote_port != rpc->conn.config.remote_port)
+ gf_log (rpc->conn.trans->name, GF_LOG_INFO,
+ "changing port to %d (from %d)",
+ config->remote_port,
+ rpc->conn.config.remote_port);
- if (config->remote_port)
rpc->conn.config.remote_port = config->remote_port;
+ }
if (config->remote_host) {
- if (rpc->conn.config.remote_host)
+ if (rpc->conn.config.remote_host) {
+ if (strcmp (rpc->conn.config.remote_host,
+ config->remote_host))
+ gf_log (rpc->conn.trans->name, GF_LOG_INFO,
+ "changing hostname to %s (from %s)",
+ config->remote_host,
+ rpc->conn.config.remote_host);
FREE (rpc->conn.config.remote_host);
+ } else {
+ gf_log (rpc->conn.trans->name, GF_LOG_INFO,
+ "setting hostname to %s",
+ config->remote_host);
+ }
+
rpc->conn.config.remote_host = gf_strdup (config->remote_host);
}
}
+
diff --git a/rpc/rpc-lib/src/rpc-clnt.h b/rpc/rpc-lib/src/rpc-clnt.h
index f5fe17f5..584963ad 100644
--- a/rpc/rpc-lib/src/rpc-clnt.h
+++ b/rpc/rpc-lib/src/rpc-clnt.h
@@ -1,24 +1,15 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
-#ifndef _RPC_CLNT_H
-#define _RPC_CLNT_H
+#ifndef __RPC_CLNT_H
+#define __RPC_CLNT_H
#include "stack.h"
#include "rpc-transport.h"
@@ -31,8 +22,10 @@ typedef enum {
RPC_CLNT_MSG
} rpc_clnt_event_t;
-#define AUTH_GLUSTERFS 5
-#define RPC_CLNT_MAX_AUTH_BYTES 1024
+
+#define SFRAME_GET_PROGNUM(sframe) (sframe->rpcreq->prog->prognum)
+#define SFRAME_GET_PROGVER(sframe) (sframe->rpcreq->prog->progver)
+#define SFRAME_GET_PROCNUM(sframe) (sframe->rpcreq->procnum)
struct xptr_clnt;
struct rpc_req;
@@ -66,6 +59,7 @@ struct saved_frame {
struct saved_frames {
int64_t count;
struct saved_frame sf;
+ struct saved_frame lk_sf;
};
@@ -84,7 +78,7 @@ typedef struct rpc_clnt_program {
int numproc;
} rpc_clnt_prog_t;
-typedef int (*rpcclnt_cb_fn) (void *data);
+typedef int (*rpcclnt_cb_fn) (struct rpc_clnt *rpc, void *mydata, void *data);
/* The descriptor for each procedure/actor that runs
* over the RPC service.
@@ -112,15 +106,17 @@ typedef struct rpcclnt_cb_program {
/* list member to link to list of registered services with rpc_clnt */
struct list_head program;
+
+ /* Needed for passing back in cb_actor */
+ void *mydata;
} rpcclnt_cb_program_t;
-#define RPC_MAX_AUTH_BYTES 400
typedef struct rpc_auth_data {
- int flavour;
- int datalen;
- char authdata[RPC_MAX_AUTH_BYTES];
+ int flavour;
+ int datalen;
+ char authdata[GF_MAX_AUTH_BYTES];
} rpc_auth_data_t;
@@ -167,7 +163,7 @@ struct rpc_req {
void *conn_private;
};
-struct rpc_clnt {
+typedef struct rpc_clnt {
pthread_mutex_t lock;
rpc_clnt_notify_t notifyfn;
rpc_clnt_connection_t conn;
@@ -184,12 +180,15 @@ struct rpc_clnt {
glusterfs_ctx_t *ctx;
int refcount;
-};
+ int auth_null;
+ char disabled;
+} rpc_clnt_t;
-struct rpc_clnt * rpc_clnt_init (struct rpc_clnt_config *config,
- dict_t *options, glusterfs_ctx_t *ctx,
- char *name);
+struct rpc_clnt *rpc_clnt_new (dict_t *options, glusterfs_ctx_t *ctx,
+ char *name, uint32_t reqpool_size);
+
+int rpc_clnt_start (struct rpc_clnt *rpc);
int rpc_clnt_register_notify (struct rpc_clnt *rpc, rpc_clnt_notify_t fn,
void *mydata);
@@ -223,12 +222,11 @@ rpc_clnt_ref (struct rpc_clnt *rpc);
struct rpc_clnt *
rpc_clnt_unref (struct rpc_clnt *rpc);
-void rpc_clnt_destroy (struct rpc_clnt *rpc);
+int rpc_clnt_connection_cleanup (rpc_clnt_connection_t *conn);
void rpc_clnt_set_connected (rpc_clnt_connection_t *conn);
void rpc_clnt_unset_connected (rpc_clnt_connection_t *conn);
-
void rpc_clnt_reconnect (void *trans_ptr);
void rpc_clnt_reconfig (struct rpc_clnt *rpc, struct rpc_clnt_config *config);
@@ -237,6 +235,12 @@ void rpc_clnt_reconfig (struct rpc_clnt *rpc, struct rpc_clnt_config *config);
* procedure handlers.
*/
int rpcclnt_cbk_program_register (struct rpc_clnt *svc,
- rpcclnt_cb_program_t *program);
+ rpcclnt_cb_program_t *program, void *mydata);
+
+void
+rpc_clnt_disable (struct rpc_clnt *rpc);
+
+char
+rpc_clnt_is_disabled (struct rpc_clnt *rpc);
#endif /* !_RPC_CLNT_H */
diff --git a/rpc/rpc-lib/src/rpc-common.c b/rpc/rpc-lib/src/rpc-common.c
deleted file mode 100644
index 941f01a2..00000000
--- a/rpc/rpc-lib/src/rpc-common.c
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
- This file is part of GlusterFS.
-
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
-*/
-
-
-
-#include "xdr-common.h"
-
-ssize_t
-xdr_serialize_generic (struct iovec outmsg, void *res, xdrproc_t proc)
-{
- ssize_t ret = -1;
- XDR xdr;
-
- if ((!outmsg.iov_base) || (!res) || (!proc))
- return -1;
-
- xdrmem_create (&xdr, outmsg.iov_base, (unsigned int)outmsg.iov_len,
- XDR_ENCODE);
-
- if (!proc (&xdr, res)) {
- ret = -1;
- goto ret;
- }
-
- ret = xdr_encoded_length (xdr);
-
-ret:
- return ret;
-}
-
-
-ssize_t
-xdr_to_generic (struct iovec inmsg, void *args, xdrproc_t proc)
-{
- XDR xdr;
- ssize_t ret = -1;
-
- if ((!inmsg.iov_base) || (!args) || (!proc))
- return -1;
-
- xdrmem_create (&xdr, inmsg.iov_base, (unsigned int)inmsg.iov_len,
- XDR_DECODE);
-
- if (!proc (&xdr, args)) {
- ret = -1;
- goto ret;
- }
-
- ret = xdr_decoded_length (xdr);
-ret:
- return ret;
-}
-
-
-bool_t
-xdr_gf_dump_req (XDR *xdrs, gf_dump_req *objp)
-{
- if (!xdr_u_quad_t (xdrs, &objp->gfs_id))
- return FALSE;
- return TRUE;
-}
-
-bool_t
-xdr_gf_prog_detail (XDR *xdrs, gf_prog_detail *objp)
-{
- if (!xdr_string (xdrs, &objp->progname, ~0))
- return FALSE;
- if (!xdr_u_quad_t (xdrs, &objp->prognum))
- return FALSE;
- if (!xdr_u_quad_t (xdrs, &objp->progver))
- return FALSE;
- if (!xdr_pointer (xdrs, (char **)&objp->next, sizeof (gf_prog_detail), (xdrproc_t) xdr_gf_prog_detail))
- return FALSE;
- return TRUE;
-}
-
-bool_t
-xdr_gf_dump_rsp (XDR *xdrs, gf_dump_rsp *objp)
-{
- if (!xdr_u_quad_t (xdrs, &objp->gfs_id))
- return FALSE;
- if (!xdr_int (xdrs, &objp->op_ret))
- return FALSE;
- if (!xdr_int (xdrs, &objp->op_errno))
- return FALSE;
- if (!xdr_pointer (xdrs, (char **)&objp->prog, sizeof (gf_prog_detail), (xdrproc_t) xdr_gf_prog_detail))
- return FALSE;
- return TRUE;
-}
-
-
-ssize_t
-xdr_serialize_dump_rsp (struct iovec outmsg, void *rsp)
-{
- return xdr_serialize_generic (outmsg, (void *)rsp,
- (xdrproc_t)xdr_gf_dump_rsp);
-}
-
-ssize_t
-xdr_to_dump_req (struct iovec inmsg, void *args)
-{
- return xdr_to_generic (inmsg, (void *)args,
- (xdrproc_t)xdr_gf_dump_req);
-}
-
-
-ssize_t
-xdr_from_dump_req (struct iovec outmsg, void *rsp)
-{
- return xdr_serialize_generic (outmsg, (void *)rsp,
- (xdrproc_t)xdr_gf_dump_req);
-}
-
-ssize_t
-xdr_to_dump_rsp (struct iovec inmsg, void *args)
-{
- return xdr_to_generic (inmsg, (void *)args,
- (xdrproc_t)xdr_gf_dump_rsp);
-}
diff --git a/rpc/rpc-lib/src/rpc-drc.c b/rpc/rpc-lib/src/rpc-drc.c
new file mode 100644
index 00000000..66d07cfe
--- /dev/null
+++ b/rpc/rpc-lib/src/rpc-drc.c
@@ -0,0 +1,811 @@
+/*
+ Copyright (c) 2013 Red Hat, Inc. <http://www.redhat.com>
+ This file is part of GlusterFS.
+
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
+*/
+
+#ifndef _CONFIG_H
+#define _CONFIG_H
+#include "config.h"
+#endif
+
+#include "rpcsvc.h"
+#ifndef RPC_DRC_H
+#include "rpc-drc.h"
+#endif
+#include "locking.h"
+#include "hashfn.h"
+#include "common-utils.h"
+#include "statedump.h"
+#include "mem-pool.h"
+
+#include <netinet/in.h>
+#include <unistd.h>
+
+/**
+ * rpcsvc_drc_op_destroy - Destroys the cached reply
+ *
+ * @param drc - the main drc structure
+ * @param reply - the cached reply to destroy
+ * @return NULL if reply is destroyed, reply otherwise
+ */
+static drc_cached_op_t *
+rpcsvc_drc_op_destroy (rpcsvc_drc_globals_t *drc, drc_cached_op_t *reply)
+{
+ GF_ASSERT (drc);
+ GF_ASSERT (reply);
+
+ if (reply->state == DRC_OP_IN_TRANSIT)
+ return reply;
+
+ iobref_unref (reply->msg.iobref);
+ if (reply->msg.rpchdr)
+ GF_FREE (reply->msg.rpchdr);
+ if (reply->msg.proghdr)
+ GF_FREE (reply->msg.proghdr);
+ if (reply->msg.progpayload)
+ GF_FREE (reply->msg.progpayload);
+
+ list_del (&reply->global_list);
+ reply->client->op_count--;
+ drc->op_count--;
+ mem_put (reply);
+ reply = NULL;
+
+ return reply;
+}
+
+/**
+ * rpcsvc_drc_op_rb_unref - This function is used in rb tree cleanup only
+ *
+ * @param reply - the cached reply to unref
+ * @param drc - the main drc structure
+ * @return void
+ */
+static void
+rpcsvc_drc_rb_op_destroy (void *reply, void *drc)
+{
+ rpcsvc_drc_op_destroy (drc, (drc_cached_op_t *)reply);
+}
+
+/**
+ * rpcsvc_remove_drc_client - Cleanup the drc client
+ *
+ * @param client - the drc client to be removed
+ * @return void
+ */
+static void
+rpcsvc_remove_drc_client (drc_client_t *client)
+{
+ rb_destroy (client->rbtree, rpcsvc_drc_rb_op_destroy);
+ list_del (&client->client_list);
+ GF_FREE (client);
+}
+
+/**
+ * rpcsvc_client_lookup - Given a sockaddr_storage, find the client if it exists
+ *
+ * @param drc - the main drc structure
+ * @param sockaddr - the network address of the client to be looked up
+ * @return drc client if it exists, NULL otherwise
+ */
+static drc_client_t *
+rpcsvc_client_lookup (rpcsvc_drc_globals_t *drc,
+ struct sockaddr_storage *sockaddr)
+{
+ drc_client_t *client = NULL;
+
+ GF_ASSERT (drc);
+ GF_ASSERT (sockaddr);
+
+ if (list_empty (&drc->clients_head))
+ return NULL;
+
+ list_for_each_entry (client, &drc->clients_head, client_list) {
+ if (gf_sock_union_equal_addr (&client->sock_union,
+ (union gf_sock_union *)sockaddr))
+ return client;
+ }
+
+ return NULL;
+}
+
+/**
+ * drc_compare_reqs - Used by rbtree to determine if incoming req matches with
+ * an existing node(cached reply) in rbtree
+ *
+ * @param item - pointer to the incoming req
+ * @param rb_node_data - pointer to an rbtree node (cached reply)
+ * @param param - drc pointer - unused here, but used in *op_destroy
+ * @return 0 if req matches reply, else (req->xid - reply->xid)
+ */
+int
+drc_compare_reqs (const void *item, const void *rb_node_data, void *param)
+{
+ int ret = -1;
+ rpcsvc_request_t *req = NULL;
+ drc_cached_op_t *reply = NULL;
+
+ GF_ASSERT (item);
+ GF_ASSERT (rb_node_data);
+ GF_ASSERT (param);
+
+ req = (rpcsvc_request_t *)item;
+ reply = (drc_cached_op_t *)rb_node_data;
+
+ ret = req->xid - reply->xid;
+ if (ret != 0)
+ return ret;
+
+ if (req->prognum == reply->prognum &&
+ req->procnum == reply->procnum &&
+ req->progver == reply->progversion)
+ return 0;
+
+ return 1;
+}
+
+/**
+ * drc_rb_calloc - used by rbtree api to allocate memory for nodes
+ *
+ * @param allocator - the libavl_allocator structure used by rbtree
+ * @param size - not needed by this function
+ * @return pointer to new cached reply (node in rbtree)
+ */
+static void *
+drc_rb_calloc (struct libavl_allocator *allocator, size_t size)
+{
+ rpcsvc_drc_globals_t *drc = NULL;
+
+ /* get the drc pointer by simple typecast, since allocator
+ * is the first member of rpcsvc_drc_globals_t
+ */
+ drc = (rpcsvc_drc_globals_t *)allocator;
+
+ return mem_get (drc->mempool);
+}
+
+/**
+ * drc_rb_free - used by rbtree api to free a node
+ *
+ * @param a - the libavl_allocator structure used by rbtree api
+ * @param block - node that needs to be freed
+ * @return void
+ */
+static void
+drc_rb_free (struct libavl_allocator *a, void *block)
+{
+ mem_put (block);
+}
+
+/**
+ * drc_init_client_cache - initialize a drc client and its rb tree
+ *
+ * @param drc - the main drc structure
+ * @param client - the drc client to be initialized
+ * @return 0 on success, -1 on failure
+ */
+static int
+drc_init_client_cache (rpcsvc_drc_globals_t *drc, drc_client_t *client)
+{
+ GF_ASSERT (drc);
+ GF_ASSERT (client);
+
+ drc->allocator.libavl_malloc = drc_rb_calloc;
+ drc->allocator.libavl_free = drc_rb_free;
+
+ client->rbtree = rb_create (drc_compare_reqs, drc,
+ (struct libavl_allocator *)drc);
+ if (!client->rbtree) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "rb tree creation failed");
+ return -1;
+ }
+
+ return 0;
+}
+
+/**
+ * rpcsvc_get_drc_client - find the drc client with given sockaddr, else
+ * allocate and initialize a new drc client
+ *
+ * @param drc - the main drc structure
+ * @param sockaddr - network address of client
+ * @return drc client on success, NULL on failure
+ */
+static drc_client_t *
+rpcsvc_get_drc_client (rpcsvc_drc_globals_t *drc,
+ struct sockaddr_storage *sockaddr)
+{
+ drc_client_t *client = NULL;
+
+ GF_ASSERT (drc);
+ GF_ASSERT (sockaddr);
+
+ client = rpcsvc_client_lookup (drc, sockaddr);
+ if (client)
+ goto out;
+
+ /* if lookup fails, allocate cache for the new client */
+ client = GF_CALLOC (1, sizeof (drc_client_t),
+ gf_common_mt_drc_client_t);
+ if (!client)
+ goto out;
+
+ client->ref = 0;
+ client->sock_union = (union gf_sock_union)*sockaddr;
+ client->op_count = 0;
+
+ if (drc_init_client_cache (drc, client)) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG,
+ "initialization of drc client failed");
+ GF_FREE (client);
+ client = NULL;
+ goto out;
+ }
+ drc->client_count++;
+
+ list_add (&client->client_list, &drc->clients_head);
+
+ out:
+ return client;
+}
+
+/**
+ * rpcsvc_need_drc - Determine if a request needs DRC service
+ *
+ * @param req - incoming request
+ * @return 1 if DRC is needed for req, 0 otherwise
+ */
+int
+rpcsvc_need_drc (rpcsvc_request_t *req)
+{
+ rpcsvc_actor_t *actor = NULL;
+ rpcsvc_drc_globals_t *drc = NULL;
+
+ GF_ASSERT (req);
+ GF_ASSERT (req->svc);
+
+ drc = req->svc->drc;
+
+ if (!drc || drc->status == DRC_UNINITIATED)
+ return 0;
+
+ actor = rpcsvc_program_actor (req);
+ if (!actor)
+ return 0;
+
+ return (actor->op_type == DRC_NON_IDEMPOTENT
+ && drc->type != DRC_TYPE_NONE);
+}
+
+/**
+ * rpcsvc_drc_client_ref - ref the drc client
+ *
+ * @param client - the drc client to ref
+ * @return client
+ */
+static drc_client_t *
+rpcsvc_drc_client_ref (drc_client_t *client)
+{
+ GF_ASSERT (client);
+ client->ref++;
+ return client;
+}
+
+/**
+ * rpcsvc_drc_client_unref - unref the drc client, and destroy
+ * the client on last unref
+ *
+ * @param drc - the main drc structure
+ * @param client - the drc client to unref
+ * @return NULL if it is the last unref, client otherwise
+ */
+static drc_client_t *
+rpcsvc_drc_client_unref (rpcsvc_drc_globals_t *drc, drc_client_t *client)
+{
+ GF_ASSERT (drc);
+ GF_ASSERT (client->ref);
+
+ client->ref--;
+ if (!client->ref) {
+ drc->client_count--;
+ rpcsvc_remove_drc_client (client);
+ client = NULL;
+ }
+
+ return client;
+}
+
+/**
+ * rpcsvc_drc_lookup - lookup a request to see if it is already cached
+ *
+ * @param req - incoming request
+ * @return cached reply of req if found, NULL otherwise
+ */
+drc_cached_op_t *
+rpcsvc_drc_lookup (rpcsvc_request_t *req)
+{
+ drc_client_t *client = NULL;
+ drc_cached_op_t *reply = NULL;
+
+ GF_ASSERT (req);
+
+ if (!req->trans->drc_client) {
+ client = rpcsvc_get_drc_client (req->svc->drc,
+ &req->trans->peerinfo.sockaddr);
+ if (!client)
+ goto out;
+ req->trans->drc_client = client;
+ }
+
+ client = rpcsvc_drc_client_ref (req->trans->drc_client);
+
+ if (client->op_count == 0)
+ goto out;
+
+ reply = rb_find (client->rbtree, req);
+
+ out:
+ if (client)
+ rpcsvc_drc_client_unref (req->svc->drc, client);
+
+ return reply;
+}
+
+/**
+ * rpcsvc_send_cached_reply - send the cached reply for the incoming request
+ *
+ * @param req - incoming request (which is a duplicate in this case)
+ * @param reply - the cached reply for req
+ * @return 0 on successful reply submission, -1 or other non-zero value otherwise
+ */
+int
+rpcsvc_send_cached_reply (rpcsvc_request_t *req, drc_cached_op_t *reply)
+{
+ int ret = 0;
+
+ GF_ASSERT (req);
+ GF_ASSERT (reply);
+
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "sending cached reply: xid: %d, "
+ "client: %s", req->xid, req->trans->peerinfo.identifier);
+
+ rpcsvc_drc_client_ref (reply->client);
+ ret = rpcsvc_transport_submit (req->trans,
+ reply->msg.rpchdr, reply->msg.rpchdrcount,
+ reply->msg.proghdr, reply->msg.proghdrcount,
+ reply->msg.progpayload, reply->msg.progpayloadcount,
+ reply->msg.iobref, req->trans_private);
+ rpcsvc_drc_client_unref (req->svc->drc, reply->client);
+
+ return ret;
+}
+
+/**
+ * rpcsvc_cache_reply - cache the reply for the processed request 'req'
+ *
+ * @param req - processed request
+ * @param iobref - iobref structure of the reply
+ * @param rpchdr - rpc header of the reply
+ * @param rpchdrcount - size of rpchdr
+ * @param proghdr - program header of the reply
+ * @param proghdrcount - size of proghdr
+ * @param payload - payload of the reply if any
+ * @param payloadcount - size of payload
+ * @return 0 on success, -1 on failure
+ */
+int
+rpcsvc_cache_reply (rpcsvc_request_t *req, struct iobref *iobref,
+ struct iovec *rpchdr, int rpchdrcount,
+ struct iovec *proghdr, int proghdrcount,
+ struct iovec *payload, int payloadcount)
+{
+ int ret = -1;
+ drc_cached_op_t *reply = NULL;
+
+ GF_ASSERT (req);
+ GF_ASSERT (req->reply);
+
+ reply = req->reply;
+
+ reply->state = DRC_OP_CACHED;
+
+ reply->msg.iobref = iobref_ref (iobref);
+
+ reply->msg.rpchdrcount = rpchdrcount;
+ reply->msg.rpchdr = iov_dup (rpchdr, rpchdrcount);
+
+ reply->msg.proghdrcount = proghdrcount;
+ reply->msg.proghdr = iov_dup (proghdr, proghdrcount);
+
+ reply->msg.progpayloadcount = payloadcount;
+ if (payloadcount)
+ reply->msg.progpayload = iov_dup (payload, payloadcount);
+
+ // rpcsvc_drc_client_unref (req->svc->drc, req->trans->drc_client);
+ // rpcsvc_drc_op_unref (req->svc->drc, reply);
+ ret = 0;
+
+ return ret;
+}
+
+/**
+ * rpcsvc_vacate_drc_entries - free up some percentage of drc cache
+ * based on the lru factor
+ *
+ * @param drc - the main drc structure
+ * @return void
+ */
+static void
+rpcsvc_vacate_drc_entries (rpcsvc_drc_globals_t *drc)
+{
+ uint32_t i = 0;
+ uint32_t n = 0;
+ drc_cached_op_t *reply = NULL;
+ drc_cached_op_t *tmp = NULL;
+ drc_client_t *client = NULL;
+
+ GF_ASSERT (drc);
+
+ n = drc->global_cache_size / drc->lru_factor;
+
+ list_for_each_entry_safe_reverse (reply, tmp, &drc->cache_head, global_list) {
+ /* Don't delete ops that are in transit */
+ if (reply->state == DRC_OP_IN_TRANSIT)
+ continue;
+
+ client = reply->client;
+
+ (void *)rb_delete (client->rbtree, reply);
+
+ rpcsvc_drc_op_destroy (drc, reply);
+ rpcsvc_drc_client_unref (drc, client);
+ i++;
+ if (i >= n)
+ break;
+ }
+}
+
+/**
+ * rpcsvc_add_op_to_cache - insert the cached op into the client rbtree and drc list
+ *
+ * @param drc - the main drc structure
+ * @param reply - the op to be inserted
+ * @return 0 on success, -1 on failure
+ */
+static int
+rpcsvc_add_op_to_cache (rpcsvc_drc_globals_t *drc, drc_cached_op_t *reply)
+{
+ drc_client_t *client = NULL;
+ drc_cached_op_t **tmp_reply = NULL;
+
+ GF_ASSERT (drc);
+ GF_ASSERT (reply);
+
+ client = reply->client;
+
+ /* cache is full, free up some space */
+ if (drc->op_count >= drc->global_cache_size)
+ rpcsvc_vacate_drc_entries (drc);
+
+ tmp_reply = (drc_cached_op_t **)rb_probe (client->rbtree, reply);
+ if (*tmp_reply != reply) {
+ /* should never happen */
+ gf_log (GF_RPCSVC, GF_LOG_ERROR,
+ "DRC failed to detect duplicates");
+ return -1;
+ } else if (*tmp_reply == NULL) {
+ /* mem alloc failed */
+ return -1;
+ }
+
+ client->op_count++;
+ list_add (&reply->global_list, &drc->cache_head);
+ drc->op_count++;
+
+ return 0;
+}
+
+/**
+ * rpcsvc_cache_request - cache the in-transition incoming request
+ *
+ * @param req - incoming request
+ * @return 0 on success, -1 on failure
+ */
+int
+rpcsvc_cache_request (rpcsvc_request_t *req)
+{
+ int ret = -1;
+ drc_client_t *client = NULL;
+ drc_cached_op_t *reply = NULL;
+ rpcsvc_drc_globals_t *drc = NULL;
+
+ GF_ASSERT (req);
+
+ drc = req->svc->drc;
+
+ client = req->trans->drc_client;
+ if (!client) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "drc client is NULL");
+ goto out;
+ }
+
+ reply = mem_get (drc->mempool);
+ if (!reply)
+ goto out;
+
+ reply->client = rpcsvc_drc_client_ref (client);
+ reply->xid = req->xid;
+ reply->prognum = req->prognum;
+ reply->progversion = req->progver;
+ reply->procnum = req->procnum;
+ reply->state = DRC_OP_IN_TRANSIT;
+ req->reply = reply;
+
+ ret = rpcsvc_add_op_to_cache (drc, reply);
+ if (ret) {
+ req->reply = NULL;
+ rpcsvc_drc_op_destroy (drc, reply);
+ rpcsvc_drc_client_unref (drc, client);
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Failed to add op to drc cache");
+ }
+
+ out:
+ return ret;
+}
+
+/**
+ *
+ * rpcsvc_drc_priv - function which dumps the drc state
+ *
+ * @param drc - the main drc structure
+ * @return 0 on success, -1 on failure
+ */
+int32_t
+rpcsvc_drc_priv (rpcsvc_drc_globals_t *drc)
+{
+ int i = 0;
+ char key[GF_DUMP_MAX_BUF_LEN] = {0};
+ drc_client_t *client = NULL;
+ char ip[INET6_ADDRSTRLEN] = {0};
+
+ if (!drc || drc->status == DRC_UNINITIATED) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "DRC is "
+ "uninitialized, not dumping its state");
+ return 0;
+ }
+
+ gf_proc_dump_add_section("rpc.drc");
+
+ if (TRY_LOCK (&drc->lock))
+ return -1;
+
+ gf_proc_dump_build_key (key, "drc", "type");
+ gf_proc_dump_write (key, "%d", drc->type);
+
+ gf_proc_dump_build_key (key, "drc", "client_count");
+ gf_proc_dump_write (key, "%d", drc->client_count);
+
+ gf_proc_dump_build_key (key, "drc", "current_cache_size");
+ gf_proc_dump_write (key, "%d", drc->op_count);
+
+ gf_proc_dump_build_key (key, "drc", "max_cache_size");
+ gf_proc_dump_write (key, "%d", drc->global_cache_size);
+
+ gf_proc_dump_build_key (key, "drc", "lru_factor");
+ gf_proc_dump_write (key, "%d", drc->lru_factor);
+
+ gf_proc_dump_build_key (key, "drc", "duplicate_request_count");
+ gf_proc_dump_write (key, "%d", drc->cache_hits);
+
+ gf_proc_dump_build_key (key, "drc", "in_transit_duplicate_requests");
+ gf_proc_dump_write (key, "%d", drc->intransit_hits);
+
+ list_for_each_entry (client, &drc->clients_head, client_list) {
+ gf_proc_dump_build_key (key, "client", "%d.ip-address", i);
+ memset (ip, 0, INET6_ADDRSTRLEN);
+ switch (client->sock_union.storage.ss_family) {
+ case AF_INET:
+ gf_proc_dump_write (key, "%s", inet_ntop (AF_INET,
+ &client->sock_union.sin.sin_addr.s_addr,
+ ip, INET_ADDRSTRLEN));
+ break;
+ case AF_INET6:
+ gf_proc_dump_write (key, "%s", inet_ntop (AF_INET6,
+ &client->sock_union.sin6.sin6_addr,
+ ip, INET6_ADDRSTRLEN));
+ break;
+ default:
+ gf_proc_dump_write (key, "%s", "N/A");
+ }
+
+ gf_proc_dump_build_key (key, "client", "%d.ref_count", i);
+ gf_proc_dump_write (key, "%d", client->ref);
+ gf_proc_dump_build_key (key, "client", "%d.op_count", i);
+ gf_proc_dump_write (key, "%d", client->op_count);
+ i++;
+ }
+
+ UNLOCK (&drc->lock);
+ return 0;
+}
+
+/**
+ * rpcsvc_drc_notify - function which is notified of RPC transport events
+ *
+ * @param svc - pointer to rpcsvc_t structure of the rpc
+ * @param xl - pointer to the xlator
+ * @param event - the event which triggered this notify
+ * @param data - the transport structure
+ * @return 0 on success, -1 on failure
+ */
+int
+rpcsvc_drc_notify (rpcsvc_t *svc, void *xl,
+ rpcsvc_event_t event, void *data)
+{
+ int ret = -1;
+ rpc_transport_t *trans = NULL;
+ drc_client_t *client = NULL;
+ rpcsvc_drc_globals_t *drc = NULL;
+
+ GF_ASSERT (svc);
+ GF_ASSERT (svc->drc);
+ GF_ASSERT (data);
+
+ drc = svc->drc;
+
+ if (drc->status == DRC_UNINITIATED ||
+ drc->type == DRC_TYPE_NONE)
+ return 0;
+
+ LOCK (&drc->lock);
+
+ trans = (rpc_transport_t *)data;
+ client = rpcsvc_get_drc_client (drc, &trans->peerinfo.sockaddr);
+ if (!client)
+ goto out;
+
+ switch (event) {
+ case RPCSVC_EVENT_ACCEPT:
+ trans->drc_client = rpcsvc_drc_client_ref (client);
+ ret = 0;
+ break;
+
+ case RPCSVC_EVENT_DISCONNECT:
+ ret = 0;
+ if (list_empty (&drc->clients_head))
+ break;
+ /* should be the last unref */
+ rpcsvc_drc_client_unref (drc, client);
+ trans->drc_client = NULL;
+ break;
+
+ default:
+ break;
+ }
+
+ out:
+ UNLOCK (&drc->lock);
+ return ret;
+}
+
+/**
+ * rpcsvc_drc_init - Initialize the duplicate request cache service
+ *
+ * @param svc - pointer to rpcsvc_t structure of the rpc
+ * @param options - the options dictionary which configures drc
+ * @return 0 on success, non-zero integer on failure
+ */
+int
+rpcsvc_drc_init (rpcsvc_t *svc, dict_t *options)
+{
+ int ret = 0;
+ uint32_t drc_type = 0;
+ uint32_t drc_size = 0;
+ uint32_t drc_factor = 0;
+ rpcsvc_drc_globals_t *drc = NULL;
+
+ GF_ASSERT (svc);
+ GF_ASSERT (options);
+
+ if (!svc->drc) {
+ drc = GF_CALLOC (1, sizeof (rpcsvc_drc_globals_t),
+ gf_common_mt_drc_globals_t);
+ if (!drc)
+ return -1;
+
+ svc->drc = drc;
+ LOCK_INIT (&drc->lock);
+ } else {
+ drc = svc->drc;
+ }
+
+ LOCK (&drc->lock);
+ if (drc->type != DRC_TYPE_NONE) {
+ ret = 0;
+ goto out;
+ }
+
+ /* Toggle DRC on/off, when more drc types(persistent/cluster)
+ are added, we shouldn't treat this as boolean */
+ ret = dict_get_str_boolean (options, "nfs.drc", _gf_false);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_INFO, "drc user options need second look");
+ ret = _gf_true;
+ }
+
+ if (ret == _gf_false) {
+ /* drc off */
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "DRC is off");
+ ret = 0;
+ goto out;
+ }
+
+ /* Specify type of DRC to be used */
+ ret = dict_get_uint32 (options, "nfs.drc-type", &drc_type);
+ if (ret) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "drc type not set."
+ " Continuing with default");
+ drc_type = DRC_DEFAULT_TYPE;
+ }
+
+ drc->type = drc_type;
+
+ /* Set the global cache size (no. of ops to cache) */
+ ret = dict_get_uint32 (options, "nfs.drc-size", &drc_size);
+ if (ret) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "drc size not set."
+ " Continuing with default size");
+ drc_size = DRC_DEFAULT_CACHE_SIZE;
+ }
+
+ drc->global_cache_size = drc_size;
+
+ /* Mempool for cached ops */
+ drc->mempool = mem_pool_new (drc_cached_op_t, drc->global_cache_size);
+ if (!drc->mempool) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get mempool for"
+ " DRC, drc-size: %d", drc->global_cache_size);
+ ret = -1;
+ goto out;
+ }
+
+ /* What percent of cache to be evicted whenever it fills up */
+ ret = dict_get_uint32 (options, "nfs.drc-lru-factor", &drc_factor);
+ if (ret) {
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "drc lru factor not set."
+ " Continuing with policy default");
+ drc_factor = DRC_DEFAULT_LRU_FACTOR;
+ }
+
+ drc->lru_factor = (drc_lru_factor_t) drc_factor;
+
+ INIT_LIST_HEAD (&drc->clients_head);
+ INIT_LIST_HEAD (&drc->cache_head);
+
+ ret = rpcsvc_register_notify (svc, rpcsvc_drc_notify, THIS);
+ if (ret) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR,
+ "registration of drc_notify function failed");
+ goto out;
+ }
+
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "drc init successful");
+ drc->status = DRC_INITIATED;
+
+ out:
+ UNLOCK (&drc->lock);
+ if (ret == -1) {
+ if (drc->mempool) {
+ mem_pool_destroy (drc->mempool);
+ drc->mempool = NULL;
+ }
+ GF_FREE (drc);
+ svc->drc = NULL;
+ }
+ return ret;
+}
diff --git a/rpc/rpc-lib/src/rpc-drc.h b/rpc/rpc-lib/src/rpc-drc.h
new file mode 100644
index 00000000..0a168899
--- /dev/null
+++ b/rpc/rpc-lib/src/rpc-drc.h
@@ -0,0 +1,100 @@
+/*
+ Copyright (c) 2013 Red Hat, Inc. <http://www.redhat.com>
+ This file is part of GlusterFS.
+
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
+*/
+
+#ifndef RPC_DRC_H
+#define RPC_DRC_H
+
+#ifndef _CONFIG_H
+#define _CONFIG_H
+#include "config.h"
+#endif
+
+#include "rpcsvc-common.h"
+#include "rpcsvc.h"
+#include "locking.h"
+#include "dict.h"
+#include "rb.h"
+
+/* per-client cache structure */
+struct drc_client {
+ uint32_t ref;
+ union gf_sock_union sock_union;
+ /* pointers to the cache */
+ struct rb_table *rbtree;
+ /* no. of ops currently cached */
+ uint32_t op_count;
+ struct list_head client_list;
+};
+
+struct drc_cached_op {
+ drc_op_state_t state;
+ uint32_t xid;
+ int prognum;
+ int progversion;
+ int procnum;
+ rpc_transport_msg_t msg;
+ drc_client_t *client;
+ struct list_head client_list;
+ struct list_head global_list;
+ int32_t ref;
+};
+
+/* global drc definitions */
+enum drc_status {
+ DRC_UNINITIATED,
+ DRC_INITIATED
+};
+typedef enum drc_status drc_status_t;
+
+struct drc_globals {
+ /* allocator must be the first member since
+ * it is used so in gf_libavl_allocator
+ */
+ struct libavl_allocator allocator;
+ drc_type_t type;
+ /* configurable size parameter */
+ uint32_t global_cache_size;
+ drc_lru_factor_t lru_factor;
+ gf_lock_t lock;
+ drc_status_t status;
+ uint32_t op_count;
+ uint64_t cache_hits;
+ uint64_t intransit_hits;
+ struct mem_pool *mempool;
+ struct list_head cache_head;
+ uint32_t client_count;
+ struct list_head clients_head;
+};
+
+int
+rpcsvc_need_drc (rpcsvc_request_t *req);
+
+drc_cached_op_t *
+rpcsvc_drc_lookup (rpcsvc_request_t *req);
+
+int
+rpcsvc_send_cached_reply (rpcsvc_request_t *req, drc_cached_op_t *reply);
+
+int
+rpcsvc_cache_reply (rpcsvc_request_t *req, struct iobref *iobref,
+ struct iovec *rpchdr, int rpchdrcount,
+ struct iovec *proghdr, int proghdrcount,
+ struct iovec *payload, int payloadcount);
+
+int
+rpcsvc_cache_request (rpcsvc_request_t *req);
+
+int32_t
+rpcsvc_drc_priv (rpcsvc_drc_globals_t *drc);
+
+int
+rpcsvc_drc_init (rpcsvc_t *svc, dict_t *options);
+
+#endif /* RPC_DRC_H */
diff --git a/rpc/rpc-lib/src/rpc-transport.c b/rpc/rpc-lib/src/rpc-transport.c
index c69237d5..89f3b3e8 100644
--- a/rpc/rpc-lib/src/rpc-transport.c
+++ b/rpc/rpc-lib/src/rpc-transport.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#include <dlfcn.h>
@@ -42,596 +33,67 @@
#define GF_OPTION_LIST_EMPTY(_opt) (_opt->value[0] == NULL)
#endif
-/* RFC 1123 & 952 */
-static char
-valid_host_name (char *address, int length)
-{
- int i = 0;
- char ret = 1;
-
- if ((length > 75) || (length == 1)) {
- ret = 0;
- goto out;
- }
-
- if (!isalnum (address[length - 1])) {
- ret = 0;
- goto out;
- }
-
- for (i = 0; i < length; i++) {
- if (!isalnum (address[i]) && (address[i] != '.')
- && (address[i] != '-')) {
- ret = 0;
- goto out;
- }
- }
-
-out:
- return ret;
-}
-static char
-valid_ipv4_address (char *address, int length)
+int
+rpc_transport_get_myaddr (rpc_transport_t *this, char *peeraddr, int addrlen,
+ struct sockaddr_storage *sa, size_t salen)
{
- int octets = 0;
- int value = 0;
- char *tmp = NULL, *ptr = NULL, *prev = NULL, *endptr = NULL;
- char ret = 1;
-
- tmp = gf_strdup (address);
- prev = strtok_r (tmp, ".", &ptr);
-
- while (prev != NULL)
- {
- octets++;
- value = strtol (prev, &endptr, 10);
- if ((value > 255) || (value < 0) || (endptr != NULL)) {
- ret = 0;
- goto out;
- }
-
- prev = strtok_r (NULL, ".", &ptr);
- }
+ int32_t ret = -1;
+ GF_VALIDATE_OR_GOTO ("rpc", this, out);
- if (octets != 4) {
- ret = 0;
- }
+ ret = this->ops->get_myaddr (this, peeraddr, addrlen, sa, salen);
out:
- GF_FREE (tmp);
return ret;
}
-
-static char
-valid_ipv6_address (char *address, int length)
+int32_t
+rpc_transport_get_myname (rpc_transport_t *this, char *hostname, int hostlen)
{
- int hex_numbers = 0;
- int value = 0;
- char *tmp = NULL, *ptr = NULL, *prev = NULL, *endptr = NULL;
- char ret = 1;
-
- tmp = gf_strdup (address);
- prev = strtok_r (tmp, ":", &ptr);
-
- while (prev != NULL)
- {
- hex_numbers++;
- value = strtol (prev, &endptr, 16);
- if ((value > 0xffff) || (value < 0)
- || (endptr != NULL && *endptr != '\0')) {
- ret = 0;
- goto out;
- }
-
- prev = strtok_r (NULL, ":", &ptr);
- }
-
- if (hex_numbers > 8) {
- ret = 0;
- }
+ int32_t ret = -1;
+ GF_VALIDATE_OR_GOTO ("rpc", this, out);
+ ret = this->ops->get_myname (this, hostname, hostlen);
out:
- GF_FREE (tmp);
return ret;
}
-
-static char
-valid_internet_address (char *address)
+int32_t
+rpc_transport_get_peername (rpc_transport_t *this, char *hostname, int hostlen)
{
- char ret = 0;
- int length = 0;
-
- if (address == NULL) {
- goto out;
- }
-
- length = strlen (address);
- if (length == 0) {
- goto out;
- }
-
- if (valid_ipv4_address (address, length)
- || valid_ipv6_address (address, length)
- || valid_host_name (address, length)) {
- ret = 1;
- }
+ int32_t ret = -1;
+ GF_VALIDATE_OR_GOTO ("rpc", this, out);
+ ret = this->ops->get_peername (this, hostname, hostlen);
out:
return ret;
}
-
-int
-__volume_option_value_validate (char *name,
- data_pair_t *pair,
- volume_option_t *opt)
-{
- int i = 0;
- int ret = -1;
- uint64_t input_size = 0;
- long long inputll = 0;
-
- /* Key is valid, validate the option */
- switch (opt->type) {
- case GF_OPTION_TYPE_XLATOR:
- break;
-
- case GF_OPTION_TYPE_PATH:
- {
- if (strstr (pair->value->data, "../")) {
- gf_log (name, GF_LOG_ERROR,
- "invalid path given '%s'",
- pair->value->data);
- ret = -1;
- goto out;
- }
-
- /* Make sure the given path is valid */
- if (pair->value->data[0] != '/') {
- gf_log (name, GF_LOG_WARNING,
- "option %s %s: '%s' is not an "
- "absolute path name",
- pair->key, pair->value->data,
- pair->value->data);
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_INT:
- {
- /* Check the range */
- if (gf_string2longlong (pair->value->data,
- &inputll) != 0) {
- gf_log (name, GF_LOG_ERROR,
- "invalid number format \"%s\" in "
- "\"option %s\"",
- pair->value->data, pair->key);
- goto out;
- }
-
- if ((opt->min == 0) && (opt->max == 0)) {
- gf_log (name, GF_LOG_DEBUG,
- "no range check required for "
- "'option %s %s'",
- pair->key, pair->value->data);
- ret = 0;
- break;
- }
- if ((inputll < opt->min) ||
- (inputll > opt->max)) {
- gf_log (name, GF_LOG_WARNING,
- "'%lld' in 'option %s %s' is out of "
- "range [%"PRId64" - %"PRId64"]",
- inputll, pair->key,
- pair->value->data,
- opt->min, opt->max);
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_SIZET:
- {
- /* Check the range */
- if (gf_string2bytesize (pair->value->data,
- &input_size) != 0) {
- gf_log (name, GF_LOG_ERROR,
- "invalid size format \"%s\" in "
- "\"option %s\"",
- pair->value->data, pair->key);
- goto out;
- }
-
- if ((opt->min == 0) && (opt->max == 0)) {
- gf_log (name, GF_LOG_DEBUG,
- "no range check required for "
- "'option %s %s'",
- pair->key, pair->value->data);
- ret = 0;
- break;
- }
- if ((input_size < opt->min) ||
- (input_size > opt->max)) {
- gf_log (name, GF_LOG_ERROR,
- "'%"PRId64"' in 'option %s %s' is "
- "out of range [%"PRId64" - %"PRId64"]",
- input_size, pair->key,
- pair->value->data,
- opt->min, opt->max);
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_BOOL:
- {
- /* Check if the value is one of
- '0|1|on|off|no|yes|true|false|enable|disable' */
- gf_boolean_t bool_value;
- if (gf_string2boolean (pair->value->data,
- &bool_value) != 0) {
- gf_log (name, GF_LOG_ERROR,
- "option %s %s: '%s' is not a valid "
- "boolean value",
- pair->key, pair->value->data,
- pair->value->data);
- goto out;
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_STR:
- {
- /* Check if the '*str' is valid */
- if (GF_OPTION_LIST_EMPTY(opt)) {
- ret = 0;
- goto out;
- }
-
- for (i = 0; (i < ZR_OPTION_MAX_ARRAY_SIZE) &&
- opt->value[i]; i++) {
- if (strcasecmp (opt->value[i],
- pair->value->data) == 0) {
- ret = 0;
- break;
- }
- }
-
- if ((i == ZR_OPTION_MAX_ARRAY_SIZE)
- || ((i < ZR_OPTION_MAX_ARRAY_SIZE)
- && (!opt->value[i]))) {
- /* enter here only if
- * 1. reached end of opt->value array and haven't
- * validated input
- * OR
- * 2. valid input list is less than
- * ZR_OPTION_MAX_ARRAY_SIZE and input has not
- * matched all possible input values.
- */
- char given_array[4096] = {0,};
- for (i = 0; (i < ZR_OPTION_MAX_ARRAY_SIZE) &&
- opt->value[i]; i++) {
- strcat (given_array, opt->value[i]);
- strcat (given_array, ", ");
- }
-
- gf_log (name, GF_LOG_ERROR,
- "option %s %s: '%s' is not valid "
- "(possible options are %s)",
- pair->key, pair->value->data,
- pair->value->data, given_array);
-
- goto out;
- }
- }
- break;
- case GF_OPTION_TYPE_PERCENT:
- {
- uint32_t percent = 0;
-
-
- /* Check if the value is valid percentage */
- if (gf_string2percent (pair->value->data,
- &percent) != 0) {
- gf_log (name, GF_LOG_ERROR,
- "invalid percent format \"%s\" "
- "in \"option %s\"",
- pair->value->data, pair->key);
- goto out;
- }
-
- if ((percent < 0) || (percent > 100)) {
- gf_log (name, GF_LOG_ERROR,
- "'%d' in 'option %s %s' is out of "
- "range [0 - 100]",
- percent, pair->key,
- pair->value->data);
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_PERCENT_OR_SIZET:
- {
- uint32_t percent = 0;
- uint64_t input_size = 0;
-
- /* Check if the value is valid percentage */
- if (gf_string2percent (pair->value->data,
- &percent) == 0) {
- if (percent > 100) {
- gf_log (name, GF_LOG_DEBUG,
- "value given was greater than 100, "
- "assuming this is actually a size");
- if (gf_string2bytesize (pair->value->data,
- &input_size) == 0) {
- /* Check the range */
- if ((opt->min == 0) &&
- (opt->max == 0)) {
- gf_log (name, GF_LOG_DEBUG,
- "no range check "
- "required for "
- "'option %s %s'",
- pair->key,
- pair->value->data);
- // It is a size
- ret = 0;
- goto out;
- }
- if ((input_size < opt->min) ||
- (input_size > opt->max)) {
- gf_log (name, GF_LOG_ERROR,
- "'%"PRId64"' in "
- "'option %s %s' is out"
- " of range [%"PRId64""
- "- %"PRId64"]",
- input_size, pair->key,
- pair->value->data,
- opt->min, opt->max);
- }
- // It is a size
- ret = 0;
- goto out;
- } else {
- // It's not a percent or size
- gf_log (name, GF_LOG_ERROR,
- "invalid number format \"%s\" "
- "in \"option %s\"",
- pair->value->data, pair->key);
- }
-
- }
- // It is a percent
- ret = 0;
- goto out;
- } else {
- if (gf_string2bytesize (pair->value->data,
- &input_size) == 0) {
- /* Check the range */
- if ((opt->min == 0) && (opt->max == 0)) {
- gf_log (name, GF_LOG_DEBUG,
- "no range check required for "
- "'option %s %s'",
- pair->key, pair->value->data);
- // It is a size
- ret = 0;
- goto out;
- }
- if ((input_size < opt->min) ||
- (input_size > opt->max)) {
- gf_log (name, GF_LOG_ERROR,
- "'%"PRId64"' in 'option %s %s'"
- " is out of range [%"PRId64" -"
- " %"PRId64"]",
- input_size, pair->key,
- pair->value->data,
- opt->min, opt->max);
- }
- } else {
- // It's not a percent or size
- gf_log (name, GF_LOG_ERROR,
- "invalid number format \"%s\" "
- "in \"option %s\"",
- pair->value->data, pair->key);
- }
- //It is a size
- ret = 0;
- goto out;
- }
-
- }
- break;
- case GF_OPTION_TYPE_TIME:
- {
- uint32_t input_time = 0;
-
- /* Check if the value is valid percentage */
- if (gf_string2time (pair->value->data,
- &input_time) != 0) {
- gf_log (name,
- GF_LOG_ERROR,
- "invalid time format \"%s\" in "
- "\"option %s\"",
- pair->value->data, pair->key);
- goto out;
- }
-
- if ((opt->min == 0) && (opt->max == 0)) {
- gf_log (name, GF_LOG_DEBUG,
- "no range check required for "
- "'option %s %s'",
- pair->key, pair->value->data);
- ret = 0;
- goto out;
- }
- if ((input_time < opt->min) ||
- (input_time > opt->max)) {
- gf_log (name, GF_LOG_ERROR,
- "'%"PRIu32"' in 'option %s %s' is "
- "out of range [%"PRId64" - %"PRId64"]",
- input_time, pair->key,
- pair->value->data,
- opt->min, opt->max);
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_DOUBLE:
- {
- double input_time = 0.0;
-
- /* Check if the value is valid double */
- if (gf_string2double (pair->value->data,
- &input_time) != 0) {
- gf_log (name,
- GF_LOG_ERROR,
- "invalid time format \"%s\" in \"option %s\"",
- pair->value->data, pair->key);
- goto out;
- }
-
- if (input_time < 0.0) {
- gf_log (name,
- GF_LOG_ERROR,
- "invalid time format \"%s\" in \"option %s\"",
- pair->value->data, pair->key);
- goto out;
- }
-
- if ((opt->min == 0) && (opt->max == 0)) {
- gf_log (name, GF_LOG_DEBUG,
- "no range check required for 'option %s %s'",
- pair->key, pair->value->data);
- ret = 0;
- goto out;
- }
- ret = 0;
- }
- break;
- case GF_OPTION_TYPE_INTERNET_ADDRESS:
- {
- if (valid_internet_address (pair->value->data)) {
- ret = 0;
- }
- }
- break;
- case GF_OPTION_TYPE_ANY:
- /* NO CHECK */
- ret = 0;
- break;
- }
-
-out:
- return ret;
-}
-
-/* FIXME: this procedure should be removed from transport */
-int
-validate_volume_options (char *name, dict_t *options, volume_option_t *opt)
-{
- int i = 0;
- int ret = -1;
- int index = 0;
- volume_option_t *trav = NULL;
- data_pair_t *pairs = NULL;
-
- if (!opt) {
- ret = 0;
- goto out;
- }
-
- /* First search for not supported options, if any report error */
- pairs = options->members_list;
- while (pairs) {
- ret = -1;
- for (index = 0;
- opt[index].key && opt[index].key[0] ; index++) {
- trav = &(opt[index]);
- for (i = 0 ;
- (i < ZR_VOLUME_MAX_NUM_KEY) &&
- trav->key[i]; i++) {
- /* Check if the key is valid */
- if (fnmatch (trav->key[i],
- pairs->key, FNM_NOESCAPE) == 0) {
- ret = 0;
- break;
- }
- }
- if (!ret) {
- if (i) {
- gf_log (name, GF_LOG_WARNING,
- "option '%s' is deprecated, "
- "preferred is '%s', continuing"
- " with correction",
- trav->key[i], trav->key[0]);
- /* TODO: some bytes lost */
- pairs->key = gf_strdup (trav->key[0]);
- }
- break;
- }
- }
- if (!ret) {
- ret = __volume_option_value_validate (name, pairs, trav);
- if (-1 == ret) {
- goto out;
- }
- }
-
- pairs = pairs->next;
- }
-
- ret = 0;
- out:
- return ret;
-}
-
-int32_t
-rpc_transport_get_myaddr (rpc_transport_t *this, char *peeraddr, int addrlen,
- struct sockaddr_storage *sa, size_t salen)
-{
- if (!this)
- return -1;
-
- return this->ops->get_myaddr (this, peeraddr, addrlen, sa, salen);
-}
-
-int32_t
-rpc_transport_get_myname (rpc_transport_t *this, char *hostname, int hostlen)
-{
- if (!this)
- return -1;
-
- return this->ops->get_myname (this, hostname, hostlen);
-}
-
-int32_t
-rpc_transport_get_peername (rpc_transport_t *this, char *hostname, int hostlen)
-{
- if (!this)
- return -1;
- return this->ops->get_peername (this, hostname, hostlen);
-}
-
int32_t
rpc_transport_get_peeraddr (rpc_transport_t *this, char *peeraddr, int addrlen,
struct sockaddr_storage *sa, size_t salen)
{
- if (!this)
- return -1;
- return this->ops->get_peeraddr (this, peeraddr, addrlen, sa, salen);
+ int32_t ret = -1;
+ GF_VALIDATE_OR_GOTO ("rpc", this, out);
+
+ ret = this->ops->get_peeraddr (this, peeraddr, addrlen, sa, salen);
+out:
+ return ret;
}
void
rpc_transport_pollin_destroy (rpc_transport_pollin_t *pollin)
{
- if (!pollin) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("rpc", pollin, out);
if (pollin->iobref) {
iobref_unref (pollin->iobref);
}
-
+
+ if (pollin->hdr_iobuf) {
+ iobuf_unref (pollin->hdr_iobuf);
+ }
+
if (pollin->private) {
/* */
GF_FREE (pollin->private);
@@ -645,16 +107,16 @@ out:
rpc_transport_pollin_t *
rpc_transport_pollin_alloc (rpc_transport_t *this, struct iovec *vector,
- int count, struct iobref *iobref, void *private)
+ int count, struct iobuf *hdr_iobuf,
+ struct iobref *iobref, void *private)
{
rpc_transport_pollin_t *msg = NULL;
msg = GF_CALLOC (1, sizeof (*msg), gf_common_mt_rpc_trans_pollin_t);
if (!msg) {
- gf_log ("rpc-transport", GF_LOG_ERROR, "out of memory");
goto out;
}
- if (count == 2) {
+ if (count > 1) {
msg->vectored = 1;
}
@@ -662,147 +124,14 @@ rpc_transport_pollin_alloc (rpc_transport_t *this, struct iovec *vector,
msg->count = count;
msg->iobref = iobref_ref (iobref);
msg->private = private;
+ if (hdr_iobuf)
+ msg->hdr_iobuf = iobuf_ref (hdr_iobuf);
out:
return msg;
}
-rpc_transport_pollin_t *
-rpc_transport_same_process_pollin_alloc (rpc_transport_t *this,
- struct iovec *rpchdr, int rpchdrcount,
- struct iovec *proghdr,
- int proghdrcount,
- struct iovec *progpayload,
- int progpayloadcount,
- rpc_transport_rsp_t *rsp,
- char is_request)
-{
- rpc_transport_pollin_t *msg = NULL;
- int rpchdrlen = 0, proghdrlen = 0;
- int progpayloadlen = 0;
- char vectored = 0;
- char *hdr = NULL, *progpayloadbuf = NULL;
- struct iobuf *iobuf = NULL;
-
- if (!rpchdr || !proghdr) {
- goto err;
- }
-
- msg = GF_CALLOC (1, sizeof (*msg), gf_common_mt_rpc_trans_pollin_t);
- if (!msg) {
- gf_log ("rpc-transport", GF_LOG_ERROR, "out of memory");
- goto err;
- }
-
- rpchdrlen = iov_length (rpchdr, rpchdrcount);
- proghdrlen = iov_length (proghdr, proghdrcount);
-
- if (progpayload) {
- vectored = 1;
- progpayloadlen = iov_length (progpayload, progpayloadcount);
- }
-
- /* FIXME: we are assuming rpchdr and proghdr will fit into
- * an iobuf (128KB)
- */
- if ((rpchdrlen + proghdrlen) > this->ctx->page_size) {
- gf_log ("rpc_transport", GF_LOG_DEBUG, "program hdr and rpc"
- " hdr together combined (%d) is bigger than "
- "iobuf size (%zu)", (rpchdrlen + proghdrlen),
- this->ctx->page_size);
- goto err;
- }
-
- if (vectored) {
- msg->iobref = iobref_new ();
- if (!msg->iobref) {
- gf_log ("rpc-transport", GF_LOG_ERROR,
- "out of memory");
- goto err;
- }
-
- iobuf = iobuf_get (this->ctx->iobuf_pool);
- if (!iobuf) {
- gf_log ("rpc_transport", GF_LOG_ERROR,
- "out of memory");
- goto err;
- }
-
- iobref_add (msg->iobref, iobuf);
- iobuf_unref (iobuf);
-
- msg->vector[0].iov_len = rpchdrlen + proghdrlen;
- msg->vector[0].iov_base = hdr = iobuf_ptr (iobuf);
-
- if (!is_request && rsp) {
- msg->vector[1] = rsp->rsp_payload[0];
- progpayloadbuf = rsp->rsp_payload[0].iov_base;
- } else {
- iobuf = iobuf_get (this->ctx->iobuf_pool);
- if (!iobuf) {
- gf_log ("rpc_transport", GF_LOG_ERROR,
- "out of memory");
- goto err;
- }
-
- iobref_add (msg->iobref, iobuf);
- iobuf_unref (iobuf);
-
- msg->vector[1].iov_base
- = progpayloadbuf = iobuf_ptr (iobuf);
- }
- msg->vector[1].iov_len = progpayloadlen;
- } else {
- if (!is_request && rsp) {
- /* FIXME: Assuming rspvec contains only one vector */
- hdr = rsp->rsphdr[0].iov_base;
- msg->vector[0] = rsp->rsphdr[0];
- } else {
- msg->iobref = iobref_new ();
- if (!msg->iobref) {
- gf_log ("rpc-transport", GF_LOG_ERROR,
- "out of memory");
- goto err;
- }
-
- iobuf = iobuf_get (this->ctx->iobuf_pool);
- if (!iobuf) {
- gf_log ("rpc_transport", GF_LOG_ERROR,
- "out of memory");
- goto err;
- }
-
- iobref_add (msg->iobref, iobuf);
- iobuf_unref (iobuf);
-
- hdr = iobuf_ptr (iobuf);
- msg->vector[0].iov_base = hdr;
- }
-
- msg->vector[0].iov_len = rpchdrlen + proghdrlen;
- }
-
- iov_unload (hdr, rpchdr, rpchdrcount);
- hdr += rpchdrlen;
- iov_unload (hdr, proghdr, proghdrcount);
-
- if (progpayload) {
- iov_unload (progpayloadbuf, progpayload,
- progpayloadcount);
- }
-
- if (is_request) {
- msg->private = rsp;
- }
- return msg;
-err:
- if (msg) {
- rpc_transport_pollin_destroy (msg);
- }
-
- return NULL;
-}
rpc_transport_t *
rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
@@ -815,30 +144,35 @@ rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
int32_t ret = -1;
int8_t is_tcp = 0, is_unix = 0, is_ibsdp = 0;
volume_opt_list_t *vol_opt = NULL;
+ gf_boolean_t bind_insecure = _gf_false;
+ xlator_t *this = NULL;
GF_VALIDATE_OR_GOTO("rpc-transport", options, fail);
GF_VALIDATE_OR_GOTO("rpc-transport", ctx, fail);
GF_VALIDATE_OR_GOTO("rpc-transport", trans_name, fail);
trans = GF_CALLOC (1, sizeof (struct rpc_transport), gf_common_mt_rpc_trans_t);
- GF_VALIDATE_OR_GOTO("rpc-transport", trans, fail);
+ if (!trans)
+ goto fail;
trans->name = gf_strdup (trans_name);
- GF_VALIDATE_OR_GOTO ("rpc-transport", trans->name, fail);
+ if (!trans->name)
+ goto fail;
trans->ctx = ctx;
type = str;
/* Backward compatibility */
- ret = dict_get_str (options, "transport-type", &type);
+ ret = dict_get_str (options, "transport-type", &type);
if (ret < 0) {
ret = dict_set_str (options, "transport-type", "socket");
if (ret < 0)
gf_log ("dict", GF_LOG_DEBUG,
"setting transport-type failed");
- gf_log ("rpc-transport", GF_LOG_WARNING,
- "missing 'option transport-type'. defaulting to "
- "\"socket\"");
+ else
+ gf_log ("rpc-transport", GF_LOG_DEBUG,
+ "missing 'option transport-type'. defaulting to "
+ "\"socket\"");
} else {
{
/* Backword compatibility to handle * /client,
@@ -876,6 +210,28 @@ rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
}
}
+ /* client-bind-insecure is for clients protocol, and
+ * bind-insecure for glusterd. Both mutually exclusive
+ */
+ ret = dict_get_str (options, "client-bind-insecure", &type);
+ if (ret)
+ ret = dict_get_str (options, "bind-insecure", &type);
+ if (ret == 0) {
+ ret = gf_string2boolean (type, &bind_insecure);
+ if (ret < 0) {
+ gf_log ("rcp-transport", GF_LOG_WARNING,
+ "bind-insecure option %s is not a"
+ " valid bool option", type);
+ goto fail;
+ }
+ if (_gf_true == bind_insecure)
+ trans->bind_insecure = 1;
+ else
+ trans->bind_insecure = 0;
+ } else {
+ trans->bind_insecure = 0;
+ }
+
ret = dict_get_str (options, "transport-type", &type);
if (ret < 0) {
gf_log ("rpc-transport", GF_LOG_ERROR,
@@ -886,22 +242,24 @@ rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
ret = gf_asprintf (&name, "%s/%s.so", RPC_TRANSPORTDIR, type);
if (-1 == ret) {
- gf_log ("rpc-transport", GF_LOG_ERROR, "asprintf failed");
goto fail;
}
+
gf_log ("rpc-transport", GF_LOG_DEBUG,
"attempt to load file %s", name);
handle = dlopen (name, RTLD_NOW|RTLD_GLOBAL);
if (handle == NULL) {
gf_log ("rpc-transport", GF_LOG_ERROR, "%s", dlerror ());
- gf_log ("rpc-transport", GF_LOG_ERROR,
+ gf_log ("rpc-transport", GF_LOG_WARNING,
"volume '%s': transport-type '%s' is not valid or "
"not found on this machine",
trans_name, type);
goto fail;
}
+ trans->dl_handle = handle;
+
trans->ops = dlsym (handle, "tops");
if (trans->ops == NULL) {
gf_log ("rpc-transport", GF_LOG_ERROR,
@@ -909,37 +267,42 @@ rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
goto fail;
}
- trans->init = dlsym (handle, "init");
+ *VOID(&(trans->init)) = dlsym (handle, "init");
if (trans->init == NULL) {
gf_log ("rpc-transport", GF_LOG_ERROR,
"dlsym (gf_rpc_transport_init) on %s", dlerror ());
goto fail;
}
- trans->fini = dlsym (handle, "fini");
+ *VOID(&(trans->fini)) = dlsym (handle, "fini");
if (trans->fini == NULL) {
gf_log ("rpc-transport", GF_LOG_ERROR,
"dlsym (gf_rpc_transport_fini) on %s", dlerror ());
goto fail;
}
+ *VOID(&(trans->reconfigure)) = dlsym (handle, "reconfigure");
+ if (trans->fini == NULL) {
+ gf_log ("rpc-transport", GF_LOG_DEBUG,
+ "dlsym (gf_rpc_transport_reconfigure) on %s", dlerror());
+ }
+
vol_opt = GF_CALLOC (1, sizeof (volume_opt_list_t),
gf_common_mt_volume_opt_list_t);
if (!vol_opt) {
- gf_log (trans_name, GF_LOG_ERROR, "out of memory");
goto fail;
}
+ this = THIS;
vol_opt->given_opt = dlsym (handle, "options");
if (vol_opt->given_opt == NULL) {
gf_log ("rpc-transport", GF_LOG_DEBUG,
"volume option validation not specified");
} else {
- /* FIXME: is adding really needed? */
- /* list_add_tail (&vol_opt->list, &xl->volume_options); */
- if (-1 ==
- validate_volume_options (trans_name, options,
- vol_opt->given_opt)) {
+ INIT_LIST_HEAD (&vol_opt->list);
+ list_add_tail (&vol_opt->list, &(this->volume_options));
+ if (xlator_options_validate_list (this, options, vol_opt,
+ NULL)) {
gf_log ("rpc-transport", GF_LOG_ERROR,
"volume option validation failed");
goto fail;
@@ -949,7 +312,7 @@ rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
trans->options = options;
pthread_mutex_init (&trans->lock, NULL);
- trans->xl = THIS;
+ trans->xl = this;
ret = trans->init (trans);
if (ret != 0) {
@@ -958,30 +321,27 @@ rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *trans_name)
goto fail;
}
- return_trans = trans;
+ return_trans = trans;
- if (name) {
- GF_FREE (name);
- }
+ GF_FREE (name);
- GF_FREE (vol_opt);
return return_trans;
fail:
if (trans) {
- if (trans->name) {
- GF_FREE (trans->name);
- }
+ GF_FREE (trans->name);
+
+ if (trans->dl_handle)
+ dlclose (trans->dl_handle);
GF_FREE (trans);
}
- if (vol_opt) {
- GF_FREE (vol_opt);
- }
+ GF_FREE (name);
- if (name) {
- GF_FREE (name);
+ if (vol_opt && !list_empty (&vol_opt->list)) {
+ list_del_init (&vol_opt->list);
+ GF_FREE (vol_opt);
}
return NULL;
@@ -1062,13 +422,17 @@ rpc_transport_destroy (rpc_transport_t *this)
GF_VALIDATE_OR_GOTO("rpc_transport", this, fail);
+ if (this->options)
+ dict_unref (this->options);
if (this->fini)
this->fini (this);
pthread_mutex_destroy (&this->lock);
- if (this->name)
- GF_FREE (this->name);
+ GF_FREE (this->name);
+
+ if (this->dl_handle)
+ dlclose (this->dl_handle);
GF_FREE (this);
fail:
@@ -1105,14 +469,17 @@ rpc_transport_unref (rpc_transport_t *this)
pthread_mutex_lock (&this->lock);
{
- refcount = --this->refcount;
+ refcount = --this->refcount;
}
pthread_mutex_unlock (&this->lock);
if (refcount == 0) {
- /* xlator_notify (this->xl, GF_EVENT_RPC_TRANSPORT_CLEANUP,
- this); */
- rpc_transport_destroy (this);
+ if (this->mydata)
+ this->notify (this, this->mydata, RPC_TRANSPORT_CLEANUP,
+ NULL);
+ this->mydata = NULL;
+ this->notify = NULL;
+ rpc_transport_destroy (this);
}
ret = 0;
@@ -1126,10 +493,7 @@ rpc_transport_notify (rpc_transport_t *this, rpc_transport_event_t event,
void *data, ...)
{
int32_t ret = -1;
-
- if (this == NULL) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("rpc", this, out);
if (this->notify != NULL) {
ret = this->notify (this, this->mydata, event, data);
@@ -1146,11 +510,8 @@ inline int
rpc_transport_register_notify (rpc_transport_t *trans,
rpc_transport_notify_t notify, void *mydata)
{
- int ret = -1;
-
- if (trans == NULL) {
- goto out;
- }
+ int32_t ret = -1;
+ GF_VALIDATE_OR_GOTO ("rpc", trans, out);
trans->notify = notify;
trans->mydata = mydata;
@@ -1159,3 +520,147 @@ rpc_transport_register_notify (rpc_transport_t *trans,
out:
return ret;
}
+
+
+
+//give negative values to skip setting that value
+//this function asserts if both the values are negative.
+//why call it if you dont set it.
+int
+rpc_transport_keepalive_options_set (dict_t *options, int32_t interval,
+ int32_t time)
+{
+ int ret = -1;
+
+ GF_ASSERT (options);
+ GF_ASSERT ((interval > 0) || (time > 0));
+
+ ret = dict_set_int32 (options,
+ "transport.socket.keepalive-interval", interval);
+ if (ret)
+ goto out;
+
+ ret = dict_set_int32 (options,
+ "transport.socket.keepalive-time", time);
+ if (ret)
+ goto out;
+out:
+ return ret;
+}
+
+int
+rpc_transport_unix_options_build (dict_t **options, char *filepath,
+ int frame_timeout)
+{
+ dict_t *dict = NULL;
+ char *fpath = NULL;
+ int ret = -1;
+
+ GF_ASSERT (filepath);
+ GF_ASSERT (options);
+
+ dict = dict_new ();
+ if (!dict)
+ goto out;
+
+ fpath = gf_strdup (filepath);
+ if (!fpath) {
+ ret = -1;
+ goto out;
+ }
+
+ ret = dict_set_dynstr (dict, "transport.socket.connect-path", fpath);
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport.address-family", "unix");
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport.socket.nodelay", "off");
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport-type", "socket");
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport.socket.keepalive", "off");
+ if (ret)
+ goto out;
+
+ if (frame_timeout > 0) {
+ ret = dict_set_int32 (dict, "frame-timeout", frame_timeout);
+ if (ret)
+ goto out;
+ }
+
+ *options = dict;
+out:
+ if (ret) {
+ GF_FREE (fpath);
+ if (dict)
+ dict_unref (dict);
+ }
+ return ret;
+}
+
+int
+rpc_transport_inet_options_build (dict_t **options, const char *hostname,
+ int port)
+{
+ dict_t *dict = NULL;
+ char *host = NULL;
+ int ret = -1;
+
+ GF_ASSERT (options);
+ GF_ASSERT (hostname);
+ GF_ASSERT (port >= 1024);
+
+ dict = dict_new ();
+ if (!dict)
+ goto out;
+
+ host = gf_strdup ((char*)hostname);
+ if (!hostname) {
+ ret = -1;
+ goto out;
+ }
+
+ ret = dict_set_dynstr (dict, "remote-host", host);
+ if (ret) {
+ gf_log (THIS->name, GF_LOG_WARNING,
+ "failed to set remote-host with %s", host);
+ goto out;
+ }
+
+ ret = dict_set_int32 (dict, "remote-port", port);
+ if (ret) {
+ gf_log (THIS->name, GF_LOG_WARNING,
+ "failed to set remote-port with %d", port);
+ goto out;
+ }
+ ret = dict_set_str (dict, "transport.address-family", "inet");
+ if (ret) {
+ gf_log (THIS->name, GF_LOG_WARNING,
+ "failed to set addr-family with inet");
+ goto out;
+ }
+
+ ret = dict_set_str (dict, "transport-type", "socket");
+ if (ret) {
+ gf_log (THIS->name, GF_LOG_WARNING,
+ "failed to set trans-type with socket");
+ goto out;
+ }
+
+ *options = dict;
+out:
+ if (ret) {
+ GF_FREE (host);
+ if (dict)
+ dict_unref (dict);
+ }
+
+ return ret;
+}
diff --git a/rpc/rpc-lib/src/rpc-transport.h b/rpc/rpc-lib/src/rpc-transport.h
index 2ba46fba..aa9df72b 100644
--- a/rpc/rpc-lib/src/rpc-transport.h
+++ b/rpc/rpc-lib/src/rpc-transport.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef __RPC_TRANSPORT_H__
@@ -25,6 +16,7 @@
#include "config.h"
#endif
+
#include <inttypes.h>
#ifdef GF_SOLARIS_HOST_OS
#include <rpc/auth.h>
@@ -34,15 +26,20 @@
#include <rpc/rpc_msg.h>
+
#ifndef MAX_IOVEC
#define MAX_IOVEC 16
#endif
+#ifndef AI_ADDRCONFIG
+#define AI_ADDRCONFIG 0
+#endif /* AI_ADDRCONFIG */
+
/* Given the 4-byte fragment header, returns non-zero if this fragment
- * is the last fragment for the RPC record being assemebled.
+ * is the last fragment for the RPC record being assembled.
* RPC Record marking standard defines a 32 bit value as the fragment
* header with the MSB signifying whether the fragment is the last
- * fragment for the record being asembled.
+ * fragment for the record being assembled.
*/
#define RPC_LASTFRAG(fraghdr) ((uint32_t)(fraghdr & 0x80000000U))
@@ -71,9 +68,14 @@ typedef struct rpc_transport rpc_transport_t;
#include "rpcsvc-common.h"
struct peer_info {
- struct sockaddr_storage sockaddr;
- socklen_t sockaddr_len;
- char identifier[UNIX_PATH_MAX];
+ struct sockaddr_storage sockaddr;
+ socklen_t sockaddr_len;
+ char identifier[UNIX_PATH_MAX];
+ // OP-VERSION of clients
+ uint32_t max_op_version;
+ uint32_t min_op_version;
+ //Volume mounted by client
+ char volname[1024];
};
typedef struct peer_info peer_info_t;
@@ -95,7 +97,7 @@ typedef enum {
* reading a single msg, this event may be
* delivered more than once.
*/
- RPC_TRANSPORT_MAP_XID_REQUEST, /* reciever of this event should send
+ RPC_TRANSPORT_MAP_XID_REQUEST, /* receiver of this event should send
* the prognum and procnum corresponding
* to xid.
*/
@@ -166,42 +168,50 @@ struct rpc_transport_pollin {
char vectored;
void *private;
struct iobref *iobref;
+ struct iobuf *hdr_iobuf;
char is_reply;
};
typedef struct rpc_transport_pollin rpc_transport_pollin_t;
typedef int (*rpc_transport_notify_t) (rpc_transport_t *, void *mydata,
rpc_transport_event_t, void *data, ...);
+
+
struct rpc_transport {
- struct rpc_transport_ops *ops;
+ struct rpc_transport_ops *ops;
rpc_transport_t *listener; /* listener transport to which
* request for creation of this
* transport came from. valid only
* on server process.
*/
- void *private;
- void *xl_private;
+
+ void *private;
+ struct _client_t *xl_private;
void *xl; /* Used for THIS */
- void *mydata;
- pthread_mutex_t lock;
- int32_t refcount;
+ void *mydata;
+ pthread_mutex_t lock;
+ int32_t refcount;
glusterfs_ctx_t *ctx;
dict_t *options;
char *name;
- void *dnscache;
- data_t *buf;
- int32_t (*init) (rpc_transport_t *this);
- void (*fini) (rpc_transport_t *this);
+ void *dnscache;
+ void *drc_client;
+ data_t *buf;
+ int32_t (*init) (rpc_transport_t *this);
+ void (*fini) (rpc_transport_t *this);
+ int (*reconfigure) (rpc_transport_t *this, dict_t *options);
rpc_transport_notify_t notify;
void *notify_data;
- peer_info_t peerinfo;
- peer_info_t myinfo;
+ peer_info_t peerinfo;
+ peer_info_t myinfo;
uint64_t total_bytes_read;
uint64_t total_bytes_write;
struct list_head list;
+ int bind_insecure;
+ void *dl_handle; /* handle of dlopen() */
};
struct rpc_transport_ops {
@@ -212,9 +222,9 @@ struct rpc_transport_ops {
rpc_transport_req_t *req);
int32_t (*submit_reply) (rpc_transport_t *this,
rpc_transport_reply_t *reply);
- int32_t (*connect) (rpc_transport_t *this, int port);
- int32_t (*listen) (rpc_transport_t *this);
- int32_t (*disconnect) (rpc_transport_t *this);
+ int32_t (*connect) (rpc_transport_t *this, int port);
+ int32_t (*listen) (rpc_transport_t *this);
+ int32_t (*disconnect) (rpc_transport_t *this);
int32_t (*get_peername) (rpc_transport_t *this, char *hostname,
int hostlen);
int32_t (*get_peeraddr) (rpc_transport_t *this, char *peeraddr,
@@ -238,6 +248,9 @@ int32_t
rpc_transport_disconnect (rpc_transport_t *this);
int32_t
+rpc_transport_destroy (rpc_transport_t *this);
+
+int32_t
rpc_transport_notify (rpc_transport_t *this, rpc_transport_event_t event,
void *data, ...);
@@ -248,9 +261,6 @@ int32_t
rpc_transport_submit_reply (rpc_transport_t *this,
rpc_transport_reply_t *reply);
-int32_t
-rpc_transport_destroy (rpc_transport_t *this);
-
rpc_transport_t *
rpc_transport_load (glusterfs_ctx_t *ctx, dict_t *options, char *name);
@@ -280,8 +290,19 @@ rpc_transport_get_myaddr (rpc_transport_t *this, char *peeraddr, int addrlen,
rpc_transport_pollin_t *
rpc_transport_pollin_alloc (rpc_transport_t *this, struct iovec *vector,
- int count, struct iobref *iobref, void *private);
+ int count, struct iobuf *hdr_iobuf,
+ struct iobref *iobref, void *private);
void
rpc_transport_pollin_destroy (rpc_transport_pollin_t *pollin);
+int
+rpc_transport_keepalive_options_set (dict_t *options, int32_t interval,
+ int32_t time);
+
+int
+rpc_transport_unix_options_build (dict_t **options, char *filepath,
+ int frame_timeout);
+
+int
+rpc_transport_inet_options_build (dict_t **options, const char *hostname, int port);
#endif /* __RPC_TRANSPORT_H__ */
diff --git a/rpc/rpc-lib/src/rpcsvc-auth.c b/rpc/rpc-lib/src/rpcsvc-auth.c
index 5cfa255b..10827945 100644
--- a/rpc/rpc-lib/src/rpcsvc-auth.c
+++ b/rpc/rpc-lib/src/rpcsvc-auth.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#include "rpcsvc.h"
@@ -29,6 +20,8 @@ rpcsvc_auth_unix_init (rpcsvc_t *svc, dict_t *options);
extern rpcsvc_auth_t *
rpcsvc_auth_glusterfs_init (rpcsvc_t *svc, dict_t *options);
+extern rpcsvc_auth_t *
+rpcsvc_auth_glusterfs_v2_init (rpcsvc_t *svc, dict_t *options);
int
rpcsvc_auth_add_initer (struct list_head *list, char *idfier,
@@ -41,7 +34,6 @@ rpcsvc_auth_add_initer (struct list_head *list, char *idfier,
new = GF_CALLOC (1, sizeof (*new), gf_common_mt_rpcsvc_auth_list);
if (!new) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Memory allocation failed");
return -1;
}
@@ -67,6 +59,16 @@ rpcsvc_auth_add_initers (rpcsvc_t *svc)
goto err;
}
+
+ ret = rpcsvc_auth_add_initer (&svc->authschemes, "auth-glusterfs-v2",
+ (rpcsvc_auth_initer_t)
+ rpcsvc_auth_glusterfs_v2_init);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR,
+ "Failed to add AUTH_GLUSTERFS-v2");
+ goto err;
+ }
+
ret = rpcsvc_auth_add_initer (&svc->authschemes, "auth-unix",
(rpcsvc_auth_initer_t)
rpcsvc_auth_unix_init);
@@ -176,6 +178,52 @@ err:
}
int
+rpcsvc_set_allow_insecure (rpcsvc_t *svc, dict_t *options)
+{
+ int ret = -1;
+ char *allow_insecure_str = NULL;
+ gf_boolean_t is_allow_insecure = _gf_false;
+
+ GF_ASSERT (svc);
+ GF_ASSERT (options);
+
+ ret = dict_get_str (options, "rpc-auth-allow-insecure",
+ &allow_insecure_str);
+ if (0 == ret) {
+ ret = gf_string2boolean (allow_insecure_str,
+ &is_allow_insecure);
+ if (0 == ret) {
+ if (_gf_true == is_allow_insecure)
+ svc->allow_insecure = 1;
+ else
+ svc->allow_insecure = 0;
+ }
+ }
+
+ return 0;
+}
+
+int
+rpcsvc_set_root_squash (rpcsvc_t *svc, dict_t *options)
+{
+ int ret = -1;
+
+ GF_ASSERT (svc);
+ GF_ASSERT (options);
+
+ ret = dict_get_str_boolean (options, "root-squash", 0);
+ if (ret != -1)
+ svc->root_squash = ret;
+ else
+ svc->root_squash = _gf_false;
+
+ if (svc->root_squash)
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "root squashing enabled ");
+
+ return 0;
+}
+
+int
rpcsvc_auth_init (rpcsvc_t *svc, dict_t *options)
{
int ret = -1;
@@ -183,6 +231,8 @@ rpcsvc_auth_init (rpcsvc_t *svc, dict_t *options)
if ((!svc) || (!options))
return -1;
+ (void) rpcsvc_set_allow_insecure (svc, options);
+ (void) rpcsvc_set_root_squash (svc, options);
ret = rpcsvc_auth_add_initers (svc);
if (ret == -1) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to add initers");
@@ -292,17 +342,17 @@ rpcsvc_authenticate (rpcsvc_request_t *req)
if (!req)
return ret;
- //minauth = rpcsvc_request_prog_minauth (req);
- minauth = 1;
+ /* FIXME use rpcsvc_request_prog_minauth() */
+ minauth = 0;
if (minauth > rpcsvc_request_cred_flavour (req)) {
- gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Auth too weak");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING, "Auth too weak");
rpcsvc_request_set_autherr (req, AUTH_TOOWEAK);
goto err;
}
auth = rpcsvc_auth_get_handler (req);
if (!auth) {
- gf_log (GF_RPCSVC, GF_LOG_DEBUG, "No auth handler found");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING, "No auth handler found");
goto err;
}
@@ -313,18 +363,13 @@ err:
return ret;
}
-
int
rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen)
{
- int count = 0;
- int gen = RPCSVC_AUTH_REJECT;
- int spec = RPCSVC_AUTH_REJECT;
- int final = RPCSVC_AUTH_REJECT;
- char *srchstr = NULL;
- char *valstr = NULL;
- gf_boolean_t boolval = _gf_false;
- int ret = 0;
+ int count = 0;
+ int result = RPCSVC_AUTH_REJECT;
+ char *srchstr = NULL;
+ int ret = 0;
struct rpcsvc_auth_list *auth = NULL;
struct rpcsvc_auth_list *tmp = NULL;
@@ -342,59 +387,27 @@ rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen)
if (count >= arrlen)
break;
- gen = gf_asprintf (&srchstr, "rpc-auth.%s", auth->name);
- if (gen == -1) {
+ result = gf_asprintf (&srchstr, "rpc-auth.%s.%s",
+ auth->name, volname);
+ if (result == -1) {
count = -1;
goto err;
}
- gen = RPCSVC_AUTH_REJECT;
- if (dict_get (svc->options, srchstr)) {
- ret = dict_get_str (svc->options, srchstr, &valstr);
- if (ret == 0) {
- ret = gf_string2boolean (valstr, &boolval);
- if (ret == 0) {
- if (boolval == _gf_true)
- gen = RPCSVC_AUTH_ACCEPT;
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Faile"
- "d to read auth val");
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Faile"
- "d to read auth val");
- }
-
+ ret = dict_get_str_boolean (svc->options, srchstr, 0xC00FFEE);
GF_FREE (srchstr);
- spec = gf_asprintf (&srchstr, "rpc-auth.%s.%s", auth->name,
- volname);
- if (spec == -1) {
- count = -1;
- goto err;
- }
-
- spec = RPCSVC_AUTH_DONTCARE;
- if (dict_get (svc->options, srchstr)) {
- ret = dict_get_str (svc->options, srchstr, &valstr);
- if (ret == 0) {
- ret = gf_string2boolean (valstr, &boolval);
- if (ret == 0) {
- if (boolval == _gf_true)
- spec = RPCSVC_AUTH_ACCEPT;
- else
- spec = RPCSVC_AUTH_REJECT;
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Faile"
- "d to read auth val");
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Faile"
- "d to read auth val");
- }
- GF_FREE (srchstr);
- final = rpcsvc_combine_gen_spec_volume_checks (gen, spec);
- if (final == RPCSVC_AUTH_ACCEPT) {
+ switch (ret) {
+ case _gf_true:
+ result = RPCSVC_AUTH_ACCEPT;
autharr[count] = auth->auth->authnum;
++count;
+ break;
+ case _gf_false:
+ result = RPCSVC_AUTH_REJECT;
+ break;
+ default:
+ result = RPCSVC_AUTH_DONTCARE;
}
}
@@ -402,16 +415,22 @@ err:
return count;
}
-
gid_t *
rpcsvc_auth_unix_auxgids (rpcsvc_request_t *req, int *arrlen)
{
if ((!req) || (!arrlen))
return NULL;
- if ((req->cred.flavour != AUTH_UNIX) ||
- (req->cred.flavour != AUTH_GLUSTERFS))
+ /* In case of AUTH_NULL auxgids are not used */
+ switch (req->cred.flavour) {
+ case AUTH_UNIX:
+ case AUTH_GLUSTERFS:
+ case AUTH_GLUSTERFS_v2:
+ break;
+ default:
+ gf_log ("rpc", GF_LOG_DEBUG, "auth type not unix or glusterfs");
return NULL;
+ }
*arrlen = req->auxgidcount;
if (*arrlen == 0)
diff --git a/rpc/rpc-lib/src/rpcsvc-common.h b/rpc/rpc-lib/src/rpcsvc-common.h
index 7e72bc3a..054e187c 100644
--- a/rpc/rpc-lib/src/rpcsvc-common.h
+++ b/rpc/rpc-lib/src/rpcsvc-common.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _RPCSVC_COMMON_H
@@ -29,6 +20,7 @@
typedef enum {
RPCSVC_EVENT_ACCEPT,
RPCSVC_EVENT_DISCONNECT,
+ RPCSVC_EVENT_TRANSPORT_DESTROY,
RPCSVC_EVENT_LISTENER_DEAD,
} rpcsvc_event_t;
@@ -38,6 +30,8 @@ struct rpcsvc_state;
typedef int (*rpcsvc_notify_t) (struct rpcsvc_state *, void *mydata,
rpcsvc_event_t, void *data);
+struct drc_globals;
+typedef struct drc_globals rpcsvc_drc_globals_t;
/* Contains global state required for all the RPC services.
*/
@@ -59,7 +53,8 @@ typedef struct rpcsvc_state {
/* Allow insecure ports. */
int allow_insecure;
-
+ gf_boolean_t register_portmap;
+ gf_boolean_t root_squash;
glusterfs_ctx_t *ctx;
/* list of connections which will listen for incoming connections */
@@ -75,7 +70,53 @@ typedef struct rpcsvc_state {
void *mydata; /* This is xlator */
rpcsvc_notify_t notifyfn;
struct mem_pool *rxpool;
+ rpcsvc_drc_globals_t *drc;
} rpcsvc_t;
+/* DRC START */
+enum drc_op_type {
+ DRC_NA = 0,
+ DRC_IDEMPOTENT = 1,
+ DRC_NON_IDEMPOTENT = 2
+};
+typedef enum drc_op_type drc_op_type_t;
+
+enum drc_type {
+ DRC_TYPE_NONE = 0,
+ DRC_TYPE_IN_MEMORY = 1
+};
+typedef enum drc_type drc_type_t;
+
+enum drc_lru_factor {
+ DRC_LRU_5_PC = 20,
+ DRC_LRU_10_PC = 10,
+ DRC_LRU_25_PC = 4,
+ DRC_LRU_50_PC = 2
+};
+typedef enum drc_lru_factor drc_lru_factor_t;
+
+enum drc_xid_state {
+ DRC_XID_MONOTONOUS = 0,
+ DRC_XID_WRAPPED = 1
+};
+typedef enum drc_xid_state drc_xid_state_t;
+
+enum drc_op_state {
+ DRC_OP_IN_TRANSIT = 0,
+ DRC_OP_CACHED = 1
+};
+typedef enum drc_op_state drc_op_state_t;
+
+enum drc_policy {
+ DRC_LRU = 0
+};
+typedef enum drc_policy drc_policy_t;
+
+/* Default policies for DRC */
+#define DRC_DEFAULT_TYPE DRC_TYPE_IN_MEMORY
+#define DRC_DEFAULT_CACHE_SIZE 0x20000
+#define DRC_DEFAULT_LRU_FACTOR DRC_LRU_25_PC
+
+/* DRC END */
#endif /* #ifndef _RPCSVC_COMMON_H */
diff --git a/rpc/rpc-lib/src/rpcsvc.c b/rpc/rpc-lib/src/rpcsvc.c
index b78b1f0e..d6f5e754 100644
--- a/rpc/rpc-lib/src/rpcsvc.c
+++ b/rpc/rpc-lib/src/rpcsvc.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _CONFIG_H
@@ -34,6 +25,10 @@
#include "iobuf.h"
#include "globals.h"
#include "xdr-common.h"
+#include "xdr-generic.h"
+#include "rpc-common-xdr.h"
+#include "syncop.h"
+#include "rpc-drc.h"
#include <errno.h>
#include <pthread.h>
@@ -48,6 +43,8 @@
#include "xdr-rpcclnt.h"
+#define ACL_PROGRAM 100227
+
struct rpcsvc_program gluster_dump_prog;
#define rpcsvc_alloc_request(svc, request) \
@@ -60,451 +57,6 @@ rpcsvc_listener_t *
rpcsvc_get_listener (rpcsvc_t *svc, uint16_t port, rpc_transport_t *trans);
int
-rpcsvc_transport_peer_check_search (dict_t *options, char *pattern, char *clstr)
-{
- int ret = -1;
- char *addrtok = NULL;
- char *addrstr = NULL;
- char *svptr = NULL;
-
- if ((!options) || (!clstr))
- return -1;
-
- if (!dict_get (options, pattern))
- return -1;
-
- ret = dict_get_str (options, pattern, &addrstr);
- if (ret < 0) {
- ret = -1;
- goto err;
- }
-
- if (!addrstr) {
- ret = -1;
- goto err;
- }
-
- addrtok = strtok_r (addrstr, ",", &svptr);
- while (addrtok) {
-
-#ifdef FNM_CASEFOLD
- ret = fnmatch (addrtok, clstr, FNM_CASEFOLD);
-#else
- ret = fnmatch (addrtok, clstr, 0);
-#endif
- if (ret == 0)
- goto err;
-
- addrtok = strtok_r (NULL, ",", &svptr);
- }
-
- ret = -1;
-err:
-
- return ret;
-}
-
-
-int
-rpcsvc_transport_peer_check_allow (dict_t *options, char *volname, char *clstr)
-{
- int ret = RPCSVC_AUTH_DONTCARE;
- char *srchstr = NULL;
- char globalrule[] = "rpc-auth.addr.allow";
-
- if ((!options) || (!clstr))
- return ret;
-
- /* If volname is NULL, then we're searching for the general rule to
- * determine the current address in clstr is allowed or not for all
- * subvolumes.
- */
- if (volname) {
- ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.allow", volname);
- if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
- ret = RPCSVC_AUTH_DONTCARE;
- goto out;
- }
- } else
- srchstr = globalrule;
-
- ret = rpcsvc_transport_peer_check_search (options, srchstr, clstr);
- if (volname)
- GF_FREE (srchstr);
-
- if (ret == 0)
- ret = RPCSVC_AUTH_ACCEPT;
- else
- ret = RPCSVC_AUTH_DONTCARE;
-out:
- return ret;
-}
-
-int
-rpcsvc_transport_peer_check_reject (dict_t *options, char *volname, char *clstr)
-{
- int ret = RPCSVC_AUTH_DONTCARE;
- char *srchstr = NULL;
- char generalrule[] = "rpc-auth.addr.reject";
-
- if ((!options) || (!clstr))
- return ret;
-
- if (volname) {
- ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.reject", volname);
- if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
- ret = RPCSVC_AUTH_REJECT;
- goto out;
- }
- } else
- srchstr = generalrule;
-
- ret = rpcsvc_transport_peer_check_search (options, srchstr, clstr);
- if (volname)
- GF_FREE (srchstr);
-
- if (ret == 0)
- ret = RPCSVC_AUTH_REJECT;
- else
- ret = RPCSVC_AUTH_DONTCARE;
-out:
- return ret;
-}
-
-
-/* This function tests the results of the allow rule and the reject rule to
- * combine them into a single result that can be used to determine if the
- * connection should be allowed to proceed.
- * Heres the test matrix we need to follow in this function.
- *
- * A - Allow, the result of the allow test. Never returns R.
- * R - Reject, result of the reject test. Never returns A.
- * Both can return D or dont care if no rule was given.
- *
- * | @allow | @reject | Result |
- * | A | R | R |
- * | D | D | D |
- * | A | D | A |
- * | D | R | R |
- */
-int
-rpcsvc_combine_allow_reject_volume_check (int allow, int reject)
-{
- int final = RPCSVC_AUTH_REJECT;
-
- /* If allowed rule allows but reject rule rejects, we stay cautious
- * and reject. */
- if ((allow == RPCSVC_AUTH_ACCEPT) && (reject == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
- /* if both are dont care, that is user did not specify for either allow
- * or reject, we leave it up to the general rule to apply, in the hope
- * that there is one.
- */
- else if ((allow == RPCSVC_AUTH_DONTCARE) &&
- (reject == RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_DONTCARE;
- /* If one is dont care, the other one applies. */
- else if ((allow == RPCSVC_AUTH_ACCEPT) &&
- (reject == RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((allow == RPCSVC_AUTH_DONTCARE) &&
- (reject == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
-
- return final;
-}
-
-
-/* Combines the result of the general rule test against, the specific rule
- * to determine final permission for the client's address.
- *
- * | @gen | @spec | Result |
- * | A | A | A |
- * | A | R | R |
- * | A | D | A |
- * | D | A | A |
- * | D | R | R |
- * | D | D | D |
- * | R | A | A |
- * | R | D | R |
- * | R | R | R |
- */
-int
-rpcsvc_combine_gen_spec_addr_checks (int gen, int spec)
-{
- int final = RPCSVC_AUTH_REJECT;
-
- if ((gen == RPCSVC_AUTH_ACCEPT) && (spec == RPCSVC_AUTH_ACCEPT))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_ACCEPT) && (spec == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
- else if ((gen == RPCSVC_AUTH_ACCEPT) && (spec == RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_DONTCARE) && (spec == RPCSVC_AUTH_ACCEPT))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_DONTCARE) && (spec == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
- else if ((gen == RPCSVC_AUTH_DONTCARE) && (spec== RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_DONTCARE;
- else if ((gen == RPCSVC_AUTH_REJECT) && (spec == RPCSVC_AUTH_ACCEPT))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_REJECT) && (spec == RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_REJECT;
- else if ((gen == RPCSVC_AUTH_REJECT) && (spec == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
-
- return final;
-}
-
-
-
-/* Combines the result of the general rule test against, the specific rule
- * to determine final test for the connection coming in for a given volume.
- *
- * | @gen | @spec | Result |
- * | A | A | A |
- * | A | R | R |
- * | A | D | A |
- * | D | A | A |
- * | D | R | R |
- * | D | D | R |, special case, we intentionally disallow this.
- * | R | A | A |
- * | R | D | R |
- * | R | R | R |
- */
-int
-rpcsvc_combine_gen_spec_volume_checks (int gen, int spec)
-{
- int final = RPCSVC_AUTH_REJECT;
-
- if ((gen == RPCSVC_AUTH_ACCEPT) && (spec == RPCSVC_AUTH_ACCEPT))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_ACCEPT) && (spec == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
- else if ((gen == RPCSVC_AUTH_ACCEPT) && (spec == RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_DONTCARE) && (spec == RPCSVC_AUTH_ACCEPT))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_DONTCARE) && (spec == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
- /* On no rule, we reject. */
- else if ((gen == RPCSVC_AUTH_DONTCARE) && (spec== RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_REJECT;
- else if ((gen == RPCSVC_AUTH_REJECT) && (spec == RPCSVC_AUTH_ACCEPT))
- final = RPCSVC_AUTH_ACCEPT;
- else if ((gen == RPCSVC_AUTH_REJECT) && (spec == RPCSVC_AUTH_DONTCARE))
- final = RPCSVC_AUTH_REJECT;
- else if ((gen == RPCSVC_AUTH_REJECT) && (spec == RPCSVC_AUTH_REJECT))
- final = RPCSVC_AUTH_REJECT;
-
- return final;
-}
-
-
-int
-rpcsvc_transport_peer_check_name (dict_t *options, char *volname,
- rpc_transport_t *trans)
-{
- int ret = RPCSVC_AUTH_REJECT;
- int aret = RPCSVC_AUTH_REJECT;
- int rjret = RPCSVC_AUTH_REJECT;
- char clstr[RPCSVC_PEER_STRLEN];
-
- if (!trans)
- return ret;
-
- ret = rpc_transport_get_peername (trans, clstr, RPCSVC_PEER_STRLEN);
- if (ret != 0) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get remote addr: "
- "%s", gai_strerror (ret));
- ret = RPCSVC_AUTH_REJECT;
- goto err;
- }
-
- aret = rpcsvc_transport_peer_check_allow (options, volname, clstr);
- rjret = rpcsvc_transport_peer_check_reject (options, volname, clstr);
-
- ret = rpcsvc_combine_allow_reject_volume_check (aret, rjret);
-
-err:
- return ret;
-}
-
-
-int
-rpcsvc_transport_peer_check_addr (dict_t *options, char *volname,
- rpc_transport_t *trans)
-{
- int ret = RPCSVC_AUTH_REJECT;
- int aret = RPCSVC_AUTH_DONTCARE;
- int rjret = RPCSVC_AUTH_REJECT;
- char clstr[RPCSVC_PEER_STRLEN];
-
- if (!trans)
- return ret;
-
- ret = rpcsvc_transport_peeraddr (trans, clstr, RPCSVC_PEER_STRLEN, NULL,
- 0);
- if (ret != 0) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get remote addr: "
- "%s", gai_strerror (ret));
- ret = RPCSVC_AUTH_REJECT;
- goto err;
- }
-
- aret = rpcsvc_transport_peer_check_allow (options, volname, clstr);
- rjret = rpcsvc_transport_peer_check_reject (options, volname, clstr);
-
- ret = rpcsvc_combine_allow_reject_volume_check (aret, rjret);
-err:
- return ret;
-}
-
-
-int
-rpcsvc_transport_check_volume_specific (dict_t *options, char *volname,
- rpc_transport_t *trans)
-{
- int namechk = RPCSVC_AUTH_REJECT;
- int addrchk = RPCSVC_AUTH_REJECT;
- gf_boolean_t namelookup = _gf_true;
- char *namestr = NULL;
- int ret = 0;
-
- if ((!options) || (!volname) || (!trans))
- return RPCSVC_AUTH_REJECT;
-
- /* Enabled by default */
- if ((dict_get (options, "rpc-auth.addr.namelookup"))) {
- ret = dict_get_str (options, "rpc-auth.addr.namelookup"
- , &namestr);
- if (ret == 0) {
- ret = gf_string2boolean (namestr, &namelookup);
- if (ret)
- gf_log ("rpcsvc", GF_LOG_DEBUG,
- "wrong option %s given for "
- "'namelookup'", namestr);
- }
- }
-
- /* We need two separate checks because the rules with addresses in them
- * can be network addresses which can be general and names can be
- * specific which will over-ride the network address rules.
- */
- if (namelookup)
- namechk = rpcsvc_transport_peer_check_name (options, volname,
- trans);
- addrchk = rpcsvc_transport_peer_check_addr (options, volname, trans);
-
- if (namelookup)
- ret = rpcsvc_combine_gen_spec_addr_checks (addrchk, namechk);
- else
- ret = addrchk;
-
- return ret;
-}
-
-
-int
-rpcsvc_transport_check_volume_general (dict_t *options, rpc_transport_t *trans)
-{
- int addrchk = RPCSVC_AUTH_REJECT;
- int namechk = RPCSVC_AUTH_REJECT;
- gf_boolean_t namelookup = _gf_true;
- char *namestr = NULL;
- int ret = 0;
-
- if ((!options) || (!trans))
- return RPCSVC_AUTH_REJECT;
-
- /* Enabled by default */
- if ((dict_get (options, "rpc-auth.addr.namelookup"))) {
- ret = dict_get_str (options, "rpc-auth.addr.namelookup"
- , &namestr);
- if (!ret) {
- ret = gf_string2boolean (namestr, &namelookup);
- if (ret)
- gf_log ("rpcsvc", GF_LOG_DEBUG,
- "wrong option %s given for "
- "'namelookup'", namestr);
- }
- }
-
- /* We need two separate checks because the rules with addresses in them
- * can be network addresses which can be general and names can be
- * specific which will over-ride the network address rules.
- */
- if (namelookup)
- namechk = rpcsvc_transport_peer_check_name (options, NULL,
- trans);
-
- addrchk = rpcsvc_transport_peer_check_addr (options, NULL, trans);
-
- if (namelookup)
- ret = rpcsvc_combine_gen_spec_addr_checks (addrchk, namechk);
- else
- ret = addrchk;
-
- return ret;
-}
-
-int
-rpcsvc_transport_peer_check (dict_t *options, char *volname,
- rpc_transport_t *trans)
-{
- int general_chk = RPCSVC_AUTH_REJECT;
- int specific_chk = RPCSVC_AUTH_REJECT;
-
- if ((!options) || (!volname) || (!trans))
- return RPCSVC_AUTH_REJECT;
-
- general_chk = rpcsvc_transport_check_volume_general (options, trans);
- specific_chk = rpcsvc_transport_check_volume_specific (options, volname,
- trans);
-
- return rpcsvc_combine_gen_spec_volume_checks (general_chk,specific_chk);
-}
-
-
-char *
-rpcsvc_volume_allowed (dict_t *options, char *volname)
-{
- char globalrule[] = "rpc-auth.addr.allow";
- char *srchstr = NULL;
- char *addrstr = NULL;
- int ret = -1;
-
- if ((!options) || (!volname))
- return NULL;
-
- ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.allow", volname);
- if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
- goto out;
- }
-
- if (!dict_get (options, srchstr)) {
- GF_FREE (srchstr);
- srchstr = globalrule;
- ret = dict_get_str (options, srchstr, &addrstr);
- if (ret)
- gf_log ("rpcsvc", GF_LOG_DEBUG,
- "failed to get the string %s", srchstr);
- } else {
- ret = dict_get_str (options, srchstr, &addrstr);
- if (ret)
- gf_log ("rpcsvc", GF_LOG_DEBUG,
- "failed to get the string %s", srchstr);
- GF_FREE (srchstr);
- }
-out:
- return addrstr;
-}
-
-
-int
rpcsvc_notify (rpc_transport_t *trans, void *mydata,
rpc_transport_event_t event, void *data, ...);
@@ -515,7 +67,6 @@ rpcsvc_notify_wrapper_alloc (void)
wrapper = GF_CALLOC (1, sizeof (*wrapper), gf_common_mt_rpcsvc_wrapper_t);
if (!wrapper) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "memory allocation failed");
goto out;
}
@@ -551,105 +102,34 @@ out:
return;
}
-
-int
-rpcsvc_conn_privport_check (rpcsvc_t *svc, char *volname,
- rpc_transport_t *trans)
+rpcsvc_vector_sizer
+rpcsvc_get_program_vector_sizer (rpcsvc_t *svc, uint32_t prognum,
+ uint32_t progver, uint32_t procnum)
{
- struct sockaddr_storage sa = {0, };
- int ret = RPCSVC_AUTH_REJECT;
- socklen_t sasize = sizeof (sa);
- char *srchstr = NULL;
- char *valstr = NULL;
- int globalinsecure = RPCSVC_AUTH_REJECT;
- int exportinsecure = RPCSVC_AUTH_DONTCARE;
- uint16_t port = 0;
- gf_boolean_t insecure = _gf_false;
-
- if ((!svc) || (!volname) || (!trans))
- return ret;
-
- ret = rpcsvc_transport_peeraddr (trans, NULL, 0, &sa,
- sasize);
- if (ret != 0) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get peer addr: %s",
- gai_strerror (ret));
- ret = RPCSVC_AUTH_REJECT;
- goto err;
- }
-
- if (sa.ss_family == AF_INET) {
- port = ((struct sockaddr_in *)&sa)->sin_port;
- } else {
- port = ((struct sockaddr_in6 *)&sa)->sin6_port;
- }
-
- gf_log (GF_RPCSVC, GF_LOG_TRACE, "Client port: %d", (int)port);
- /* If the port is already a privileged one, dont bother with checking
- * options.
- */
- if (port <= 1024) {
- ret = RPCSVC_AUTH_ACCEPT;
- goto err;
- }
+ rpcsvc_program_t *program = NULL;
+ char found = 0;
- /* Disabled by default */
- if ((dict_get (svc->options, "rpc-auth.ports.insecure"))) {
- ret = dict_get_str (svc->options, "rpc-auth.ports.insecure"
- , &srchstr);
- if (ret == 0) {
- ret = gf_string2boolean (srchstr, &insecure);
- if (ret == 0) {
- if (insecure == _gf_true)
- globalinsecure = RPCSVC_AUTH_ACCEPT;
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to"
- " read rpc-auth.ports.insecure value");
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to"
- " read rpc-auth.ports.insecure value");
- }
+ if (!svc)
+ return NULL;
- /* Disabled by default */
- ret = gf_asprintf (&srchstr, "rpc-auth.ports.%s.insecure", volname);
- if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
- ret = RPCSVC_AUTH_REJECT;
- goto err;
+ pthread_mutex_lock (&svc->rpclock);
+ {
+ list_for_each_entry (program, &svc->programs, program) {
+ if ((program->prognum == prognum)
+ && (program->progver == progver)) {
+ found = 1;
+ break;
+ }
+ }
}
+ pthread_mutex_unlock (&svc->rpclock);
- if (dict_get (svc->options, srchstr)) {
- ret = dict_get_str (svc->options, srchstr, &valstr);
- if (ret == 0) {
- ret = gf_string2boolean (srchstr, &insecure);
- if (ret == 0) {
- if (insecure == _gf_true)
- exportinsecure = RPCSVC_AUTH_ACCEPT;
- else
- exportinsecure = RPCSVC_AUTH_REJECT;
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to"
- " read rpc-auth.ports.insecure value");
- } else
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to"
- " read rpc-auth.ports.insecure value");
- }
-
- ret = rpcsvc_combine_gen_spec_volume_checks (globalinsecure,
- exportinsecure);
- if (ret == RPCSVC_AUTH_ACCEPT)
- gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Unprivileged port allowed");
+ if (found)
+ return program->actors[procnum].vector_sizer;
else
- gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Unprivileged port not"
- " allowed");
-
-err:
- if (srchstr)
- GF_FREE (srchstr);
- return ret;
+ return NULL;
}
-
/* This needs to change to returning errors, since
* we need to return RPC specific error messages when some
* of the pointers below are NULL.
@@ -685,26 +165,35 @@ rpcsvc_program_actor (rpcsvc_request_t *req)
if (!found) {
if (err != PROG_MISMATCH) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR,
- "RPC program not available");
+ /* log in DEBUG when nfs clients try to see if
+ * ACL requests are accepted by nfs server
+ */
+ gf_log (GF_RPCSVC, (req->prognum == ACL_PROGRAM) ?
+ GF_LOG_DEBUG : GF_LOG_WARNING,
+ "RPC program not available (req %u %u)",
+ req->prognum, req->progver);
err = PROG_UNAVAIL;
goto err;
}
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC program version not"
- " available");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING,
+ "RPC program version not available (req %u %u)",
+ req->prognum, req->progver);
goto err;
}
req->prog = program;
if (!program->actors) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC System error");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING,
+ "RPC Actor not found for program %s %d",
+ program->progname, program->prognum);
err = SYSTEM_ERR;
goto err;
}
if ((req->procnum < 0) || (req->procnum >= program->numactors)) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC Program procedure not"
- " available");
+ " available for procedure %d in %s", req->procnum,
+ program->progname);
err = PROC_UNAVAIL;
goto err;
}
@@ -712,12 +201,15 @@ rpcsvc_program_actor (rpcsvc_request_t *req)
actor = &program->actors[req->procnum];
if (!actor->actor) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC Program procedure not"
- " available");
+ " available for procedure %d in %s", req->procnum,
+ program->progname);
err = PROC_UNAVAIL;
actor = NULL;
goto err;
}
+ req->synctask = program->synctask;
+
err = SUCCESS;
gf_log (GF_RPCSVC, GF_LOG_TRACE, "Actor found: %s - %s",
program->progname, actor->procname);
@@ -730,9 +222,9 @@ err:
/* this procedure can only pass 4 arguments to registered notifyfn. To send more
- * arguements call wrapper->notify directly.
+ * arguments call wrapper->notify directly.
*/
-inline void
+static inline void
rpcsvc_program_notify (rpcsvc_listener_t *listener, rpcsvc_event_t event,
void *data)
{
@@ -755,7 +247,7 @@ out:
}
-inline int
+static inline int
rpcsvc_accept (rpcsvc_t *svc, rpc_transport_t *listen_trans,
rpc_transport_t *new_trans)
{
@@ -785,9 +277,12 @@ rpcsvc_request_destroy (rpcsvc_request_t *req)
iobref_unref (req->iobref);
}
+ if (req->hdr_iobuf)
+ iobuf_unref (req->hdr_iobuf);
+
rpc_transport_unref (req->trans);
- mem_put (req->svc->rxpool, req);
+ mem_put (req);
out:
return;
@@ -800,6 +295,8 @@ rpcsvc_request_init (rpcsvc_t *svc, rpc_transport_t *trans,
struct iovec progmsg, rpc_transport_pollin_t *msg,
rpcsvc_request_t *req)
{
+ int i = 0;
+
if ((!trans) || (!callmsg)|| (!req) || (!msg))
return NULL;
@@ -814,7 +311,11 @@ rpcsvc_request_init (rpcsvc_t *svc, rpc_transport_t *trans,
req->msg[0] = progmsg;
req->iobref = iobref_ref (msg->iobref);
if (msg->vectored) {
- req->msg[1] = msg->vector[1];
+ /* msg->vector[2] is defined in structure. prevent a
+ out of bound access */
+ for (i = 1; i < min (msg->count, 2); i++) {
+ req->msg[i] = msg->vector[i];
+ }
}
req->svc = svc;
@@ -860,7 +361,6 @@ rpcsvc_request_create (rpcsvc_t *svc, rpc_transport_t *trans,
*/
rpcsvc_alloc_request (svc, req);
if (!req) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to alloc request");
goto err;
}
@@ -871,15 +371,17 @@ rpcsvc_request_create (rpcsvc_t *svc, rpc_transport_t *trans,
req->cred.authdata,req->verf.authdata);
if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC call decoding failed");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING, "RPC call decoding failed");
rpcsvc_request_seterr (req, GARBAGE_ARGS);
+ req->trans = rpc_transport_ref (trans);
+ req->svc = svc;
goto err;
}
ret = -1;
rpcsvc_request_init (svc, trans, &rpcmsg, progmsg, msg, req);
- gf_log (GF_RPCSVC, GF_LOG_TRACE, "recieved rpc-message (XID: 0x%lx, "
+ gf_log (GF_RPCSVC, GF_LOG_TRACE, "received rpc-message (XID: 0x%lx, "
"Ver: %ld, Program: %ld, ProgVers: %ld, Proc: %ld) from"
" rpc-transport (%s)", rpc_call_xid (&rpcmsg),
rpc_call_rpcvers (&rpcmsg), rpc_call_program (&rpcmsg),
@@ -887,7 +389,14 @@ rpcsvc_request_create (rpcsvc_t *svc, rpc_transport_t *trans,
trans->name);
if (rpc_call_rpcvers (&rpcmsg) != 2) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC version not supported");
+ /* LOG- TODO: print rpc version, also print the peerinfo
+ from transport */
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "RPC version not supported "
+ "(XID: 0x%lx, Ver: %ld, Prog: %ld, ProgVers: %ld, "
+ "Proc: %ld) from trans (%s)", rpc_call_xid (&rpcmsg),
+ rpc_call_rpcvers (&rpcmsg), rpc_call_program (&rpcmsg),
+ rpc_call_progver (&rpcmsg), rpc_call_progproc (&rpcmsg),
+ trans->name);
rpcsvc_request_seterr (req, RPC_MISMATCH);
goto err;
}
@@ -899,7 +408,12 @@ rpcsvc_request_create (rpcsvc_t *svc, rpc_transport_t *trans,
* error happened.
*/
rpcsvc_request_seterr (req, AUTH_ERROR);
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed authentication");
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "auth failed on request. "
+ "(XID: 0x%lx, Ver: %ld, Prog: %ld, ProgVers: %ld, "
+ "Proc: %ld) from trans (%s)", rpc_call_xid (&rpcmsg),
+ rpc_call_rpcvers (&rpcmsg), rpc_call_program (&rpcmsg),
+ rpc_call_progver (&rpcmsg), rpc_call_progproc (&rpcmsg),
+ trans->name);
ret = -1;
goto err;
}
@@ -909,12 +423,13 @@ rpcsvc_request_create (rpcsvc_t *svc, rpc_transport_t *trans,
* since we are not handling authentication failures for now.
*/
req->rpc_status = MSG_ACCEPTED;
+ req->reply = NULL;
ret = 0;
err:
if (ret == -1) {
ret = rpcsvc_error_reply (req);
if (ret)
- gf_log ("rpcsvc", GF_LOG_DEBUG,
+ gf_log ("rpcsvc", GF_LOG_WARNING,
"failed to queue error reply");
req = NULL;
}
@@ -924,13 +439,39 @@ err:
int
+rpcsvc_check_and_reply_error (int ret, call_frame_t *frame, void *opaque)
+{
+ rpcsvc_request_t *req = NULL;
+
+ req = opaque;
+
+ if (ret)
+ gf_log ("rpcsvc", GF_LOG_ERROR,
+ "rpc actor failed to complete successfully");
+
+ if (ret == RPCSVC_ACTOR_ERROR) {
+ ret = rpcsvc_error_reply (req);
+ if (ret)
+ gf_log ("rpcsvc", GF_LOG_WARNING,
+ "failed to queue error reply");
+ }
+
+ return 0;
+}
+
+int
rpcsvc_handle_rpc_call (rpcsvc_t *svc, rpc_transport_t *trans,
rpc_transport_pollin_t *msg)
{
- rpcsvc_actor_t *actor = NULL;
- rpcsvc_request_t *req = NULL;
- int ret = -1;
- uint16_t port = 0;
+ rpcsvc_actor_t *actor = NULL;
+ rpcsvc_actor actor_fn = NULL;
+ rpcsvc_request_t *req = NULL;
+ int ret = -1;
+ uint16_t port = 0;
+ gf_boolean_t is_unix = _gf_false;
+ gf_boolean_t unprivileged = _gf_false;
+ drc_cached_op_t *reply = NULL;
+ rpcsvc_drc_globals_t *drc = NULL;
if (!trans || !svc)
return -1;
@@ -943,9 +484,11 @@ rpcsvc_handle_rpc_call (rpcsvc_t *svc, rpc_transport_t *trans,
case AF_INET6:
port = ((struct sockaddr_in6 *)&trans->peerinfo.sockaddr)->sin6_port;
break;
-
+ case AF_UNIX:
+ is_unix = _gf_true;
+ break;
default:
- gf_log (GF_RPCSVC, GF_LOG_DEBUG,
+ gf_log (GF_RPCSVC, GF_LOG_ERROR,
"invalid address family (%d)",
trans->peerinfo.sockaddr.ss_family);
return -1;
@@ -953,19 +496,18 @@ rpcsvc_handle_rpc_call (rpcsvc_t *svc, rpc_transport_t *trans,
- port = ntohs (port);
+ if (is_unix == _gf_false) {
+ port = ntohs (port);
- gf_log ("rpcsvc", GF_LOG_TRACE, "Client port: %d", (int)port);
+ gf_log ("rpcsvc", GF_LOG_TRACE, "Client port: %d", (int)port);
- if (port > 1024) { //Non-privilaged user, fail request
- gf_log ("glusterd", GF_LOG_ERROR, "Request received from non-"
- "privileged port. Failing request");
- return -1;
+ if (port > 1024)
+ unprivileged = _gf_true;
}
req = rpcsvc_request_create (svc, trans, msg);
if (!req)
- goto err;
+ goto out;
if (!rpcsvc_request_accepted (req))
goto err_reply;
@@ -974,38 +516,85 @@ rpcsvc_handle_rpc_call (rpcsvc_t *svc, rpc_transport_t *trans,
if (!actor)
goto err_reply;
- if (actor && (req->rpc_err == SUCCESS)) {
+ if (0 == svc->allow_insecure && unprivileged && !actor->unprivileged) {
+ /* Non-privileged user, fail request */
+ gf_log ("glusterd", GF_LOG_ERROR,
+ "Request received from non-"
+ "privileged port. Failing request");
+ rpcsvc_request_destroy (req);
+ return -1;
+ }
+
+ /* DRC */
+ if (rpcsvc_need_drc (req)) {
+ drc = req->svc->drc;
+
+ LOCK (&drc->lock);
+ reply = rpcsvc_drc_lookup (req);
+
+ /* retransmission of completed request, send cached reply */
+ if (reply && reply->state == DRC_OP_CACHED) {
+ gf_log (GF_RPCSVC, GF_LOG_INFO, "duplicate request:"
+ " XID: 0x%x", req->xid);
+ ret = rpcsvc_send_cached_reply (req, reply);
+ drc->cache_hits++;
+ UNLOCK (&drc->lock);
+ goto out;
+
+ } /* retransmitted request, original op in transit, drop it */
+ else if (reply && reply->state == DRC_OP_IN_TRANSIT) {
+ gf_log (GF_RPCSVC, GF_LOG_INFO, "op in transit,"
+ " discarding. XID: 0x%x", req->xid);
+ ret = 0;
+ drc->intransit_hits++;
+ rpcsvc_request_destroy (req);
+ UNLOCK (&drc->lock);
+ goto out;
+
+ } /* fresh request, cache it as in-transit and proceed */
+ else {
+ ret = rpcsvc_cache_request (req);
+ }
+ UNLOCK (&drc->lock);
+ }
+
+ if (req->rpc_err == SUCCESS) {
/* Before going to xlator code, set the THIS properly */
THIS = svc->mydata;
- if (req->count == 2) {
- if (actor->vector_actor) {
- ret = actor->vector_actor (req, &req->msg[1], 1,
- req->iobref);
- } else {
- rpcsvc_request_seterr (req, PROC_UNAVAIL);
- gf_log (GF_RPCSVC, GF_LOG_ERROR,
- "No vectored handler present");
- ret = RPCSVC_ACTOR_ERROR;
- }
- } else if (actor->actor) {
- ret = actor->actor (req);
+ actor_fn = actor->actor;
+
+ if (!actor_fn) {
+ rpcsvc_request_seterr (req, PROC_UNAVAIL);
+ /* LOG TODO: print more info about procnum,
+ prognum etc, also print transport info */
+ gf_log (GF_RPCSVC, GF_LOG_ERROR,
+ "No vectored handler present");
+ ret = RPCSVC_ACTOR_ERROR;
+ goto err_reply;
}
- }
-err_reply:
- if (ret == RPCSVC_ACTOR_ERROR) {
- ret = rpcsvc_error_reply (req);
+ if (req->synctask) {
+ if (msg->hdr_iobuf)
+ req->hdr_iobuf = iobuf_ref (msg->hdr_iobuf);
+
+ ret = synctask_new (THIS->ctx->env,
+ (synctask_fn_t) actor_fn,
+ rpcsvc_check_and_reply_error, NULL,
+ req);
+ } else {
+ ret = actor_fn (req);
+ }
}
- if (ret)
- gf_log ("rpcsvc", GF_LOG_DEBUG, "failed to queue error reply");
+err_reply:
+ ret = rpcsvc_check_and_reply_error (ret, NULL, req);
/* No need to propagate error beyond this function since the reply
* has now been queued. */
ret = 0;
-err:
+out:
return ret;
}
@@ -1017,46 +606,49 @@ rpcsvc_handle_disconnect (rpcsvc_t *svc, rpc_transport_t *trans)
rpcsvc_notify_wrapper_t *wrappers = NULL, *wrapper;
int32_t ret = -1, i = 0, wrapper_count = 0;
rpcsvc_listener_t *listener = NULL;
-
+
event = (trans->listener == NULL) ? RPCSVC_EVENT_LISTENER_DEAD
: RPCSVC_EVENT_DISCONNECT;
-
+
pthread_mutex_lock (&svc->rpclock);
{
+ if (!svc->notify_count)
+ goto unlock;
+
wrappers = GF_CALLOC (svc->notify_count, sizeof (*wrapper),
gf_common_mt_rpcsvc_wrapper_t);
if (!wrappers) {
goto unlock;
}
-
+
list_for_each_entry (wrapper, &svc->notify, list) {
if (wrapper->notify) {
wrappers[i++] = *wrapper;
}
}
-
+
wrapper_count = i;
}
unlock:
pthread_mutex_unlock (&svc->rpclock);
-
+
if (wrappers) {
for (i = 0; i < wrapper_count; i++) {
wrappers[i].notify (svc, wrappers[i].data,
event, trans);
}
-
+
GF_FREE (wrappers);
}
-
+
if (event == RPCSVC_EVENT_LISTENER_DEAD) {
listener = rpcsvc_get_listener (svc, -1, trans->listener);
rpcsvc_listener_destroy (listener);
}
-
+
return ret;
}
-
+
int
rpcsvc_notify (rpc_transport_t *trans, void *mydata,
@@ -1066,6 +658,7 @@ rpcsvc_notify (rpc_transport_t *trans, void *mydata,
rpc_transport_pollin_t *msg = NULL;
rpc_transport_t *new_trans = NULL;
rpcsvc_t *svc = NULL;
+ rpcsvc_listener_t *listener = NULL;
svc = mydata;
if (svc == NULL) {
@@ -1095,13 +688,20 @@ rpcsvc_notify (rpc_transport_t *trans, void *mydata,
/* do nothing, no need for rpcsvc to handle this, client should
* handle this event
*/
+ /* print info about transport too : LOG TODO */
gf_log ("rpcsvc", GF_LOG_CRITICAL,
"got CONNECT event, which should have not come");
ret = 0;
break;
case RPC_TRANSPORT_CLEANUP:
- /* FIXME: think about this later */
+ listener = rpcsvc_get_listener (svc, -1, trans->listener);
+ if (listener == NULL) {
+ goto out;
+ }
+
+ rpcsvc_program_notify (listener, RPCSVC_EVENT_TRANSPORT_DESTROY,
+ trans);
ret = 0;
break;
@@ -1135,7 +735,7 @@ rpcsvc_record_build_header (char *recordstart, size_t rlen,
*/
ret = rpc_reply_to_xdr (&reply, recordstart, rlen, &replyhdr);
if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to create RPC reply");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING, "Failed to create RPC reply");
goto err;
}
@@ -1155,7 +755,7 @@ err:
return txrecord;
}
-inline int
+static inline int
rpcsvc_get_callid (rpcsvc_t *rpc)
{
return GF_UNIVERSAL_ANSWER;
@@ -1206,7 +806,7 @@ rpcsvc_callback_build_header (char *recordstart, size_t rlen,
ret = rpc_request_to_xdr (request, recordstart, rlen, &requesthdr);
if (ret == -1) {
- gf_log ("rpcsvc", GF_LOG_DEBUG,
+ gf_log ("rpcsvc", GF_LOG_WARNING,
"Failed to create RPC request");
goto out;
}
@@ -1238,34 +838,36 @@ rpcsvc_callback_build_record (rpcsvc_t *rpc, int prognum, int progver,
char *record = NULL;
struct iovec recordhdr = {0, };
size_t pagesize = 0;
+ size_t xdr_size = 0;
int ret = -1;
if ((!rpc) || (!recbuf)) {
goto out;
}
+ /* Fill the rpc structure and XDR it into the buffer got above. */
+ ret = rpcsvc_fill_callback (prognum, progver, procnum, payload, xid,
+ &request);
+ if (ret == -1) {
+ gf_log ("rpcsvc", GF_LOG_WARNING, "cannot build a rpc-request "
+ "xid (%"PRIu64")", xid);
+ goto out;
+ }
+
/* First, try to get a pointer into the buffer which the RPC
* layer can use.
*/
- request_iob = iobuf_get (rpc->ctx->iobuf_pool);
+ xdr_size = xdr_sizeof ((xdrproc_t)xdr_callmsg, &request);
+
+ request_iob = iobuf_get2 (rpc->ctx->iobuf_pool, (xdr_size + payload));
if (!request_iob) {
- gf_log ("rpcsvc", GF_LOG_ERROR, "Failed to get iobuf");
goto out;
}
- pagesize = ((struct iobuf_pool *)rpc->ctx->iobuf_pool)->page_size;
+ pagesize = iobuf_pagesize (request_iob);
record = iobuf_ptr (request_iob); /* Now we have it. */
- /* Fill the rpc structure and XDR it into the buffer got above. */
- ret = rpcsvc_fill_callback (prognum, progver, procnum, payload, xid,
- &request);
- if (ret == -1) {
- gf_log ("rpcsvc", GF_LOG_DEBUG, "cannot build a rpc-request "
- "xid (%"PRIu64")", xid);
- goto out;
- }
-
recordhdr = rpcsvc_callback_build_header (record, pagesize, &request,
payload);
@@ -1314,7 +916,7 @@ rpcsvc_callback_submit (rpcsvc_t *rpc, rpc_transport_t *trans,
proglen, callid,
&rpchdr);
if (!request_iob) {
- gf_log ("rpcsvc", GF_LOG_DEBUG,
+ gf_log ("rpcsvc", GF_LOG_WARNING,
"cannot build rpc-record");
goto out;
}
@@ -1326,7 +928,7 @@ rpcsvc_callback_submit (rpcsvc_t *rpc, rpc_transport_t *trans,
ret = rpc_transport_submit_request (trans, &req);
if (ret == -1) {
- gf_log ("rpc-clnt", GF_LOG_DEBUG,
+ gf_log ("rpcsvc", GF_LOG_WARNING,
"transmission of rpc-request failed");
goto out;
}
@@ -1339,21 +941,22 @@ out:
return ret;
}
-inline int
-rpcsvc_transport_submit (rpc_transport_t *trans, struct iovec *hdrvec,
- int hdrcount, struct iovec *proghdr, int proghdrcount,
- struct iovec *progpayload, int progpayloadcount,
- struct iobref *iobref, void *priv)
+int
+rpcsvc_transport_submit (rpc_transport_t *trans, struct iovec *rpchdr,
+ int rpchdrcount, struct iovec *proghdr,
+ int proghdrcount, struct iovec *progpayload,
+ int progpayloadcount, struct iobref *iobref,
+ void *priv)
{
int ret = -1;
rpc_transport_reply_t reply = {{0, }};
- if ((!trans) || (!hdrvec) || (!hdrvec->iov_base)) {
+ if ((!trans) || (!rpchdr) || (!rpchdr->iov_base)) {
goto out;
}
- reply.msg.rpchdr = hdrvec;
- reply.msg.rpchdrcount = hdrcount;
+ reply.msg.rpchdr = rpchdr;
+ reply.msg.rpchdrcount = rpchdrcount;
reply.msg.proghdr = proghdr;
reply.msg.proghdrcount = proghdrcount;
reply.msg.progpayload = progpayload;
@@ -1371,27 +974,31 @@ out:
int
rpcsvc_fill_reply (rpcsvc_request_t *req, struct rpc_msg *reply)
{
+ int ret = -1;
rpcsvc_program_t *prog = NULL;
if ((!req) || (!reply))
- return -1;
-
- prog = rpcsvc_request_program (req);
- if (!prog)
- return -1;
+ goto out;
+ ret = 0;
rpc_fill_empty_reply (reply, req->xid);
-
- if (req->rpc_status == MSG_DENIED)
+ if (req->rpc_status == MSG_DENIED) {
rpc_fill_denied_reply (reply, req->rpc_err, req->auth_err);
- else if (req->rpc_status == MSG_ACCEPTED)
- rpc_fill_accepted_reply (reply, req->rpc_err, prog->proglowvers,
- prog->proghighvers, req->verf.flavour,
- req->verf.datalen,
+ goto out;
+ }
+
+ prog = rpcsvc_request_program (req);
+
+ if (req->rpc_status == MSG_ACCEPTED)
+ rpc_fill_accepted_reply (reply, req->rpc_err,
+ (prog) ? prog->proglowvers : 0,
+ (prog) ? prog->proghighvers: 0,
+ req->verf.flavour, req->verf.datalen,
req->verf.authdata);
else
gf_log (GF_RPCSVC, GF_LOG_ERROR, "Invalid rpc_status value");
- return 0;
+out:
+ return ret;
}
@@ -1405,13 +1012,14 @@ rpcsvc_fill_reply (rpcsvc_request_t *req, struct rpc_msg *reply)
*/
struct iobuf *
rpcsvc_record_build_record (rpcsvc_request_t *req, size_t payload,
- struct iovec *recbuf)
+ size_t hdrlen, struct iovec *recbuf)
{
struct rpc_msg reply;
struct iobuf *replyiob = NULL;
char *record = NULL;
struct iovec recordhdr = {0, };
size_t pagesize = 0;
+ size_t xdr_size = 0;
rpcsvc_t *svc = NULL;
int ret = -1;
@@ -1419,20 +1027,25 @@ rpcsvc_record_build_record (rpcsvc_request_t *req, size_t payload,
return NULL;
svc = req->svc;
- replyiob = iobuf_get (svc->ctx->iobuf_pool);
- pagesize = iobpool_pagesize ((struct iobuf_pool *)svc->ctx->iobuf_pool);
- if (!replyiob) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get iobuf");
- goto err_exit;
- }
-
- record = iobuf_ptr (replyiob); /* Now we have it. */
/* Fill the rpc structure and XDR it into the buffer got above. */
ret = rpcsvc_fill_reply (req, &reply);
if (ret)
goto err_exit;
+ xdr_size = xdr_sizeof ((xdrproc_t)xdr_replymsg, &reply);
+
+ /* Payload would include 'readv' size etc too, where as
+ that comes as another payload iobuf */
+ replyiob = iobuf_get2 (svc->ctx->iobuf_pool, (xdr_size + hdrlen));
+ if (!replyiob) {
+ goto err_exit;
+ }
+
+ pagesize = iobuf_pagesize (replyiob);
+
+ record = iobuf_ptr (replyiob); /* Now we have it. */
+
recordhdr = rpcsvc_record_build_header (record, pagesize, reply,
payload);
if (!recordhdr.iov_base) {
@@ -1487,7 +1100,9 @@ rpcsvc_submit_generic (rpcsvc_request_t *req, struct iovec *proghdr,
struct iovec recordhdr = {0, };
rpc_transport_t *trans = NULL;
size_t msglen = 0;
+ size_t hdrlen = 0;
char new_iobref = 0;
+ rpcsvc_drc_globals_t *drc = NULL;
if ((!req) || (!req->trans))
return -1;
@@ -1505,7 +1120,7 @@ rpcsvc_submit_generic (rpcsvc_request_t *req, struct iovec *proghdr,
gf_log (GF_RPCSVC, GF_LOG_TRACE, "Tx message: %zu", msglen);
/* Build the buffer containing the encoded RPC reply. */
- replyiob = rpcsvc_record_build_record (req, msglen, &recordhdr);
+ replyiob = rpcsvc_record_build_record (req, msglen, hdrlen, &recordhdr);
if (!replyiob) {
gf_log (GF_RPCSVC, GF_LOG_ERROR,"Reply record creation failed");
goto disconnect_exit;
@@ -1514,8 +1129,6 @@ rpcsvc_submit_generic (rpcsvc_request_t *req, struct iovec *proghdr,
if (!iobref) {
iobref = iobref_new ();
if (!iobref) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "memory allocation "
- "failed");
goto disconnect_exit;
}
@@ -1524,21 +1137,35 @@ rpcsvc_submit_generic (rpcsvc_request_t *req, struct iovec *proghdr,
iobref_add (iobref, replyiob);
+ /* cache the request in the duplicate request cache for appropriate ops */
+ if (req->reply) {
+ drc = req->svc->drc;
+
+ LOCK (&drc->lock);
+ ret = rpcsvc_cache_reply (req, iobref, &recordhdr, 1,
+ proghdr, hdrcount,
+ payload, payloadcount);
+ UNLOCK (&drc->lock);
+ }
+
ret = rpcsvc_transport_submit (trans, &recordhdr, 1, proghdr, hdrcount,
payload, payloadcount, iobref,
req->trans_private);
if (ret == -1) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "failed to submit message "
- "(XID: 0x%lx, Program: %s, ProgVers: %d, Proc: %d) to "
- "rpc-transport (%s)", req->xid, req->prog->progname,
- req->prog->progver, req->procnum, trans->name);
+ "(XID: 0x%x, Program: %s, ProgVers: %d, Proc: %d) to "
+ "rpc-transport (%s)", req->xid,
+ req->prog ? req->prog->progname : "(not matched)",
+ req->prog ? req->prog->progver : 0,
+ req->procnum, trans->name);
} else {
gf_log (GF_RPCSVC, GF_LOG_TRACE,
- "submitted reply for rpc-message (XID: 0x%lx, "
+ "submitted reply for rpc-message (XID: 0x%x, "
"Program: %s, ProgVers: %d, Proc: %d) to rpc-transport "
- "(%s)", req->xid, req->prog->progname,
- req->prog->progver, req->procnum, trans->name);
+ "(%s)", req->xid, req->prog ? req->prog->progname: "-",
+ req->prog ? req->prog->progver : 0,
+ req->procnum, trans->name);
}
disconnect_exit:
@@ -1564,6 +1191,8 @@ rpcsvc_error_reply (rpcsvc_request_t *req)
if (!req)
return -1;
+ gf_log_callingfn ("", GF_LOG_DEBUG, "sending a RPC error reply");
+
/* At this point the req should already have been filled with the
* appropriate RPC error numbers.
*/
@@ -1594,21 +1223,30 @@ out:
}
-inline int
+static inline int
rpcsvc_program_unregister_portmap (rpcsvc_program_t *prog)
{
+ int ret = 0;
+
if (!prog)
- return -1;
+ goto out;
if (!(pmap_unset(prog->prognum, prog->progver))) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "Could not unregister with"
" portmap");
- return -1;
+ goto out;
}
- return 0;
+ ret = 0;
+out:
+ return ret;
}
+int
+rpcsvc_register_portmap_enabled (rpcsvc_t *svc)
+{
+ return svc->register_portmap;
+}
int32_t
rpcsvc_get_listener_port (rpcsvc_listener_t *listener)
@@ -1621,11 +1259,11 @@ rpcsvc_get_listener_port (rpcsvc_listener_t *listener)
switch (listener->trans->myinfo.sockaddr.ss_family) {
case AF_INET:
- listener_port = ((struct sockaddr_in6 *)&listener->trans->myinfo.sockaddr)->sin6_port;
+ listener_port = ((struct sockaddr_in *)&listener->trans->myinfo.sockaddr)->sin_port;
break;
case AF_INET6:
- listener_port = ((struct sockaddr_in *)&listener->trans->myinfo.sockaddr)->sin_port;
+ listener_port = ((struct sockaddr_in6 *)&listener->trans->myinfo.sockaddr)->sin6_port;
break;
default:
@@ -1647,7 +1285,7 @@ rpcsvc_get_listener (rpcsvc_t *svc, uint16_t port, rpc_transport_t *trans)
{
rpcsvc_listener_t *listener = NULL;
char found = 0;
- uint32_t listener_port = 0;
+ uint32_t listener_port = 0;
if (!svc) {
goto out;
@@ -1667,7 +1305,7 @@ rpcsvc_get_listener (rpcsvc_t *svc, uint16_t port, rpc_transport_t *trans)
listener_port = rpcsvc_get_listener_port (listener);
if (listener_port == -1) {
- gf_log (GF_RPCSVC, GF_LOG_DEBUG,
+ gf_log (GF_RPCSVC, GF_LOG_ERROR,
"invalid port for listener %s",
listener->trans->name);
continue;
@@ -1710,28 +1348,44 @@ rpcsvc_submit_message (rpcsvc_request_t *req, struct iovec *proghdr,
int
-rpcsvc_program_unregister (rpcsvc_t *svc, rpcsvc_program_t *prog)
+rpcsvc_program_unregister (rpcsvc_t *svc, rpcsvc_program_t *program)
{
int ret = -1;
-
- if (!svc || !prog) {
+ rpcsvc_program_t *prog = NULL;
+ if (!svc || !program) {
goto out;
}
- ret = rpcsvc_program_unregister_portmap (prog);
+ ret = rpcsvc_program_unregister_portmap (program);
if (ret == -1) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "portmap unregistration of"
" program failed");
goto out;
}
+ pthread_mutex_lock (&svc->rpclock);
+ {
+ list_for_each_entry (prog, &svc->programs, program) {
+ if ((prog->prognum == program->prognum)
+ && (prog->progver == program->progver)) {
+ break;
+ }
+ }
+ }
+ pthread_mutex_unlock (&svc->rpclock);
+
+ if (prog == NULL) {
+ ret = -1;
+ goto out;
+ }
+
gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Program unregistered: %s, Num: %d,"
" Ver: %d, Port: %d", prog->progname, prog->prognum,
prog->progver, prog->progport);
pthread_mutex_lock (&svc->rpclock);
{
- list_del (&prog->program);
+ list_del_init (&prog->program);
}
pthread_mutex_unlock (&svc->rpclock);
@@ -1739,8 +1393,8 @@ rpcsvc_program_unregister (rpcsvc_t *svc, rpcsvc_program_t *prog)
out:
if (ret == -1) {
gf_log (GF_RPCSVC, GF_LOG_ERROR, "Program unregistration failed"
- ": %s, Num: %d, Ver: %d, Port: %d", prog->progname,
- prog->prognum, prog->progver, prog->progport);
+ ": %s, Num: %d, Ver: %d, Port: %d", program->progname,
+ program->prognum, program->progver, program->progport);
}
return ret;
@@ -1779,21 +1433,21 @@ rpcsvc_transport_create (rpcsvc_t *svc, dict_t *options, char *name)
trans = rpc_transport_load (svc->ctx, options, name);
if (!trans) {
- gf_log (GF_RPCSVC, GF_LOG_DEBUG, "cannot create listener, "
+ gf_log (GF_RPCSVC, GF_LOG_WARNING, "cannot create listener, "
"initing the transport failed");
goto out;
}
ret = rpc_transport_listen (trans);
if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_DEBUG,
+ gf_log (GF_RPCSVC, GF_LOG_WARNING,
"listening on transport failed");
goto out;
}
ret = rpc_transport_register_notify (trans, rpcsvc_notify, svc);
if (ret == -1) {
- gf_log (GF_RPCSVC, GF_LOG_DEBUG, "registering notify failed");
+ gf_log (GF_RPCSVC, GF_LOG_WARNING, "registering notify failed");
goto out;
}
@@ -1815,7 +1469,6 @@ rpcsvc_listener_alloc (rpcsvc_t *svc, rpc_transport_t *trans)
listener = GF_CALLOC (1, sizeof (*listener),
gf_common_mt_rpcsvc_listener_t);
if (!listener) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "memory allocation failed");
goto out;
}
@@ -1839,7 +1492,7 @@ rpcsvc_create_listener (rpcsvc_t *svc, dict_t *options, char *name)
{
rpc_transport_t *trans = NULL;
rpcsvc_listener_t *listener = NULL;
- int32_t ret = -1;
+ int32_t ret = -1;
if (!svc || !options) {
goto out;
@@ -1847,6 +1500,7 @@ rpcsvc_create_listener (rpcsvc_t *svc, dict_t *options, char *name)
trans = rpcsvc_transport_create (svc, options, name);
if (!trans) {
+ /* LOG TODO */
goto out;
}
@@ -1894,13 +1548,11 @@ rpcsvc_create_listeners (rpcsvc_t *svc, dict_t *options, char *name)
/* duplicate transport_type, since following dict_set will free it */
transport_type = gf_strdup (transport_type);
if (transport_type == NULL) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "out of memory");
goto out;
}
str = gf_strdup (transport_type);
if (str == NULL) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -1909,7 +1561,6 @@ rpcsvc_create_listeners (rpcsvc_t *svc, dict_t *options, char *name)
while (ptr != NULL) {
tmp = gf_strdup (ptr);
if (tmp == NULL) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -1932,6 +1583,7 @@ rpcsvc_create_listeners (rpcsvc_t *svc, dict_t *options, char *name)
}
GF_FREE (transport_name);
+ transport_name = NULL;
count++;
}
@@ -1943,17 +1595,13 @@ rpcsvc_create_listeners (rpcsvc_t *svc, dict_t *options, char *name)
transport_type = NULL;
out:
- if (str != NULL) {
- GF_FREE (str);
- }
+ GF_FREE (str);
- if (transport_type != NULL) {
- GF_FREE (transport_type);
- }
+ GF_FREE (transport_type);
- if (tmp != NULL) {
- GF_FREE (tmp);
- }
+ GF_FREE (tmp);
+
+ GF_FREE (transport_name);
return count;
}
@@ -2047,7 +1695,6 @@ rpcsvc_program_register (rpcsvc_t *svc, rpcsvc_program_t *program)
newprog = GF_CALLOC (1, sizeof(*newprog),gf_common_mt_rpcsvc_program_t);
if (newprog == NULL) {
- gf_log (GF_RPCSVC, GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -2076,7 +1723,6 @@ out:
return ret;
}
-
static void
free_prog_details (gf_dump_rsp *rsp)
{
@@ -2125,38 +1771,41 @@ static int
rpcsvc_dump (rpcsvc_request_t *req)
{
char rsp_buf[8 * 1024] = {0,};
- gf_dump_rsp rsp = {0,};
- struct iovec iov = {0,};
- int op_errno = EINVAL;
- int ret = -1;
+ gf_dump_rsp rsp = {0,};
+ struct iovec iov = {0,};
+ int op_errno = EINVAL;
+ int ret = -1;
+ uint32_t dump_rsp_len = 0;
if (!req)
- goto fail;
+ goto sendrsp;
ret = build_prog_details (req, &rsp);
if (ret < 0) {
op_errno = -ret;
- goto fail;
+ goto sendrsp;
}
-fail:
+ op_errno = 0;
+
+sendrsp:
rsp.op_errno = gf_errno_to_error (op_errno);
rsp.op_ret = ret;
+ dump_rsp_len = xdr_sizeof ((xdrproc_t) xdr_gf_dump_rsp,
+ &rsp);
+
iov.iov_base = rsp_buf;
- iov.iov_len = (8 * 1024);
+ iov.iov_len = dump_rsp_len;
- ret = xdr_serialize_dump_rsp (iov, &rsp);
+ ret = xdr_serialize_generic (iov, &rsp, (xdrproc_t)xdr_gf_dump_rsp);
if (ret < 0) {
- if (req)
- req->rpc_err = GARBAGE_ARGS;
- op_errno = EINVAL;
- goto fail;
+ ret = RPCSVC_ACTOR_ERROR;
+ } else {
+ rpcsvc_submit_generic (req, &iov, 1, NULL, 0, NULL);
+ ret = 0;
}
- ret = rpcsvc_submit_generic (req, &iov, 1, NULL, 0,
- NULL);
-
free_prog_details (&rsp);
return ret;
@@ -2165,18 +1814,95 @@ fail:
int
rpcsvc_init_options (rpcsvc_t *svc, dict_t *options)
{
+ char *optstr = NULL;
+ int ret = -1;
+
+ if ((!svc) || (!options))
+ return -1;
+
svc->memfactor = RPCSVC_DEFAULT_MEMFACTOR;
- return 0;
+
+ svc->register_portmap = _gf_true;
+ if (dict_get (options, "rpc.register-with-portmap")) {
+ ret = dict_get_str (options, "rpc.register-with-portmap",
+ &optstr);
+ if (ret < 0) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to parse "
+ "dict");
+ goto out;
+ }
+
+ ret = gf_string2boolean (optstr, &svc->register_portmap);
+ if (ret < 0) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to parse bool "
+ "string");
+ goto out;
+ }
+ }
+
+ if (!svc->register_portmap)
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Portmap registration "
+ "disabled");
+
+ ret = 0;
+out:
+ return ret;
}
+int
+rpcsvc_transport_unix_options_build (dict_t **options, char *filepath)
+{
+ dict_t *dict = NULL;
+ char *fpath = NULL;
+ int ret = -1;
+
+ GF_ASSERT (filepath);
+ GF_ASSERT (options);
+
+ dict = dict_new ();
+ if (!dict)
+ goto out;
+
+ fpath = gf_strdup (filepath);
+ if (!fpath) {
+ ret = -1;
+ goto out;
+ }
+
+ ret = dict_set_dynstr (dict, "transport.socket.listen-path", fpath);
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport.address-family", "unix");
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport.socket.nodelay", "off");
+ if (ret)
+ goto out;
+
+ ret = dict_set_str (dict, "transport-type", "socket");
+ if (ret)
+ goto out;
+
+ *options = dict;
+out:
+ if (ret) {
+ GF_FREE (fpath);
+ if (dict)
+ dict_unref (dict);
+ }
+ return ret;
+}
/* The global RPC service initializer.
*/
rpcsvc_t *
-rpcsvc_init (glusterfs_ctx_t *ctx, dict_t *options)
+rpcsvc_init (xlator_t *xl, glusterfs_ctx_t *ctx, dict_t *options,
+ uint32_t poolcount)
{
rpcsvc_t *svc = NULL;
- int ret = -1, poolcount = 0;
+ int ret = -1;
if ((!ctx) || (!options))
return NULL;
@@ -2197,7 +1923,8 @@ rpcsvc_init (glusterfs_ctx_t *ctx, dict_t *options)
goto free_svc;
}
- poolcount = RPCSVC_POOLCOUNT_MULT * svc->memfactor;
+ if (!poolcount)
+ poolcount = RPCSVC_POOLCOUNT_MULT * svc->memfactor;
gf_log (GF_RPCSVC, GF_LOG_TRACE, "rx pool: %d", poolcount);
svc->rxpool = mem_pool_new (rpcsvc_request_t, poolcount);
@@ -2217,6 +1944,7 @@ rpcsvc_init (glusterfs_ctx_t *ctx, dict_t *options)
ret = -1;
svc->options = options;
svc->ctx = ctx;
+ svc->mydata = xl;
gf_log (GF_RPCSVC, GF_LOG_DEBUG, "RPC service inited.");
gluster_dump_prog.options = options;
@@ -2227,6 +1955,7 @@ rpcsvc_init (glusterfs_ctx_t *ctx, dict_t *options)
"failed to register DUMP program");
goto free_svc;
}
+
ret = 0;
free_svc:
if (ret == -1) {
@@ -2238,10 +1967,334 @@ free_svc:
}
+int
+rpcsvc_transport_peer_check_search (dict_t *options, char *pattern,
+ char *ip, char *hostname)
+{
+ int ret = -1;
+ char *addrtok = NULL;
+ char *addrstr = NULL;
+ char *dup_addrstr = NULL;
+ char *svptr = NULL;
+
+ if ((!options) || (!ip))
+ return -1;
+
+ ret = dict_get_str (options, pattern, &addrstr);
+ if (ret < 0) {
+ ret = -1;
+ goto err;
+ }
+
+ if (!addrstr) {
+ ret = -1;
+ goto err;
+ }
+
+ dup_addrstr = gf_strdup (addrstr);
+ addrtok = strtok_r (dup_addrstr, ",", &svptr);
+ while (addrtok) {
+
+ /* CASEFOLD not present on Solaris */
+#ifdef FNM_CASEFOLD
+ ret = fnmatch (addrtok, ip, FNM_CASEFOLD);
+#else
+ ret = fnmatch (addrtok, ip, 0);
+#endif
+ if (ret == 0)
+ goto err;
+
+ /* compare hostnames if applicable */
+ if (hostname) {
+#ifdef FNM_CASEFOLD
+ ret = fnmatch (addrtok, hostname, FNM_CASEFOLD);
+#else
+ ret = fnmatch (addrtok, hostname, 0);
+#endif
+ if (ret == 0)
+ goto err;
+ }
+
+ addrtok = strtok_r (NULL, ",", &svptr);
+ }
+
+ ret = -1;
+err:
+ GF_FREE (dup_addrstr);
+
+ return ret;
+}
+
+
+int
+rpcsvc_transport_peer_check_allow (dict_t *options, char *volname,
+ char *ip, char *hostname)
+{
+ int ret = RPCSVC_AUTH_DONTCARE;
+ char *srchstr = NULL;
+
+ if ((!options) || (!ip) || (!volname))
+ return ret;
+
+ ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.allow", volname);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
+ ret = RPCSVC_AUTH_DONTCARE;
+ goto out;
+ }
+
+ ret = rpcsvc_transport_peer_check_search (options, srchstr,
+ ip, hostname);
+ GF_FREE (srchstr);
+
+ if (ret == 0)
+ ret = RPCSVC_AUTH_ACCEPT;
+ else
+ ret = RPCSVC_AUTH_REJECT;
+out:
+ return ret;
+}
+
+int
+rpcsvc_transport_peer_check_reject (dict_t *options, char *volname,
+ char *ip, char *hostname)
+{
+ int ret = RPCSVC_AUTH_DONTCARE;
+ char *srchstr = NULL;
+
+ if ((!options) || (!ip) || (!volname))
+ return ret;
+
+ ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.reject",
+ volname);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
+ ret = RPCSVC_AUTH_REJECT;
+ goto out;
+ }
+
+ ret = rpcsvc_transport_peer_check_search (options, srchstr,
+ ip, hostname);
+ GF_FREE (srchstr);
+
+ if (ret == 0)
+ ret = RPCSVC_AUTH_REJECT;
+ else
+ ret = RPCSVC_AUTH_DONTCARE;
+out:
+ return ret;
+}
+
+
+/* Combines rpc auth's allow and reject options.
+ * Order of checks is important.
+ * First, REJECT if either rejects.
+ * If neither rejects, ACCEPT if either accepts.
+ * If neither accepts, DONTCARE
+ */
+int
+rpcsvc_combine_allow_reject_volume_check (int allow, int reject)
+{
+ if (allow == RPCSVC_AUTH_REJECT ||
+ reject == RPCSVC_AUTH_REJECT)
+ return RPCSVC_AUTH_REJECT;
+
+ if (allow == RPCSVC_AUTH_ACCEPT ||
+ reject == RPCSVC_AUTH_ACCEPT)
+ return RPCSVC_AUTH_ACCEPT;
+
+ return RPCSVC_AUTH_DONTCARE;
+}
+
+int
+rpcsvc_auth_check (dict_t *options, char *volname,
+ rpc_transport_t *trans)
+{
+ int ret = RPCSVC_AUTH_REJECT;
+ int accept = RPCSVC_AUTH_REJECT;
+ int reject = RPCSVC_AUTH_REJECT;
+ char *hostname = NULL;
+ char *ip = NULL;
+ char client_ip[RPCSVC_PEER_STRLEN] = {0};
+ char *allow_str = NULL;
+ char *reject_str = NULL;
+ char *srchstr = NULL;
+
+ if (!options || !volname || !trans)
+ return ret;
+
+ ret = rpcsvc_transport_peername (trans, client_ip, RPCSVC_PEER_STRLEN);
+ if (ret != 0) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get remote addr: "
+ "%s", gai_strerror (ret));
+ return RPCSVC_AUTH_REJECT;
+ }
+
+ /* Accept if its the default case: Allow all, Reject none
+ * The default volfile always contains a 'allow *' rule
+ * for each volume. If allow rule is missing (which implies
+ * there is some bad volfile generating code doing this), we
+ * assume no one is allowed mounts, and thus, we reject mounts.
+ */
+ ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.allow", volname);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
+ return RPCSVC_AUTH_REJECT;
+ }
+
+ ret = dict_get_str (options, srchstr, &allow_str);
+ GF_FREE (srchstr);
+ if (ret < 0)
+ return RPCSVC_AUTH_REJECT;
+
+ ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.reject", volname);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
+ return RPCSVC_AUTH_REJECT;
+ }
+
+ ret = dict_get_str (options, srchstr, &reject_str);
+ GF_FREE (srchstr);
+ if (reject_str == NULL && !strcmp ("*", allow_str))
+ return RPCSVC_AUTH_ACCEPT;
+
+ /* Non-default rule, authenticate */
+ if (!get_host_name (client_ip, &ip))
+ ip = client_ip;
+
+ /* addr-namelookup disabled by default */
+ ret = dict_get_str_boolean (options, "rpc-auth.addr.namelookup", 0);
+ if (ret == _gf_true) {
+ ret = gf_get_hostname_from_ip (ip, &hostname);
+ if (ret) {
+ if (hostname)
+ GF_FREE (hostname);
+ /* failed to get hostname, but hostname auth
+ * is enabled, so authentication will not be
+ * 100% correct. reject mounts
+ */
+ return RPCSVC_AUTH_REJECT;
+ }
+ }
+
+ accept = rpcsvc_transport_peer_check_allow (options, volname,
+ ip, hostname);
+
+ reject = rpcsvc_transport_peer_check_reject (options, volname,
+ ip, hostname);
+
+ if (hostname)
+ GF_FREE (hostname);
+ return rpcsvc_combine_allow_reject_volume_check (accept, reject);
+}
+
+int
+rpcsvc_transport_privport_check (rpcsvc_t *svc, char *volname,
+ rpc_transport_t *trans)
+{
+ union gf_sock_union sock_union;
+ int ret = RPCSVC_AUTH_REJECT;
+ socklen_t sinsize = sizeof (&sock_union.sin);
+ char *srchstr = NULL;
+ char *valstr = NULL;
+ uint16_t port = 0;
+ gf_boolean_t insecure = _gf_false;
+
+ memset (&sock_union, 0, sizeof (sock_union));
+
+ if ((!svc) || (!volname) || (!trans))
+ return ret;
+
+ ret = rpcsvc_transport_peeraddr (trans, NULL, 0, &sock_union.storage,
+ sinsize);
+ if (ret != 0) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to get peer addr: %s",
+ gai_strerror (ret));
+ ret = RPCSVC_AUTH_REJECT;
+ goto err;
+ }
+
+ port = ntohs (sock_union.sin.sin_port);
+ gf_log (GF_RPCSVC, GF_LOG_TRACE, "Client port: %d", (int)port);
+ /* If the port is already a privileged one, dont bother with checking
+ * options.
+ */
+ if (port <= 1024) {
+ ret = RPCSVC_AUTH_ACCEPT;
+ goto err;
+ }
+
+ /* Disabled by default */
+ ret = gf_asprintf (&srchstr, "rpc-auth.ports.%s.insecure", volname);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
+ ret = RPCSVC_AUTH_REJECT;
+ goto err;
+ }
+
+ ret = dict_get_str (svc->options, srchstr, &valstr);
+ if (ret) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to"
+ " read rpc-auth.ports.insecure value");
+ goto err;
+ }
+
+ ret = gf_string2boolean (valstr, &insecure);
+ if (ret) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "Failed to"
+ " convert rpc-auth.ports.insecure value");
+ goto err;
+ }
+
+ ret = insecure ? RPCSVC_AUTH_ACCEPT : RPCSVC_AUTH_REJECT;
+
+ if (ret == RPCSVC_AUTH_ACCEPT)
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Unprivileged port allowed");
+ else
+ gf_log (GF_RPCSVC, GF_LOG_DEBUG, "Unprivileged port not"
+ " allowed");
+
+err:
+ if (srchstr)
+ GF_FREE (srchstr);
+
+ return ret;
+}
+
+
+char *
+rpcsvc_volume_allowed (dict_t *options, char *volname)
+{
+ char globalrule[] = "rpc-auth.addr.allow";
+ char *srchstr = NULL;
+ char *addrstr = NULL;
+ int ret = -1;
+
+ if ((!options) || (!volname))
+ return NULL;
+
+ ret = gf_asprintf (&srchstr, "rpc-auth.addr.%s.allow", volname);
+ if (ret == -1) {
+ gf_log (GF_RPCSVC, GF_LOG_ERROR, "asprintf failed");
+ goto out;
+ }
+
+ if (!dict_get (options, srchstr))
+ ret = dict_get_str (options, globalrule, &addrstr);
+ else
+ ret = dict_get_str (options, srchstr, &addrstr);
+
+out:
+ GF_FREE (srchstr);
+
+ return addrstr;
+}
+
+
rpcsvc_actor_t gluster_dump_actors[] = {
- [GF_DUMP_NULL] = {"NULL", GF_DUMP_NULL, NULL, NULL, NULL },
- [GF_DUMP_DUMP] = {"DUMP", GF_DUMP_DUMP, rpcsvc_dump, NULL, NULL },
- [GF_DUMP_MAXVALUE] = {"MAXVALUE", GF_DUMP_MAXVALUE, NULL, NULL, NULL },
+ [GF_DUMP_NULL] = {"NULL", GF_DUMP_NULL, NULL, NULL, 0, DRC_NA},
+ [GF_DUMP_DUMP] = {"DUMP", GF_DUMP_DUMP, rpcsvc_dump, NULL, 0, DRC_NA},
+ [GF_DUMP_MAXVALUE] = {"MAXVALUE", GF_DUMP_MAXVALUE, NULL, NULL, 0, DRC_NA},
};
diff --git a/rpc/rpc-lib/src/rpcsvc.h b/rpc/rpc-lib/src/rpcsvc.h
index fca7d047..67ff74be 100644
--- a/rpc/rpc-lib/src/rpcsvc.h
+++ b/rpc/rpc-lib/src/rpcsvc.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _RPCSVC_H
@@ -34,6 +25,7 @@
#include "iobuf.h"
#include "xdr-rpc.h"
#include "glusterfs.h"
+#include "xlator.h"
#include "rpcsvc-common.h"
#include <pthread.h>
@@ -42,20 +34,24 @@
#include <rpc/rpc_msg.h>
#include "compat.h"
-#ifndef NGRPS
-#define NGRPS 16
-#endif /* !NGRPS */
+#ifndef MAX_IOVEC
+#define MAX_IOVEC 16
+#endif
#define GF_RPCSVC "rpc-service"
#define RPCSVC_THREAD_STACK_SIZE ((size_t)(1024 * GF_UNIT_KB))
#define RPCSVC_FRAGHDR_SIZE 4 /* 4-byte RPC fragment header size */
-#define RPCSVC_DEFAULT_LISTEN_PORT 6996
-#define RPCSVC_DEFAULT_MEMFACTOR 15
+#define RPCSVC_DEFAULT_LISTEN_PORT GF_DEFAULT_BASE_PORT
+#define RPCSVC_DEFAULT_MEMFACTOR 8
#define RPCSVC_EVENTPOOL_SIZE_MULT 1024
-#define RPCSVC_POOLCOUNT_MULT 35
+#define RPCSVC_POOLCOUNT_MULT 64
#define RPCSVC_CONN_READ (128 * GF_UNIT_KB)
#define RPCSVC_PAGE_SIZE (128 * GF_UNIT_KB)
+#define RPC_ROOT_UID 0
+#define RPC_ROOT_GID 0
+#define RPC_NOBODY_UID 65534
+#define RPC_NOBODY_GID 65534
/* RPC Record States */
#define RPCSVC_READ_FRAGHDR 1
@@ -110,8 +106,6 @@
#define AUTH_KERB 4 /* kerberos style */
#endif /* */
-#define AUTH_GLUSTERFS 5
-
typedef struct rpcsvc_program rpcsvc_program_t;
struct rpcsvc_notify_wrapper {
@@ -138,15 +132,17 @@ struct rpcsvc_config {
int max_block_size;
};
-#define RPCSVC_MAX_AUTH_BYTES 400
typedef struct rpcsvc_auth_data {
int flavour;
int datalen;
- char authdata[RPCSVC_MAX_AUTH_BYTES];
+ char authdata[GF_MAX_AUTH_BYTES];
} rpcsvc_auth_data_t;
#define rpcsvc_auth_flavour(au) ((au).flavour)
+typedef struct drc_client drc_client_t;
+typedef struct drc_cached_op drc_cached_op_t;
+
/* The container for the RPC call handed up to an actor.
* Dynamically allocated. Lives till the call reply is completely
* transmitted.
@@ -179,13 +175,13 @@ struct rpcsvc_request {
gid_t gid;
pid_t pid;
- uint64_t lk_owner;
+ gf_lkowner_t lk_owner;
uint64_t gfs_id;
- /* Might want to move this to AUTH_UNIX specifix state since this array
- * is not available for every authenticatino scheme.
+ /* Might want to move this to AUTH_UNIX specific state since this array
+ * is not available for every authentication scheme.
*/
- gid_t auxgids[NGRPS];
+ gid_t auxgids[GF_MAX_AUX_GROUPS];
int auxgidcount;
@@ -193,7 +189,7 @@ struct rpcsvc_request {
* by the program actors. This is the buffer that will need to
* be de-xdred by the actor.
*/
- struct iovec msg[2];
+ struct iovec msg[MAX_IOVEC];
int count;
struct iobref *iobref;
@@ -212,8 +208,8 @@ struct rpcsvc_request {
int auth_err;
/* There can be cases of RPC requests where the reply needs to
- * be built from multiple sources. For eg. where even the NFS reply can
- * contain a payload, as in the NFSv3 read reply. Here the RPC header
+ * be built from multiple sources. E.g. where even the NFS reply
+ * can contain a payload, as in the NFSv3 read reply. Here the RPC header
* ,NFS header and the read data are brought together separately from
* different buffers, so we need to stage the buffers temporarily here
* before all of them get added to the connection's transmission list.
@@ -235,6 +231,9 @@ struct rpcsvc_request {
*/
rpcsvc_auth_data_t verf;
+ /* Execute this request's actor function as a synctask? */
+ gf_boolean_t synctask;
+
/* Container for a RPC program wanting to store a temp
* request-specific item.
*/
@@ -242,29 +241,56 @@ struct rpcsvc_request {
/* Container for transport to store request-specific item */
void *trans_private;
+
+ /* we need to ref the 'iobuf' in case of 'synctasking' it */
+ struct iobuf *hdr_iobuf;
+
+ /* pointer to cached reply for use in DRC */
+ drc_cached_op_t *reply;
};
#define rpcsvc_request_program(req) ((rpcsvc_program_t *)((req)->prog))
-#define rpcsvc_request_program_private(req) (((rpcsvc_program_t *)((req)->program))->private)
+#define rpcsvc_request_procnum(req) (((req)->procnum))
+#define rpcsvc_request_program_private(req) (((rpcsvc_program_t *)((req)->prog))->private)
#define rpcsvc_request_accepted(req) ((req)->rpc_status == MSG_ACCEPTED)
#define rpcsvc_request_accepted_success(req) ((req)->rpc_err == SUCCESS)
-#define rpcsvc_request_uid(req) ((req)->uid)
-#define rpcsvc_request_gid(req) ((req)->gid)
#define rpcsvc_request_prog_minauth(req) (rpcsvc_request_program(req)->min_auth)
#define rpcsvc_request_cred_flavour(req) (rpcsvc_auth_flavour(req->cred))
#define rpcsvc_request_verf_flavour(req) (rpcsvc_auth_flavour(req->verf))
-
+#define rpcsvc_request_service(req) ((req)->svc)
#define rpcsvc_request_uid(req) ((req)->uid)
#define rpcsvc_request_gid(req) ((req)->gid)
#define rpcsvc_request_private(req) ((req)->private)
#define rpcsvc_request_xid(req) ((req)->xid)
#define rpcsvc_request_set_private(req,prv) (req)->private = (void *)(prv)
+#define rpcsvc_request_iobref_ref(req) (iobref_ref ((req)->iobref))
#define rpcsvc_request_record_ref(req) (iobuf_ref ((req)->recordiob))
#define rpcsvc_request_record_unref(req) (iobuf_unref ((req)->recordiob))
-
+#define rpcsvc_request_record_iob(req) ((req)->recordiob)
+#define rpcsvc_request_set_vecstate(req, state) ((req)->vecstate = state)
+#define rpcsvc_request_vecstate(req) ((req)->vecstate)
+#define rpcsvc_request_transport(req) ((req)->trans)
+#define rpcsvc_request_transport_ref(req) (rpc_transport_ref((req)->trans))
+#define RPC_AUTH_ROOT_SQUASH(req) \
+ do { \
+ int gidcount = 0; \
+ if (req->svc->root_squash) { \
+ if (req->uid == RPC_ROOT_UID) \
+ req->uid = RPC_NOBODY_UID; \
+ if (req->gid == RPC_ROOT_GID) \
+ req->gid = RPC_NOBODY_GID; \
+ for (gidcount = 0; gidcount < req->auxgidcount; \
+ ++gidcount) { \
+ if (!req->auxgids[gidcount]) \
+ req->auxgids[gidcount] = \
+ RPC_NOBODY_GID; \
+ } \
+ } \
+ } while (0);
#define RPCSVC_ACTOR_SUCCESS 0
#define RPCSVC_ACTOR_ERROR (-1)
+#define RPCSVC_ACTOR_IGNORE (-2)
/* Functor for every type of protocol actor
* must be defined like this.
@@ -279,10 +305,8 @@ struct rpcsvc_request {
*
*/
typedef int (*rpcsvc_actor) (rpcsvc_request_t *req);
-typedef int (*rpcsvc_vector_actor) (rpcsvc_request_t *req, struct iovec *vec,
- int count, struct iobref *iobref);
-typedef int (*rpcsvc_vector_sizer) (rpcsvc_request_t *req, ssize_t *readsize,
- int *newiob);
+typedef int (*rpcsvc_vector_sizer) (int state, ssize_t *readsize,
+ char *base_addr, char *curr_addr);
/* Every protocol actor will also need to specify the function the RPC layer
* will use to serialize or encode the message into XDR format just before
@@ -296,7 +320,6 @@ typedef void *(*rpcsvc_encode_reply) (void *msg);
*/
typedef void (*rpcsvc_deallocate_reply) (void *msg);
-
#define RPCSVC_NAME_MAX 32
/* The descriptor for each procedure/actor that runs
* over the RPC service.
@@ -312,11 +335,13 @@ typedef struct rpcsvc_actor_desc {
* the XDR scheme, RPC cannot guarantee memory aligned addresses for
* the resulting message-specific structures. Allowing a specialized
* handler for letting the RPC program read the data from the network
- * directly into its alligned buffers.
+ * directly into its aligned buffers.
*/
- rpcsvc_vector_actor vector_actor;
rpcsvc_vector_sizer vector_sizer;
+ /* Can actor be ran on behalf an unprivileged requestor? */
+ gf_boolean_t unprivileged;
+ drc_op_type_t op_type;
} rpcsvc_actor_t;
/* Describes a program and its version along with the function pointers
@@ -371,6 +396,9 @@ struct rpcsvc_program {
*/
int min_auth;
+ /* Execute actor function as a synctask? */
+ gf_boolean_t synctask;
+
/* list member to link to list of registered services with rpcsvc */
struct list_head program;
};
@@ -406,11 +434,15 @@ rpcsvc_listener_destroy (rpcsvc_listener_t *listener);
extern int
rpcsvc_program_register_portmap (rpcsvc_program_t *newprog, uint32_t port);
+extern int
+rpcsvc_register_portmap_enabled (rpcsvc_t *svc);
+
/* Inits the global RPC service data structures.
* Called in main.
*/
extern rpcsvc_t *
-rpcsvc_init (glusterfs_ctx_t *ctx, dict_t *options);
+rpcsvc_init (xlator_t *xl, glusterfs_ctx_t *ctx, dict_t *options,
+ uint32_t poolcount);
int
rpcsvc_register_notify (rpcsvc_t *svc, rpcsvc_notify_t notify, void *mydata);
@@ -422,6 +454,13 @@ int
rpcsvc_unregister_notify (rpcsvc_t *svc, rpcsvc_notify_t notify, void *mydata);
int
+rpcsvc_transport_submit (rpc_transport_t *trans, struct iovec *rpchdr,
+ int rpchdrcount, struct iovec *proghdr,
+ int proghdrcount, struct iovec *progpayload,
+ int progpayloadcount, struct iobref *iobref,
+ void *priv);
+
+int
rpcsvc_submit_message (rpcsvc_request_t *req, struct iovec *proghdr,
int hdrcount, struct iovec *payload, int payloadcount,
struct iobref *iobref);
@@ -442,17 +481,18 @@ rpcsvc_error_reply (rpcsvc_request_t *req);
extern int
rpcsvc_transport_peername (rpc_transport_t *trans, char *hostname, int hostlen);
-extern inline int
+extern int
rpcsvc_transport_peeraddr (rpc_transport_t *trans, char *addrstr, int addrlen,
struct sockaddr_storage *returnsa, socklen_t sasize);
extern int
-rpcsvc_transport_peer_check (dict_t *options, char *volname,
- rpc_transport_t *trans);
+rpcsvc_auth_check (dict_t *options, char *volname,
+ rpc_transport_t *trans);
extern int
rpcsvc_transport_privport_check (rpcsvc_t *svc, char *volname,
rpc_transport_t *trans);
+
#define rpcsvc_request_seterr(req, err) (req)->rpc_err = err
#define rpcsvc_request_set_autherr(req, err) (req)->auth_err = err
@@ -524,9 +564,6 @@ rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen);
extern gid_t *
rpcsvc_auth_unix_auxgids (rpcsvc_request_t *req, int *arrlen);
-extern int
-rpcsvc_combine_gen_spec_volume_checks (int gen, int spec);
-
extern char *
rpcsvc_volume_allowed (dict_t *options, char *volname);
@@ -534,4 +571,20 @@ int rpcsvc_callback_submit (rpcsvc_t *rpc, rpc_transport_t *trans,
rpcsvc_cbk_program_t *prog, int procnum,
struct iovec *proghdr, int proghdrcount);
+rpcsvc_actor_t *
+rpcsvc_program_actor (rpcsvc_request_t *req);
+
+int
+rpcsvc_transport_unix_options_build (dict_t **options, char *filepath);
+int
+rpcsvc_set_allow_insecure (rpcsvc_t *svc, dict_t *options);
+int
+rpcsvc_set_root_squash (rpcsvc_t *svc, dict_t *options);
+int
+rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen);
+char *
+rpcsvc_volume_allowed (dict_t *options, char *volname);
+rpcsvc_vector_sizer
+rpcsvc_get_program_vector_sizer (rpcsvc_t *svc, uint32_t prognum,
+ uint32_t progver, uint32_t procnum);
#endif
diff --git a/rpc/rpc-lib/src/xdr-common.h b/rpc/rpc-lib/src/xdr-common.h
index 13ac41ce..34dc9c6a 100644
--- a/rpc/rpc-lib/src/xdr-common.h
+++ b/rpc/rpc-lib/src/xdr-common.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _XDR_COMMON_H_
@@ -30,15 +21,20 @@
#include <rpc/xdr.h>
#include <sys/uio.h>
-enum {
+#ifdef __NetBSD__
+#include <dirent.h>
+#endif /* __NetBSD__ */
+
+enum gf_dump_procnum {
GF_DUMP_NULL,
GF_DUMP_DUMP,
GF_DUMP_MAXVALUE,
-} gf_dump_procnum_t;
+};
#define GLUSTER_DUMP_PROGRAM 123451501 /* Completely random */
#define GLUSTER_DUMP_VERSION 1
+#define GF_MAX_AUTH_BYTES 2048
#if GF_DARWIN_HOST_OS
#define xdr_u_quad_t xdr_u_int64_t
@@ -47,6 +43,14 @@ enum {
#define uint64_t u_int64_t
#endif
+#if defined(__NetBSD__)
+#define xdr_u_quad_t xdr_u_int64_t
+#define xdr_quad_t xdr_int64_t
+#define xdr_uint32_t xdr_u_int32_t
+#define xdr_uint64_t xdr_u_int64_t
+#endif
+
+
#if GF_SOLARIS_HOST_OS
#define u_quad_t uint64_t
#define quad_t int64_t
@@ -55,57 +59,9 @@ enum {
#define xdr_uint32_t xdr_uint32_t
#endif
-struct auth_glusterfs_parms {
- uint64_t lk_owner;
- u_int pid;
- u_int uid;
- u_int gid;
- u_int ngrps;
- u_int groups[16];
-} __attribute__((packed));
-typedef struct auth_glusterfs_parms auth_glusterfs_parms;
-
-struct gf_dump_req {
- uint64_t gfs_id;
-} __attribute__((packed));
-typedef struct gf_dump_req gf_dump_req;
-
-struct gf_prog_detail {
- char *progname;
- uint64_t prognum;
- uint64_t progver;
- struct gf_prog_detail *next;
-} __attribute__((packed));
-typedef struct gf_prog_detail gf_prog_detail;
-
-struct gf_dump_rsp {
- uint64_t gfs_id;
- int op_ret;
- int op_errno;
- struct gf_prog_detail *prog;
-}__attribute__((packed));
-typedef struct gf_dump_rsp gf_dump_rsp;
-
-extern bool_t
-xdr_auth_glusterfs_parms (XDR *xdrs, auth_glusterfs_parms *objp);
-extern bool_t xdr_gf_dump_req (XDR *, gf_dump_req*);
-extern bool_t xdr_gf_prog_detail (XDR *, gf_prog_detail*);
-extern bool_t xdr_gf_dump_rsp (XDR *, gf_dump_rsp*);
-
-ssize_t
-xdr_serialize_dump_rsp (struct iovec outmsg, void *rsp);
-ssize_t
-xdr_to_dump_req (struct iovec inmsg, void *args);
-ssize_t
-xdr_from_dump_req (struct iovec outmsg, void *rsp);
-ssize_t
-xdr_to_dump_rsp (struct iovec inmsg, void *args);
-
-#define XDR_BYTES_PER_UNIT 4
-
/* Returns the address of the byte that follows the
* last byte used for decoding the previous xdr component.
- * For eg, once the RPC call for NFS has been decoded, thie macro will return
+ * E.g. once the RPC call for NFS has been decoded, the macro will return
* the address from which the NFS header starts.
*/
#define xdr_decoded_remaining_addr(xdr) ((&xdr)->x_private)
diff --git a/rpc/rpc-lib/src/xdr-rpc.c b/rpc/rpc-lib/src/xdr-rpc.c
index 26e7fa57..adb48a53 100644
--- a/rpc/rpc-lib/src/xdr-rpc.c
+++ b/rpc/rpc-lib/src/xdr-rpc.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _CONFIG_H
@@ -33,6 +24,7 @@
#include "xdr-rpc.h"
#include "xdr-common.h"
#include "logging.h"
+#include "common-utils.h"
/* Decodes the XDR format in msgbuf into rpc_msg.
* The remaining payload is returned into payload.
@@ -42,11 +34,12 @@ xdr_to_rpc_call (char *msgbuf, size_t len, struct rpc_msg *call,
struct iovec *payload, char *credbytes, char *verfbytes)
{
XDR xdr;
- char opaquebytes[MAX_AUTH_BYTES];
+ char opaquebytes[GF_MAX_AUTH_BYTES];
struct opaque_auth *oa = NULL;
+ int ret = -1;
- if ((!msgbuf) || (!call))
- return -1;
+ GF_VALIDATE_OR_GOTO ("rpc", msgbuf, out);
+ GF_VALIDATE_OR_GOTO ("rpc", call, out);
memset (call, 0, sizeof (*call));
@@ -63,15 +56,19 @@ xdr_to_rpc_call (char *msgbuf, size_t len, struct rpc_msg *call,
oa->oa_base = verfbytes;
xdrmem_create (&xdr, msgbuf, len, XDR_DECODE);
- if (!xdr_callmsg (&xdr, call))
- return -1;
+ if (!xdr_callmsg (&xdr, call)) {
+ gf_log ("rpc", GF_LOG_WARNING, "failed to decode call msg");
+ goto out;
+ }
if (payload) {
payload->iov_base = xdr_decoded_remaining_addr (xdr);
payload->iov_len = xdr_decoded_remaining_len (xdr);
}
- return 0;
+ ret = 0;
+out:
+ return ret;
}
@@ -85,8 +82,9 @@ true_func (XDR *s, caddr_t *a)
int
rpc_fill_empty_reply (struct rpc_msg *reply, uint32_t xid)
{
- if (!reply)
- return -1;
+ int ret = -1;
+
+ GF_VALIDATE_OR_GOTO ("rpc", reply, out);
/* Setting to 0 also results in reply verifier flavor to be
* set to AUTH_NULL which is what we want right now.
@@ -95,19 +93,22 @@ rpc_fill_empty_reply (struct rpc_msg *reply, uint32_t xid)
reply->rm_xid = xid;
reply->rm_direction = REPLY;
- return 0;
+ ret = 0;
+out:
+ return ret;
}
int
rpc_fill_denied_reply (struct rpc_msg *reply, int rjstat, int auth_err)
{
- if (!reply)
- return -1;
+ int ret = -1;
+
+ GF_VALIDATE_OR_GOTO ("rpc", reply, out);
reply->rm_reply.rp_stat = MSG_DENIED;
reply->rjcted_rply.rj_stat = rjstat;
if (rjstat == RPC_MISMATCH) {
- /* No problem with hardocoding
+ /* No problem with hardcoding
* RPC version numbers. We only support
* v2 anyway.
*/
@@ -116,7 +117,9 @@ rpc_fill_denied_reply (struct rpc_msg *reply, int rjstat, int auth_err)
} else if (rjstat == AUTH_ERROR)
reply->rjcted_rply.rj_why = auth_err;
- return 0;
+ ret = 0;
+out:
+ return ret;
}
@@ -124,8 +127,9 @@ int
rpc_fill_accepted_reply (struct rpc_msg *reply, int arstat, int proglow,
int proghigh, int verf, int len, char *vdata)
{
- if (!reply)
- return -1;
+ int ret = -1;
+
+ GF_VALIDATE_OR_GOTO ("rpc", reply, out);
reply->rm_reply.rp_stat = MSG_ACCEPTED;
reply->acpted_rply.ar_stat = arstat;
@@ -145,26 +149,34 @@ rpc_fill_accepted_reply (struct rpc_msg *reply, int arstat, int proglow,
reply->acpted_rply.ar_results.where = NULL;
}
- return 0;
+ ret = 0;
+out:
+ return ret;
}
int
rpc_reply_to_xdr (struct rpc_msg *reply, char *dest, size_t len,
struct iovec *dst)
{
- XDR xdr;
+ XDR xdr;
+ int ret = -1;
- if ((!dest) || (!reply) || (!dst))
- return -1;
+ GF_VALIDATE_OR_GOTO ("rpc", reply, out);
+ GF_VALIDATE_OR_GOTO ("rpc", dest, out);
+ GF_VALIDATE_OR_GOTO ("rpc", dst, out);
xdrmem_create (&xdr, dest, len, XDR_ENCODE);
- if (!xdr_replymsg(&xdr, reply))
- return -1;
+ if (!xdr_replymsg(&xdr, reply)) {
+ gf_log ("rpc", GF_LOG_WARNING, "failed to encode reply msg");
+ goto out;
+ }
dst->iov_base = dest;
dst->iov_len = xdr_encoded_length (xdr);
- return 0;
+ ret = 0;
+out:
+ return ret;
}
@@ -173,9 +185,12 @@ xdr_to_auth_unix_cred (char *msgbuf, int msglen, struct authunix_parms *au,
char *machname, gid_t *gids)
{
XDR xdr;
+ int ret = -1;
- if ((!msgbuf) || (!machname) || (!gids) || (!au))
- return -1;
+ GF_VALIDATE_OR_GOTO ("rpc", msgbuf, out);
+ GF_VALIDATE_OR_GOTO ("rpc", machname, out);
+ GF_VALIDATE_OR_GOTO ("rpc", gids, out);
+ GF_VALIDATE_OR_GOTO ("rpc", au, out);
au->aup_machname = machname;
#ifdef GF_DARWIN_HOST_OS
@@ -186,8 +201,12 @@ xdr_to_auth_unix_cred (char *msgbuf, int msglen, struct authunix_parms *au,
xdrmem_create (&xdr, msgbuf, msglen, XDR_DECODE);
- if (!xdr_authunix_parms (&xdr, au))
- return -1;
+ if (!xdr_authunix_parms (&xdr, au)) {
+ gf_log ("rpc", GF_LOG_WARNING, "failed to decode auth unix parms");
+ goto out;
+ }
- return 0;
+ ret = 0;
+out:
+ return ret;
}
diff --git a/rpc/rpc-lib/src/xdr-rpc.h b/rpc/rpc-lib/src/xdr-rpc.h
index d504391d..f5f4a941 100644
--- a/rpc/rpc-lib/src/xdr-rpc.h
+++ b/rpc/rpc-lib/src/xdr-rpc.h
@@ -1,23 +1,14 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
-#ifndef _XDR_RPC_H
+#ifndef _XDR_RPC_H_
#define _XDR_RPC_H_
#ifndef _CONFIG_H
@@ -39,6 +30,14 @@
#include <rpc/xdr.h>
#include <sys/uio.h>
+#include "xdr-common.h"
+
+typedef enum {
+ AUTH_GLUSTERFS = 5,
+ AUTH_GLUSTERFS_v2 = 390039, /* using a number from 'unused' range,
+ from the list available in RFC5531 */
+} gf_rpc_authtype_t;
+
/* Converts a given network buffer from its XDR format to a structure
* that contains everything an RPC call needs to work.
*/
@@ -62,7 +61,7 @@ rpc_reply_to_xdr (struct rpc_msg *reply, char *dest, size_t len,
extern int
xdr_to_auth_unix_cred (char *msgbuf, int msglen, struct authunix_parms *au,
char *machname, gid_t *gids);
-/* Macros that simplify accesing the members of an RPC call structure. */
+/* Macros that simplify accessing the members of an RPC call structure. */
#define rpc_call_xid(call) ((call)->rm_xid)
#define rpc_call_direction(call) ((call)->rm_direction)
#define rpc_call_rpcvers(call) ((call)->ru.RM_cmb.cb_rpcvers)
diff --git a/rpc/rpc-lib/src/xdr-rpcclnt.c b/rpc/rpc-lib/src/xdr-rpcclnt.c
index 933887b0..810d1961 100644
--- a/rpc/rpc-lib/src/xdr-rpcclnt.c
+++ b/rpc/rpc-lib/src/xdr-rpcclnt.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _CONFIG_H
@@ -34,6 +25,7 @@
#include "xdr-rpc.h"
#include "xdr-common.h"
#include "logging.h"
+#include "common-utils.h"
/* Decodes the XDR format in msgbuf into rpc_msg.
* The remaining payload is returned into payload.
@@ -43,12 +35,10 @@ xdr_to_rpc_reply (char *msgbuf, size_t len, struct rpc_msg *reply,
struct iovec *payload, char *verfbytes)
{
XDR xdr;
- int ret = -1;
+ int ret = -EINVAL;
- if ((!msgbuf) || (!reply)) {
- ret = -EINVAL;
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("rpc", msgbuf, out);
+ GF_VALIDATE_OR_GOTO ("rpc", reply, out);
memset (reply, 0, sizeof (struct rpc_msg));
@@ -58,6 +48,7 @@ xdr_to_rpc_reply (char *msgbuf, size_t len, struct rpc_msg *reply,
xdrmem_create (&xdr, msgbuf, len, XDR_DECODE);
if (!xdr_replymsg (&xdr, reply)) {
+ gf_log ("rpc", GF_LOG_WARNING, "failed to decode reply msg");
ret = -errno;
goto out;
}
@@ -71,13 +62,6 @@ out:
return ret;
}
-#if 0
-bool_t
-true_func (XDR *s, caddr_t *a)
-{
- return TRUE;
-}
-#endif
int
rpc_request_to_xdr (struct rpc_msg *request, char *dest, size_t len,
@@ -86,12 +70,13 @@ rpc_request_to_xdr (struct rpc_msg *request, char *dest, size_t len,
XDR xdr;
int ret = -1;
- if ((!dest) || (!request) || (!dst)) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("rpc", dest, out);
+ GF_VALIDATE_OR_GOTO ("rpc", request, out);
+ GF_VALIDATE_OR_GOTO ("rpc", dst, out);
xdrmem_create (&xdr, dest, len, XDR_ENCODE);
if (!xdr_callmsg (&xdr, request)) {
+ gf_log ("rpc", GF_LOG_WARNING, "failed to encode call msg");
goto out;
}
@@ -112,13 +97,14 @@ auth_unix_cred_to_xdr (struct authunix_parms *au, char *dest, size_t len,
XDR xdr;
int ret = -1;
- if (!au || !dest || !iov) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("rpc", au, out);
+ GF_VALIDATE_OR_GOTO ("rpc", dest, out);
+ GF_VALIDATE_OR_GOTO ("rpc", iov, out);
xdrmem_create (&xdr, dest, len, XDR_DECODE);
if (!xdr_authunix_parms (&xdr, au)) {
+ gf_log ("rpc", GF_LOG_WARNING, "failed to decode authunix parms");
goto out;
}
diff --git a/rpc/rpc-lib/src/xdr-rpcclnt.h b/rpc/rpc-lib/src/xdr-rpcclnt.h
index c0d925ee..c08d872f 100644
--- a/rpc/rpc-lib/src/xdr-rpcclnt.h
+++ b/rpc/rpc-lib/src/xdr-rpcclnt.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _XDR_RPCCLNT_H
@@ -33,7 +24,7 @@
#include <rpc/rpc_msg.h>
#include <rpc/auth_unix.h>
-/* Macros that simplify accesing the members of an RPC call structure. */
+/* Macros that simplify accessing the members of an RPC call structure. */
#define rpc_reply_xid(reply) ((reply)->rm_xid)
#define rpc_reply_status(reply) ((reply)->ru.RM_rmb.rp_stat)
#define rpc_accepted_reply_status(reply) ((reply)->acpted_rply.ar_stat)
diff --git a/rpc/rpc-transport/rdma/src/Makefile.am b/rpc/rpc-transport/rdma/src/Makefile.am
index bc888b17..2bf7cf23 100644
--- a/rpc/rpc-transport/rdma/src/Makefile.am
+++ b/rpc/rpc-transport/rdma/src/Makefile.am
@@ -3,18 +3,20 @@
transport_LTLIBRARIES = rdma.la
transportdir = $(libdir)/glusterfs/$(PACKAGE_VERSION)/rpc-transport
-rdma_la_LDFLAGS = -module -avoidversion
+rdma_la_LDFLAGS = -module -avoid-version
rdma_la_SOURCES = rdma.c name.c
rdma_la_LIBADD = $(top_builddir)/libglusterfs/src/libglusterfs.la \
- -libverbs
+ -libverbs -lrdmacm
noinst_HEADERS = rdma.h name.h
-I$(top_srcdir)/libglusterfs/src -I$(top_srcdir)/rpc/rpc-lib/src/ \
-I$(top_srcdir)/xlators/protocol/lib/src/ -shared -nostartfiles $(GF_CFLAGS)
-AM_CFLAGS = -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Wall -D$(GF_HOST_OS) \
+AM_CPPFLAGS = $(GF_CPPFLAGS) \
-I$(top_srcdir)/libglusterfs/src -I$(top_srcdir)/rpc/rpc-lib/src/ \
- -I$(top_srcdir)/xlators/protocol/lib/src -shared -nostartfiles $(GF_CFLAGS)
+ -I$(top_srcdir)/rpc/xdr/src
+
+AM_CFLAGS = -Wall $(GF_CFLAGS)
CLEANFILES = *~
diff --git a/rpc/rpc-transport/rdma/src/name.c b/rpc/rpc-transport/rdma/src/name.c
index 3142cb6c..c57428ad 100644
--- a/rpc/rpc-transport/rdma/src/name.c
+++ b/rpc/rpc-transport/rdma/src/name.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2008-2009 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#include <sys/types.h>
@@ -22,12 +13,7 @@
#include <errno.h>
#include <netdb.h>
#include <string.h>
-
-#ifdef CLIENT_PORT_CEILING
-#undef CLIENT_PORT_CEILING
-#endif
-
-#define CLIENT_PORT_CEILING 1024
+#include <rdma/rdma_cma.h>
#ifndef AF_INET_SDP
#define AF_INET_SDP 27
@@ -35,37 +21,54 @@
#include "rpc-transport.h"
#include "rdma.h"
+#include "common-utils.h"
+
int32_t
-gf_resolve_ip6 (const char *hostname,
- uint16_t port,
- int family,
- void **dnscache,
+gf_resolve_ip6 (const char *hostname,
+ uint16_t port,
+ int family,
+ void **dnscache,
struct addrinfo **addr_info);
static int32_t
-af_inet_bind_to_port_lt_ceiling (int fd, struct sockaddr *sockaddr,
+af_inet_bind_to_port_lt_ceiling (struct rdma_cm_id *cm_id,
+ struct sockaddr *sockaddr,
socklen_t sockaddr_len, int ceiling)
{
- int32_t ret = -1;
- /* struct sockaddr_in sin = {0, }; */
- uint16_t port = ceiling - 1;
+ int32_t ret = -1;
+ uint16_t port = ceiling - 1;
+ // by default assume none of the ports are blocked and all are available
+ gf_boolean_t ports[1024] = {_gf_false,};
+ int i = 0;
+
+ ret = gf_process_reserved_ports (ports);
+ if (ret != 0) {
+ for (i = 0; i < 1024; i++)
+ ports[i] = _gf_false;
+ }
while (port)
{
switch (sockaddr->sa_family)
{
case AF_INET6:
- ((struct sockaddr_in6 *)sockaddr)->sin6_port = htons (port);
+ ((struct sockaddr_in6 *)sockaddr)->sin6_port
+ = htons (port);
break;
case AF_INET_SDP:
case AF_INET:
- ((struct sockaddr_in *)sockaddr)->sin_port = htons (port);
+ ((struct sockaddr_in *)sockaddr)->sin_port
+ = htons (port);
break;
}
-
- ret = bind (fd, sockaddr, sockaddr_len);
+ // ignore the reserved ports
+ if (ports[port] == _gf_true) {
+ port--;
+ continue;
+ }
+ ret = rdma_bind_addr (cm_id, sockaddr);
if (ret == 0)
break;
@@ -79,23 +82,22 @@ af_inet_bind_to_port_lt_ceiling (int fd, struct sockaddr *sockaddr,
return ret;
}
+#if 0
static int32_t
-af_unix_client_bind (rpc_transport_t *this,
- struct sockaddr *sockaddr,
- socklen_t sockaddr_len,
- int sock)
+af_unix_client_bind (rpc_transport_t *this, struct sockaddr *sockaddr,
+ socklen_t sockaddr_len, struct rdma_cm_id *cm_id)
{
data_t *path_data = NULL;
struct sockaddr_un *addr = NULL;
int32_t ret = -1;
- path_data = dict_get (this->options,
+ path_data = dict_get (this->options,
"transport.rdma.bind-path");
if (path_data) {
char *path = data_to_str (path_data);
if (!path || strlen (path) > UNIX_PATH_MAX) {
gf_log (this->name, GF_LOG_DEBUG,
- "transport.rdma.bind-path not specfied "
+ "transport.rdma.bind-path not specified "
"for unix socket, letting connect to assign "
"default value");
goto err;
@@ -106,7 +108,7 @@ af_unix_client_bind (rpc_transport_t *this,
ret = bind (sock, (struct sockaddr *)addr, sockaddr_len);
if (ret == -1) {
gf_log (this->name, GF_LOG_ERROR,
- "cannot bind to unix-domain socket %d (%s)",
+ "cannot bind to unix-domain socket %d (%s)",
sock, strerror (errno));
goto err;
}
@@ -115,30 +117,31 @@ af_unix_client_bind (rpc_transport_t *this,
err:
return ret;
}
+#endif
static int32_t
client_fill_address_family (rpc_transport_t *this, struct sockaddr *sockaddr)
{
data_t *address_family_data = NULL;
- address_family_data = dict_get (this->options,
+ address_family_data = dict_get (this->options,
"transport.address-family");
if (!address_family_data) {
data_t *remote_host_data = NULL, *connect_path_data = NULL;
remote_host_data = dict_get (this->options, "remote-host");
- connect_path_data = dict_get (this->options,
+ connect_path_data = dict_get (this->options,
"transport.rdma.connect-path");
- if (!(remote_host_data || connect_path_data) ||
+ if (!(remote_host_data || connect_path_data) ||
(remote_host_data && connect_path_data)) {
gf_log (this->name, GF_LOG_ERROR,
"address-family not specified and not able to "
"determine the same from other options "
- "(remote-host:%s and connect-path:%s)",
- data_to_str (remote_host_data),
+ "(remote-host:%s and connect-path:%s)",
+ data_to_str (remote_host_data),
data_to_str (connect_path_data));
return -1;
- }
+ }
if (remote_host_data) {
gf_log (this->name, GF_LOG_DEBUG,
@@ -162,13 +165,11 @@ client_fill_address_family (rpc_transport_t *this, struct sockaddr *sockaddr)
sockaddr->sa_family = AF_INET6;
} else if (!strcasecmp (address_family, "inet-sdp")) {
sockaddr->sa_family = AF_INET_SDP;
- } else if (!strcasecmp (address_family, "inet/inet6")
- || !strcasecmp (address_family, "inet6/inet")) {
- sockaddr->sa_family = AF_UNSPEC;
} else {
gf_log (this->name, GF_LOG_ERROR,
- "unknown address-family (%s) specified",
+ "unknown address-family (%s) specified",
address_family);
+ sockaddr->sa_family = AF_UNSPEC;
return -1;
}
}
@@ -177,8 +178,8 @@ client_fill_address_family (rpc_transport_t *this, struct sockaddr *sockaddr)
}
static int32_t
-af_inet_client_get_remote_sockaddr (rpc_transport_t *this,
- struct sockaddr *sockaddr,
+af_inet_client_get_remote_sockaddr (rpc_transport_t *this,
+ struct sockaddr *sockaddr,
socklen_t *sockaddr_len,
int16_t remote_port)
{
@@ -193,7 +194,7 @@ af_inet_client_get_remote_sockaddr (rpc_transport_t *this,
if (remote_host_data == NULL)
{
gf_log (this->name, GF_LOG_ERROR,
- "option remote-host missing in volume %s",
+ "option remote-host missing in volume %s",
this->name);
ret = -1;
goto err;
@@ -203,7 +204,7 @@ af_inet_client_get_remote_sockaddr (rpc_transport_t *this,
if (remote_host == NULL)
{
gf_log (this->name, GF_LOG_ERROR,
- "option remote-host has data NULL in volume %s",
+ "option remote-host has data NULL in volume %s",
this->name);
ret = -1;
goto err;
@@ -238,7 +239,7 @@ af_inet_client_get_remote_sockaddr (rpc_transport_t *this,
/* TODO: gf_resolve is a blocking call. kick in some
non blocking dns techniques */
ret = gf_resolve_ip6 (remote_host, remote_port,
- sockaddr->sa_family,
+ sockaddr->sa_family,
&this->dnscache, &addr_info);
if (ret == -1) {
gf_log (this->name, GF_LOG_ERROR,
@@ -254,8 +255,8 @@ err:
}
static int32_t
-af_unix_client_get_remote_sockaddr (rpc_transport_t *this,
- struct sockaddr *sockaddr,
+af_unix_client_get_remote_sockaddr (rpc_transport_t *this,
+ struct sockaddr *sockaddr,
socklen_t *sockaddr_len)
{
struct sockaddr_un *sockaddr_un = NULL;
@@ -263,7 +264,7 @@ af_unix_client_get_remote_sockaddr (rpc_transport_t *this,
data_t *connect_path_data = NULL;
int32_t ret = 0;
- connect_path_data = dict_get (this->options,
+ connect_path_data = dict_get (this->options,
"transport.rdma.connect-path");
if (!connect_path_data) {
gf_log (this->name, GF_LOG_ERROR,
@@ -311,7 +312,7 @@ af_unix_server_get_local_sockaddr (rpc_transport_t *this,
struct sockaddr_un *sunaddr = (struct sockaddr_un *)addr;
- listen_path_data = dict_get (this->options,
+ listen_path_data = dict_get (this->options,
"transport.rdma.listen-path");
if (!listen_path_data) {
gf_log (this->name, GF_LOG_ERROR,
@@ -342,9 +343,9 @@ err:
return ret;
}
-static int32_t
-af_inet_server_get_local_sockaddr (rpc_transport_t *this,
- struct sockaddr *addr,
+static int32_t
+af_inet_server_get_local_sockaddr (rpc_transport_t *this,
+ struct sockaddr *addr,
socklen_t *addr_len)
{
struct addrinfo hints, *res = 0;
@@ -360,31 +361,31 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
listen_host_data = dict_get (options,
"transport.rdma.bind-address");
- if (listen_port_data)
- {
+ if (listen_port_data) {
listen_port = data_to_uint16 (listen_port_data);
} else {
- if (addr->sa_family == AF_INET6) {
- struct sockaddr_in6 *in = (struct sockaddr_in6 *) addr;
- in->sin6_addr = in6addr_any;
- in->sin6_port = htons(listen_port);
- *addr_len = sizeof(struct sockaddr_in6);
+ listen_port = GF_DEFAULT_RDMA_LISTEN_PORT;
+
+ if (addr->sa_family == AF_INET6) {
+ struct sockaddr_in6 *in = (struct sockaddr_in6 *) addr;
+ in->sin6_addr = in6addr_any;
+ in->sin6_port = htons(listen_port);
+ *addr_len = sizeof(struct sockaddr_in6);
goto out;
- } else if (addr->sa_family == AF_INET) {
- struct sockaddr_in *in = (struct sockaddr_in *) addr;
- in->sin_addr.s_addr = htonl(INADDR_ANY);
- in->sin_port = htons(listen_port);
- *addr_len = sizeof(struct sockaddr_in);
+ } else if (addr->sa_family == AF_INET) {
+ struct sockaddr_in *in = (struct sockaddr_in *) addr;
+ in->sin_addr.s_addr = htonl(INADDR_ANY);
+ in->sin_port = htons(listen_port);
+ *addr_len = sizeof(struct sockaddr_in);
goto out;
- }
- }
+ }
+ }
if (listen_port == (uint16_t) -1)
listen_port = GF_DEFAULT_RDMA_LISTEN_PORT;
- if (listen_host_data)
- {
+ if (listen_host_data) {
listen_host = data_to_str (listen_host_data);
}
@@ -398,9 +399,8 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
ret = getaddrinfo(listen_host, service, &hints, &res);
if (ret != 0) {
- gf_log (this->name,
- GF_LOG_ERROR,
- "getaddrinfo failed for host %s, service %s (%s)",
+ gf_log (this->name, GF_LOG_ERROR,
+ "getaddrinfo failed for host %s, service %s (%s)",
listen_host, service, gai_strerror (ret));
ret = -1;
goto out;
@@ -416,10 +416,8 @@ out:
}
int32_t
-gf_rdma_client_bind (rpc_transport_t *this,
- struct sockaddr *sockaddr,
- socklen_t *sockaddr_len,
- int sock)
+gf_rdma_client_bind (rpc_transport_t *this, struct sockaddr *sockaddr,
+ socklen_t *sockaddr_len, struct rdma_cm_id *cm_id)
{
int ret = 0;
@@ -431,22 +429,24 @@ gf_rdma_client_bind (rpc_transport_t *this,
*sockaddr_len = sizeof (struct sockaddr_in);
case AF_INET6:
- ret = af_inet_bind_to_port_lt_ceiling (sock, sockaddr,
- *sockaddr_len,
- CLIENT_PORT_CEILING);
+ ret = af_inet_bind_to_port_lt_ceiling (cm_id, sockaddr,
+ *sockaddr_len,
+ GF_CLIENT_PORT_CEILING);
if (ret == -1) {
gf_log (this->name, GF_LOG_WARNING,
- "cannot bind inet socket (%d) to port "
- "less than %d (%s)",
- sock, CLIENT_PORT_CEILING, strerror (errno));
+ "cannot bind rdma_cm_id to port "
+ "less than %d (%s)", GF_CLIENT_PORT_CEILING,
+ strerror (errno));
ret = 0;
}
break;
case AF_UNIX:
*sockaddr_len = sizeof (struct sockaddr_un);
- ret = af_unix_client_bind (this, (struct sockaddr *)sockaddr,
+#if 0
+ ret = af_unix_client_bind (this, (struct sockaddr *)sockaddr,
*sockaddr_len, sock);
+#endif
break;
default:
@@ -473,7 +473,7 @@ gf_rdma_client_get_remote_sockaddr (rpc_transport_t *this,
ret = -1;
goto err;
}
-
+
switch (sockaddr->sa_family)
{
case AF_INET_SDP:
@@ -483,7 +483,7 @@ gf_rdma_client_get_remote_sockaddr (rpc_transport_t *this,
case AF_INET:
case AF_INET6:
case AF_UNSPEC:
- ret = af_inet_client_get_remote_sockaddr (this,
+ ret = af_inet_client_get_remote_sockaddr (this,
sockaddr,
sockaddr_len,
remote_port);
@@ -495,8 +495,8 @@ gf_rdma_client_get_remote_sockaddr (rpc_transport_t *this,
break;
case AF_UNIX:
- ret = af_unix_client_get_remote_sockaddr (this,
- sockaddr,
+ ret = af_unix_client_get_remote_sockaddr (this,
+ sockaddr,
sockaddr_len);
break;
@@ -505,21 +505,21 @@ gf_rdma_client_get_remote_sockaddr (rpc_transport_t *this,
"unknown address-family %d", sockaddr->sa_family);
ret = -1;
}
-
+
err:
return ret;
}
int32_t
gf_rdma_server_get_local_sockaddr (rpc_transport_t *this,
- struct sockaddr *addr,
- socklen_t *addr_len)
+ struct sockaddr *addr,
+ socklen_t *addr_len)
{
data_t *address_family_data = NULL;
int32_t ret = 0;
char is_inet_sdp = 0;
- address_family_data = dict_get (this->options,
+ address_family_data = dict_get (this->options,
"transport.address-family");
if (address_family_data) {
char *address_family = NULL;
@@ -533,21 +533,19 @@ gf_rdma_server_get_local_sockaddr (rpc_transport_t *this,
addr->sa_family = AF_INET_SDP;
} else if (!strcasecmp (address_family, "unix")) {
addr->sa_family = AF_UNIX;
- } else if (!strcasecmp (address_family, "inet/inet6")
- || !strcasecmp (address_family, "inet6/inet")) {
- addr->sa_family = AF_UNSPEC;
} else {
gf_log (this->name, GF_LOG_ERROR,
- "unknown address family (%s) specified",
+ "unknown address family (%s) specified",
address_family);
+ addr->sa_family = AF_UNSPEC;
ret = -1;
goto err;
}
} else {
gf_log (this->name, GF_LOG_DEBUG,
"option address-family not specified, defaulting "
- "to inet/inet6");
- addr->sa_family = AF_UNSPEC;
+ "to inet");
+ addr->sa_family = AF_INET;
}
switch (addr->sa_family)
@@ -574,60 +572,53 @@ err:
return ret;
}
-int32_t
-fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *addr,
+int32_t
+fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *addr,
int32_t addr_len, char *identifier)
{
int32_t ret = 0, tmpaddr_len = 0;
char service[NI_MAXSERV], host[NI_MAXHOST];
- struct sockaddr_storage tmpaddr;
+ union gf_sock_union sock_union;
- memset (&tmpaddr, 0, sizeof (tmpaddr));
- tmpaddr = *addr;
+ memset (&sock_union, 0, sizeof (sock_union));
+ sock_union.storage = *addr;
tmpaddr_len = addr_len;
- if (((struct sockaddr *) &tmpaddr)->sa_family == AF_INET6) {
+ if (sock_union.sa.sa_family == AF_INET6) {
int32_t one_to_four, four_to_eight, twelve_to_sixteen;
int16_t eight_to_ten, ten_to_twelve;
-
+
one_to_four = four_to_eight = twelve_to_sixteen = 0;
eight_to_ten = ten_to_twelve = 0;
-
- one_to_four = ((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr.s6_addr32[0];
- four_to_eight = ((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr.s6_addr32[1];
+
+ one_to_four = sock_union.sin6.sin6_addr.s6_addr32[0];
+ four_to_eight = sock_union.sin6.sin6_addr.s6_addr32[1];
#ifdef GF_SOLARIS_HOST_OS
- eight_to_ten = S6_ADDR16(((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr)[4];
+ eight_to_ten = S6_ADDR16(sock_union.sin6.sin6_addr)[4];
#else
- eight_to_ten = ((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr.s6_addr16[4];
+ eight_to_ten = sock_union.sin6.sin6_addr.s6_addr16[4];
#endif
#ifdef GF_SOLARIS_HOST_OS
- ten_to_twelve = S6_ADDR16(((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr)[5];
+ ten_to_twelve = S6_ADDR16(sock_union.sin6.sin6_addr)[5];
#else
- ten_to_twelve = ((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr.s6_addr16[5];
+ ten_to_twelve = sock_union.sin6.sin6_addr.s6_addr16[5];
#endif
- twelve_to_sixteen = ((struct sockaddr_in6 *)
- &tmpaddr)->sin6_addr.s6_addr32[3];
+ twelve_to_sixteen = sock_union.sin6.sin6_addr.s6_addr32[3];
/* ipv4 mapped ipv6 address has
bits 0-80: 0
bits 80-96: 0xffff
- bits 96-128: ipv4 address
+ bits 96-128: ipv4 address
*/
-
+
if (one_to_four == 0 &&
four_to_eight == 0 &&
eight_to_ten == 0 &&
ten_to_twelve == -1) {
- struct sockaddr_in *in_ptr = (struct sockaddr_in *)&tmpaddr;
- memset (&tmpaddr, 0, sizeof (tmpaddr));
-
+ struct sockaddr_in *in_ptr = &sock_union.sin;
+ memset (&sock_union, 0, sizeof (sock_union));
+
in_ptr->sin_family = AF_INET;
in_ptr->sin_port = ((struct sockaddr_in6 *)addr)->sin6_port;
in_ptr->sin_addr.s_addr = twelve_to_sixteen;
@@ -635,7 +626,7 @@ fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *add
}
}
- ret = getnameinfo ((struct sockaddr *) &tmpaddr,
+ ret = getnameinfo (&sock_union.sa,
tmpaddr_len,
host, sizeof (host),
service, sizeof (service),
@@ -666,8 +657,8 @@ gf_rdma_get_transport_identifiers (rpc_transport_t *this)
case AF_INET:
case AF_INET6:
{
- ret = fill_inet6_inet_identifiers (this,
- &this->myinfo.sockaddr,
+ ret = fill_inet6_inet_identifiers (this,
+ &this->myinfo.sockaddr,
this->myinfo.sockaddr_len,
this->myinfo.identifier);
if (ret == -1) {
@@ -705,7 +696,7 @@ gf_rdma_get_transport_identifiers (rpc_transport_t *this)
break;
default:
- gf_log (this->name, GF_LOG_ERROR,
+ gf_log (this->name, GF_LOG_ERROR,
"unknown address family (%d)",
((struct sockaddr *) &this->myinfo.sockaddr)->sa_family);
ret = -1;
diff --git a/rpc/rpc-transport/rdma/src/name.h b/rpc/rpc-transport/rdma/src/name.h
index 32c30145..742fc5fc 100644
--- a/rpc/rpc-transport/rdma/src/name.h
+++ b/rpc/rpc-transport/rdma/src/name.h
@@ -1,35 +1,23 @@
/*
- Copyright (c) 2008-2009 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _IB_VERBS_NAME_H
#define _IB_VERBS_NAME_H
-#include <sys/socket.h>
-#include <sys/un.h>
+#include <rdma/rdma_cma.h>
#include "compat.h"
int32_t
-gf_rdma_client_bind (rpc_transport_t *this,
- struct sockaddr *sockaddr,
- socklen_t *sockaddr_len,
- int sock);
+gf_rdma_client_bind (rpc_transport_t *this, struct sockaddr *sockaddr,
+ socklen_t *sockaddr_len, struct rdma_cm_id *cm_id);
int32_t
gf_rdma_client_get_remote_sockaddr (rpc_transport_t *this,
diff --git a/rpc/rpc-transport/rdma/src/rdma.c b/rpc/rpc-transport/rdma/src/rdma.c
index 00664963..8ef7d1e3 100644
--- a/rpc/rpc-transport/rdma/src/rdma.c
+++ b/rpc/rpc-transport/rdma/src/rdma.c
@@ -1,23 +1,13 @@
/*
- Copyright (c) 2006-2009 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
-
#ifndef _CONFIG_H
#define _CONFIG_H
#include "config.h"
@@ -30,119 +20,47 @@
#include "name.h"
#include "byte-order.h"
#include "xlator.h"
+#include "xdr-rpc.h"
#include <signal.h>
-#define RDMA_LOG_NAME "rpc-transport/rdma"
+#define GF_RDMA_LOG_NAME "rpc-transport/rdma"
static int32_t
-__rdma_ioq_churn (rdma_peer_t *peer);
+__gf_rdma_ioq_churn (gf_rdma_peer_t *peer);
-rdma_post_t *
-rdma_post_ref (rdma_post_t *post);
+gf_rdma_post_t *
+gf_rdma_post_ref (gf_rdma_post_t *post);
int
-rdma_post_unref (rdma_post_t *post);
-
-int32_t
-gf_resolve_ip6 (const char *hostname,
- uint16_t port,
- int family,
- void **dnscache,
- struct addrinfo **addr_info);
-
-static uint16_t
-rdma_get_local_lid (struct ibv_context *context,
- int32_t port)
-{
- struct ibv_port_attr attr;
+gf_rdma_post_unref (gf_rdma_post_t *post);
- if (ibv_query_port (context, port, &attr))
- return 0;
+static void *
+gf_rdma_send_completion_proc (void *data);
- return attr.lid;
-}
+static void *
+gf_rdma_recv_completion_proc (void *data);
-static const char *
-get_port_state_str(enum ibv_port_state pstate)
-{
- switch (pstate) {
- case IBV_PORT_DOWN: return "PORT_DOWN";
- case IBV_PORT_INIT: return "PORT_INIT";
- case IBV_PORT_ARMED: return "PORT_ARMED";
- case IBV_PORT_ACTIVE: return "PORT_ACTIVE";
- case IBV_PORT_ACTIVE_DEFER: return "PORT_ACTIVE_DEFER";
- default: return "invalid state";
- }
-}
+void *
+gf_rdma_async_event_thread (void *context);
static int32_t
-ib_check_active_port (struct ibv_context *ctx, uint8_t port)
-{
- struct ibv_port_attr port_attr;
-
- int32_t ret = 0;
- const char *state_str = NULL;
-
- if (!ctx) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "Error in supplied context");
- return -1;
- }
-
- ret = ibv_query_port (ctx, port, &port_attr);
-
- if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "Failed to query port %u properties", port);
- return -1;
- }
-
- state_str = get_port_state_str (port_attr.state);
- gf_log (RDMA_LOG_NAME, GF_LOG_TRACE,
- "Infiniband PORT: (%u) STATE: (%s)",
- port, state_str);
-
- if (port_attr.state == IBV_PORT_ACTIVE)
- return 0;
-
- return -1;
-}
+gf_rdma_create_qp (rpc_transport_t *this);
static int32_t
-ib_get_active_port (struct ibv_context *ib_ctx)
-{
- struct ibv_device_attr ib_device_attr;
+__gf_rdma_teardown (rpc_transport_t *this);
- int32_t ret = -1;
- uint8_t ib_port = 0;
-
- if (!ib_ctx) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "Error in supplied context");
- return -1;
- }
- if (ibv_query_device (ib_ctx, &ib_device_attr)) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "Failed to query device properties");
- return -1;
- }
+static int32_t
+gf_rdma_teardown (rpc_transport_t *this);
- for (ib_port = 1; ib_port <= ib_device_attr.phys_port_cnt; ++ib_port) {
- ret = ib_check_active_port (ib_ctx, ib_port);
- if (ret == 0)
- return ib_port;
+static int32_t
+gf_rdma_disconnect (rpc_transport_t *this);
- gf_log (RDMA_LOG_NAME, GF_LOG_TRACE,
- "Port:(%u) not active", ib_port);
- continue;
- }
- return ret;
-}
+static void
+gf_rdma_cm_handle_disconnect (rpc_transport_t *this);
static void
-rdma_put_post (rdma_queue_t *queue,
- rdma_post_t *post)
+gf_rdma_put_post (gf_rdma_queue_t *queue, gf_rdma_post_t *post)
{
post->ctx.is_request = 0;
@@ -167,16 +85,16 @@ rdma_put_post (rdma_queue_t *queue,
}
-static rdma_post_t *
-rdma_new_post (rdma_device_t *device, int32_t len, rdma_post_type_t type)
+static gf_rdma_post_t *
+gf_rdma_new_post (rpc_transport_t *this, gf_rdma_device_t *device, int32_t len,
+ gf_rdma_post_type_t type)
{
- rdma_post_t *post = NULL;
- int ret = -1;
+ gf_rdma_post_t *post = NULL;
+ int ret = -1;
- post = (rdma_post_t *) GF_CALLOC (1, sizeof (*post),
- gf_common_mt_rdma_post_t);
+ post = (gf_rdma_post_t *) GF_CALLOC (1, sizeof (*post),
+ gf_common_mt_rdma_post_t);
if (post == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -186,7 +104,7 @@ rdma_new_post (rdma_device_t *device, int32_t len, rdma_post_type_t type)
post->buf = valloc (len);
if (!post->buf) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
+ gf_log_nomem (GF_RDMA_LOG_NAME, GF_LOG_ERROR, len);
goto out;
}
@@ -195,8 +113,9 @@ rdma_new_post (rdma_device_t *device, int32_t len, rdma_post_type_t type)
post->buf_size,
IBV_ACCESS_LOCAL_WRITE);
if (!post->mr) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "memory registration failed");
+ gf_log (this->name, GF_LOG_WARNING,
+ "memory registration failed (%s)",
+ strerror (errno));
goto out;
}
@@ -206,9 +125,7 @@ rdma_new_post (rdma_device_t *device, int32_t len, rdma_post_type_t type)
ret = 0;
out:
if (ret != 0) {
- if (post->buf != NULL) {
- free (post->buf);
- }
+ free (post->buf);
GF_FREE (post);
post = NULL;
@@ -218,17 +135,17 @@ out:
}
-static rdma_post_t *
-rdma_get_post (rdma_queue_t *queue)
+static gf_rdma_post_t *
+gf_rdma_get_post (gf_rdma_queue_t *queue)
{
- rdma_post_t *post;
+ gf_rdma_post_t *post = NULL;
pthread_mutex_lock (&queue->lock);
{
post = queue->passive_posts.next;
if (post == &queue->passive_posts)
post = NULL;
-
+
if (post) {
if (post->prev)
post->prev->next = post->next;
@@ -248,7 +165,7 @@ rdma_get_post (rdma_queue_t *queue)
}
void
-rdma_destroy_post (rdma_post_t *post)
+gf_rdma_destroy_post (gf_rdma_post_t *post)
{
ibv_dereg_mr (post->mr);
free (post->buf);
@@ -257,10 +174,12 @@ rdma_destroy_post (rdma_post_t *post)
static int32_t
-__rdma_quota_get (rdma_peer_t *peer)
+__gf_rdma_quota_get (gf_rdma_peer_t *peer)
{
- int32_t ret = -1;
- rdma_private_t *priv = peer->trans->private;
+ int32_t ret = -1;
+ gf_rdma_private_t *priv = NULL;
+
+ priv = peer->trans->private;
if (priv->connected && peer->quota > 0) {
ret = peer->quota--;
@@ -269,25 +188,9 @@ __rdma_quota_get (rdma_peer_t *peer)
return ret;
}
-/*
- static int32_t
- rdma_quota_get (rdma_peer_t *peer)
- {
- int32_t ret = -1;
- rdma_private_t *priv = peer->trans->private;
-
- pthread_mutex_lock (&priv->write_mutex);
- {
- ret = __rdma_quota_get (peer);
- }
- pthread_mutex_unlock (&priv->write_mutex);
-
- return ret;
- }
-*/
-static void
-__rdma_ioq_entry_free (rdma_ioq_t *entry)
+static void
+__gf_rdma_ioq_entry_free (gf_rdma_ioq_t *entry)
{
list_del_init (&entry->list);
@@ -300,48 +203,923 @@ __rdma_ioq_entry_free (rdma_ioq_t *entry)
iobref_unref (entry->msg.request.rsp_iobref);
entry->msg.request.rsp_iobref = NULL;
}
- /* TODO: use mem-pool */
- mem_put (entry->pool, entry);
+
+ mem_put (entry);
}
static void
-__rdma_ioq_flush (rdma_peer_t *peer)
+__gf_rdma_ioq_flush (gf_rdma_peer_t *peer)
{
- rdma_ioq_t *entry = NULL, *dummy = NULL;
+ gf_rdma_ioq_t *entry = NULL, *dummy = NULL;
list_for_each_entry_safe (entry, dummy, &peer->ioq, list) {
- __rdma_ioq_entry_free (entry);
+ __gf_rdma_ioq_entry_free (entry);
}
}
static int32_t
-__rdma_disconnect (rpc_transport_t *this)
+__gf_rdma_disconnect (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
- int32_t ret = 0;
+ gf_rdma_private_t *priv = NULL;
- if (priv->connected || priv->tcp_connected) {
- fcntl (priv->sock, F_SETFL, O_NONBLOCK);
- if (shutdown (priv->sock, SHUT_RDWR) != 0) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_DEBUG,
- "shutdown () - error: %s",
- strerror (errno));
- ret = -errno;
- priv->tcp_connected = 0;
+ priv = this->private;
+
+ if (priv->connected) {
+ rdma_disconnect (priv->peer.cm_id);
+ }
+
+ return 0;
+}
+
+
+static void
+gf_rdma_queue_init (gf_rdma_queue_t *queue)
+{
+ pthread_mutex_init (&queue->lock, NULL);
+
+ queue->active_posts.next = &queue->active_posts;
+ queue->active_posts.prev = &queue->active_posts;
+ queue->passive_posts.next = &queue->passive_posts;
+ queue->passive_posts.prev = &queue->passive_posts;
+}
+
+
+static void
+__gf_rdma_destroy_queue (gf_rdma_post_t *post)
+{
+ gf_rdma_post_t *tmp = NULL;
+
+ while (post->next != post) {
+ tmp = post->next;
+
+ post->next = post->next->next;
+ post->next->prev = post;
+
+ gf_rdma_destroy_post (tmp);
+ }
+}
+
+
+static void
+gf_rdma_destroy_queue (gf_rdma_queue_t *queue)
+{
+ if (queue == NULL) {
+ goto out;
+ }
+
+ pthread_mutex_lock (&queue->lock);
+ {
+ if (queue->passive_count > 0) {
+ __gf_rdma_destroy_queue (&queue->passive_posts);
+ queue->passive_count = 0;
+ }
+
+ if (queue->active_count > 0) {
+ __gf_rdma_destroy_queue (&queue->active_posts);
+ queue->active_count = 0;
}
}
-
+ pthread_mutex_unlock (&queue->lock);
+
+out:
+ return;
+}
+
+
+static void
+gf_rdma_destroy_posts (rpc_transport_t *this)
+{
+ gf_rdma_device_t *device = NULL;
+ gf_rdma_private_t *priv = NULL;
+
+ if (this == NULL) {
+ goto out;
+ }
+
+ priv = this->private;
+ device = priv->device;
+
+ gf_rdma_destroy_queue (&device->sendq);
+ gf_rdma_destroy_queue (&device->recvq);
+
+out:
+ return;
+}
+
+
+static int32_t
+__gf_rdma_create_posts (rpc_transport_t *this, int32_t count, int32_t size,
+ gf_rdma_queue_t *q, gf_rdma_post_type_t type)
+{
+ int32_t i = 0;
+ int32_t ret = 0;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+
+ priv = this->private;
+ device = priv->device;
+
+ for (i=0 ; i<count ; i++) {
+ gf_rdma_post_t *post = NULL;
+
+ post = gf_rdma_new_post (this, device, size + 2048, type);
+ if (!post) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "post creation failed");
+ ret = -1;
+ break;
+ }
+
+ gf_rdma_put_post (q, post);
+ }
return ret;
}
static int32_t
-rdma_post_send (struct ibv_qp *qp,
- rdma_post_t *post,
- int32_t len)
+gf_rdma_post_recv (struct ibv_srq *srq,
+ gf_rdma_post_t *post)
+{
+ struct ibv_sge list = {
+ .addr = (unsigned long) post->buf,
+ .length = post->buf_size,
+ .lkey = post->mr->lkey
+ };
+
+ struct ibv_recv_wr wr = {
+ .wr_id = (unsigned long) post,
+ .sg_list = &list,
+ .num_sge = 1,
+ }, *bad_wr;
+
+ gf_rdma_post_ref (post);
+
+ return ibv_post_srq_recv (srq, &wr, &bad_wr);
+}
+
+
+static int32_t
+gf_rdma_create_posts (rpc_transport_t *this)
+{
+ int32_t i = 0, ret = 0;
+ gf_rdma_post_t *post = NULL;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_options_t *options = NULL;
+ gf_rdma_device_t *device = NULL;
+
+ priv = this->private;
+ options = &priv->options;
+ device = priv->device;
+
+ ret = __gf_rdma_create_posts (this, options->send_count,
+ options->send_size,
+ &device->sendq, GF_RDMA_SEND_POST);
+ if (!ret)
+ ret = __gf_rdma_create_posts (this, options->recv_count,
+ options->recv_size,
+ &device->recvq,
+ GF_RDMA_RECV_POST);
+
+ if (!ret) {
+ for (i=0 ; i<options->recv_count ; i++) {
+ post = gf_rdma_get_post (&device->recvq);
+ if (gf_rdma_post_recv (device->srq, post) != 0) {
+ ret = -1;
+ break;
+ }
+ }
+ }
+
+ if (ret)
+ gf_rdma_destroy_posts (this);
+
+ return ret;
+}
+
+
+static void
+gf_rdma_destroy_cq (rpc_transport_t *this)
+{
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+
+ priv = this->private;
+ device = priv->device;
+
+ if (device->recv_cq)
+ ibv_destroy_cq (device->recv_cq);
+ device->recv_cq = NULL;
+
+ if (device->send_cq)
+ ibv_destroy_cq (device->send_cq);
+ device->send_cq = NULL;
+
+ return;
+}
+
+
+static int32_t
+gf_rdma_create_cq (rpc_transport_t *this)
+{
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_options_t *options = NULL;
+ gf_rdma_device_t *device = NULL;
+ uint64_t send_cqe = 0;
+ int32_t ret = 0;
+ struct ibv_device_attr device_attr = {{0}, };
+
+ priv = this->private;
+ options = &priv->options;
+ device = priv->device;
+
+ device->recv_cq = ibv_create_cq (priv->device->context,
+ options->recv_count * 2,
+ device,
+ device->recv_chan,
+ 0);
+ if (!device->recv_cq) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "creation of CQ for device %s failed",
+ device->device_name);
+ ret = -1;
+ goto out;
+ } else if (ibv_req_notify_cq (device->recv_cq, 0)) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "ibv_req_notify_cq on recv CQ of device %s failed",
+ device->device_name);
+ ret = -1;
+ goto out;
+ }
+
+ do {
+ ret = ibv_query_device (priv->device->context, &device_attr);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "ibv_query_device on %s returned %d (%s)",
+ priv->device->device_name, ret,
+ (ret > 0) ? strerror (ret) : "");
+ ret = -1;
+ goto out;
+ }
+
+ send_cqe = options->send_count * 128;
+ send_cqe = (send_cqe > device_attr.max_cqe)
+ ? device_attr.max_cqe : send_cqe;
+
+ /* TODO: make send_cq size dynamically adaptive */
+ device->send_cq = ibv_create_cq (priv->device->context,
+ send_cqe, device,
+ device->send_chan, 0);
+ if (!device->send_cq) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "creation of send_cq for device %s failed",
+ device->device_name);
+ ret = -1;
+ goto out;
+ }
+
+ if (ibv_req_notify_cq (device->send_cq, 0)) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "ibv_req_notify_cq on send_cq for device %s"
+ " failed", device->device_name);
+ ret = -1;
+ goto out;
+ }
+ } while (0);
+
+out:
+ if (ret != 0)
+ gf_rdma_destroy_cq (this);
+
+ return ret;
+}
+
+
+static gf_rdma_device_t *
+gf_rdma_get_device (rpc_transport_t *this, struct ibv_context *ibctx,
+ char *device_name)
+{
+ glusterfs_ctx_t *ctx = NULL;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_options_t *options = NULL;
+ int32_t ret = 0;
+ int32_t i = 0;
+ gf_rdma_device_t *trav = NULL, *device = NULL;
+ gf_rdma_ctx_t *rdma_ctx = NULL;
+
+ priv = this->private;
+ options = &priv->options;
+ ctx = this->ctx;
+ rdma_ctx = ctx->ib;
+
+ trav = rdma_ctx->device;
+
+ while (trav) {
+ if (!strcmp (trav->device_name, device_name))
+ break;
+ trav = trav->next;
+ }
+
+ if (!trav) {
+ trav = GF_CALLOC (1, sizeof (*trav),
+ gf_common_mt_rdma_device_t);
+ if (trav == NULL) {
+ goto out;
+ }
+
+ priv->device = trav;
+ trav->context = ibctx;
+
+ trav->request_ctx_pool
+ = mem_pool_new (gf_rdma_request_context_t,
+ GF_RDMA_POOL_SIZE);
+ if (trav->request_ctx_pool == NULL) {
+ goto out;
+ }
+
+ trav->ioq_pool
+ = mem_pool_new (gf_rdma_ioq_t, GF_RDMA_POOL_SIZE);
+ if (trav->ioq_pool == NULL) {
+ goto out;
+ }
+
+ trav->reply_info_pool = mem_pool_new (gf_rdma_reply_info_t,
+ GF_RDMA_POOL_SIZE);
+ if (trav->reply_info_pool == NULL) {
+ goto out;
+ }
+
+ trav->device_name = gf_strdup (device_name);
+
+ trav->next = rdma_ctx->device;
+ rdma_ctx->device = trav;
+
+ trav->send_chan = ibv_create_comp_channel (trav->context);
+ if (!trav->send_chan) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create send completion channel for "
+ "device (%s)", device_name);
+ goto out;
+ }
+
+ trav->recv_chan = ibv_create_comp_channel (trav->context);
+ if (!trav->recv_chan) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create recv completion channel for "
+ "device (%s)", device_name);
+
+ /* TODO: cleanup current mess */
+ goto out;
+ }
+
+ if (gf_rdma_create_cq (this) < 0) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create CQ for device (%s)",
+ device_name);
+ goto out;
+ }
+
+ /* protection domain */
+ trav->pd = ibv_alloc_pd (trav->context);
+
+ if (!trav->pd) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not allocate protection domain for "
+ "device (%s)", device_name);
+ goto out;
+ }
+
+ struct ibv_srq_init_attr attr = {
+ .attr = {
+ .max_wr = options->recv_count,
+ .max_sge = 1,
+ .srq_limit = 10
+ }
+ };
+ trav->srq = ibv_create_srq (trav->pd, &attr);
+
+ if (!trav->srq) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create SRQ for device (%s)",
+ device_name);
+ goto out;
+ }
+
+ /* queue init */
+ gf_rdma_queue_init (&trav->sendq);
+ gf_rdma_queue_init (&trav->recvq);
+
+ if (gf_rdma_create_posts (this) < 0) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not allocate posts for device (%s)",
+ device_name);
+ goto out;
+ }
+
+ /* completion threads */
+ ret = pthread_create (&trav->send_thread,
+ NULL,
+ gf_rdma_send_completion_proc,
+ trav->send_chan);
+ if (ret) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create send completion thread for "
+ "device (%s)", device_name);
+ goto out;
+ }
+
+ ret = pthread_create (&trav->recv_thread,
+ NULL,
+ gf_rdma_recv_completion_proc,
+ trav->recv_chan);
+ if (ret) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create recv completion thread "
+ "for device (%s)", device_name);
+ return NULL;
+ }
+
+ ret = pthread_create (&trav->async_event_thread,
+ NULL,
+ gf_rdma_async_event_thread,
+ ibctx);
+ if (ret) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create async_event_thread");
+ return NULL;
+ }
+
+ /* qpreg */
+ pthread_mutex_init (&trav->qpreg.lock, NULL);
+ for (i=0; i<42; i++) {
+ trav->qpreg.ents[i].next = &trav->qpreg.ents[i];
+ trav->qpreg.ents[i].prev = &trav->qpreg.ents[i];
+ }
+ }
+
+ device = trav;
+ trav = NULL;
+out:
+
+ if (trav != NULL) {
+ gf_rdma_destroy_posts (this);
+ mem_pool_destroy (trav->ioq_pool);
+ mem_pool_destroy (trav->request_ctx_pool);
+ mem_pool_destroy (trav->reply_info_pool);
+ ibv_dealloc_pd (trav->pd);
+ gf_rdma_destroy_cq (this);
+ ibv_destroy_comp_channel (trav->recv_chan);
+ ibv_destroy_comp_channel (trav->send_chan);
+ GF_FREE ((char *)trav->device_name);
+ GF_FREE (trav);
+ }
+
+ return device;
+}
+
+
+static rpc_transport_t *
+gf_rdma_transport_new (rpc_transport_t *listener, struct rdma_cm_id *cm_id)
+{
+ gf_rdma_private_t *listener_priv = NULL, *priv = NULL;
+ rpc_transport_t *this = NULL, *new = NULL;
+ gf_rdma_options_t *options = NULL;
+ char *device_name = NULL;
+
+ listener_priv = listener->private;
+
+ this = GF_CALLOC (1, sizeof (rpc_transport_t),
+ gf_common_mt_rpc_transport_t);
+ if (this == NULL) {
+ goto out;
+ }
+
+ this->listener = listener;
+
+ priv = GF_CALLOC (1, sizeof (gf_rdma_private_t),
+ gf_common_mt_rdma_private_t);
+ if (priv == NULL) {
+ goto out;
+ }
+
+ this->private = priv;
+ priv->options = listener_priv->options;
+
+ priv->listener = listener;
+ priv->entity = GF_RDMA_SERVER;
+
+ options = &priv->options;
+
+ this->ops = listener->ops;
+ this->init = listener->init;
+ this->fini = listener->fini;
+ this->ctx = listener->ctx;
+ this->name = gf_strdup (listener->name);
+ this->notify = listener->notify;
+ this->mydata = listener->mydata;
+
+ this->myinfo.sockaddr_len = sizeof (cm_id->route.addr.src_addr);
+ memcpy (&this->myinfo.sockaddr, &cm_id->route.addr.src_addr,
+ this->myinfo.sockaddr_len);
+
+ this->peerinfo.sockaddr_len = sizeof (cm_id->route.addr.dst_addr);
+ memcpy (&this->peerinfo.sockaddr, &cm_id->route.addr.dst_addr,
+ this->peerinfo.sockaddr_len);
+
+ priv->peer.trans = this;
+ gf_rdma_get_transport_identifiers (this);
+
+ device_name = (char *)ibv_get_device_name (cm_id->verbs->device);
+ if (device_name == NULL) {
+ gf_log (listener->name, GF_LOG_WARNING,
+ "cannot get device name (peer:%s me:%s)",
+ this->peerinfo.identifier, this->myinfo.identifier);
+ goto out;
+ }
+
+ priv->device = gf_rdma_get_device (this, cm_id->verbs,
+ device_name);
+ if (priv->device == NULL) {
+ gf_log (listener->name, GF_LOG_WARNING,
+ "cannot get infiniband device %s (peer:%s me:%s)",
+ device_name, this->peerinfo.identifier,
+ this->myinfo.identifier);
+ goto out;
+ }
+
+ priv->peer.send_count = options->send_count;
+ priv->peer.recv_count = options->recv_count;
+ priv->peer.send_size = options->send_size;
+ priv->peer.recv_size = options->recv_size;
+ priv->peer.cm_id = cm_id;
+ INIT_LIST_HEAD (&priv->peer.ioq);
+
+ pthread_mutex_init (&priv->write_mutex, NULL);
+ pthread_mutex_init (&priv->recv_mutex, NULL);
+
+ cm_id->context = this;
+
+ new = rpc_transport_ref (this);
+ this = NULL;
+out:
+ if (this != NULL) {
+ if (this->private != NULL) {
+ GF_FREE (this->private);
+ }
+
+ if (this->name != NULL) {
+ GF_FREE (this->name);
+ }
+
+ GF_FREE (this);
+ }
+
+ return new;
+}
+
+
+static int
+gf_rdma_cm_handle_connect_request (struct rdma_cm_event *event)
+{
+ int ret = -1;
+ rpc_transport_t *this = NULL, *listener = NULL;
+ struct rdma_cm_id *child_cm_id = NULL, *listener_cm_id = NULL;
+ struct rdma_conn_param conn_param = {0, };
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_options_t *options = NULL;
+
+ child_cm_id = event->id;
+ listener_cm_id = event->listen_id;
+
+ listener = listener_cm_id->context;
+ priv = listener->private;
+ options = &priv->options;
+
+ this = gf_rdma_transport_new (listener, child_cm_id);
+ if (this == NULL) {
+ gf_log (listener->name, GF_LOG_WARNING,
+ "could not create a transport for incoming connection"
+ " (me.name:%s me.identifier:%s)", listener->name,
+ listener->myinfo.identifier);
+ rdma_destroy_id (child_cm_id);
+ goto out;
+ }
+
+ gf_log (listener->name, GF_LOG_TRACE,
+ "got a connect request (me:%s peer:%s)",
+ listener->myinfo.identifier, this->peerinfo.identifier);
+
+ ret = gf_rdma_create_qp (this);
+ if (ret < 0) {
+ gf_log (listener->name, GF_LOG_WARNING,
+ "could not create QP (peer:%s me:%s)",
+ this->peerinfo.identifier, this->myinfo.identifier);
+ gf_rdma_cm_handle_disconnect (this);
+ goto out;
+ }
+
+ conn_param.responder_resources = 1;
+ conn_param.initiator_depth = 1;
+ conn_param.retry_count = options->attr_retry_cnt;
+ conn_param.rnr_retry_count = options->attr_rnr_retry;
+
+ ret = rdma_accept(child_cm_id, &conn_param);
+ if (ret < 0) {
+ gf_log (listener->name, GF_LOG_WARNING, "rdma_accept failed "
+ "peer:%s me:%s (%s)", this->peerinfo.identifier,
+ this->myinfo.identifier, strerror (errno));
+ gf_rdma_cm_handle_disconnect (this);
+ goto out;
+ }
+
+ ret = 0;
+
+out:
+ return ret;
+}
+
+
+static int
+gf_rdma_cm_handle_route_resolved (struct rdma_cm_event *event)
+{
+ struct rdma_conn_param conn_param = {0, };
+ int ret = 0;
+ rpc_transport_t *this = NULL;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_peer_t *peer = NULL;
+ gf_rdma_options_t *options = NULL;
+
+ if (event == NULL) {
+ goto out;
+ }
+
+ this = event->id->context;
+
+ priv = this->private;
+ peer = &priv->peer;
+ options = &priv->options;
+
+ ret = gf_rdma_create_qp (this);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "could not create QP (peer:%s me:%s)",
+ this->peerinfo.identifier, this->myinfo.identifier);
+ gf_rdma_cm_handle_disconnect (this);
+ goto out;
+ }
+
+ memset(&conn_param, 0, sizeof conn_param);
+ conn_param.responder_resources = 1;
+ conn_param.initiator_depth = 1;
+ conn_param.retry_count = options->attr_retry_cnt;
+ conn_param.rnr_retry_count = options->attr_rnr_retry;
+
+ ret = rdma_connect(peer->cm_id, &conn_param);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "rdma_connect failed (%s)", strerror (errno));
+ gf_rdma_cm_handle_disconnect (this);
+ goto out;
+ }
+
+ gf_log (this->name, GF_LOG_TRACE, "route resolved (me:%s peer:%s)",
+ this->myinfo.identifier, this->peerinfo.identifier);
+
+ ret = 0;
+out:
+ return ret;
+}
+
+
+static int
+gf_rdma_cm_handle_addr_resolved (struct rdma_cm_event *event)
+{
+ rpc_transport_t *this = NULL;
+ gf_rdma_peer_t *peer = NULL;
+ gf_rdma_private_t *priv = NULL;
+ int ret = 0;
+
+ this = event->id->context;
+
+ priv = this->private;
+ peer = &priv->peer;
+
+ GF_ASSERT (peer->cm_id == event->id);
+
+ this->myinfo.sockaddr_len = sizeof (peer->cm_id->route.addr.src_addr);
+ memcpy (&this->myinfo.sockaddr, &peer->cm_id->route.addr.src_addr,
+ this->myinfo.sockaddr_len);
+
+ this->peerinfo.sockaddr_len = sizeof (peer->cm_id->route.addr.dst_addr);
+ memcpy (&this->peerinfo.sockaddr, &peer->cm_id->route.addr.dst_addr,
+ this->peerinfo.sockaddr_len);
+
+ gf_rdma_get_transport_identifiers (this);
+
+ ret = rdma_resolve_route(peer->cm_id, 2000);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "rdma_resolve_route failed (me:%s peer:%s) (%s)",
+ this->myinfo.identifier, this->peerinfo.identifier,
+ strerror (errno));
+ gf_rdma_cm_handle_disconnect (this);
+ }
+
+ gf_log (this->name, GF_LOG_TRACE, "Address resolved (me:%s peer:%s)",
+ this->myinfo.identifier, this->peerinfo.identifier);
+
+ return ret;
+}
+
+
+static void
+gf_rdma_cm_handle_disconnect (rpc_transport_t *this)
+{
+ gf_rdma_private_t *priv = NULL;
+ char need_unref = 0, connected = 0;
+
+ priv = this->private;
+ gf_log (this->name, GF_LOG_DEBUG,
+ "peer disconnected, cleaning up");
+
+ pthread_mutex_lock (&priv->write_mutex);
+ {
+ if (priv->peer.cm_id != NULL) {
+ need_unref = 1;
+ connected = priv->connected;
+ priv->connected = 0;
+ }
+
+ __gf_rdma_teardown (this);
+ }
+ pthread_mutex_unlock (&priv->write_mutex);
+
+ if (connected) {
+ rpc_transport_notify (this, RPC_TRANSPORT_DISCONNECT, this);
+ }
+
+ if (need_unref)
+ rpc_transport_unref (this);
+
+}
+
+
+static int
+gf_rdma_cm_handle_event_established (struct rdma_cm_event *event)
+{
+ rpc_transport_t *this = NULL;
+ gf_rdma_private_t *priv = NULL;
+ struct rdma_cm_id *cm_id = NULL;
+ int ret = 0;
+
+ cm_id = event->id;
+ this = cm_id->context;
+ priv = this->private;
+
+ priv->connected = 1;
+
+ pthread_mutex_lock (&priv->write_mutex);
+ {
+ priv->peer.quota = 1;
+ priv->peer.quota_set = 0;
+ }
+ pthread_mutex_unlock (&priv->write_mutex);
+
+ if (priv->entity == GF_RDMA_CLIENT) {
+ ret = rpc_transport_notify (this, RPC_TRANSPORT_CONNECT, this);
+
+ } else if (priv->entity == GF_RDMA_SERVER) {
+ ret = rpc_transport_notify (priv->listener,
+ RPC_TRANSPORT_ACCEPT, this);
+ }
+
+ if (ret < 0) {
+ gf_rdma_disconnect (this);
+ }
+
+ gf_log (this->name, GF_LOG_TRACE,
+ "recieved event RDMA_CM_EVENT_ESTABLISHED (me:%s peer:%s)",
+ this->myinfo.identifier, this->peerinfo.identifier);
+
+ return ret;
+}
+
+
+static int
+gf_rdma_cm_handle_event_error (rpc_transport_t *this)
+{
+ gf_rdma_private_t *priv = NULL;
+
+ priv = this->private;
+
+ if (priv->entity != GF_RDMA_SERVER_LISTENER) {
+ gf_rdma_cm_handle_disconnect (this);
+ }
+
+ return 0;
+}
+
+
+static int
+gf_rdma_cm_handle_device_removal (struct rdma_cm_event *event)
+{
+ return 0;
+}
+
+
+static void *
+gf_rdma_cm_event_handler (void *data)
+{
+ struct rdma_cm_event *event = NULL;
+ int ret = 0;
+ rpc_transport_t *this = NULL;
+ struct rdma_event_channel *event_channel = NULL;
+
+ event_channel = data;
+
+ while (1) {
+ ret = rdma_get_cm_event (event_channel, &event);
+ if (ret != 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma_cm_get_event failed (%s)",
+ strerror (errno));
+ break;
+ }
+
+ switch (event->event) {
+ case RDMA_CM_EVENT_ADDR_RESOLVED:
+ gf_rdma_cm_handle_addr_resolved (event);
+ break;
+
+ case RDMA_CM_EVENT_ROUTE_RESOLVED:
+ gf_rdma_cm_handle_route_resolved (event);
+ break;
+
+ case RDMA_CM_EVENT_CONNECT_REQUEST:
+ gf_rdma_cm_handle_connect_request (event);
+ break;
+
+ case RDMA_CM_EVENT_ESTABLISHED:
+ gf_rdma_cm_handle_event_established (event);
+ break;
+
+ case RDMA_CM_EVENT_ADDR_ERROR:
+ case RDMA_CM_EVENT_ROUTE_ERROR:
+ case RDMA_CM_EVENT_CONNECT_ERROR:
+ case RDMA_CM_EVENT_UNREACHABLE:
+ case RDMA_CM_EVENT_REJECTED:
+ this = event->id->context;
+
+ gf_log (this->name, GF_LOG_WARNING,
+ "cma event %s, error %d (me:%s peer:%s)\n",
+ rdma_event_str(event->event), event->status,
+ this->myinfo.identifier,
+ this->peerinfo.identifier);
+
+ rdma_ack_cm_event (event);
+ event = NULL;
+
+ gf_rdma_cm_handle_event_error (this);
+ continue;
+
+ case RDMA_CM_EVENT_DISCONNECTED:
+ this = event->id->context;
+
+ gf_log (this->name, GF_LOG_DEBUG,
+ "recieved disconnect (me:%s peer:%s)\n",
+ this->myinfo.identifier,
+ this->peerinfo.identifier);
+
+ rdma_ack_cm_event (event);
+ event = NULL;
+
+ gf_rdma_cm_handle_disconnect (this);
+ continue;
+
+ case RDMA_CM_EVENT_DEVICE_REMOVAL:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "device removed");
+ gf_rdma_cm_handle_device_removal (event);
+ break;
+
+ default:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "unhandled event: %s, ignoring",
+ rdma_event_str(event->event));
+ break;
+ }
+
+ rdma_ack_cm_event (event);
+ }
+
+ return NULL;
+}
+
+
+static int32_t
+gf_rdma_post_send (struct ibv_qp *qp, gf_rdma_post_t *post, int32_t len)
{
struct ibv_sge list = {
.addr = (unsigned long) post->buf,
@@ -358,65 +1136,70 @@ rdma_post_send (struct ibv_qp *qp,
}, *bad_wr;
if (!qp)
- return -1;
+ return EINVAL;
return ibv_post_send (qp, &wr, &bad_wr);
}
int
-__rdma_encode_error(rdma_peer_t *peer, rdma_reply_info_t *reply_info,
- struct iovec *rpchdr, uint32_t *ptr,
- rdma_errcode_t err)
+__gf_rdma_encode_error(gf_rdma_peer_t *peer, gf_rdma_reply_info_t *reply_info,
+ struct iovec *rpchdr, gf_rdma_header_t *hdr,
+ gf_rdma_errcode_t err)
{
- uint32_t *startp = NULL;
struct rpc_msg *rpc_msg = NULL;
- startp = ptr;
if (reply_info != NULL) {
- *ptr++ = hton32(reply_info->rm_xid);
+ hdr->rm_xid = hton32(reply_info->rm_xid);
} else {
rpc_msg = rpchdr[0].iov_base; /* assume rpchdr contains
* only one vector.
* (which is true)
*/
- *ptr++ = rpc_msg->rm_xid;
+ hdr->rm_xid = rpc_msg->rm_xid;
}
- *ptr++ = hton32(RDMA_VERSION);
- *ptr++ = hton32(peer->send_count);
- *ptr++ = hton32(RDMA_ERROR);
- *ptr++ = hton32(err);
- if (err == ERR_VERS) {
- *ptr++ = hton32(RDMA_VERSION);
- *ptr++ = hton32(RDMA_VERSION);
- }
+ hdr->rm_vers = hton32(GF_RDMA_VERSION);
+ hdr->rm_credit = hton32(peer->send_count);
+ hdr->rm_type = hton32(GF_RDMA_ERROR);
+ hdr->rm_body.rm_error.rm_type = hton32(err);
+ if (err == ERR_VERS) {
+ hdr->rm_body.rm_error.rm_version.gf_rdma_vers_low
+ = hton32(GF_RDMA_VERSION);
+ hdr->rm_body.rm_error.rm_version.gf_rdma_vers_high
+ = hton32(GF_RDMA_VERSION);
+ }
- return (int)((unsigned long)ptr - (unsigned long)startp);
+ return sizeof (*hdr);
}
int32_t
-__rdma_send_error (rdma_peer_t *peer, rdma_ioq_t *entry, rdma_post_t *post,
- rdma_reply_info_t *reply_info, rdma_errcode_t err)
+__gf_rdma_send_error (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_post_t *post, gf_rdma_reply_info_t *reply_info,
+ gf_rdma_errcode_t err)
{
- int32_t ret = -1, len;
+ int32_t ret = -1, len = 0;
- len = __rdma_encode_error (peer, reply_info, entry->rpchdr,
- (uint32_t *)post->buf, err);
+ len = __gf_rdma_encode_error (peer, reply_info, entry->rpchdr,
+ (gf_rdma_header_t *)post->buf, err);
if (len == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
+ "encode error returned -1");
goto out;
}
- rdma_post_ref (post);
+ gf_rdma_post_ref (post);
- ret = rdma_post_send (peer->qp, post, len);
+ ret = gf_rdma_post_send (peer->qp, post, len);
if (!ret) {
ret = len;
} else {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "ibv_post_send failed with ret = %d", ret);
- rdma_post_unref (post);
- __rdma_disconnect (peer->trans);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "gf_rdma_post_send (to %s) failed with ret = %d (%s)",
+ peer->trans->peerinfo.identifier, ret,
+ (ret > 0) ? strerror (ret) : "");
+ gf_rdma_post_unref (post);
+ __gf_rdma_disconnect (peer->trans);
ret = -1;
}
@@ -426,23 +1209,24 @@ out:
int32_t
-__rdma_create_read_chunks_from_vector (rdma_peer_t *peer,
- rdma_read_chunk_t **readch_ptr,
- int32_t *pos, struct iovec *vector,
- int count,
- rdma_request_context_t *request_ctx)
+__gf_rdma_create_read_chunks_from_vector (gf_rdma_peer_t *peer,
+ gf_rdma_read_chunk_t **readch_ptr,
+ int32_t *pos, struct iovec *vector,
+ int count,
+ gf_rdma_request_context_t *request_ctx)
{
- int i = 0;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
- struct ibv_mr *mr = NULL;
- rdma_read_chunk_t *readch = NULL;
- int32_t ret = -1;
-
- if ((peer == NULL) || (readch_ptr == NULL) || (*readch_ptr == NULL)
- || (request_ctx == NULL) || (vector == NULL)) {
- goto out;
- }
+ int i = 0;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+ struct ibv_mr *mr = NULL;
+ gf_rdma_read_chunk_t *readch = NULL;
+ int32_t ret = -1;
+
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, peer, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, readch_ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, *readch_ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, request_ctx, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, vector, out);
priv = peer->trans->private;
device = priv->device;
@@ -456,8 +1240,10 @@ __rdma_create_read_chunks_from_vector (rdma_peer_t *peer,
vector[i].iov_len,
IBV_ACCESS_REMOTE_READ);
if (!mr) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "memory registration failed");
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "memory registration failed (%s) (peer:%s)",
+ strerror (errno),
+ peer->trans->peerinfo.identifier);
goto out;
}
@@ -482,76 +1268,72 @@ out:
int32_t
-__rdma_create_read_chunks (rdma_peer_t *peer, rdma_ioq_t *entry,
- rdma_chunktype_t type, uint32_t **ptr,
- rdma_request_context_t *request_ctx)
+__gf_rdma_create_read_chunks (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_chunktype_t type, uint32_t **ptr,
+ gf_rdma_request_context_t *request_ctx)
{
int32_t ret = -1;
- rdma_device_t *device = NULL;
- rdma_private_t *priv = NULL;
int pos = 0;
- if ((peer == NULL) || (entry == NULL) || (ptr == NULL)
- || (*ptr == NULL) || (request_ctx == NULL)) {
- goto out;
- }
-
- priv = peer->trans->private;
- device = priv->device;
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, peer, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, entry, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, *ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, request_ctx, out);
request_ctx->iobref = iobref_ref (entry->iobref);
- if (type == rdma_areadch) {
+ if (type == gf_rdma_areadch) {
pos = 0;
- ret = __rdma_create_read_chunks_from_vector (peer,
- (rdma_read_chunk_t **)ptr,
- &pos,
- entry->rpchdr,
- entry->rpchdr_count,
- request_ctx);
+ ret = __gf_rdma_create_read_chunks_from_vector (peer,
+ (gf_rdma_read_chunk_t **)ptr,
+ &pos,
+ entry->rpchdr,
+ entry->rpchdr_count,
+ request_ctx);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "cannot create read chunks from vector, "
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot create read chunks from vector "
"entry->rpchdr");
goto out;
}
- ret = __rdma_create_read_chunks_from_vector (peer,
- (rdma_read_chunk_t **)ptr,
- &pos,
- entry->proghdr,
- entry->proghdr_count,
- request_ctx);
+ ret = __gf_rdma_create_read_chunks_from_vector (peer,
+ (gf_rdma_read_chunk_t **)ptr,
+ &pos,
+ entry->proghdr,
+ entry->proghdr_count,
+ request_ctx);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "cannot create read chunks from vector, "
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot create read chunks from vector "
"entry->proghdr");
}
if (entry->prog_payload_count != 0) {
- ret = __rdma_create_read_chunks_from_vector (peer,
- (rdma_read_chunk_t **)ptr,
- &pos,
- entry->prog_payload,
- entry->prog_payload_count,
- request_ctx);
+ ret = __gf_rdma_create_read_chunks_from_vector (peer,
+ (gf_rdma_read_chunk_t **)ptr,
+ &pos,
+ entry->prog_payload,
+ entry->prog_payload_count,
+ request_ctx);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "cannot create read chunks from vector,"
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot create read chunks from vector"
" entry->prog_payload");
}
}
} else {
pos = iov_length (entry->rpchdr, entry->rpchdr_count);
- ret = __rdma_create_read_chunks_from_vector (peer,
- (rdma_read_chunk_t **)ptr,
- &pos,
- entry->prog_payload,
- entry->prog_payload_count,
- request_ctx);
+ ret = __gf_rdma_create_read_chunks_from_vector (peer,
+ (gf_rdma_read_chunk_t **)ptr,
+ &pos,
+ entry->prog_payload,
+ entry->prog_payload_count,
+ request_ctx);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "cannot create read chunks from vector, "
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot create read chunks from vector "
"entry->prog_payload");
}
}
@@ -565,36 +1347,39 @@ out:
int32_t
-__rdma_create_write_chunks_from_vector (rdma_peer_t *peer,
- rdma_write_chunk_t **writech_ptr,
- struct iovec *vector, int count,
- rdma_request_context_t *request_ctx)
+__gf_rdma_create_write_chunks_from_vector (gf_rdma_peer_t *peer,
+ gf_rdma_write_chunk_t **writech_ptr,
+ struct iovec *vector, int count,
+ gf_rdma_request_context_t *request_ctx)
{
- int i = 0;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
- struct ibv_mr *mr = NULL;
- rdma_write_chunk_t *writech = NULL;
- int32_t ret = -1;
-
- if ((peer == NULL) || (writech_ptr == NULL) || (*writech_ptr == NULL)
- || (request_ctx == NULL) || (vector == NULL)) {
- goto out;
- }
+ int i = 0;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+ struct ibv_mr *mr = NULL;
+ gf_rdma_write_chunk_t *writech = NULL;
+ int32_t ret = -1;
+
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, peer, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, writech_ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, *writech_ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, request_ctx, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, vector, out);
writech = *writech_ptr;
priv = peer->trans->private;
device = priv->device;
-
+
for (i = 0; i < count; i++) {
mr = ibv_reg_mr (device->pd, vector[i].iov_base,
vector[i].iov_len,
IBV_ACCESS_REMOTE_WRITE
| IBV_ACCESS_LOCAL_WRITE);
if (!mr) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "memory registration failed");
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "memory registration failed (%s) (peer:%s)",
+ strerror (errno),
+ peer->trans->peerinfo.identifier);
goto out;
}
@@ -617,22 +1402,23 @@ out:
int32_t
-__rdma_create_write_chunks (rdma_peer_t *peer, rdma_ioq_t *entry,
- rdma_chunktype_t chunk_type, uint32_t **ptr,
- rdma_request_context_t *request_ctx)
+__gf_rdma_create_write_chunks (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_chunktype_t chunk_type, uint32_t **ptr,
+ gf_rdma_request_context_t *request_ctx)
{
- int32_t ret = -1;
- rdma_write_array_t *warray = NULL;
+ int32_t ret = -1;
+ gf_rdma_write_array_t *warray = NULL;
- if ((peer == NULL) || (entry == NULL) || (ptr == NULL)
- || (*ptr == NULL) || (request_ctx == NULL)) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, peer, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, *ptr, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, request_ctx, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, entry, out);
- if ((chunk_type == rdma_replych)
+ if ((chunk_type == gf_rdma_replych)
&& ((entry->msg.request.rsphdr_count != 1) ||
(entry->msg.request.rsphdr_vec[0].iov_base == NULL))) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
(entry->msg.request.rsphdr_count == 1)
? "chunktype specified as reply chunk but the vector "
"specifying the buffer to be used for holding reply"
@@ -643,32 +1429,32 @@ __rdma_create_write_chunks (rdma_peer_t *peer, rdma_ioq_t *entry,
}
/*
- if ((chunk_type == rdma_writech)
- && ((entry->msg.request.rsphdr_count == 0)
- || (entry->msg.request.rsphdr_vec[0].iov_base == NULL))) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "vector specifying buffer to hold the program's reply "
- "header should also be provided when buffers are "
- "provided for holding the program's payload in reply");
- goto out;
- }
+ if ((chunk_type == gf_rdma_writech)
+ && ((entry->msg.request.rsphdr_count == 0)
+ || (entry->msg.request.rsphdr_vec[0].iov_base == NULL))) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
+ "vector specifying buffer to hold the program's reply "
+ "header should also be provided when buffers are "
+ "provided for holding the program's payload in reply");
+ goto out;
+ }
*/
- if (chunk_type == rdma_writech) {
- warray = (rdma_write_array_t *)*ptr;
+ if (chunk_type == gf_rdma_writech) {
+ warray = (gf_rdma_write_array_t *)*ptr;
warray->wc_discrim = hton32 (1);
warray->wc_nchunks
= hton32 (entry->msg.request.rsp_payload_count);
*ptr = (uint32_t *)&warray->wc_array[0];
- ret = __rdma_create_write_chunks_from_vector (peer,
- (rdma_write_chunk_t **)ptr,
- entry->msg.request.rsp_payload,
- entry->msg.request.rsp_payload_count,
- request_ctx);
+ ret = __gf_rdma_create_write_chunks_from_vector (peer,
+ (gf_rdma_write_chunk_t **)ptr,
+ entry->msg.request.rsp_payload,
+ entry->msg.request.rsp_payload_count,
+ request_ctx);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"cannot create write chunks from vector "
"entry->rpc_payload");
goto out;
@@ -682,23 +1468,23 @@ __rdma_create_write_chunks (rdma_peer_t *peer, rdma_ioq_t *entry,
**ptr = 0;
*ptr = *ptr + 1;
} else {
- /* no write chunklist */
+ /* no write chunklist */
**ptr = 0;
*ptr = *ptr + 1;
- warray = (rdma_write_array_t *)*ptr;
+ warray = (gf_rdma_write_array_t *)*ptr;
warray->wc_discrim = hton32 (1);
warray->wc_nchunks = hton32 (entry->msg.request.rsphdr_count);
*ptr = (uint32_t *)&warray->wc_array[0];
-
- ret = __rdma_create_write_chunks_from_vector (peer,
- (rdma_write_chunk_t **)ptr,
- entry->msg.request.rsphdr_vec,
- entry->msg.request.rsphdr_count,
- request_ctx);
+
+ ret = __gf_rdma_create_write_chunks_from_vector (peer,
+ (gf_rdma_write_chunk_t **)ptr,
+ entry->msg.request.rsphdr_vec,
+ entry->msg.request.rsphdr_count,
+ request_ctx);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"cannot create write chunks from vector "
"entry->rpchdr");
goto out;
@@ -714,8 +1500,8 @@ out:
}
-inline void
-__rdma_deregister_mr (struct ibv_mr **mr, int count)
+static inline void
+__gf_rdma_deregister_mr (struct ibv_mr **mr, int count)
{
int i = 0;
@@ -733,15 +1519,15 @@ out:
static int32_t
-__rdma_quota_put (rdma_peer_t *peer)
+__gf_rdma_quota_put (gf_rdma_peer_t *peer)
{
- int32_t ret;
+ int32_t ret = 0;
peer->quota++;
ret = peer->quota;
if (!list_empty (&peer->ioq)) {
- ret = __rdma_ioq_churn (peer);
+ ret = __gf_rdma_ioq_churn (peer);
}
return ret;
@@ -749,14 +1535,15 @@ __rdma_quota_put (rdma_peer_t *peer)
static int32_t
-rdma_quota_put (rdma_peer_t *peer)
+gf_rdma_quota_put (gf_rdma_peer_t *peer)
{
- int32_t ret;
- rdma_private_t *priv = peer->trans->private;
+ int32_t ret = 0;
+ gf_rdma_private_t *priv = NULL;
+ priv = peer->trans->private;
pthread_mutex_lock (&priv->write_mutex);
{
- ret = __rdma_quota_put (peer);
+ ret = __gf_rdma_quota_put (peer);
}
pthread_mutex_unlock (&priv->write_mutex);
@@ -766,11 +1553,11 @@ rdma_quota_put (rdma_peer_t *peer)
/* to be called with priv->mutex held */
void
-__rdma_request_context_destroy (rdma_request_context_t *context)
+__gf_rdma_request_context_destroy (gf_rdma_request_context_t *context)
{
- rdma_peer_t *peer = NULL;
- rdma_private_t *priv = NULL;
- int32_t ret = 0;
+ gf_rdma_peer_t *peer = NULL;
+ gf_rdma_private_t *priv = NULL;
+ int32_t ret = 0;
if (context == NULL) {
goto out;
@@ -778,18 +1565,18 @@ __rdma_request_context_destroy (rdma_request_context_t *context)
peer = context->peer;
- __rdma_deregister_mr (context->mr, context->mr_count);
+ __gf_rdma_deregister_mr (context->mr, context->mr_count);
priv = peer->trans->private;
if (priv->connected) {
- ret = __rdma_quota_put (peer);
+ ret = __gf_rdma_quota_put (peer);
if (ret < 0) {
gf_log ("rdma", GF_LOG_DEBUG,
"failed to send "
"message");
- mem_put (context->pool, context);
- __rdma_disconnect (peer->trans);
+ mem_put (context);
+ __gf_rdma_disconnect (peer->trans);
goto out;
}
}
@@ -804,57 +1591,38 @@ __rdma_request_context_destroy (rdma_request_context_t *context)
context->rsp_iobref = NULL;
}
- mem_put (context->pool, context);
+ mem_put (context);
out:
return;
}
-
void
-rdma_post_context_destroy (rdma_post_context_t *ctx)
+gf_rdma_post_context_destroy (gf_rdma_post_context_t *ctx)
{
if (ctx == NULL) {
goto out;
}
- __rdma_deregister_mr (ctx->mr, ctx->mr_count);
+ __gf_rdma_deregister_mr (ctx->mr, ctx->mr_count);
if (ctx->iobref != NULL) {
iobref_unref (ctx->iobref);
}
+ if (ctx->hdr_iobuf != NULL) {
+ iobuf_unref (ctx->hdr_iobuf);
+ }
+
memset (ctx, 0, sizeof (*ctx));
out:
return;
}
-static int32_t
-rdma_post_recv (struct ibv_srq *srq,
- rdma_post_t *post)
-{
- struct ibv_sge list = {
- .addr = (unsigned long) post->buf,
- .length = post->buf_size,
- .lkey = post->mr->lkey
- };
-
- struct ibv_recv_wr wr = {
- .wr_id = (unsigned long) post,
- .sg_list = &list,
- .num_sge = 1,
- }, *bad_wr;
-
- rdma_post_ref (post);
-
- return ibv_post_srq_recv (srq, &wr, &bad_wr);
-}
-
-
int
-rdma_post_unref (rdma_post_t *post)
+gf_rdma_post_unref (gf_rdma_post_t *post)
{
int refcount = -1;
@@ -869,11 +1637,11 @@ rdma_post_unref (rdma_post_t *post)
pthread_mutex_unlock (&post->lock);
if (refcount == 0) {
- rdma_post_context_destroy (&post->ctx);
- if (post->type == RDMA_SEND_POST) {
- rdma_put_post (&post->device->sendq, post);
+ gf_rdma_post_context_destroy (&post->ctx);
+ if (post->type == GF_RDMA_SEND_POST) {
+ gf_rdma_put_post (&post->device->sendq, post);
} else {
- rdma_post_recv (post->device->srq, post);
+ gf_rdma_post_recv (post->device->srq, post);
}
}
out:
@@ -882,7 +1650,7 @@ out:
int
-rdma_post_get_refcount (rdma_post_t *post)
+gf_rdma_post_get_refcount (gf_rdma_post_t *post)
{
int refcount = -1;
@@ -900,8 +1668,8 @@ out:
return refcount;
}
-rdma_post_t *
-rdma_post_ref (rdma_post_t *post)
+gf_rdma_post_t *
+gf_rdma_post_ref (gf_rdma_post_t *post)
{
if (post == NULL) {
goto out;
@@ -912,38 +1680,39 @@ rdma_post_ref (rdma_post_t *post)
post->refcount++;
}
pthread_mutex_unlock (&post->lock);
-
+
out:
return post;
}
int32_t
-__rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
- rdma_post_t *post)
+__gf_rdma_ioq_churn_request (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_post_t *post)
{
- rdma_chunktype_t rtype = rdma_noch, wtype = rdma_noch;
- uint64_t send_size = 0;
- rdma_header_t *hdr = NULL;
- struct rpc_msg *rpc_msg = NULL;
- uint32_t *chunkptr = NULL;
- char *buf = NULL;
- int32_t ret = 0;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
- int chunk_count = 0;
- rdma_request_context_t *request_ctx = NULL;
- uint32_t prog_payload_length = 0, len = 0;
- struct rpc_req *rpc_req = NULL;
-
- if ((peer == NULL) || (entry == NULL) || (post == NULL)) {
- goto out;
- }
+ gf_rdma_chunktype_t rtype = gf_rdma_noch;
+ gf_rdma_chunktype_t wtype = gf_rdma_noch;
+ uint64_t send_size = 0;
+ gf_rdma_header_t *hdr = NULL;
+ struct rpc_msg *rpc_msg = NULL;
+ uint32_t *chunkptr = NULL;
+ char *buf = NULL;
+ int32_t ret = 0;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+ int chunk_count = 0;
+ gf_rdma_request_context_t *request_ctx = NULL;
+ uint32_t prog_payload_length = 0, len = 0;
+ struct rpc_req *rpc_req = NULL;
+
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, peer, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, entry, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, post, out);
if ((entry->msg.request.rsphdr_count != 0)
&& (entry->msg.request.rsp_payload_count != 0)) {
ret = -1;
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"both write-chunklist and reply-chunk cannot be "
"present");
goto out;
@@ -953,11 +1722,11 @@ __rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
priv = peer->trans->private;
device = priv->device;
- hdr = (rdma_header_t *)post->buf;
+ hdr = (gf_rdma_header_t *)post->buf;
send_size = iov_length (entry->rpchdr, entry->rpchdr_count)
+ iov_length (entry->proghdr, entry->proghdr_count)
- + RDMA_MAX_HEADER_SIZE;
+ + GLUSTERFS_RDMA_MAX_HEADER_SIZE;
if (entry->prog_payload_count != 0) {
prog_payload_length
@@ -965,57 +1734,60 @@ __rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
entry->prog_payload_count);
}
- if (send_size > RDMA_INLINE_THRESHOLD) {
- rtype = rdma_areadch;
- } else if ((send_size + prog_payload_length) < RDMA_INLINE_THRESHOLD) {
- rtype = rdma_noch;
+ if (send_size > GLUSTERFS_RDMA_INLINE_THRESHOLD) {
+ rtype = gf_rdma_areadch;
+ } else if ((send_size + prog_payload_length)
+ < GLUSTERFS_RDMA_INLINE_THRESHOLD) {
+ rtype = gf_rdma_noch;
} else if (entry->prog_payload_count != 0) {
- rtype = rdma_readch;
+ rtype = gf_rdma_readch;
}
if (entry->msg.request.rsphdr_count != 0) {
- wtype = rdma_replych;
+ wtype = gf_rdma_replych;
} else if (entry->msg.request.rsp_payload_count != 0) {
- wtype = rdma_writech;
+ wtype = gf_rdma_writech;
}
- if (rtype == rdma_readch) {
+ if (rtype == gf_rdma_readch) {
chunk_count += entry->prog_payload_count;
- } else if (rtype == rdma_areadch) {
+ } else if (rtype == gf_rdma_areadch) {
chunk_count += entry->rpchdr_count;
chunk_count += entry->proghdr_count;
}
- if (wtype == rdma_writech) {
+ if (wtype == gf_rdma_writech) {
chunk_count += entry->msg.request.rsp_payload_count;
- } else if (wtype == rdma_replych) {
+ } else if (wtype == gf_rdma_replych) {
chunk_count += entry->msg.request.rsphdr_count;
}
- if (chunk_count > RDMA_MAX_SEGMENTS) {
+ if (chunk_count > GF_RDMA_MAX_SEGMENTS) {
ret = -1;
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"chunk count(%d) exceeding maximum allowed RDMA "
- "segment count(%d)", chunk_count, RDMA_MAX_SEGMENTS);
+ "segment count(%d)", chunk_count, GF_RDMA_MAX_SEGMENTS);
goto out;
}
- request_ctx = mem_get (priv->request_ctx_pool);
- if (request_ctx == NULL) {
- ret = -1;
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- goto out;
- }
+ if ((wtype != gf_rdma_noch) || (rtype != gf_rdma_noch)) {
+ request_ctx = mem_get (device->request_ctx_pool);
+ if (request_ctx == NULL) {
+ ret = -1;
+ goto out;
+ }
+
+ memset (request_ctx, 0, sizeof (*request_ctx));
- memset (request_ctx, 0, sizeof (*request_ctx));
+ request_ctx->pool = device->request_ctx_pool;
+ request_ctx->peer = peer;
- request_ctx->pool = priv->request_ctx_pool;
- request_ctx->peer = peer;
+ entry->msg.request.rpc_req->conn_private = request_ctx;
- entry->msg.request.rpc_req->conn_private = request_ctx;
- if (entry->msg.request.rsp_iobref != NULL) {
- request_ctx->rsp_iobref
- = iobref_ref (entry->msg.request.rsp_iobref);
+ if (entry->msg.request.rsp_iobref != NULL) {
+ request_ctx->rsp_iobref
+ = iobref_ref (entry->msg.request.rsp_iobref);
+ }
}
rpc_msg = (struct rpc_msg *) entry->rpchdr[0].iov_base;
@@ -1024,20 +1796,21 @@ __rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
* since rpc_msg->rm_xid is already
* hton32ed value of actual xid
*/
- hdr->rm_vers = hton32 (RDMA_VERSION);
+ hdr->rm_vers = hton32 (GF_RDMA_VERSION);
hdr->rm_credit = hton32 (peer->send_count);
- if (rtype == rdma_areadch) {
- hdr->rm_type = hton32 (RDMA_NOMSG);
+ if (rtype == gf_rdma_areadch) {
+ hdr->rm_type = hton32 (GF_RDMA_NOMSG);
} else {
- hdr->rm_type = hton32 (RDMA_MSG);
+ hdr->rm_type = hton32 (GF_RDMA_MSG);
}
chunkptr = &hdr->rm_body.rm_chunks[0];
- if (rtype != rdma_noch) {
- ret = __rdma_create_read_chunks (peer, entry, rtype, &chunkptr,
- request_ctx);
+ if (rtype != gf_rdma_noch) {
+ ret = __gf_rdma_create_read_chunks (peer, entry, rtype,
+ &chunkptr,
+ request_ctx);
if (ret != 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"creation of read chunks failed");
goto out;
}
@@ -1045,11 +1818,12 @@ __rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
*chunkptr++ = 0; /* no read chunks */
}
- if (wtype != rdma_noch) {
- ret = __rdma_create_write_chunks (peer, entry, wtype, &chunkptr,
- request_ctx);
+ if (wtype != gf_rdma_noch) {
+ ret = __gf_rdma_create_write_chunks (peer, entry, wtype,
+ &chunkptr,
+ request_ctx);
if (ret != 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"creation of write/reply chunk failed");
goto out;
}
@@ -1060,14 +1834,14 @@ __rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
buf = (char *)chunkptr;
- if (rtype != rdma_areadch) {
+ if (rtype != gf_rdma_areadch) {
iov_unload (buf, entry->rpchdr, entry->rpchdr_count);
buf += iov_length (entry->rpchdr, entry->rpchdr_count);
iov_unload (buf, entry->proghdr, entry->proghdr_count);
buf += iov_length (entry->proghdr, entry->proghdr_count);
- if (rtype != rdma_readch) {
+ if (rtype != gf_rdma_readch) {
iov_unload (buf, entry->prog_payload,
entry->prog_payload_count);
buf += iov_length (entry->prog_payload,
@@ -1076,17 +1850,19 @@ __rdma_ioq_churn_request (rdma_peer_t *peer, rdma_ioq_t *entry,
}
len = buf - post->buf;
-
- rdma_post_ref (post);
- ret = rdma_post_send (peer->qp, post, len);
+ gf_rdma_post_ref (post);
+
+ ret = gf_rdma_post_send (peer->qp, post, len);
if (!ret) {
ret = len;
} else {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "ibv_post_send failed with ret = %d", ret);
- rdma_post_unref (post);
- __rdma_disconnect (peer->trans);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "gf_rdma_post_send (to %s) failed with ret = %d (%s)",
+ peer->trans->peerinfo.identifier, ret,
+ (ret > 0) ? strerror (ret) : "");
+ gf_rdma_post_unref (post);
+ __gf_rdma_disconnect (peer->trans);
ret = -1;
}
@@ -1094,7 +1870,10 @@ out:
if (ret == -1) {
rpc_req = entry->msg.request.rpc_req;
- __rdma_request_context_destroy (rpc_req->conn_private);
+ if (request_ctx != NULL) {
+ __gf_rdma_request_context_destroy (rpc_req->conn_private);
+ }
+
rpc_req->conn_private = NULL;
}
@@ -1102,9 +1881,9 @@ out:
}
-inline void
-__rdma_fill_reply_header (rdma_header_t *header, struct iovec *rpchdr,
- rdma_reply_info_t *reply_info, int credits)
+static inline void
+__gf_rdma_fill_reply_header (gf_rdma_header_t *header, struct iovec *rpchdr,
+ gf_rdma_reply_info_t *reply_info, int credits)
{
struct rpc_msg *rpc_msg = NULL;
@@ -1112,14 +1891,14 @@ __rdma_fill_reply_header (rdma_header_t *header, struct iovec *rpchdr,
header->rm_xid = hton32 (reply_info->rm_xid);
} else {
rpc_msg = rpchdr[0].iov_base; /* assume rpchdr contains
- * only one vector.
- * (which is true)
- */
+ * only one vector.
+ * (which is true)
+ */
header->rm_xid = rpc_msg->rm_xid;
}
- header->rm_type = hton32 (RDMA_MSG);
- header->rm_vers = hton32 (RDMA_VERSION);
+ header->rm_type = hton32 (GF_RDMA_MSG);
+ header->rm_vers = hton32 (GF_RDMA_VERSION);
header->rm_credit = hton32 (credits);
header->rm_body.rm_chunks[0] = 0; /* no read chunks */
@@ -1131,36 +1910,36 @@ __rdma_fill_reply_header (rdma_header_t *header, struct iovec *rpchdr,
int32_t
-__rdma_send_reply_inline (rdma_peer_t *peer, rdma_ioq_t *entry,
- rdma_post_t *post, rdma_reply_info_t *reply_info)
+__gf_rdma_send_reply_inline (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_post_t *post,
+ gf_rdma_reply_info_t *reply_info)
{
- rdma_header_t *header = NULL;
- int32_t send_size = 0, ret = 0;
- char *buf = NULL;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
-
- priv = peer->trans->private;
- device = priv->device;
+ gf_rdma_header_t *header = NULL;
+ int32_t send_size = 0, ret = 0;
+ char *buf = NULL;
send_size = iov_length (entry->rpchdr, entry->rpchdr_count)
+ iov_length (entry->proghdr, entry->proghdr_count)
+ iov_length (entry->prog_payload, entry->prog_payload_count)
- + sizeof (rdma_header_t); /*
- * remember, no chunklists in the
- * reply
- */
-
- if (send_size > RDMA_INLINE_THRESHOLD) {
- ret = __rdma_send_error (peer, entry, post, reply_info,
- ERR_CHUNK);
+ + sizeof (gf_rdma_header_t); /*
+ * remember, no chunklists in the
+ * reply
+ */
+
+ if (send_size > GLUSTERFS_RDMA_INLINE_THRESHOLD) {
+ ret = __gf_rdma_send_error (peer, entry, post, reply_info,
+ ERR_CHUNK);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "msg size (%d) is greater than maximum size "
+ "of msg that can be sent inlined (%d)",
+ send_size, GLUSTERFS_RDMA_INLINE_THRESHOLD);
goto out;
}
- header = (rdma_header_t *)post->buf;
+ header = (gf_rdma_header_t *)post->buf;
- __rdma_fill_reply_header (header, entry->rpchdr, reply_info,
- peer->send_count);
+ __gf_rdma_fill_reply_header (header, entry->rpchdr, reply_info,
+ peer->send_count);
buf = (char *)&header->rm_body.rm_chunks[3];
@@ -1173,7 +1952,7 @@ __rdma_send_reply_inline (rdma_peer_t *peer, rdma_ioq_t *entry,
iov_unload (buf, entry->proghdr, entry->proghdr_count);
buf += iov_length (entry->proghdr, entry->proghdr_count);
}
-
+
if (entry->prog_payload_count != 0) {
iov_unload (buf, entry->prog_payload,
entry->prog_payload_count);
@@ -1181,16 +1960,18 @@ __rdma_send_reply_inline (rdma_peer_t *peer, rdma_ioq_t *entry,
entry->prog_payload_count);
}
- rdma_post_ref (post);
+ gf_rdma_post_ref (post);
- ret = rdma_post_send (peer->qp, post, (buf - post->buf));
+ ret = gf_rdma_post_send (peer->qp, post, (buf - post->buf));
if (!ret) {
ret = send_size;
} else {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "ibv_post_send failed with ret = %d", ret);
- rdma_post_unref (post);
- __rdma_disconnect (peer->trans);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "posting send (to %s) failed with ret = %d (%s)",
+ peer->trans->peerinfo.identifier, ret,
+ (ret > 0) ? strerror (ret) : "");
+ gf_rdma_post_unref (post);
+ __gf_rdma_disconnect (peer->trans);
ret = -1;
}
@@ -1200,17 +1981,18 @@ out:
int32_t
-__rdma_reply_encode_write_chunks (rdma_peer_t *peer, uint32_t payload_size,
- rdma_post_t *post,
- rdma_reply_info_t *reply_info,
- uint32_t **ptr)
+__gf_rdma_reply_encode_write_chunks (gf_rdma_peer_t *peer,
+ uint32_t payload_size,
+ gf_rdma_post_t *post,
+ gf_rdma_reply_info_t *reply_info,
+ uint32_t **ptr)
{
- uint32_t chunk_size = 0;
- int32_t ret = -1;
- rdma_write_array_t *target_array = NULL;
- int i = 0;
+ uint32_t chunk_size = 0;
+ int32_t ret = -1;
+ gf_rdma_write_array_t *target_array = NULL;
+ int i = 0;
- target_array = (rdma_write_array_t *)*ptr;
+ target_array = (gf_rdma_write_array_t *)*ptr;
for (i = 0; i < reply_info->wc_array->wc_nchunks; i++) {
chunk_size +=
@@ -1218,7 +2000,7 @@ __rdma_reply_encode_write_chunks (rdma_peer_t *peer, uint32_t payload_size,
}
if (chunk_size < payload_size) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
"length of payload (%d) is exceeding the total "
"write chunk length (%d)", payload_size, chunk_size);
goto out;
@@ -1233,7 +2015,7 @@ __rdma_reply_encode_write_chunks (rdma_peer_t *peer, uint32_t payload_size,
target_array->wc_array[i].wc_target.rs_length
= hton32 (min (payload_size,
- reply_info->wc_array->wc_array[i].wc_target.rs_length));
+ reply_info->wc_array->wc_array[i].wc_target.rs_length));
}
target_array->wc_nchunks = hton32 (i);
@@ -1249,18 +2031,17 @@ out:
inline int32_t
-__rdma_register_local_mr_for_rdma (rdma_peer_t *peer,
- struct iovec *vector, int count,
- rdma_post_context_t *ctx)
+__gf_rdma_register_local_mr_for_rdma (gf_rdma_peer_t *peer,
+ struct iovec *vector, int count,
+ gf_rdma_post_context_t *ctx)
{
- int i = 0;
- int32_t ret = -1;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
+ int i = 0;
+ int32_t ret = -1;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
- if ((ctx == NULL) || (vector == NULL)) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, ctx, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, vector, out);
priv = peer->trans->private;
device = priv->device;
@@ -1281,6 +2062,9 @@ __rdma_register_local_mr_for_rdma (rdma_peer_t *peer,
vector[i].iov_len,
IBV_ACCESS_LOCAL_WRITE);
if (ctx->mr[ctx->mr_count] == NULL) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "registering memory for IBV_ACCESS_LOCAL_WRITE "
+ "failed (%s)", strerror (errno));
goto out;
}
@@ -1296,15 +2080,15 @@ out:
* 2. modifies vec
*/
int32_t
-__rdma_write (rdma_peer_t *peer, rdma_post_t *post, struct iovec *vec,
- uint32_t xfer_len, int *idx, rdma_write_chunk_t *writech)
+__gf_rdma_write (gf_rdma_peer_t *peer, gf_rdma_post_t *post, struct iovec *vec,
+ uint32_t xfer_len, int *idx, gf_rdma_write_chunk_t *writech)
{
- int size = 0, num_sge = 0, i = 0;
- int32_t ret = -1;
- struct ibv_sge *sg_list = NULL;
- struct ibv_send_wr wr = {
- .opcode = IBV_WR_RDMA_WRITE,
- .send_flags = IBV_SEND_SIGNALED,
+ int size = 0, num_sge = 0, i = 0;
+ int32_t ret = -1;
+ struct ibv_sge *sg_list = NULL;
+ struct ibv_send_wr wr = {
+ .opcode = IBV_WR_RDMA_WRITE,
+ .send_flags = IBV_SEND_SIGNALED,
}, *bad_wr;
if ((peer == NULL) || (writech == NULL) || (idx == NULL)
@@ -1318,9 +2102,9 @@ __rdma_write (rdma_peer_t *peer, rdma_post_t *post, struct iovec *vec,
num_sge = i - *idx;
- sg_list = GF_CALLOC (num_sge, sizeof (struct ibv_sge), gf_common_mt_sge);
+ sg_list = GF_CALLOC (num_sge, sizeof (struct ibv_sge),
+ gf_common_mt_sge);
if (sg_list == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
ret = -1;
goto out;
}
@@ -1345,11 +2129,19 @@ __rdma_write (rdma_peer_t *peer, rdma_post_t *post, struct iovec *vec,
wr.sg_list = sg_list;
wr.num_sge = num_sge;
- wr.wr_id = (unsigned long) rdma_post_ref (post);
+ wr.wr_id = (unsigned long) gf_rdma_post_ref (post);
wr.wr.rdma.rkey = writech->wc_target.rs_handle;
wr.wr.rdma.remote_addr = writech->wc_target.rs_offset;
ret = ibv_post_send(peer->qp, &wr, &bad_wr);
+ if (ret) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma write to "
+ "client (%s) failed with ret = %d (%s)",
+ peer->trans->peerinfo.identifier, ret,
+ (ret > 0) ? strerror (ret) : "");
+ ret = -1;
+ }
GF_FREE (sg_list);
out:
@@ -1358,14 +2150,15 @@ out:
int32_t
-__rdma_do_rdma_write (rdma_peer_t *peer, rdma_post_t *post,
- struct iovec *vector, int count, struct iobref *iobref,
- rdma_reply_info_t *reply_info)
+__gf_rdma_do_gf_rdma_write (gf_rdma_peer_t *peer, gf_rdma_post_t *post,
+ struct iovec *vector, int count,
+ struct iobref *iobref,
+ gf_rdma_reply_info_t *reply_info)
{
- int i = 0, payload_idx = 0;
- uint32_t payload_size = 0, xfer_len = 0;
- int32_t ret = -1;
-
+ int i = 0, payload_idx = 0;
+ uint32_t payload_size = 0, xfer_len = 0;
+ int32_t ret = -1;
+
if (count != 0) {
payload_size = iov_length (vector, count);
}
@@ -1375,9 +2168,11 @@ __rdma_do_rdma_write (rdma_peer_t *peer, rdma_post_t *post,
goto out;
}
- ret = __rdma_register_local_mr_for_rdma (peer, vector, count,
- &post->ctx);
+ ret = __gf_rdma_register_local_mr_for_rdma (peer, vector, count,
+ &post->ctx);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "registering memory region for rdma failed");
goto out;
}
@@ -1389,9 +2184,13 @@ __rdma_do_rdma_write (rdma_peer_t *peer, rdma_post_t *post,
xfer_len = min (payload_size,
reply_info->wc_array->wc_array[i].wc_target.rs_length);
- ret = __rdma_write (peer, post, vector, xfer_len, &payload_idx,
- &reply_info->wc_array->wc_array[i]);
+ ret = __gf_rdma_write (peer, post, vector, xfer_len,
+ &payload_idx,
+ &reply_info->wc_array->wc_array[i]);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma write to client (%s) failed",
+ peer->trans->peerinfo.identifier);
goto out;
}
@@ -1406,44 +2205,41 @@ out:
int32_t
-__rdma_send_reply_type_nomsg (rdma_peer_t *peer, rdma_ioq_t *entry,
- rdma_post_t *post, rdma_reply_info_t *reply_info)
+__gf_rdma_send_reply_type_nomsg (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_post_t *post,
+ gf_rdma_reply_info_t *reply_info)
{
- rdma_header_t *header = NULL;
- char *buf = NULL;
- uint32_t payload_size = 0;
- int count = 0, i = 0;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
- int32_t ret = 0;
- struct iovec vector[MAX_IOVEC];
-
- priv = peer->trans->private;
- device = priv->device;
+ gf_rdma_header_t *header = NULL;
+ char *buf = NULL;
+ uint32_t payload_size = 0;
+ int count = 0, i = 0;
+ int32_t ret = 0;
+ struct iovec vector[MAX_IOVEC];
- header = (rdma_header_t *)post->buf;
+ header = (gf_rdma_header_t *)post->buf;
- __rdma_fill_reply_header (header, entry->rpchdr, reply_info,
- peer->send_count);
+ __gf_rdma_fill_reply_header (header, entry->rpchdr, reply_info,
+ peer->send_count);
- header->rm_type = hton32 (RDMA_NOMSG);
+ header->rm_type = hton32 (GF_RDMA_NOMSG);
payload_size = iov_length (entry->rpchdr, entry->rpchdr_count) +
iov_length (entry->proghdr, entry->proghdr_count);
/* encode reply chunklist */
buf = (char *)&header->rm_body.rm_chunks[2];
- ret = __rdma_reply_encode_write_chunks (peer, payload_size, post,
- reply_info, (uint32_t **)&buf);
+ ret = __gf_rdma_reply_encode_write_chunks (peer, payload_size, post,
+ reply_info,
+ (uint32_t **)&buf);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"encoding write chunks failed");
- ret = __rdma_send_error (peer, entry, post, reply_info,
- ERR_CHUNK);
+ ret = __gf_rdma_send_error (peer, entry, post, reply_info,
+ ERR_CHUNK);
goto out;
}
- rdma_post_ref (post);
+ gf_rdma_post_ref (post);
for (i = 0; i < entry->rpchdr_count; i++) {
vector[count++] = entry->rpchdr[i];
@@ -1453,20 +2249,24 @@ __rdma_send_reply_type_nomsg (rdma_peer_t *peer, rdma_ioq_t *entry,
vector[count++] = entry->proghdr[i];
}
- ret = __rdma_do_rdma_write (peer, post, vector, count, entry->iobref,
- reply_info);
+ ret = __gf_rdma_do_gf_rdma_write (peer, post, vector, count,
+ entry->iobref, reply_info);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "rdma write to client failed");
- rdma_post_unref (post);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma write to peer (%s) failed",
+ peer->trans->peerinfo.identifier);
+ gf_rdma_post_unref (post);
goto out;
}
- ret = rdma_post_send (peer->qp, post, (buf - post->buf));
- if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "rdma send to client failed");
- rdma_post_unref (post);
+ ret = gf_rdma_post_send (peer->qp, post, (buf - post->buf));
+ if (ret) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "posting a send request to client (%s) failed with "
+ "ret = %d (%s)", peer->trans->peerinfo.identifier, ret,
+ (ret > 0) ? strerror (ret) : "");
+ ret = -1;
+ gf_rdma_post_unref (post);
} else {
ret = payload_size;
}
@@ -1477,66 +2277,64 @@ out:
int32_t
-__rdma_send_reply_type_msg (rdma_peer_t *peer, rdma_ioq_t *entry,
- rdma_post_t *post, rdma_reply_info_t *reply_info)
+__gf_rdma_send_reply_type_msg (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_post_t *post,
+ gf_rdma_reply_info_t *reply_info)
{
- rdma_header_t *header = NULL;
- int32_t send_size = 0, ret = 0;
- char *ptr = NULL;
- uint32_t payload_size = 0;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
-
- priv = peer->trans->private;
- device = priv->device;
+ gf_rdma_header_t *header = NULL;
+ int32_t send_size = 0, ret = 0;
+ char *ptr = NULL;
+ uint32_t payload_size = 0;
send_size = iov_length (entry->rpchdr, entry->rpchdr_count)
+ iov_length (entry->proghdr, entry->proghdr_count)
- + RDMA_MAX_HEADER_SIZE;
+ + GLUSTERFS_RDMA_MAX_HEADER_SIZE;
- if (send_size > RDMA_INLINE_THRESHOLD) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ if (send_size > GLUSTERFS_RDMA_INLINE_THRESHOLD) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"client has provided only write chunks, but the "
"combined size of rpc and program header (%d) is "
"exceeding the size of msg that can be sent using "
- "RDMA send (%d)", send_size, RDMA_INLINE_THRESHOLD);
+ "RDMA send (%d)", send_size,
+ GLUSTERFS_RDMA_INLINE_THRESHOLD);
- ret = __rdma_send_error (peer, entry, post, reply_info,
- ERR_CHUNK);
+ ret = __gf_rdma_send_error (peer, entry, post, reply_info,
+ ERR_CHUNK);
goto out;
}
- header = (rdma_header_t *)post->buf;
+ header = (gf_rdma_header_t *)post->buf;
- __rdma_fill_reply_header (header, entry->rpchdr, reply_info,
- peer->send_count);
+ __gf_rdma_fill_reply_header (header, entry->rpchdr, reply_info,
+ peer->send_count);
payload_size = iov_length (entry->prog_payload,
entry->prog_payload_count);
ptr = (char *)&header->rm_body.rm_chunks[1];
- ret = __rdma_reply_encode_write_chunks (peer, payload_size, post,
- reply_info, (uint32_t **)&ptr);
+ ret = __gf_rdma_reply_encode_write_chunks (peer, payload_size, post,
+ reply_info,
+ (uint32_t **)&ptr);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"encoding write chunks failed");
- ret = __rdma_send_error (peer, entry, post, reply_info,
- ERR_CHUNK);
+ ret = __gf_rdma_send_error (peer, entry, post, reply_info,
+ ERR_CHUNK);
goto out;
}
*(uint32_t *)ptr = 0; /* terminate reply chunklist */
ptr += sizeof (uint32_t);
- rdma_post_ref (post);
+ gf_rdma_post_ref (post);
- ret = __rdma_do_rdma_write (peer, post, entry->prog_payload,
- entry->prog_payload_count, entry->iobref,
- reply_info);
+ ret = __gf_rdma_do_gf_rdma_write (peer, post, entry->prog_payload,
+ entry->prog_payload_count,
+ entry->iobref, reply_info);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "rdma write to client failed");
- rdma_post_unref (post);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING, "rdma write to peer "
+ "(%s) failed", peer->trans->peerinfo.identifier);
+ gf_rdma_post_unref (post);
goto out;
}
@@ -1546,11 +2344,14 @@ __rdma_send_reply_type_msg (rdma_peer_t *peer, rdma_ioq_t *entry,
iov_unload (ptr, entry->proghdr, entry->proghdr_count);
ptr += iov_length (entry->proghdr, entry->proghdr_count);
- ret = rdma_post_send (peer->qp, post, (ptr - post->buf));
- if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "rdma send to client failed");
- rdma_post_unref (post);
+ ret = gf_rdma_post_send (peer->qp, post, (ptr - post->buf));
+ if (ret) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma send to client (%s) failed with ret = %d (%s)",
+ peer->trans->peerinfo.identifier, ret,
+ (ret > 0) ? strerror (ret) : "");
+ gf_rdma_post_unref (post);
+ ret = -1;
} else {
ret = send_size + payload_size;
}
@@ -1561,7 +2362,7 @@ out:
void
-rdma_reply_info_destroy (rdma_reply_info_t *reply_info)
+gf_rdma_reply_info_destroy (gf_rdma_reply_info_t *reply_info)
{
if (reply_info == NULL) {
goto out;
@@ -1572,28 +2373,27 @@ rdma_reply_info_destroy (rdma_reply_info_t *reply_info)
reply_info->wc_array = NULL;
}
- mem_put (reply_info->pool, reply_info);
+ mem_put (reply_info);
out:
return;
}
-rdma_reply_info_t *
-rdma_reply_info_alloc (rdma_peer_t *peer)
+gf_rdma_reply_info_t *
+gf_rdma_reply_info_alloc (gf_rdma_peer_t *peer)
{
- rdma_reply_info_t *reply_info = NULL;
- rdma_private_t *priv = NULL;
+ gf_rdma_reply_info_t *reply_info = NULL;
+ gf_rdma_private_t *priv = NULL;
priv = peer->trans->private;
- reply_info = mem_get (priv->reply_info_pool);
+ reply_info = mem_get (priv->device->reply_info_pool);
if (reply_info == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto out;
}
memset (reply_info, 0, sizeof (*reply_info));
- reply_info->pool = priv->reply_info_pool;
+ reply_info->pool = priv->device->reply_info_pool;
out:
return reply_info;
@@ -1601,15 +2401,16 @@ out:
int32_t
-__rdma_ioq_churn_reply (rdma_peer_t *peer, rdma_ioq_t *entry, rdma_post_t *post)
+__gf_rdma_ioq_churn_reply (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry,
+ gf_rdma_post_t *post)
{
- rdma_reply_info_t *reply_info = NULL;
- int32_t ret = -1;
- rdma_chunktype_t type = rdma_noch;
+ gf_rdma_reply_info_t *reply_info = NULL;
+ int32_t ret = -1;
+ gf_rdma_chunktype_t type = gf_rdma_noch;
- if ((peer == NULL) || (entry == NULL) || (post == NULL)) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, peer, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, entry, out);
+ GF_VALIDATE_OR_GOTO (GF_RDMA_LOG_NAME, post, out);
reply_info = entry->msg.reply_info;
if (reply_info != NULL) {
@@ -1617,29 +2418,47 @@ __rdma_ioq_churn_reply (rdma_peer_t *peer, rdma_ioq_t *entry, rdma_post_t *post)
}
switch (type) {
- case rdma_noch:
- ret = __rdma_send_reply_inline (peer, entry, post, reply_info);
+ case gf_rdma_noch:
+ ret = __gf_rdma_send_reply_inline (peer, entry, post,
+ reply_info);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "failed to send reply to peer (%s) as an "
+ "inlined rdma msg",
+ peer->trans->peerinfo.identifier);
+ }
break;
- case rdma_replych:
- ret = __rdma_send_reply_type_nomsg (peer, entry, post,
- reply_info);
+ case gf_rdma_replych:
+ ret = __gf_rdma_send_reply_type_nomsg (peer, entry, post,
+ reply_info);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "failed to send reply to peer (%s) as "
+ "RDMA_NOMSG", peer->trans->peerinfo.identifier);
+ }
break;
- case rdma_writech:
- ret = __rdma_send_reply_type_msg (peer, entry, post,
- reply_info);
+ case gf_rdma_writech:
+ ret = __gf_rdma_send_reply_type_msg (peer, entry, post,
+ reply_info);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "failed to send reply with write chunks "
+ "to peer (%s)",
+ peer->trans->peerinfo.identifier);
+ }
break;
default:
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "invalid chunktype (%d) specified for sending reply",
- type);
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "invalid chunktype (%d) specified for sending reply "
+ " (peer:%s)", type, peer->trans->peerinfo.identifier);
break;
}
if (reply_info != NULL) {
- rdma_reply_info_destroy (reply_info);
+ gf_rdma_reply_info_destroy (reply_info);
}
out:
return ret;
@@ -1647,41 +2466,57 @@ out:
int32_t
-__rdma_ioq_churn_entry (rdma_peer_t *peer, rdma_ioq_t *entry)
+__gf_rdma_ioq_churn_entry (gf_rdma_peer_t *peer, gf_rdma_ioq_t *entry)
{
- int32_t ret = 0, quota = 0;
- rdma_private_t *priv = NULL;
- rdma_device_t *device = NULL;
- rdma_options_t *options = NULL;
- rdma_post_t *post = NULL;
+ int32_t ret = 0, quota = 0;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+ gf_rdma_options_t *options = NULL;
+ gf_rdma_post_t *post = NULL;
priv = peer->trans->private;
options = &priv->options;
device = priv->device;
- quota = __rdma_quota_get (peer);
+ quota = __gf_rdma_quota_get (peer);
if (quota > 0) {
- post = rdma_get_post (&device->sendq);
- if (post == NULL) {
- post = rdma_new_post (device,
- (options->send_size + 2048),
- RDMA_SEND_POST);
+ post = gf_rdma_get_post (&device->sendq);
+ if (post == NULL) {
+ post = gf_rdma_new_post (peer->trans, device,
+ (options->send_size + 2048),
+ GF_RDMA_SEND_POST);
}
if (post == NULL) {
ret = -1;
+ gf_log_callingfn (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "not able to get a post to send msg");
goto out;
}
if (entry->is_request) {
- ret = __rdma_ioq_churn_request (peer, entry, post);
+ ret = __gf_rdma_ioq_churn_request (peer, entry, post);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "failed to process request ioq entry "
+ "to peer(%s)",
+ peer->trans->peerinfo.identifier);
+ }
} else {
- ret = __rdma_ioq_churn_reply (peer, entry, post);
+ ret = __gf_rdma_ioq_churn_reply (peer, entry, post);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "failed to process reply ioq entry "
+ "to peer (%s)",
+ peer->trans->peerinfo.identifier);
+ }
}
if (ret != 0) {
- __rdma_ioq_entry_free (entry);
+ __gf_rdma_ioq_entry_free (entry);
}
+ } else {
+ ret = 0;
}
out:
@@ -1690,17 +2525,17 @@ out:
static int32_t
-__rdma_ioq_churn (rdma_peer_t *peer)
+__gf_rdma_ioq_churn (gf_rdma_peer_t *peer)
{
- rdma_ioq_t *entry = NULL;
- int32_t ret = 0;
+ gf_rdma_ioq_t *entry = NULL;
+ int32_t ret = 0;
while (!list_empty (&peer->ioq))
{
/* pick next entry */
entry = peer->ioq_next;
- ret = __rdma_ioq_churn_entry (peer, entry);
+ ret = __gf_rdma_ioq_churn_entry (peer, entry);
if (ret <= 0)
break;
@@ -1708,7 +2543,7 @@ __rdma_ioq_churn (rdma_peer_t *peer)
/*
list_for_each_entry_safe (entry, dummy, &peer->ioq, list) {
- ret = __rdma_ioq_churn_entry (peer, entry);
+ ret = __gf_rdma_ioq_churn_entry (peer, entry);
if (ret <= 0) {
break;
}
@@ -1720,28 +2555,35 @@ __rdma_ioq_churn (rdma_peer_t *peer)
static int32_t
-rdma_writev (rpc_transport_t *this,
- rdma_ioq_t *entry)
+gf_rdma_writev (rpc_transport_t *this, gf_rdma_ioq_t *entry)
{
- int32_t ret = 0, need_append = 1;
- rdma_private_t *priv = this->private;
- rdma_peer_t *peer = NULL;
+ int32_t ret = 0, need_append = 1;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_peer_t *peer = NULL;
+ priv = this->private;
pthread_mutex_lock (&priv->write_mutex);
{
if (!priv->connected) {
- gf_log (this->name, GF_LOG_DEBUG,
- "rdma is not connected to post a "
- "send request");
+ gf_log (this->name, GF_LOG_WARNING,
+ "rdma is not connected to peer (%s)",
+ this->peerinfo.identifier);
ret = -1;
goto unlock;
}
peer = &priv->peer;
if (list_empty (&peer->ioq)) {
- ret = __rdma_ioq_churn_entry (peer, entry);
- if (ret > 0) {
+ ret = __gf_rdma_ioq_churn_entry (peer, entry);
+ if (ret != 0) {
need_append = 0;
+
+ if (ret < 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "processing ioq entry destined "
+ "to (%s) failed",
+ this->peerinfo.identifier);
+ }
}
}
@@ -1755,27 +2597,26 @@ unlock:
}
-rdma_ioq_t *
-rdma_ioq_new (rpc_transport_t *this, rpc_transport_data_t *data)
+gf_rdma_ioq_t *
+gf_rdma_ioq_new (rpc_transport_t *this, rpc_transport_data_t *data)
{
- rdma_ioq_t *entry = NULL;
- int count = 0, i = 0;
- rpc_transport_msg_t *msg = NULL;
- rdma_private_t *priv = NULL;
+ gf_rdma_ioq_t *entry = NULL;
+ int count = 0, i = 0;
+ rpc_transport_msg_t *msg = NULL;
+ gf_rdma_private_t *priv = NULL;
if ((data == NULL) || (this == NULL)) {
goto out;
}
priv = this->private;
- /* TODO: use mem-pool */
- entry = mem_get (priv->ioq_pool);
+
+ entry = mem_get (priv->device->ioq_pool);
if (entry == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto out;
}
memset (entry, 0, sizeof (*entry));
- entry->pool = priv->ioq_pool;
+ entry->pool = priv->device->ioq_pool;
if (data->is_request) {
msg = &data->data.req.msg;
@@ -1792,7 +2633,7 @@ rdma_ioq_new (rpc_transport_t *this, rpc_transport_data_t *data)
if (data->data.req.rsp.rsp_payload_count != 0) {
for (i = 0; i < data->data.req.rsp.rsp_payload_count;
i++) {
- entry->msg.request.rsp_payload[i]
+ entry->msg.request.rsp_payload[i]
= data->data.req.rsp.rsp_payload[i];
}
@@ -1815,7 +2656,7 @@ rdma_ioq_new (rpc_transport_t *this, rpc_transport_data_t *data)
count = msg->rpchdrcount + msg->proghdrcount + msg->progpayloadcount;
- assert (count <= MAX_IOVEC);
+ GF_ASSERT (count <= MAX_IOVEC);
if (msg->rpchdr != NULL) {
memcpy (&entry->rpchdr[0], msg->rpchdr,
@@ -1847,12 +2688,11 @@ out:
int32_t
-rdma_submit_request (rpc_transport_t *this,
- rpc_transport_req_t *req)
+gf_rdma_submit_request (rpc_transport_t *this, rpc_transport_req_t *req)
{
- int32_t ret = 0;
- rdma_ioq_t *entry = NULL;
- rpc_transport_data_t data = {0, };
+ int32_t ret = 0;
+ gf_rdma_ioq_t *entry = NULL;
+ rpc_transport_data_t data = {0, };
if (req == NULL) {
goto out;
@@ -1861,16 +2701,22 @@ rdma_submit_request (rpc_transport_t *this,
data.is_request = 1;
data.data.req = *req;
- entry = rdma_ioq_new (this, &data);
+ entry = gf_rdma_ioq_new (this, &data);
if (entry == NULL) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "getting a new ioq entry failed (peer:%s)",
+ this->peerinfo.identifier);
goto out;
}
- ret = rdma_writev (this, entry);
+ ret = gf_rdma_writev (this, entry);
if (ret > 0) {
ret = 0;
} else if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "sending request to peer (%s) failed",
+ this->peerinfo.identifier);
rpc_transport_disconnect (this);
}
@@ -1879,11 +2725,11 @@ out:
}
int32_t
-rdma_submit_reply (rpc_transport_t *this, rpc_transport_reply_t *reply)
+gf_rdma_submit_reply (rpc_transport_t *this, rpc_transport_reply_t *reply)
{
- int32_t ret = 0;
- rdma_ioq_t *entry = NULL;
- rpc_transport_data_t data = {0, };
+ int32_t ret = 0;
+ gf_rdma_ioq_t *entry = NULL;
+ rpc_transport_data_t data = {0, };
if (reply == NULL) {
goto out;
@@ -1891,15 +2737,21 @@ rdma_submit_reply (rpc_transport_t *this, rpc_transport_reply_t *reply)
data.data.reply = *reply;
- entry = rdma_ioq_new (this, &data);
+ entry = gf_rdma_ioq_new (this, &data);
if (entry == NULL) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "getting a new ioq entry failed (peer:%s)",
+ this->peerinfo.identifier);
goto out;
}
- ret = rdma_writev (this, entry);
+ ret = gf_rdma_writev (this, entry);
if (ret > 0) {
ret = 0;
} else if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "sending request to peer (%s) failed",
+ this->peerinfo.identifier);
rpc_transport_disconnect (this);
}
@@ -1907,179 +2759,15 @@ out:
return ret;
}
-#if 0
-static int
-rdma_receive (rpc_transport_t *this, char **hdr_p, size_t *hdrlen_p,
- struct iobuf **iobuf_p)
-{
- rdma_private_t *priv = this->private;
- /* TODO: return error if !priv->connected, check with locks */
- /* TODO: boundry checks for data_ptr/offset */
- char *copy_from = NULL;
- rdma_header_t *header = NULL;
- uint32_t size1, size2, data_len = 0;
- char *hdr = NULL;
- struct iobuf *iobuf = NULL;
- int32_t ret = 0;
-
- pthread_mutex_lock (&priv->recv_mutex);
- {
-/*
- while (!priv->data_ptr)
- pthread_cond_wait (&priv->recv_cond, &priv->recv_mutex);
-*/
-
- copy_from = priv->data_ptr + priv->data_offset;
-
- priv->data_ptr = NULL;
- data_len = priv->data_len;
- pthread_cond_broadcast (&priv->recv_cond);
- }
- pthread_mutex_unlock (&priv->recv_mutex);
-
- header = (rdma_header_t *)copy_from;
- if (strcmp (header->colonO, ":O")) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "%s: corrupt header received", this->name);
- ret = -1;
- goto err;
- }
-
- size1 = ntoh32 (header->size1);
- size2 = ntoh32 (header->size2);
-
- if (data_len != (size1 + size2 + sizeof (*header))) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "%s: sizeof data read from transport is not equal "
- "to the size specified in the header",
- this->name);
- ret = -1;
- goto err;
- }
-
- copy_from += sizeof (*header);
-
- if (size1) {
- hdr = GF_CALLOC (1, size1, gf_common_mt_char);
- if (!hdr) {
- gf_log (this->name, GF_LOG_ERROR,
- "unable to allocate header for peer %s",
- this->peerinfo.identifier);
- ret = -ENOMEM;
- goto err;
- }
- memcpy (hdr, copy_from, size1);
- copy_from += size1;
- *hdr_p = hdr;
- }
- *hdrlen_p = size1;
-
- if (size2) {
- iobuf = iobuf_get (this->ctx->iobuf_pool);
- if (!iobuf) {
- gf_log (this->name, GF_LOG_ERROR,
- "unable to allocate IO buffer for peer %s",
- this->peerinfo.identifier);
- ret = -ENOMEM;
- goto err;
- }
- memcpy (iobuf->ptr, copy_from, size2);
- *iobuf_p = iobuf;
- }
-
-err:
- return ret;
-}
-#endif
-
-
-static void
-rdma_destroy_cq (rpc_transport_t *this)
-{
- rdma_private_t *priv = this->private;
- rdma_device_t *device = priv->device;
-
- if (device->recv_cq)
- ibv_destroy_cq (device->recv_cq);
- device->recv_cq = NULL;
-
- if (device->send_cq)
- ibv_destroy_cq (device->send_cq);
- device->send_cq = NULL;
-
- return;
-}
-
-
-static int32_t
-rdma_create_cq (rpc_transport_t *this)
-{
- rdma_private_t *priv = this->private;
- rdma_options_t *options = &priv->options;
- rdma_device_t *device = priv->device;
- int32_t ret = 0;
-
- device->recv_cq = ibv_create_cq (priv->device->context,
- options->recv_count * 2,
- device,
- device->recv_chan,
- 0);
- if (!device->recv_cq) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: creation of CQ failed",
- this->name);
- ret = -1;
- } else if (ibv_req_notify_cq (device->recv_cq, 0)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: ibv_req_notify_cq on CQ failed",
- this->name);
- ret = -1;
- }
-
- do {
- /* TODO: make send_cq size dynamically adaptive */
- device->send_cq = ibv_create_cq (priv->device->context,
- options->send_count * 1024,
- device,
- device->send_chan,
- 0);
- if (!device->send_cq) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: creation of send_cq failed",
- this->name);
- ret = -1;
- break;
- }
-
- if (ibv_req_notify_cq (device->send_cq, 0)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: ibv_req_notify_cq on send_cq failed",
- this->name);
- ret = -1;
- break;
- }
- } while (0);
-
- if (ret != 0)
- rdma_destroy_cq (this);
-
- return ret;
-}
-
static int
-rdma_register_peer (rdma_device_t *device,
- int32_t qp_num,
- rdma_peer_t *peer)
+gf_rdma_register_peer (gf_rdma_device_t *device, int32_t qp_num,
+ gf_rdma_peer_t *peer)
{
- struct _qpent *ent = NULL;
- rdma_qpreg_t *qpreg = NULL;
- int32_t hash = 0;
- int ret = -1;
+ struct _qpent *ent = NULL;
+ gf_rdma_qpreg_t *qpreg = NULL;
+ int32_t hash = 0;
+ int ret = -1;
qpreg = &device->qpreg;
hash = qp_num % 42;
@@ -2099,7 +2787,6 @@ rdma_register_peer (rdma_device_t *device,
ent = (struct _qpent *) GF_CALLOC (1, sizeof (*ent),
gf_common_mt_qpent);
if (ent == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto unlock;
}
@@ -2121,37 +2808,41 @@ unlock:
static void
-rdma_unregister_peer (rdma_device_t *device,
- int32_t qp_num)
+gf_rdma_unregister_peer (gf_rdma_device_t *device, int32_t qp_num)
{
- struct _qpent *ent;
- rdma_qpreg_t *qpreg = &device->qpreg;
- int32_t hash = qp_num % 42;
+ struct _qpent *ent = NULL;
+ gf_rdma_qpreg_t *qpreg = NULL;
+ int32_t hash = 0;
+
+ qpreg = &device->qpreg;
+ hash = qp_num % 42;
pthread_mutex_lock (&qpreg->lock);
- ent = qpreg->ents[hash].next;
- while ((ent != &qpreg->ents[hash]) && (ent->qp_num != qp_num))
- ent = ent->next;
- if (ent->qp_num != qp_num) {
- pthread_mutex_unlock (&qpreg->lock);
- return;
- }
- ent->prev->next = ent->next;
- ent->next->prev = ent->prev;
- /* TODO: unref reg->peer */
- GF_FREE (ent);
- qpreg->count--;
+ {
+ ent = qpreg->ents[hash].next;
+ while ((ent != &qpreg->ents[hash]) && (ent->qp_num != qp_num))
+ ent = ent->next;
+ if (ent->qp_num != qp_num) {
+ pthread_mutex_unlock (&qpreg->lock);
+ return;
+ }
+ ent->prev->next = ent->next;
+ ent->next->prev = ent->prev;
+ /* TODO: unref reg->peer */
+ GF_FREE (ent);
+ qpreg->count--;
+ }
pthread_mutex_unlock (&qpreg->lock);
}
-static rdma_peer_t *
-__rdma_lookup_peer (rdma_device_t *device, int32_t qp_num)
+static gf_rdma_peer_t *
+__gf_rdma_lookup_peer (gf_rdma_device_t *device, int32_t qp_num)
{
- struct _qpent *ent = NULL;
- rdma_peer_t *peer = NULL;
- rdma_qpreg_t *qpreg = NULL;
- int32_t hash = 0;
+ struct _qpent *ent = NULL;
+ gf_rdma_peer_t *peer = NULL;
+ gf_rdma_qpreg_t *qpreg = NULL;
+ int32_t hash = 0;
qpreg = &device->qpreg;
hash = qp_num % 42;
@@ -2166,34 +2857,16 @@ __rdma_lookup_peer (rdma_device_t *device, int32_t qp_num)
return peer;
}
-/*
- static rdma_peer_t *
- rdma_lookup_peer (rdma_device_t *device,
- int32_t qp_num)
- {
- rdma_qpreg_t *qpreg = NULL;
- rdma_peer_t *peer = NULL;
-
- qpreg = &device->qpreg;
- pthread_mutex_lock (&qpreg->lock);
- {
- peer = __rdma_lookup_peer (device, qp_num);
- }
- pthread_mutex_unlock (&qpreg->lock);
-
- return peer;
- }
-*/
-
static void
-__rdma_destroy_qp (rpc_transport_t *this)
+__gf_rdma_destroy_qp (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
+ gf_rdma_private_t *priv = NULL;
+ priv = this->private;
if (priv->peer.qp) {
- rdma_unregister_peer (priv->device, priv->peer.qp->qp_num);
- ibv_destroy_qp (priv->peer.qp);
+ gf_rdma_unregister_peer (priv->device, priv->peer.qp->qp_num);
+ rdma_destroy_qp (priv->peer.cm_id);
}
priv->peer.qp = NULL;
@@ -2202,15 +2875,38 @@ __rdma_destroy_qp (rpc_transport_t *this)
static int32_t
-rdma_create_qp (rpc_transport_t *this)
+gf_rdma_create_qp (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
- rdma_options_t *options = &priv->options;
- rdma_device_t *device = priv->device;
- int32_t ret = 0;
- rdma_peer_t *peer;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_device_t *device = NULL;
+ int32_t ret = 0;
+ gf_rdma_peer_t *peer = NULL;
+ char *device_name = NULL;
+
+ priv = this->private;
peer = &priv->peer;
+
+ device_name = (char *)ibv_get_device_name (peer->cm_id->verbs->device);
+ if (device_name == NULL) {
+ ret = -1;
+ gf_log (this->name, GF_LOG_WARNING, "cannot get device_name");
+ goto out;
+ }
+
+ device = gf_rdma_get_device (this, peer->cm_id->verbs,
+ device_name);
+ if (device == NULL) {
+ ret = -1;
+ gf_log (this->name, GF_LOG_WARNING, "cannot get device for "
+ "device %s", device_name);
+ goto out;
+ }
+
+ if (priv->device == NULL) {
+ priv->device = device;
+ }
+
struct ibv_qp_init_attr init_attr = {
.send_cq = device->send_cq,
.recv_cq = device->recv_cq,
@@ -2218,339 +2914,79 @@ rdma_create_qp (rpc_transport_t *this)
.cap = {
.max_send_wr = peer->send_count,
.max_recv_wr = peer->recv_count,
- .max_send_sge = 1,
+ .max_send_sge = 2,
.max_recv_sge = 1
},
.qp_type = IBV_QPT_RC
};
-
- struct ibv_qp_attr attr = {
- .qp_state = IBV_QPS_INIT,
- .pkey_index = 0,
- .port_num = options->port,
- .qp_access_flags
- = IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE
- };
-
- peer->qp = ibv_create_qp (device->pd, &init_attr);
- if (!peer->qp) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_CRITICAL,
- "%s: could not create QP",
- this->name);
- ret = -1;
- goto out;
- } else if (ibv_modify_qp (peer->qp, &attr,
- IBV_QP_STATE |
- IBV_QP_PKEY_INDEX |
- IBV_QP_PORT |
- IBV_QP_ACCESS_FLAGS)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: failed to modify QP to INIT state",
- this->name);
+
+ ret = rdma_create_qp(peer->cm_id, device->pd, &init_attr);
+ if (ret != 0) {
+ gf_log (peer->trans->name, GF_LOG_CRITICAL,
+ "%s: could not create QP (%s)", this->name,
+ strerror (errno));
ret = -1;
goto out;
}
- peer->local_lid = rdma_get_local_lid (device->context,
- options->port);
- peer->local_qpn = peer->qp->qp_num;
- peer->local_psn = lrand48 () & 0xffffff;
+ peer->qp = peer->cm_id->qp;
- ret = rdma_register_peer (device, peer->qp->qp_num, peer);
+ ret = gf_rdma_register_peer (device, peer->qp->qp_num, peer);
out:
if (ret == -1)
- __rdma_destroy_qp (this);
+ __gf_rdma_destroy_qp (this);
return ret;
}
-static void
-rdma_destroy_posts (rpc_transport_t *this)
-{
-
-}
-
-
static int32_t
-__rdma_create_posts (rpc_transport_t *this, int32_t count, int32_t size,
- rdma_queue_t *q, rdma_post_type_t type)
+__gf_rdma_teardown (rpc_transport_t *this)
{
- int32_t i;
- int32_t ret = 0;
- rdma_private_t *priv = this->private;
- rdma_device_t *device = priv->device;
-
- for (i=0 ; i<count ; i++) {
- rdma_post_t *post;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_peer_t *peer = NULL;
- post = rdma_new_post (device, size + 2048, type);
- if (!post) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: post creation failed",
- this->name);
- ret = -1;
- break;
- }
-
- rdma_put_post (q, post);
- }
- return ret;
-}
-
-
-static int32_t
-rdma_create_posts (rpc_transport_t *this)
-{
- int32_t i, ret;
- rdma_post_t *post = NULL;
- rdma_private_t *priv = this->private;
- rdma_options_t *options = &priv->options;
- rdma_device_t *device = priv->device;
-
- ret = __rdma_create_posts (this, options->send_count,
- options->send_size,
- &device->sendq, RDMA_SEND_POST);
- if (!ret)
- ret = __rdma_create_posts (this, options->recv_count,
- options->recv_size,
- &device->recvq, RDMA_RECV_POST);
+ priv = this->private;
+ peer = &priv->peer;
- if (!ret) {
- for (i=0 ; i<options->recv_count ; i++) {
- post = rdma_get_post (&device->recvq);
- if (rdma_post_recv (device->srq, post) != 0) {
- ret = -1;
- break;
- }
- }
+ if (peer->cm_id->qp != NULL) {
+ __gf_rdma_destroy_qp (this);
}
- if (ret)
- rdma_destroy_posts (this);
-
- return ret;
-}
-
-
-static int32_t
-rdma_connect_qp (rpc_transport_t *this)
-{
- rdma_private_t *priv = this->private;
- rdma_options_t *options = &priv->options;
- struct ibv_qp_attr attr = {
- .qp_state = IBV_QPS_RTR,
- .path_mtu = options->mtu,
- .dest_qp_num = priv->peer.remote_qpn,
- .rq_psn = priv->peer.remote_psn,
- .max_dest_rd_atomic = 1,
- .min_rnr_timer = 12,
- .qp_access_flags
- = IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE,
- .ah_attr = {
- .is_global = 0,
- .dlid = priv->peer.remote_lid,
- .sl = 0,
- .src_path_bits = 0,
- .port_num = options->port
- }
- };
- if (ibv_modify_qp (priv->peer.qp, &attr,
- IBV_QP_STATE |
- IBV_QP_AV |
- IBV_QP_PATH_MTU |
- IBV_QP_DEST_QPN |
- IBV_QP_RQ_PSN |
- IBV_QP_MAX_DEST_RD_ATOMIC |
- IBV_QP_MIN_RNR_TIMER)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_CRITICAL,
- "Failed to modify QP to RTR\n");
- return -1;
+ if (!list_empty (&priv->peer.ioq)) {
+ __gf_rdma_ioq_flush (peer);
}
- /* TODO: make timeout and retry_cnt configurable from options */
- attr.qp_state = IBV_QPS_RTS;
- attr.timeout = 14;
- attr.retry_cnt = 7;
- attr.rnr_retry = 7;
- attr.sq_psn = priv->peer.local_psn;
- attr.max_rd_atomic = 1;
- if (ibv_modify_qp (priv->peer.qp, &attr,
- IBV_QP_STATE |
- IBV_QP_TIMEOUT |
- IBV_QP_RETRY_CNT |
- IBV_QP_RNR_RETRY |
- IBV_QP_SQ_PSN |
- IBV_QP_MAX_QP_RD_ATOMIC)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_CRITICAL,
- "Failed to modify QP to RTS\n");
- return -1;
+ if (peer->cm_id != NULL) {
+ rdma_destroy_id (peer->cm_id);
+ peer->cm_id = NULL;
}
+ /* TODO: decrement cq size */
return 0;
}
+
static int32_t
-__rdma_teardown (rpc_transport_t *this)
+gf_rdma_teardown (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
-
- __rdma_destroy_qp (this);
+ int32_t ret = 0;
+ gf_rdma_private_t *priv = NULL;
- if (!list_empty (&priv->peer.ioq)) {
- __rdma_ioq_flush (&priv->peer);
+ if (this == NULL) {
+ goto out;
}
- /* TODO: decrement cq size */
- return 0;
-}
-
-/*
- * return value:
- * 0 = success (completed)
- * -1 = error
- * > 0 = incomplete
- */
-
-static int
-__tcp_rwv (rpc_transport_t *this, struct iovec *vector, int count,
- struct iovec **pending_vector, int *pending_count,
- int write)
-{
- rdma_private_t *priv = NULL;
- int sock = -1;
- int ret = -1;
- struct iovec *opvector = vector;
- int opcount = count;
- int moved = 0;
-
priv = this->private;
- sock = priv->sock;
- while (opcount)
+ pthread_mutex_lock (&priv->write_mutex);
{
- if (write)
- {
- ret = writev (sock, opvector, opcount);
-
- if (ret == 0 || (ret == -1 && errno == EAGAIN))
- {
- /* done for now */
- break;
- }
- }
- else
- {
- ret = readv (sock, opvector, opcount);
-
- if (ret == -1 && errno == EAGAIN)
- {
- /* done for now */
- break;
- }
- }
-
- if (ret == 0)
- {
- gf_log (this->name, GF_LOG_DEBUG,
- "EOF from peer %s", this->peerinfo.identifier);
- opcount = -1;
- errno = ENOTCONN;
- break;
- }
-
- if (ret == -1)
- {
- if (errno == EINTR)
- continue;
-
- gf_log (this->name, GF_LOG_DEBUG,
- "%s failed (%s)", write ? "writev" : "readv",
- strerror (errno));
- if (write && !priv->connected &&
- (errno == ECONNREFUSED))
- gf_log (this->name, GF_LOG_ERROR,
- "possible mismatch of 'rpc-transport-type'"
- " in protocol server and client. "
- "check volume file");
- opcount = -1;
- break;
- }
-
- moved = 0;
-
- while (moved < ret)
- {
- if ((ret - moved) >= opvector[0].iov_len)
- {
- moved += opvector[0].iov_len;
- opvector++;
- opcount--;
- }
- else
- {
- opvector[0].iov_len -= (ret - moved);
- opvector[0].iov_base += (ret - moved);
- moved += (ret - moved);
- }
- while (opcount && !opvector[0].iov_len)
- {
- opvector++;
- opcount--;
- }
- }
- }
-
- if (pending_vector)
- *pending_vector = opvector;
-
- if (pending_count)
- *pending_count = opcount;
-
- return opcount;
-}
-
-
-static int
-__tcp_readv (rpc_transport_t *this, struct iovec *vector, int count,
- struct iovec **pending_vector, int *pending_count)
-{
- int ret = -1;
-
- ret = __tcp_rwv (this, vector, count,
- pending_vector, pending_count, 0);
-
- return ret;
-}
-
-
-static int
-__tcp_writev (rpc_transport_t *this, struct iovec *vector, int count,
- struct iovec **pending_vector, int *pending_count)
-{
- int ret = -1;
- rdma_private_t *priv = this->private;
-
- ret = __tcp_rwv (this, vector, count, pending_vector,
- pending_count, 1);
-
- if (ret > 0) {
- /* TODO: Avoid multiple calls when socket is already
- registered for POLLOUT */
- priv->idx = event_select_on (this->ctx->event_pool,
- priv->sock, priv->idx, -1, 1);
- } else if (ret == 0) {
- priv->idx = event_select_on (this->ctx->event_pool,
- priv->sock,
- priv->idx, -1, 0);
+ ret = __gf_rdma_teardown (this);
}
+ pthread_mutex_unlock (&priv->write_mutex);
+out:
return ret;
}
@@ -2562,12 +2998,12 @@ __tcp_writev (rpc_transport_t *this, struct iovec *vector, int count,
* event is sent to upper layers.
*/
int32_t
-rdma_get_write_chunklist (char **ptr, rdma_write_array_t **write_ary)
+gf_rdma_get_write_chunklist (char **ptr, gf_rdma_write_array_t **write_ary)
{
- rdma_write_array_t *from = NULL, *to = NULL;
- int32_t ret = -1, size = 0, i = 0;
+ gf_rdma_write_array_t *from = NULL, *to = NULL;
+ int32_t ret = -1, size = 0, i = 0;
- from = (rdma_write_array_t *) *ptr;
+ from = (gf_rdma_write_array_t *) *ptr;
if (from->wc_discrim == 0) {
ret = 0;
goto out;
@@ -2576,11 +3012,10 @@ rdma_get_write_chunklist (char **ptr, rdma_write_array_t **write_ary)
from->wc_nchunks = ntoh32 (from->wc_nchunks);
size = sizeof (*from)
- + (sizeof (rdma_write_chunk_t) * from->wc_nchunks);
+ + (sizeof (gf_rdma_write_chunk_t) * from->wc_nchunks);
to = GF_CALLOC (1, size, gf_common_mt_char);
if (to == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
ret = -1;
goto out;
}
@@ -2611,13 +3046,13 @@ out:
* rdma-reads and hence readchunk-list can point to memory held by post.
*/
int32_t
-rdma_get_read_chunklist (char **ptr, rdma_read_chunk_t **readch)
+gf_rdma_get_read_chunklist (char **ptr, gf_rdma_read_chunk_t **readch)
{
- int32_t ret = -1;
- rdma_read_chunk_t *chunk = NULL;
- int i = 0;
+ int32_t ret = -1;
+ gf_rdma_read_chunk_t *chunk = NULL;
+ int i = 0;
- chunk = (rdma_read_chunk_t *)*ptr;
+ chunk = (gf_rdma_read_chunk_t *)*ptr;
if (chunk[0].rc_discrim == 0) {
ret = 0;
goto out;
@@ -2643,49 +3078,52 @@ out:
inline int32_t
-rdma_decode_error_msg (rdma_peer_t *peer, rdma_post_t *post,
- size_t bytes_in_post)
+gf_rdma_decode_error_msg (gf_rdma_peer_t *peer, gf_rdma_post_t *post,
+ size_t bytes_in_post)
{
- rdma_header_t *header = NULL;
- struct iobuf *iobuf = NULL;
- struct iobref *iobref = NULL;
- int32_t ret = -1;
-
- header = (rdma_header_t *)post->buf;
+ gf_rdma_header_t *header = NULL;
+ struct iobuf *iobuf = NULL;
+ struct iobref *iobref = NULL;
+ int32_t ret = -1;
+ struct rpc_msg rpc_msg = {0, };
+
+ header = (gf_rdma_header_t *)post->buf;
header->rm_body.rm_error.rm_type
= ntoh32 (header->rm_body.rm_error.rm_type);
if (header->rm_body.rm_error.rm_type == ERR_VERS) {
- header->rm_body.rm_error.rm_version.rdma_vers_low =
- ntoh32 (header->rm_body.rm_error.rm_version.rdma_vers_low);
- header->rm_body.rm_error.rm_version.rdma_vers_high =
- ntoh32 (header->rm_body.rm_error.rm_version.rdma_vers_high);
+ header->rm_body.rm_error.rm_version.gf_rdma_vers_low =
+ ntoh32 (header->rm_body.rm_error.rm_version.gf_rdma_vers_low);
+ header->rm_body.rm_error.rm_version.gf_rdma_vers_high =
+ ntoh32 (header->rm_body.rm_error.rm_version.gf_rdma_vers_high);
}
- iobuf = iobuf_get (peer->trans->ctx->iobuf_pool);
+ rpc_msg.rm_xid = header->rm_xid;
+ rpc_msg.rm_direction = REPLY;
+ rpc_msg.rm_reply.rp_stat = MSG_DENIED;
+
+ iobuf = iobuf_get2 (peer->trans->ctx->iobuf_pool, bytes_in_post);
if (iobuf == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
ret = -1;
goto out;
}
post->ctx.iobref = iobref = iobref_new ();
if (iobref == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
ret = -1;
goto out;
}
iobref_add (iobref, iobuf);
iobuf_unref (iobuf);
- /*
- * FIXME: construct an appropriate rpc-msg here, what is being sent
- * to rpc is not correct.
- */
- post->ctx.vector[0].iov_base = iobuf_ptr (iobuf);
- post->ctx.vector[0].iov_len = bytes_in_post;
-
- memcpy (post->ctx.vector[0].iov_base, (char *)post->buf,
- post->ctx.vector[0].iov_len);
+
+ ret = rpc_reply_to_xdr (&rpc_msg, iobuf_ptr (iobuf),
+ iobuf_pagesize (iobuf), &post->ctx.vector[0]);
+ if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "Failed to create RPC reply");
+ goto out;
+ }
+
post->ctx.count = 1;
iobuf = NULL;
@@ -2707,32 +3145,34 @@ out:
int32_t
-rdma_decode_msg (rdma_peer_t *peer, rdma_post_t *post,
- rdma_read_chunk_t **readch, size_t bytes_in_post)
+gf_rdma_decode_msg (gf_rdma_peer_t *peer, gf_rdma_post_t *post,
+ gf_rdma_read_chunk_t **readch, size_t bytes_in_post)
{
- int32_t ret = -1;
- rdma_header_t *header = NULL;
- rdma_reply_info_t *reply_info = NULL;
- char *ptr = NULL;
- rdma_write_array_t *write_ary = NULL;
- size_t header_len = 0;
- struct iobuf *iobuf = NULL;
- struct iobref *iobref = NULL;
+ int32_t ret = -1;
+ gf_rdma_header_t *header = NULL;
+ gf_rdma_reply_info_t *reply_info = NULL;
+ char *ptr = NULL;
+ gf_rdma_write_array_t *write_ary = NULL;
+ size_t header_len = 0;
- header = (rdma_header_t *)post->buf;
+ header = (gf_rdma_header_t *)post->buf;
ptr = (char *)&header->rm_body.rm_chunks[0];
- ret = rdma_get_read_chunklist (&ptr, readch);
+ ret = gf_rdma_get_read_chunklist (&ptr, readch);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot get read chunklist from msg");
goto out;
}
/* skip terminator of read-chunklist */
ptr = ptr + sizeof (uint32_t);
- ret = rdma_get_write_chunklist (&ptr, &write_ary);
+ ret = gf_rdma_get_write_chunklist (&ptr, &write_ary);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot get write chunklist from msg");
goto out;
}
@@ -2740,29 +3180,35 @@ rdma_decode_msg (rdma_peer_t *peer, rdma_post_t *post,
ptr = ptr + sizeof (uint32_t);
if (write_ary != NULL) {
- reply_info = rdma_reply_info_alloc (peer);
+ reply_info = gf_rdma_reply_info_alloc (peer);
if (reply_info == NULL) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "reply_info_alloc failed");
ret = -1;
goto out;
}
- reply_info->type = rdma_writech;
+ reply_info->type = gf_rdma_writech;
reply_info->wc_array = write_ary;
reply_info->rm_xid = header->rm_xid;
} else {
- ret = rdma_get_write_chunklist (&ptr, &write_ary);
+ ret = gf_rdma_get_write_chunklist (&ptr, &write_ary);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot get reply chunklist from msg");
goto out;
}
if (write_ary != NULL) {
- reply_info = rdma_reply_info_alloc (peer);
+ reply_info = gf_rdma_reply_info_alloc (peer);
if (reply_info == NULL) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "reply_info_alloc_failed");
ret = -1;
goto out;
}
- reply_info->type = rdma_replych;
+ reply_info->type = gf_rdma_replych;
reply_info->wc_array = write_ary;
reply_info->rm_xid = header->rm_xid;
}
@@ -2770,33 +3216,21 @@ rdma_decode_msg (rdma_peer_t *peer, rdma_post_t *post,
/* skip terminator of reply chunk */
ptr = ptr + sizeof (uint32_t);
- if (header->rm_type != RDMA_NOMSG) {
- iobuf = iobuf_get (peer->trans->ctx->iobuf_pool);
- if (iobuf == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- ret = -1;
- goto out;
- }
+ if (header->rm_type != GF_RDMA_NOMSG) {
+ header_len = (long)ptr - (long)post->buf;
+ post->ctx.vector[0].iov_len = (bytes_in_post - header_len);
- post->ctx.iobref = iobref = iobref_new ();
- if (iobref == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
+ post->ctx.hdr_iobuf = iobuf_get2 (peer->trans->ctx->iobuf_pool,
+ (bytes_in_post - header_len));
+ if (post->ctx.hdr_iobuf == NULL) {
ret = -1;
goto out;
}
- iobref_add (iobref, iobuf);
- iobuf_unref (iobuf);
-
- header_len = (long)ptr - (long)post->buf;
- post->ctx.vector[0].iov_base = iobuf_ptr (iobuf);
- post->ctx.vector[0].iov_len = bytes_in_post - header_len;
+ post->ctx.vector[0].iov_base = iobuf_ptr (post->ctx.hdr_iobuf);
memcpy (post->ctx.vector[0].iov_base, ptr,
post->ctx.vector[0].iov_len);
post->ctx.count = 1;
-
- iobuf = NULL;
- iobref = NULL;
}
post->ctx.reply_info = reply_info;
@@ -2807,17 +3241,7 @@ out:
*readch = NULL;
}
- if (write_ary != NULL) {
- GF_FREE (write_ary);
- }
-
- if (iobuf != NULL) {
- iobuf_unref (iobuf);
- }
-
- if (iobref != NULL) {
- iobref_unref (iobref);
- }
+ GF_FREE (write_ary);
}
return ret;
@@ -2826,13 +3250,13 @@ out:
/* Assumes only one of either write-chunklist or a reply chunk is present */
int32_t
-rdma_decode_header (rdma_peer_t *peer, rdma_post_t *post,
- rdma_read_chunk_t **readch, size_t bytes_in_post)
+gf_rdma_decode_header (gf_rdma_peer_t *peer, gf_rdma_post_t *post,
+ gf_rdma_read_chunk_t **readch, size_t bytes_in_post)
{
- int32_t ret = -1;
- rdma_header_t *header = NULL;
+ int32_t ret = -1;
+ gf_rdma_header_t *header = NULL;
- header = (rdma_header_t *)post->buf;
+ header = (gf_rdma_header_t *)post->buf;
header->rm_xid = ntoh32 (header->rm_xid);
header->rm_vers = ntoh32 (header->rm_vers);
@@ -2840,31 +3264,39 @@ rdma_decode_header (rdma_peer_t *peer, rdma_post_t *post,
header->rm_type = ntoh32 (header->rm_type);
switch (header->rm_type) {
- case RDMA_MSG:
- case RDMA_NOMSG:
- ret = rdma_decode_msg (peer, post, readch, bytes_in_post);
+ case GF_RDMA_MSG:
+ case GF_RDMA_NOMSG:
+ ret = gf_rdma_decode_msg (peer, post, readch, bytes_in_post);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot decode msg of type (%d)",
+ header->rm_type);
+ }
+
break;
- case RDMA_MSGP:
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "rdma msg of msg-type RDMA_MSGP should not have been "
- "recieved");
+ case GF_RDMA_MSGP:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma msg of msg-type GF_RDMA_MSGP should not have "
+ "been received");
ret = -1;
break;
- case RDMA_DONE:
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "rdma msg of msg-type RDMA_DONE should not have been "
- "recieved");
+ case GF_RDMA_DONE:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma msg of msg-type GF_RDMA_DONE should not have "
+ "been received");
ret = -1;
break;
- case RDMA_ERROR:
- /* ret = rdma_decode_error_msg (peer, post, bytes_in_post); */
+ case GF_RDMA_ERROR:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "received a msg of type RDMA_ERROR");
+ ret = gf_rdma_decode_error_msg (peer, post, bytes_in_post);
break;
default:
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"unknown rdma msg-type (%d)", header->rm_type);
}
@@ -2873,15 +3305,17 @@ rdma_decode_header (rdma_peer_t *peer, rdma_post_t *post,
int32_t
-__rdma_read (rdma_peer_t *peer, rdma_post_t *post, struct iovec *to,
- rdma_read_chunk_t *readch)
+__gf_rdma_read (gf_rdma_peer_t *peer, gf_rdma_post_t *post, struct iovec *to,
+ gf_rdma_read_chunk_t *readch)
{
- int32_t ret = -1;
+ int32_t ret = -1;
struct ibv_sge list = {0, };
- struct ibv_send_wr wr = {0, }, *bad_wr = NULL;
+ struct ibv_send_wr wr = {0, }, *bad_wr = NULL;
- ret = __rdma_register_local_mr_for_rdma (peer, to, 1, &post->ctx);
+ ret = __gf_rdma_register_local_mr_for_rdma (peer, to, 1, &post->ctx);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "registering local memory for rdma read failed");
goto out;
}
@@ -2889,7 +3323,7 @@ __rdma_read (rdma_peer_t *peer, rdma_post_t *post, struct iovec *to,
list.length = to->iov_len;
list.lkey = post->ctx.mr[post->ctx.mr_count - 1]->lkey;
- wr.wr_id = (unsigned long) rdma_post_ref (post);
+ wr.wr_id = (unsigned long) gf_rdma_post_ref (post);
wr.sg_list = &list;
wr.num_sge = 1;
wr.opcode = IBV_WR_RDMA_READ;
@@ -2898,8 +3332,14 @@ __rdma_read (rdma_peer_t *peer, rdma_post_t *post, struct iovec *to,
wr.wr.rdma.rkey = readch->rc_target.rs_handle;
ret = ibv_post_send (peer->qp, &wr, &bad_wr);
- if (ret == -1) {
- rdma_post_unref (post);
+ if (ret) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma read from client "
+ "(%s) failed with ret = %d (%s)",
+ peer->trans->peerinfo.identifier,
+ ret, (ret > 0) ? strerror (ret) : "");
+ ret = -1;
+ gf_rdma_post_unref (post);
}
out:
return ret;
@@ -2907,13 +3347,14 @@ out:
int32_t
-rdma_do_reads (rdma_peer_t *peer, rdma_post_t *post, rdma_read_chunk_t *readch)
+gf_rdma_do_reads (gf_rdma_peer_t *peer, gf_rdma_post_t *post,
+ gf_rdma_read_chunk_t *readch)
{
- int32_t ret = -1, i = 0;
- size_t size = 0;
- char *ptr = NULL;
- struct iobuf *iobuf = NULL;
- rdma_private_t *priv = NULL;
+ int32_t ret = -1, i = 0, count = 0;
+ size_t size = 0;
+ char *ptr = NULL;
+ struct iobuf *iobuf = NULL;
+ gf_rdma_private_t *priv = NULL;
priv = peer->trans->private;
@@ -2922,26 +3363,16 @@ rdma_do_reads (rdma_peer_t *peer, rdma_post_t *post, rdma_read_chunk_t *readch)
}
if (i == 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"message type specified as rdma-read but there are no "
"rdma read-chunks present");
goto out;
}
- post->ctx.rdma_reads = i;
-
- if (size > peer->trans->ctx->page_size) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "total size of rdma-read (%lu) is greater than "
- "page-size (%lu). This is not supported till variable "
- "sized iobufs are implemented", (unsigned long)size,
- (unsigned long)peer->trans->ctx->page_size);
- goto out;
- }
+ post->ctx.gf_rdma_reads = i;
- iobuf = iobuf_get (peer->trans->ctx->iobuf_pool);
+ iobuf = iobuf_get2 (peer->trans->ctx->iobuf_pool, size);
if (iobuf == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -2949,7 +3380,6 @@ rdma_do_reads (rdma_peer_t *peer, rdma_post_t *post, rdma_read_chunk_t *readch)
post->ctx.iobref = iobref_new ();
if (post->ctx.iobref == NULL) {
iobuf_unref (iobuf);
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto out;
}
}
@@ -2963,22 +3393,29 @@ rdma_do_reads (rdma_peer_t *peer, rdma_post_t *post, rdma_read_chunk_t *readch)
pthread_mutex_lock (&priv->write_mutex);
{
if (!priv->connected) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "transport not connected to peer (%s), "
+ "not doing rdma reads",
+ peer->trans->peerinfo.identifier);
goto unlock;
}
for (i = 0; readch[i].rc_discrim != 0; i++) {
- post->ctx.vector[post->ctx.count].iov_base = ptr;
- post->ctx.vector[post->ctx.count].iov_len
+ count = post->ctx.count++;
+ post->ctx.vector[count].iov_base = ptr;
+ post->ctx.vector[count].iov_len
= readch[i].rc_target.rs_length;
- ret = __rdma_read (peer, post,
- &post->ctx.vector[post->ctx.count],
- &readch[i]);
+ ret = __gf_rdma_read (peer, post,
+ &post->ctx.vector[count],
+ &readch[i]);
if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma read from peer (%s) failed",
+ peer->trans->peerinfo.identifier);
goto unlock;
}
- post->ctx.count++;
ptr += readch[i].rc_target.rs_length;
}
@@ -2999,28 +3436,43 @@ out:
int32_t
-rdma_pollin_notify (rdma_peer_t *peer, rdma_post_t *post)
+gf_rdma_pollin_notify (gf_rdma_peer_t *peer, gf_rdma_post_t *post)
{
- int32_t ret = -1;
- enum msg_type msg_type = 0;
- struct rpc_req *rpc_req = NULL;
- rdma_request_context_t *request_context = NULL;
- rpc_request_info_t request_info = {0, };
- rdma_private_t *priv = NULL;
- uint32_t *ptr = NULL;
- rpc_transport_pollin_t *pollin = NULL;
+ int32_t ret = -1;
+ enum msg_type msg_type = 0;
+ struct rpc_req *rpc_req = NULL;
+ gf_rdma_request_context_t *request_context = NULL;
+ rpc_request_info_t request_info = {0, };
+ gf_rdma_private_t *priv = NULL;
+ uint32_t *ptr = NULL;
+ rpc_transport_pollin_t *pollin = NULL;
if ((peer == NULL) || (post == NULL)) {
goto out;
}
+ if (post->ctx.iobref == NULL) {
+ post->ctx.iobref = iobref_new ();
+ if (post->ctx.iobref == NULL) {
+ goto out;
+ }
+
+ /* handling the case where both hdr and payload of
+ * GF_FOP_READ_CBK were received in a single iobuf
+ * because of server sending entire msg as inline without
+ * doing rdma writes.
+ */
+ if (post->ctx.hdr_iobuf)
+ iobref_add (post->ctx.iobref, post->ctx.hdr_iobuf);
+ }
+
pollin = rpc_transport_pollin_alloc (peer->trans,
post->ctx.vector,
post->ctx.count,
+ post->ctx.hdr_iobuf,
post->ctx.iobref,
post->ctx.reply_info);
if (pollin == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
goto out;
}
@@ -3034,15 +3486,15 @@ rdma_pollin_notify (rdma_peer_t *peer, rdma_post_t *post)
RPC_TRANSPORT_MAP_XID_REQUEST,
&request_info);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
"cannot get request information from rpc "
- "layer");
+ "layer");
goto out;
}
rpc_req = request_info.rpc_req;
if (rpc_req == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
"rpc request structure not found");
ret = -1;
goto out;
@@ -3052,17 +3504,25 @@ rdma_pollin_notify (rdma_peer_t *peer, rdma_post_t *post)
rpc_req->conn_private = NULL;
priv = peer->trans->private;
- pthread_mutex_lock (&priv->write_mutex);
- {
- __rdma_request_context_destroy (request_context);
+ if (request_context != NULL) {
+ pthread_mutex_lock (&priv->write_mutex);
+ {
+ __gf_rdma_request_context_destroy (request_context);
+ }
+ pthread_mutex_unlock (&priv->write_mutex);
+ } else {
+ gf_rdma_quota_put (peer);
}
- pthread_mutex_unlock (&priv->write_mutex);
pollin->is_reply = 1;
}
ret = rpc_transport_notify (peer->trans, RPC_TRANSPORT_MSG_RECEIVED,
pollin);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "transport_notify failed");
+ }
out:
if (pollin != NULL) {
@@ -3075,19 +3535,19 @@ out:
int32_t
-rdma_recv_reply (rdma_peer_t *peer, rdma_post_t *post)
+gf_rdma_recv_reply (gf_rdma_peer_t *peer, gf_rdma_post_t *post)
{
- int32_t ret = -1;
- rdma_header_t *header = NULL;
- rdma_reply_info_t *reply_info = NULL;
- rdma_write_array_t *wc_array = NULL;
- int i = 0;
- uint32_t *ptr = NULL;
- rdma_request_context_t *ctx = NULL;
- rpc_request_info_t request_info = {0, };
- struct rpc_req *rpc_req = NULL;
-
- header = (rdma_header_t *)post->buf;
+ int32_t ret = -1;
+ gf_rdma_header_t *header = NULL;
+ gf_rdma_reply_info_t *reply_info = NULL;
+ gf_rdma_write_array_t *wc_array = NULL;
+ int i = 0;
+ uint32_t *ptr = NULL;
+ gf_rdma_request_context_t *ctx = NULL;
+ rpc_request_info_t request_info = {0, };
+ struct rpc_req *rpc_req = NULL;
+
+ header = (gf_rdma_header_t *)post->buf;
reply_info = post->ctx.reply_info;
/* no write chunklist, just notify upper layers */
@@ -3098,7 +3558,7 @@ rdma_recv_reply (rdma_peer_t *peer, rdma_post_t *post)
wc_array = reply_info->wc_array;
- if (header->rm_type == RDMA_NOMSG) {
+ if (header->rm_type == GF_RDMA_NOMSG) {
post->ctx.vector[0].iov_base
= (void *)(long)wc_array->wc_array[0].wc_target.rs_offset;
post->ctx.vector[0].iov_len
@@ -3112,7 +3572,7 @@ rdma_recv_reply (rdma_peer_t *peer, rdma_post_t *post)
post->ctx.vector[i + 1].iov_len
= wc_array->wc_array[i].wc_target.rs_length;
}
-
+
post->ctx.count += wc_array->wc_nchunks;
}
@@ -3123,34 +3583,36 @@ rdma_recv_reply (rdma_peer_t *peer, rdma_post_t *post)
RPC_TRANSPORT_MAP_XID_REQUEST,
&request_info);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "cannot get request information from rpc "
- "layer");
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "cannot get request information (peer:%s) from rpc "
+ "layer", peer->trans->peerinfo.identifier);
goto out;
}
rpc_req = request_info.rpc_req;
if (rpc_req == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"rpc request structure not found");
ret = -1;
goto out;
}
ctx = rpc_req->conn_private;
- if ((post->ctx.iobref != NULL) && (ctx->rsp_iobref != NULL)) {
- iobref_merge (post->ctx.iobref, ctx->rsp_iobref);
- } else if (post->ctx.iobref == NULL) {
+ if ((post->ctx.iobref == NULL) && ctx->rsp_iobref) {
post->ctx.iobref = iobref_ref (ctx->rsp_iobref);
}
ret = 0;
- rdma_reply_info_destroy (reply_info);
+ gf_rdma_reply_info_destroy (reply_info);
out:
if (ret == 0) {
- ret = rdma_pollin_notify (peer, post);
+ ret = gf_rdma_pollin_notify (peer, post);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "pollin notify failed");
+ }
}
return ret;
@@ -3158,17 +3620,22 @@ out:
inline int32_t
-rdma_recv_request (rdma_peer_t *peer, rdma_post_t *post,
- rdma_read_chunk_t *readch)
+gf_rdma_recv_request (gf_rdma_peer_t *peer, gf_rdma_post_t *post,
+ gf_rdma_read_chunk_t *readch)
{
int32_t ret = -1;
-
+
if (readch != NULL) {
- ret = rdma_do_reads (peer, post, readch);
+ ret = gf_rdma_do_reads (peer, post, readch);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma read from peer (%s) failed",
+ peer->trans->peerinfo.identifier);
+ }
} else {
- ret = rdma_pollin_notify (peer, post);
+ ret = gf_rdma_pollin_notify (peer, post);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"pollin notification failed");
}
}
@@ -3177,38 +3644,59 @@ rdma_recv_request (rdma_peer_t *peer, rdma_post_t *post,
}
void
-rdma_process_recv (rdma_peer_t *peer, struct ibv_wc *wc)
+gf_rdma_process_recv (gf_rdma_peer_t *peer, struct ibv_wc *wc)
{
- rdma_post_t *post = NULL;
- rdma_read_chunk_t *readch = NULL;
- int ret = -1;
- uint32_t *ptr = NULL;
- enum msg_type msg_type = 0;
- rdma_header_t *header = NULL;
-
- post = (rdma_post_t *) (long) wc->wr_id;
+ gf_rdma_post_t *post = NULL;
+ gf_rdma_read_chunk_t *readch = NULL;
+ int ret = -1;
+ uint32_t *ptr = NULL;
+ enum msg_type msg_type = 0;
+ gf_rdma_header_t *header = NULL;
+ gf_rdma_private_t *priv = NULL;
+
+ post = (gf_rdma_post_t *) (long) wc->wr_id;
if (post == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"no post found in successful work completion element");
goto out;
}
- ret = rdma_decode_header (peer, post, &readch, wc->byte_len);
+ ret = gf_rdma_decode_header (peer, post, &readch, wc->byte_len);
if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"decoding of header failed");
goto out;
}
- header = (rdma_header_t *)post->buf;
+ header = (gf_rdma_header_t *)post->buf;
+
+ priv = peer->trans->private;
+
+ pthread_mutex_lock (&priv->write_mutex);
+ {
+ if (!priv->peer.quota_set) {
+ priv->peer.quota_set = 1;
+
+ /* Initially peer.quota is set to 1 as per RFC 5666. We
+ * have to account for the quota used while sending
+ * first msg (which may or may not be returned to pool
+ * at this point) while deriving peer.quota from
+ * header->rm_credit. Hence the arithmatic below,
+ * instead of directly setting it to header->rm_credit.
+ */
+ priv->peer.quota = header->rm_credit
+ - ( 1 - priv->peer.quota);
+ }
+ }
+ pthread_mutex_unlock (&priv->write_mutex);
switch (header->rm_type) {
- case RDMA_MSG:
+ case GF_RDMA_MSG:
ptr = (uint32_t *)post->ctx.vector[0].iov_base;
msg_type = ntoh32 (*(ptr + 1));
break;
- case RDMA_NOMSG:
+ case GF_RDMA_NOMSG:
if (readch != NULL) {
msg_type = CALL;
} else {
@@ -3216,31 +3704,49 @@ rdma_process_recv (rdma_peer_t *peer, struct ibv_wc *wc)
}
break;
- case RDMA_ERROR:
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "an error has happened while transmission of msg, "
- "disconnecting the transport");
- rpc_transport_disconnect (peer->trans);
- goto out;
-
-/* ret = rdma_pollin_notify (peer, post);
- if (ret == -1) {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "pollin notification failed");
- }
- goto out;
-*/
+ case GF_RDMA_ERROR:
+ if (header->rm_body.rm_error.rm_type == ERR_CHUNK) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "peer (%s), couldn't encode or decode the msg "
+ "properly or write chunks were not provided "
+ "for replies that were bigger than "
+ "RDMA_INLINE_THRESHOLD (%d)",
+ peer->trans->peerinfo.identifier,
+ GLUSTERFS_RDMA_INLINE_THRESHOLD);
+ ret = gf_rdma_pollin_notify (peer, post);
+ if (ret == -1) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
+ "pollin notification failed");
+ }
+ goto out;
+ } else {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
+ "an error has happened while transmission of "
+ "msg, disconnecting the transport");
+ ret = -1;
+ goto out;
+ }
default:
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"invalid rdma msg-type (%d)", header->rm_type);
- break;
+ goto out;
}
if (msg_type == CALL) {
- ret = rdma_recv_request (peer, post, readch);
+ ret = gf_rdma_recv_request (peer, post, readch);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "receiving a request from peer (%s) failed",
+ peer->trans->peerinfo.identifier);
+ }
} else {
- ret = rdma_recv_reply (peer, post);
+ ret = gf_rdma_recv_reply (peer, post);
+ if (ret < 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "receiving a reply from peer (%s) failed",
+ peer->trans->peerinfo.identifier);
+ }
}
out:
@@ -3251,14 +3757,50 @@ out:
return;
}
+void *
+gf_rdma_async_event_thread (void *context)
+{
+ struct ibv_async_event event;
+ int ret;
+
+ while (1) {
+ do {
+ ret = ibv_get_async_event((struct ibv_context *)context,
+ &event);
+
+ if (ret && errno != EINTR) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "Error getting event (%s)",
+ strerror (errno));
+ }
+ } while(ret && errno == EINTR);
+
+ switch (event.event_type) {
+ case IBV_EVENT_SRQ_LIMIT_REACHED:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "recieved srq_limit reached");
+ break;
+
+ default:
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
+ "event (%d) recieved", event.event_type);
+ break;
+ }
+
+ ibv_ack_async_event(&event);
+ }
+
+ return 0;
+}
+
static void *
-rdma_recv_completion_proc (void *data)
+gf_rdma_recv_completion_proc (void *data)
{
struct ibv_comp_channel *chan = NULL;
- rdma_device_t *device = NULL;;
- rdma_post_t *post = NULL;
- rdma_peer_t *peer = NULL;
+ gf_rdma_device_t *device = NULL;;
+ gf_rdma_post_t *post = NULL;
+ gf_rdma_peer_t *peer = NULL;
struct ibv_cq *event_cq = NULL;
struct ibv_wc wc = {0, };
void *event_ctx = NULL;
@@ -3269,36 +3811,36 @@ rdma_recv_completion_proc (void *data)
while (1) {
ret = ibv_get_cq_event (chan, &event_cq, &event_ctx);
if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
"ibv_get_cq_event failed, terminating recv "
"thread %d (%d)", ret, errno);
continue;
}
device = event_ctx;
-
+
ret = ibv_req_notify_cq (event_cq, 0);
if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
"ibv_req_notify_cq on %s failed, terminating "
"recv thread: %d (%d)",
device->device_name, ret, errno);
continue;
}
- device = (rdma_device_t *) event_ctx;
+ device = (gf_rdma_device_t *) event_ctx;
while ((ret = ibv_poll_cq (event_cq, 1, &wc)) > 0) {
- post = (rdma_post_t *) (long) wc.wr_id;
+ post = (gf_rdma_post_t *) (long) wc.wr_id;
pthread_mutex_lock (&device->qpreg.lock);
{
- peer = __rdma_lookup_peer (device,
- wc.qp_num);
+ peer = __gf_rdma_lookup_peer (device,
+ wc.qp_num);
/*
* keep a refcount on transport so that it
- * doesnot get freed because of some error
+ * does not get freed because of some error
* indicated by wc.status till we are done
* with usage of peer and thereby that of trans.
*/
@@ -3309,7 +3851,7 @@ rdma_recv_completion_proc (void *data)
pthread_mutex_unlock (&device->qpreg.lock);
if (wc.status != IBV_WC_SUCCESS) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
"recv work request on `%s' returned "
"error (%d)", device->device_name,
wc.status);
@@ -3319,26 +3861,26 @@ rdma_recv_completion_proc (void *data)
}
if (post) {
- rdma_post_unref (post);
+ gf_rdma_post_unref (post);
}
continue;
}
if (peer) {
- rdma_process_recv (peer, &wc);
+ gf_rdma_process_recv (peer, &wc);
rpc_transport_unref (peer->trans);
} else {
- gf_log (RDMA_LOG_NAME,
+ gf_log (GF_RDMA_LOG_NAME,
GF_LOG_DEBUG,
"could not lookup peer for qp_num: %d",
wc.qp_num);
}
- rdma_post_unref (post);
+ gf_rdma_post_unref (post);
}
-
+
if (ret < 0) {
- gf_log (RDMA_LOG_NAME,
+ gf_log (GF_RDMA_LOG_NAME,
GF_LOG_ERROR,
"ibv_poll_cq on `%s' returned error "
"(ret = %d, errno = %d)",
@@ -3353,11 +3895,11 @@ rdma_recv_completion_proc (void *data)
void
-rdma_handle_failed_send_completion (rdma_peer_t *peer, struct ibv_wc *wc)
+gf_rdma_handle_failed_send_completion (gf_rdma_peer_t *peer, struct ibv_wc *wc)
{
- rdma_post_t *post = NULL;
- rdma_device_t *device = NULL;
- rdma_private_t *priv = NULL;
+ gf_rdma_post_t *post = NULL;
+ gf_rdma_device_t *device = NULL;
+ gf_rdma_private_t *priv = NULL;
if (peer != NULL) {
priv = peer->trans->private;
@@ -3367,13 +3909,13 @@ rdma_handle_failed_send_completion (rdma_peer_t *peer, struct ibv_wc *wc)
}
- post = (rdma_post_t *) (long) wc->wr_id;
+ post = (gf_rdma_post_t *) (long) wc->wr_id;
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"send work request on `%s' returned error "
"wc.status = %d, wc.vendor_err = %d, post->buf = %p, "
"wc.byte_len = %d, post->reused = %d",
- (device != NULL) ? device->device_name : NULL, wc->status,
+ (device != NULL) ? device->device_name : NULL, wc->status,
wc->vendor_err, post->buf, wc->byte_len, post->reused);
if (wc->status == IBV_WC_RETRY_EXC_ERR) {
@@ -3394,20 +3936,22 @@ rdma_handle_failed_send_completion (rdma_peer_t *peer, struct ibv_wc *wc)
void
-rdma_handle_successful_send_completion (rdma_peer_t *peer, struct ibv_wc *wc)
+gf_rdma_handle_successful_send_completion (gf_rdma_peer_t *peer,
+ struct ibv_wc *wc)
{
- rdma_post_t *post = NULL;
- int reads = 0, ret = 0;
+ gf_rdma_post_t *post = NULL;
+ int reads = 0, ret = 0;
+ gf_rdma_header_t *header = NULL;
if (wc->opcode != IBV_WC_RDMA_READ) {
goto out;
}
- post = (rdma_post_t *)(long) wc->wr_id;
+ post = (gf_rdma_post_t *)(long) wc->wr_id;
pthread_mutex_lock (&post->lock);
{
- reads = --post->ctx.rdma_reads;
+ reads = --post->ctx.gf_rdma_reads;
}
pthread_mutex_unlock (&post->lock);
@@ -3416,8 +3960,15 @@ rdma_handle_successful_send_completion (rdma_peer_t *peer, struct ibv_wc *wc)
goto out;
}
- ret = rdma_pollin_notify (peer, post);
- if (ret == -1) {
+ header = (gf_rdma_header_t *)post->buf;
+
+ if (header->rm_type == GF_RDMA_NOMSG) {
+ post->ctx.count = 1;
+ post->ctx.vector[0].iov_len += post->ctx.vector[1].iov_len;
+ }
+
+ ret = gf_rdma_pollin_notify (peer, post);
+ if ((ret == -1) && (peer != NULL)) {
rpc_transport_disconnect (peer->trans);
}
@@ -3427,33 +3978,33 @@ out:
static void *
-rdma_send_completion_proc (void *data)
+gf_rdma_send_completion_proc (void *data)
{
struct ibv_comp_channel *chan = NULL;
- rdma_post_t *post = NULL;
- rdma_peer_t *peer = NULL;
+ gf_rdma_post_t *post = NULL;
+ gf_rdma_peer_t *peer = NULL;
struct ibv_cq *event_cq = NULL;
void *event_ctx = NULL;
- rdma_device_t *device = NULL;
+ gf_rdma_device_t *device = NULL;
struct ibv_wc wc = {0, };
char is_request = 0;
int32_t ret = 0, quota_ret = 0;
-
+
chan = data;
while (1) {
ret = ibv_get_cq_event (chan, &event_cq, &event_ctx);
if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
"ibv_get_cq_event on failed, terminating "
"send thread: %d (%d)", ret, errno);
continue;
}
-
+
device = event_ctx;
ret = ibv_req_notify_cq (event_cq, 0);
if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
"ibv_req_notify_cq on %s failed, terminating "
"send thread: %d (%d)",
device->device_name, ret, errno);
@@ -3461,11 +4012,11 @@ rdma_send_completion_proc (void *data)
}
while ((ret = ibv_poll_cq (event_cq, 1, &wc)) > 0) {
- post = (rdma_post_t *) (long) wc.wr_id;
+ post = (gf_rdma_post_t *) (long) wc.wr_id;
pthread_mutex_lock (&device->qpreg.lock);
{
- peer = __rdma_lookup_peer (device, wc.qp_num);
+ peer = __gf_rdma_lookup_peer (device, wc.qp_num);
/*
* keep a refcount on transport so that it
@@ -3480,32 +4031,32 @@ rdma_send_completion_proc (void *data)
pthread_mutex_unlock (&device->qpreg.lock);
if (wc.status != IBV_WC_SUCCESS) {
- rdma_handle_failed_send_completion (peer, &wc);
+ gf_rdma_handle_failed_send_completion (peer, &wc);
} else {
- rdma_handle_successful_send_completion (peer,
- &wc);
+ gf_rdma_handle_successful_send_completion (peer,
+ &wc);
}
if (post) {
is_request = post->ctx.is_request;
- ret = rdma_post_unref (post);
+ ret = gf_rdma_post_unref (post);
if ((ret == 0)
&& (wc.status == IBV_WC_SUCCESS)
&& !is_request
- && (post->type == RDMA_SEND_POST)
+ && (post->type == GF_RDMA_SEND_POST)
&& (peer != NULL)) {
- /* An RDMA_RECV_POST can end up in
- * rdma_send_completion_proc for
+ /* An GF_RDMA_RECV_POST can end up in
+ * gf_rdma_send_completion_proc for
* rdma-reads, and we do not take
- * quota for getting an RDMA_RECV_POST.
+ * quota for getting an GF_RDMA_RECV_POST.
*/
- /*
+ /*
* if it is request, quota is returned
* after reply has come.
*/
- quota_ret = rdma_quota_put (peer);
+ quota_ret = gf_rdma_quota_put (peer);
if (quota_ret < 0) {
gf_log ("rdma", GF_LOG_DEBUG,
"failed to send "
@@ -3517,21 +4068,21 @@ rdma_send_completion_proc (void *data)
if (peer) {
rpc_transport_unref (peer->trans);
} else {
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_DEBUG,
"could not lookup peer for qp_num: %d",
wc.qp_num);
}
}
if (ret < 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_ERROR,
"ibv_poll_cq on `%s' returned error (ret = %d,"
" errno = %d)",
device->device_name, ret, errno);
continue;
}
- ibv_ack_cq_events (event_cq, 1);
+ ibv_ack_cq_events (event_cq, 1);
}
return NULL;
@@ -3539,19 +4090,24 @@ rdma_send_completion_proc (void *data)
static void
-rdma_options_init (rpc_transport_t *this)
+gf_rdma_options_init (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
- rdma_options_t *options = &priv->options;
- int32_t mtu;
- data_t *temp;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_options_t *options = NULL;
+ int32_t mtu = 0;
+ data_t *temp = NULL;
/* TODO: validate arguments from options below */
- options->send_size = this->ctx->page_size * 4; /* 512 KB */
- options->recv_size = this->ctx->page_size * 4; /* 512 KB */
- options->send_count = 32;
- options->recv_count = 32;
+ priv = this->private;
+ options = &priv->options;
+ options->send_size = GLUSTERFS_RDMA_INLINE_THRESHOLD;/*this->ctx->page_size * 4; 512 KB*/
+ options->recv_size = GLUSTERFS_RDMA_INLINE_THRESHOLD;/*this->ctx->page_size * 4; 512 KB*/
+ options->send_count = 4096;
+ options->recv_count = 4096;
+ options->attr_timeout = GF_RDMA_TIMEOUT;
+ options->attr_retry_cnt = GF_RDMA_RETRY_CNT;
+ options->attr_rnr_retry = GF_RDMA_RNR_RETRY;
temp = dict_get (this->options,
"transport.rdma.work-request-send-count");
@@ -3561,9 +4117,24 @@ rdma_options_init (rpc_transport_t *this)
temp = dict_get (this->options,
"transport.rdma.work-request-recv-count");
if (temp)
- options->recv_count = data_to_int32 (temp);
+ options->recv_count = data_to_int32 (temp);
+
+ temp = dict_get (this->options, "transport.rdma.attr-timeout");
+
+ if (temp)
+ options->attr_timeout = data_to_uint8 (temp);
+
+ temp = dict_get (this->options, "transport.rdma.attr-retry-cnt");
+
+ if (temp)
+ options->attr_retry_cnt = data_to_uint8 (temp);
+
+ temp = dict_get (this->options, "transport.rdma.attr-rnr-retry");
- options->port = 0;
+ if (temp)
+ options->attr_rnr_retry = data_to_uint8 (temp);
+
+ options->port = 1;
temp = dict_get (this->options,
"transport.rdma.port");
if (temp)
@@ -3587,12 +4158,12 @@ rdma_options_init (rpc_transport_t *this)
break;
default:
if (temp)
- gf_log (RDMA_LOG_NAME, GF_LOG_WARNING,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
"%s: unrecognized MTU value '%s', defaulting "
"to '2048'", this->name,
data_to_str (temp));
else
- gf_log (RDMA_LOG_NAME, GF_LOG_TRACE,
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_TRACE,
"%s: defaulting MTU to '2048'",
this->name);
options->mtu = IBV_MTU_2048;
@@ -3607,1079 +4178,250 @@ rdma_options_init (rpc_transport_t *this)
return;
}
-static void
-rdma_queue_init (rdma_queue_t *queue)
-{
- pthread_mutex_init (&queue->lock, NULL);
- queue->active_posts.next = &queue->active_posts;
- queue->active_posts.prev = &queue->active_posts;
- queue->passive_posts.next = &queue->passive_posts;
- queue->passive_posts.prev = &queue->passive_posts;
-}
-
-
-static rdma_device_t *
-rdma_get_device (rpc_transport_t *this,
- struct ibv_context *ibctx)
+gf_rdma_ctx_t *
+__gf_rdma_ctx_create (void)
{
- glusterfs_ctx_t *ctx = NULL;
- rdma_private_t *priv = NULL;
- rdma_options_t *options = NULL;
- char *device_name = NULL;
- uint32_t port = 0;
- uint8_t active_port = 0;
- int32_t ret = 0;
- int32_t i = 0;
-
- rdma_device_t *trav = NULL;
+ gf_rdma_ctx_t *rdma_ctx = NULL;
+ int ret = -1;
- priv = this->private;
- options = &priv->options;
- device_name = priv->options.device_name;
- ctx = this->ctx;
- trav = ctx->ib;
- port = priv->options.port;
-
- while (trav) {
- if ((!strcmp (trav->device_name, device_name)) &&
- (trav->port == port))
- break;
- trav = trav->next;
+ rdma_ctx = GF_CALLOC (1, sizeof (*rdma_ctx), gf_common_mt_char);
+ if (rdma_ctx == NULL) {
+ goto out;
}
- if (!trav) {
-
- trav = GF_CALLOC (1, sizeof (*trav),
- gf_common_mt_rdma_device_t);
- if (trav == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- return NULL;
- }
-
- priv->device = trav;
-
- trav->context = ibctx;
-
- ret = ib_get_active_port (trav->context);
-
- if (ret < 0) {
- if (!port) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "Failed to find any active ports and "
- "none specified in volume file,"
- " exiting");
- return NULL;
- }
- }
-
- active_port = ret;
-
- if (port) {
- ret = ib_check_active_port (trav->context, port);
- if (ret < 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_WARNING,
- "On device %s: provided port:%u is "
- "found to be offline, continuing to "
- "use the same port", device_name, port);
- }
- } else {
- priv->options.port = active_port;
- port = active_port;
- gf_log (RDMA_LOG_NAME, GF_LOG_TRACE,
- "Port unspecified in volume file using active "
- "port: %u", port);
- }
-
- trav->device_name = gf_strdup (device_name);
- trav->port = port;
-
- trav->next = ctx->ib;
- ctx->ib = trav;
-
- trav->send_chan = ibv_create_comp_channel (trav->context);
- if (!trav->send_chan) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not create send completion channel",
- device_name);
- /* TODO: cleanup current mess */
- return NULL;
- }
-
- trav->recv_chan = ibv_create_comp_channel (trav->context);
- if (!trav->recv_chan) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "could not create recv completion channel");
- /* TODO: cleanup current mess */
- return NULL;
- }
-
- if (rdma_create_cq (this) < 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not create CQ",
- this->name);
- return NULL;
- }
-
- /* protection domain */
- trav->pd = ibv_alloc_pd (trav->context);
-
- if (!trav->pd) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not allocate protection domain",
- this->name);
- return NULL;
- }
-
- struct ibv_srq_init_attr attr = {
- .attr = {
- .max_wr = options->recv_count,
- .max_sge = 1
- }
- };
- trav->srq = ibv_create_srq (trav->pd, &attr);
-
- if (!trav->srq) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not create SRQ",
- this->name);
- return NULL;
- }
+ rdma_ctx->rdma_cm_event_channel = rdma_create_event_channel ();
+ if (rdma_ctx->rdma_cm_event_channel == NULL) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "rdma_cm event channel creation failed (%s)",
+ strerror (errno));
+ goto out;
+ }
- /* queue init */
- rdma_queue_init (&trav->sendq);
- rdma_queue_init (&trav->recvq);
+ ret = pthread_create (&rdma_ctx->rdma_cm_thread, NULL,
+ gf_rdma_cm_event_handler,
+ rdma_ctx->rdma_cm_event_channel);
+ if (ret != 0) {
+ gf_log (GF_RDMA_LOG_NAME, GF_LOG_WARNING,
+ "creation of thread to handle rdma-cm events "
+ "failed (%s)", strerror (ret));
+ goto out;
+ }
- if (rdma_create_posts (this) < 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not allocate posts",
- this->name);
- return NULL;
+out:
+ if (ret < 0) {
+ if (rdma_ctx->rdma_cm_event_channel != NULL) {
+ rdma_destroy_event_channel (rdma_ctx->rdma_cm_event_channel);
}
- /* completion threads */
- ret = pthread_create (&trav->send_thread,
- NULL,
- rdma_send_completion_proc,
- trav->send_chan);
- if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "could not create send completion thread");
- return NULL;
- }
- ret = pthread_create (&trav->recv_thread,
- NULL,
- rdma_recv_completion_proc,
- trav->recv_chan);
- if (ret) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "could not create recv completion thread");
- return NULL;
- }
-
- /* qpreg */
- pthread_mutex_init (&trav->qpreg.lock, NULL);
- for (i=0; i<42; i++) {
- trav->qpreg.ents[i].next = &trav->qpreg.ents[i];
- trav->qpreg.ents[i].prev = &trav->qpreg.ents[i];
- }
+ GF_FREE (rdma_ctx);
+ rdma_ctx = NULL;
}
- return trav;
+
+ return rdma_ctx;
}
-static int32_t
-rdma_init (rpc_transport_t *this)
+static int32_t
+gf_rdma_init (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
- rdma_options_t *options = &priv->options;
- struct ibv_device **dev_list;
- struct ibv_context *ib_ctx = NULL;
- int32_t ret = 0;
-
- rdma_options_init (this);
-
- {
- dev_list = ibv_get_device_list (NULL);
-
- if (!dev_list) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_CRITICAL,
- "Failed to get IB devices");
- ret = -1;
- goto cleanup;
- }
-
- if (!*dev_list) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_CRITICAL,
- "No IB devices found");
- ret = -1;
- goto cleanup;
- }
+ gf_rdma_private_t *priv = NULL;
+ int32_t ret = 0;
+ glusterfs_ctx_t *ctx = NULL;
+ gf_rdma_options_t *options = NULL;
- if (!options->device_name) {
- if (*dev_list) {
- options->device_name =
- gf_strdup (ibv_get_device_name (*dev_list));
- } else {
- gf_log (RDMA_LOG_NAME, GF_LOG_CRITICAL,
- "IB device list is empty. Check for "
- "'ib_uverbs' module");
- return -1;
- goto cleanup;
- }
- }
+ ctx= this->ctx;
- while (*dev_list) {
- if (!strcmp (ibv_get_device_name (*dev_list),
- options->device_name)) {
- ib_ctx = ibv_open_device (*dev_list);
-
- if (!ib_ctx) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "Failed to get infiniband"
- "device context");
- ret = -1;
- goto cleanup;
- }
- break;
- }
- ++dev_list;
- }
+ priv = this->private;
- priv->device = rdma_get_device (this, ib_ctx);
+ ibv_fork_init ();
+ gf_rdma_options_init (this);
- if (!priv->device) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "could not create rdma device for %s",
- options->device_name);
- ret = -1;
- goto cleanup;
- }
- }
+ options = &priv->options;
+ priv->peer.send_count = options->send_count;
+ priv->peer.recv_count = options->recv_count;
+ priv->peer.send_size = options->send_size;
+ priv->peer.recv_size = options->recv_size;
priv->peer.trans = this;
INIT_LIST_HEAD (&priv->peer.ioq);
-
- pthread_mutex_init (&priv->read_mutex, NULL);
+
pthread_mutex_init (&priv->write_mutex, NULL);
pthread_mutex_init (&priv->recv_mutex, NULL);
pthread_cond_init (&priv->recv_cond, NULL);
- priv->request_ctx_pool = mem_pool_new (rdma_request_context_t,
- RDMA_POOL_SIZE);
- if (priv->request_ctx_pool == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- GF_FREE (priv);
- return -1;
- }
-
- priv->ioq_pool = mem_pool_new (rdma_ioq_t, RDMA_POOL_SIZE);
- if (priv->ioq_pool == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- mem_pool_destroy (priv->request_ctx_pool);
- GF_FREE (priv);
- return -1;
- }
-
- priv->reply_info_pool = mem_pool_new (rdma_reply_info_t,
- RDMA_POOL_SIZE);
- if (priv->reply_info_pool == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- mem_pool_destroy (priv->request_ctx_pool);
- mem_pool_destroy (priv->ioq_pool);
- GF_FREE (priv);
- return -1;
- }
-
-
-cleanup:
- if (-1 == ret) {
- if (ib_ctx)
- ibv_close_device (ib_ctx);
- }
-
- if (dev_list)
- ibv_free_device_list (dev_list);
-
- return ret;
-}
-
-
-static int32_t
-rdma_disconnect (rpc_transport_t *this)
-{
- rdma_private_t *priv = this->private;
- int32_t ret = 0;
-
- pthread_mutex_lock (&priv->write_mutex);
+ pthread_mutex_lock (&ctx->lock);
{
- ret = __rdma_disconnect (this);
- }
- pthread_mutex_unlock (&priv->write_mutex);
-
- return ret;
-}
-
-
-static int32_t
-__tcp_connect_finish (int fd)
-{
- int ret = -1;
- int optval = 0;
- socklen_t optlen = sizeof (int);
-
- ret = getsockopt (fd, SOL_SOCKET, SO_ERROR,
- (void *)&optval, &optlen);
-
- if (ret == 0 && optval)
- {
- errno = optval;
- ret = -1;
- }
-
- return ret;
-}
-
-static inline void
-rdma_fill_handshake_data (char *buf, struct rdma_nbio *nbio,
- rdma_private_t *priv)
-{
- sprintf (buf,
- "QP1:RECV_BLKSIZE=%08x:SEND_BLKSIZE=%08x\n"
- "QP1:LID=%04x:QPN=%06x:PSN=%06x\n",
- priv->peer.recv_size,
- priv->peer.send_size,
- priv->peer.local_lid,
- priv->peer.local_qpn,
- priv->peer.local_psn);
-
- nbio->vector.iov_base = buf;
- nbio->vector.iov_len = strlen (buf) + 1;
- nbio->count = 1;
- return;
-}
-
-static inline void
-rdma_fill_handshake_ack (char *buf, struct rdma_nbio *nbio)
-{
- sprintf (buf, "DONE\n");
- nbio->vector.iov_base = buf;
- nbio->vector.iov_len = strlen (buf) + 1;
- nbio->count = 1;
- return;
-}
-
-static int
-rdma_handshake_pollin (rpc_transport_t *this)
-{
- int ret = 0;
- rdma_private_t *priv = this->private;
- char *buf = priv->handshake.incoming.buf;
- int32_t recv_buf_size, send_buf_size;
- socklen_t sock_len;
-
- if (priv->handshake.incoming.state == RDMA_HANDSHAKE_COMPLETE) {
- return -1;
- }
-
- pthread_mutex_lock (&priv->write_mutex);
- {
- while (priv->handshake.incoming.state != RDMA_HANDSHAKE_COMPLETE)
- {
- switch (priv->handshake.incoming.state)
- {
- case RDMA_HANDSHAKE_START:
- buf = priv->handshake.incoming.buf = GF_CALLOC (1, 256, gf_common_mt_char);
- rdma_fill_handshake_data (buf, &priv->handshake.incoming, priv);
- buf[0] = 0;
- priv->handshake.incoming.state = RDMA_HANDSHAKE_RECEIVING_DATA;
- break;
-
- case RDMA_HANDSHAKE_RECEIVING_DATA:
- ret = __tcp_readv (this,
- &priv->handshake.incoming.vector,
- priv->handshake.incoming.count,
- &priv->handshake.incoming.pending_vector,
- &priv->handshake.incoming.pending_count);
- if (ret == -1) {
- goto unlock;
- }
-
- if (ret > 0) {
- gf_log (this->name, GF_LOG_TRACE,
- "partial header read on NB socket. continue later");
- goto unlock;
- }
-
- if (!ret) {
- priv->handshake.incoming.state = RDMA_HANDSHAKE_RECEIVED_DATA;
- }
- break;
-
- case RDMA_HANDSHAKE_RECEIVED_DATA:
- ret = sscanf (buf,
- "QP1:RECV_BLKSIZE=%08x:SEND_BLKSIZE=%08x\n"
- "QP1:LID=%04x:QPN=%06x:PSN=%06x\n",
- &recv_buf_size,
- &send_buf_size,
- &priv->peer.remote_lid,
- &priv->peer.remote_qpn,
- &priv->peer.remote_psn);
-
- if ((ret != 5) && (strncmp (buf, "QP1:", 4))) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_CRITICAL,
- "%s: remote-host(%s)'s "
- "transport type is different",
- this->name,
- this->peerinfo.identifier);
- ret = -1;
- goto unlock;
- }
-
- if (recv_buf_size < priv->peer.recv_size)
- priv->peer.recv_size = recv_buf_size;
- if (send_buf_size < priv->peer.send_size)
- priv->peer.send_size = send_buf_size;
-
- gf_log (RDMA_LOG_NAME, GF_LOG_TRACE,
- "%s: transacted recv_size=%d "
- "send_size=%d",
- this->name, priv->peer.recv_size,
- priv->peer.send_size);
-
- priv->peer.quota = priv->peer.send_count;
-
- if (rdma_connect_qp (this)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_ERROR,
- "%s: failed to connect with "
- "remote QP", this->name);
- ret = -1;
- goto unlock;
- }
- rdma_fill_handshake_ack (buf, &priv->handshake.incoming);
- buf[0] = 0;
- priv->handshake.incoming.state = RDMA_HANDSHAKE_RECEIVING_ACK;
- break;
-
- case RDMA_HANDSHAKE_RECEIVING_ACK:
- ret = __tcp_readv (this,
- &priv->handshake.incoming.vector,
- priv->handshake.incoming.count,
- &priv->handshake.incoming.pending_vector,
- &priv->handshake.incoming.pending_count);
- if (ret == -1) {
- goto unlock;
- }
-
- if (ret > 0) {
- gf_log (this->name, GF_LOG_TRACE,
- "partial header read on NB "
- "socket. continue later");
- goto unlock;
- }
-
- if (!ret) {
- priv->handshake.incoming.state = RDMA_HANDSHAKE_RECEIVED_ACK;
- }
- break;
-
- case RDMA_HANDSHAKE_RECEIVED_ACK:
- if (strncmp (buf, "DONE", 4)) {
- gf_log (RDMA_LOG_NAME,
- GF_LOG_DEBUG,
- "%s: handshake-3 did not "
- "return 'DONE' (%s)",
- this->name, buf);
- ret = -1;
- goto unlock;
- }
- ret = 0;
- priv->connected = 1;
- sock_len = sizeof (struct sockaddr_storage);
- getpeername (priv->sock,
- (struct sockaddr *) &this->peerinfo.sockaddr,
- &sock_len);
-
- GF_FREE (priv->handshake.incoming.buf);
- priv->handshake.incoming.buf = NULL;
- priv->handshake.incoming.state = RDMA_HANDSHAKE_COMPLETE;
+ if (ctx->ib == NULL) {
+ ctx->ib = __gf_rdma_ctx_create ();
+ if (ctx->ib == NULL) {
+ ret = -1;
}
}
}
-unlock:
- pthread_mutex_unlock (&priv->write_mutex);
-
- if (ret == -1) {
- rpc_transport_disconnect (this);
- } else {
- ret = 0;
- }
-
-
- if (!ret && priv->connected) {
- if (priv->is_server) {
- ret = rpc_transport_notify (priv->listener,
- RPC_TRANSPORT_ACCEPT,
- this);
- } else {
- ret = rpc_transport_notify (this, RPC_TRANSPORT_CONNECT,
- this);
- }
- }
+ pthread_mutex_unlock (&ctx->lock);
return ret;
}
-static int
-rdma_handshake_pollout (rpc_transport_t *this)
-{
- rdma_private_t *priv = this->private;
- char *buf = priv->handshake.outgoing.buf;
- int32_t ret = 0;
-
- if (priv->handshake.outgoing.state == RDMA_HANDSHAKE_COMPLETE) {
- return 0;
- }
-
- pthread_mutex_unlock (&priv->write_mutex);
- {
- while (priv->handshake.outgoing.state != RDMA_HANDSHAKE_COMPLETE)
- {
- switch (priv->handshake.outgoing.state)
- {
- case RDMA_HANDSHAKE_START:
- buf = priv->handshake.outgoing.buf = GF_CALLOC (1, 256, gf_common_mt_char);
- rdma_fill_handshake_data (buf, &priv->handshake.outgoing, priv);
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_SENDING_DATA;
- break;
-
- case RDMA_HANDSHAKE_SENDING_DATA:
- ret = __tcp_writev (this,
- &priv->handshake.outgoing.vector,
- priv->handshake.outgoing.count,
- &priv->handshake.outgoing.pending_vector,
- &priv->handshake.outgoing.pending_count);
- if (ret == -1) {
- goto unlock;
- }
-
- if (ret > 0) {
- gf_log (this->name, GF_LOG_TRACE,
- "partial header read on NB socket. continue later");
- goto unlock;
- }
-
- if (!ret) {
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_SENT_DATA;
- }
- break;
-
- case RDMA_HANDSHAKE_SENT_DATA:
- rdma_fill_handshake_ack (buf, &priv->handshake.outgoing);
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_SENDING_ACK;
- break;
-
- case RDMA_HANDSHAKE_SENDING_ACK:
- ret = __tcp_writev (this,
- &priv->handshake.outgoing.vector,
- priv->handshake.outgoing.count,
- &priv->handshake.outgoing.pending_vector,
- &priv->handshake.outgoing.pending_count);
- if (ret == -1) {
- goto unlock;
- }
-
- if (ret > 0) {
- gf_log (this->name, GF_LOG_TRACE,
- "partial header read on NB "
- "socket. continue later");
- goto unlock;
- }
-
- if (!ret) {
- GF_FREE (priv->handshake.outgoing.buf);
- priv->handshake.outgoing.buf = NULL;
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_COMPLETE;
- }
- break;
- }
- }
- }
-unlock:
- pthread_mutex_unlock (&priv->write_mutex);
-
- if (ret == -1) {
- rpc_transport_disconnect (this);
- } else {
- ret = 0;
- }
-
- return ret;
-}
-
-static int
-rdma_handshake_pollerr (rpc_transport_t *this)
+static int32_t
+gf_rdma_disconnect (rpc_transport_t *this)
{
- rdma_private_t *priv = this->private;
- int32_t ret = 0;
- char need_unref = 0, connected = 0;
-
- gf_log (RDMA_LOG_NAME, GF_LOG_DEBUG,
- "%s: peer disconnected, cleaning up",
- this->name);
+ gf_rdma_private_t *priv = NULL;
+ int32_t ret = 0;
- pthread_mutex_lock (&priv->write_mutex);
- {
- __rdma_teardown (this);
-
- connected = priv->connected;
- if (priv->sock != -1) {
- event_unregister (this->ctx->event_pool,
- priv->sock, priv->idx);
- need_unref = 1;
-
- if (close (priv->sock) != 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "close () - error: %s",
- strerror (errno));
- ret = -errno;
- }
- priv->tcp_connected = priv->connected = 0;
- priv->sock = -1;
- }
-
- if (priv->handshake.incoming.buf) {
- GF_FREE (priv->handshake.incoming.buf);
- priv->handshake.incoming.buf = NULL;
- }
-
- priv->handshake.incoming.state = RDMA_HANDSHAKE_START;
-
- if (priv->handshake.outgoing.buf) {
- GF_FREE (priv->handshake.outgoing.buf);
- priv->handshake.outgoing.buf = NULL;
- }
-
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_START;
- }
- pthread_mutex_unlock (&priv->write_mutex);
-
- if (connected) {
- rpc_transport_notify (this, RPC_TRANSPORT_DISCONNECT, this);
- }
-
- if (need_unref)
- rpc_transport_unref (this);
-
- return 0;
-}
-
-
-static int
-tcp_connect_finish (rpc_transport_t *this)
-{
- rdma_private_t *priv = this->private;
- int error = 0, ret = 0;
+ priv = this->private;
+ gf_log_callingfn (this->name, GF_LOG_WARNING,
+ "disconnect called (peer:%s)",
+ this->peerinfo.identifier);
pthread_mutex_lock (&priv->write_mutex);
{
- ret = __tcp_connect_finish (priv->sock);
-
- if (!ret) {
- this->myinfo.sockaddr_len =
- sizeof (this->myinfo.sockaddr);
- ret = getsockname (priv->sock,
- (struct sockaddr *)&this->myinfo.sockaddr,
- &this->myinfo.sockaddr_len);
- if (ret == -1)
- {
- gf_log (this->name, GF_LOG_ERROR,
- "getsockname on new client-socket %d "
- "failed (%s)",
- priv->sock, strerror (errno));
- close (priv->sock);
- error = 1;
- goto unlock;
- }
-
- gf_rdma_get_transport_identifiers (this);
- priv->tcp_connected = 1;
- }
-
- if (ret == -1 && errno != EINPROGRESS) {
- gf_log (this->name, GF_LOG_ERROR,
- "tcp connect to %s failed (%s)",
- this->peerinfo.identifier, strerror (errno));
- error = 1;
- }
+ ret = __gf_rdma_disconnect (this);
}
-unlock:
pthread_mutex_unlock (&priv->write_mutex);
- if (error) {
- rpc_transport_disconnect (this);
- }
-
return ret;
}
-static int
-rdma_event_handler (int fd, int idx, void *data,
- int poll_in, int poll_out, int poll_err)
-{
- rpc_transport_t *this = data;
- rdma_private_t *priv = this->private;
- rdma_options_t *options = NULL;
- int ret = 0;
-
- if (!priv->tcp_connected) {
- ret = tcp_connect_finish (this);
- if (priv->tcp_connected) {
- options = &priv->options;
-
- priv->peer.send_count = options->send_count;
- priv->peer.recv_count = options->recv_count;
- priv->peer.send_size = options->send_size;
- priv->peer.recv_size = options->recv_size;
-
- if ((ret = rdma_create_qp (this)) < 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not create QP",
- this->name);
- rpc_transport_disconnect (this);
- }
- }
- }
-
- if (!ret && poll_out && priv->tcp_connected) {
- ret = rdma_handshake_pollout (this);
- }
-
- if (!ret && poll_in && priv->tcp_connected) {
- if (priv->handshake.incoming.state == RDMA_HANDSHAKE_COMPLETE) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: pollin received on tcp socket (peer: %s) "
- "after handshake is complete",
- this->name, this->peerinfo.identifier);
- rdma_handshake_pollerr (this);
- return 0;
- }
- ret = rdma_handshake_pollin (this);
- }
-
- if (ret < 0 || poll_err) {
- ret = rdma_handshake_pollerr (this);
- }
-
- return 0;
-}
-
-static int
-__tcp_nonblock (int fd)
-{
- int flags = 0;
- int ret = -1;
-
- flags = fcntl (fd, F_GETFL);
-
- if (flags != -1)
- ret = fcntl (fd, F_SETFL, flags | O_NONBLOCK);
-
- return ret;
-}
static int32_t
-rdma_connect (struct rpc_transport *this, int port)
+gf_rdma_connect (struct rpc_transport *this, int port)
{
- dict_t *options = this->options;
-
- rdma_private_t *priv = this->private;
-
- int32_t ret = 0;
- gf_boolean_t non_blocking = 1;
- struct sockaddr_storage sockaddr;
- socklen_t sockaddr_len = 0;
+ gf_rdma_private_t *priv = NULL;
+ int32_t ret = 0;
+ union gf_sock_union sock_union = {{0, }, };
+ socklen_t sockaddr_len = 0;
+ gf_rdma_peer_t *peer = NULL;
+ gf_rdma_ctx_t *rdma_ctx = NULL;
+ gf_boolean_t connected = _gf_false;
- if (priv->connected) {
- return 0;
- }
+ priv = this->private;
- if (dict_get (options, "non-blocking-io")) {
- char *nb_connect = data_to_str (dict_get (this->options,
- "non-blocking-io"));
-
- if (gf_string2boolean (nb_connect, &non_blocking) == -1) {
- gf_log (this->name, GF_LOG_ERROR,
- "'non-blocking-io' takes only boolean "
- "options, not taking any action");
- non_blocking = 1;
- }
- }
+ peer = &priv->peer;
+
+ rpc_transport_ref (this);
ret = gf_rdma_client_get_remote_sockaddr (this,
- (struct sockaddr *)&sockaddr,
+ &sock_union.sa,
&sockaddr_len, port);
if (ret != 0) {
gf_log (this->name, GF_LOG_DEBUG,
"cannot get remote address to connect");
- return ret;
+ goto out;
}
+ rdma_ctx = this->ctx->ib;
+
pthread_mutex_lock (&priv->write_mutex);
{
- if (priv->sock != -1) {
- ret = 0;
+ if (peer->cm_id != NULL) {
+ ret = -1;
+ errno = EINPROGRESS;
+ connected = _gf_true;
goto unlock;
}
-
- priv->sock = socket (((struct sockaddr *)&sockaddr)->sa_family,
- SOCK_STREAM, 0);
-
- if (priv->sock == -1) {
+
+ priv->entity = GF_RDMA_CLIENT;
+
+ ret = rdma_create_id (rdma_ctx->rdma_cm_event_channel,
+ &peer->cm_id, this, RDMA_PS_TCP);
+ if (ret != 0) {
gf_log (this->name, GF_LOG_ERROR,
- "socket () - error: %s", strerror (errno));
+ "creation of rdma_cm_id failed (%s)",
+ strerror (errno));
ret = -errno;
goto unlock;
}
- gf_log (this->name, GF_LOG_TRACE,
- "socket fd = %d", priv->sock);
-
- memcpy (&this->peerinfo.sockaddr, &sockaddr, sockaddr_len);
+ memcpy (&this->peerinfo.sockaddr, &sock_union.storage,
+ sockaddr_len);
this->peerinfo.sockaddr_len = sockaddr_len;
if (port > 0)
- ((struct sockaddr_in *) (&sockaddr))->sin_port
- = htons (port);
+ sock_union.sin.sin_port = htons (port);
- ((struct sockaddr *) &this->myinfo.sockaddr)->sa_family =
+ ((struct sockaddr *) &this->myinfo.sockaddr)->sa_family =
((struct sockaddr *)&this->peerinfo.sockaddr)->sa_family;
- if (non_blocking)
- {
- ret = __tcp_nonblock (priv->sock);
-
- if (ret == -1)
- {
- gf_log (this->name, GF_LOG_ERROR,
- "could not set socket %d to non "
- "blocking mode (%s)",
- priv->sock, strerror (errno));
- close (priv->sock);
- priv->sock = -1;
- goto unlock;
- }
- }
-
ret = gf_rdma_client_bind (this,
(struct sockaddr *)&this->myinfo.sockaddr,
- &this->myinfo.sockaddr_len, priv->sock);
- if (ret == -1)
- {
+ &this->myinfo.sockaddr_len,
+ peer->cm_id);
+ if (ret != 0) {
gf_log (this->name, GF_LOG_WARNING,
"client bind failed: %s", strerror (errno));
- close (priv->sock);
- priv->sock = -1;
goto unlock;
}
- ret = connect (priv->sock,
- (struct sockaddr *)&this->peerinfo.sockaddr,
- this->peerinfo.sockaddr_len);
- if (ret == -1 && errno != EINPROGRESS)
- {
- gf_log (this->name, GF_LOG_ERROR,
- "connection attempt failed (%s)",
+ ret = rdma_resolve_addr (peer->cm_id, NULL, &sock_union.sa,
+ 2000);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "rdma_resolve_addr failed (%s)",
strerror (errno));
- close (priv->sock);
- priv->sock = -1;
goto unlock;
}
- priv->tcp_connected = priv->connected = 0;
-
- rpc_transport_ref (this);
-
- priv->handshake.incoming.state = RDMA_HANDSHAKE_START;
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_START;
-
- priv->idx = event_register (this->ctx->event_pool,
- priv->sock, rdma_event_handler,
- this, 1, 1);
+ priv->connected = 0;
}
unlock:
pthread_mutex_unlock (&priv->write_mutex);
- return ret;
-}
-
-static int
-rdma_server_event_handler (int fd, int idx, void *data,
- int poll_in, int poll_out, int poll_err)
-{
- int32_t main_sock = -1;
- rpc_transport_t *this, *trans = data;
- rdma_private_t *priv = NULL;
- rdma_private_t *trans_priv = (rdma_private_t *) trans->private;
- rdma_options_t *options = NULL;
-
- if (!poll_in) {
- return 0;
- }
-
- this = GF_CALLOC (1, sizeof (rpc_transport_t),
- gf_common_mt_rpc_transport_t);
- if (this == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- return -1;
- }
-
- this->listener = trans;
-
- priv = GF_CALLOC (1, sizeof (rdma_private_t),
- gf_common_mt_rdma_private_t);
- if (priv == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- GF_FREE (priv);
- return -1;
- }
- this->private = priv;
- /* Copy all the rdma related values in priv, from trans_priv
- as other than QP, all the values remain same */
- priv->device = trans_priv->device;
- priv->options = trans_priv->options;
- priv->request_ctx_pool = mem_pool_new (rdma_request_context_t,
- RDMA_POOL_SIZE);
- priv->is_server = 1;
- priv->listener = trans;
- if (priv->request_ctx_pool == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- GF_FREE (priv);
- return -1;
- }
-
- priv->ioq_pool = mem_pool_new (rdma_ioq_t, RDMA_POOL_SIZE);
- if (priv->ioq_pool == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- mem_pool_destroy (priv->request_ctx_pool);
- GF_FREE (priv);
- return -1;
- }
-
- priv->reply_info_pool = mem_pool_new (rdma_reply_info_t,
- RDMA_POOL_SIZE);
- if (priv->reply_info_pool == NULL) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR, "out of memory");
- mem_pool_destroy (priv->request_ctx_pool);
- mem_pool_destroy (priv->ioq_pool);
- GF_FREE (priv);
- return -1;
- }
-
- options = &priv->options;
+out:
+ if (ret != 0) {
+ if (!connected) {
+ gf_rdma_teardown (this);
+ }
- this->ops = trans->ops;
- this->init = trans->init;
- this->fini = trans->fini;
- this->ctx = trans->ctx;
- this->name = gf_strdup (trans->name);
- this->notify = trans->notify;
- this->mydata = trans->mydata;
-
- memcpy (&this->myinfo.sockaddr, &trans->myinfo.sockaddr,
- trans->myinfo.sockaddr_len);
- this->myinfo.sockaddr_len = trans->myinfo.sockaddr_len;
-
- main_sock = (trans_priv)->sock;
- this->peerinfo.sockaddr_len = sizeof (this->peerinfo.sockaddr);
- priv->sock = accept (main_sock,
- (struct sockaddr *)&this->peerinfo.sockaddr,
- &this->peerinfo.sockaddr_len);
- if (priv->sock == -1) {
- gf_log ("rdma/server", GF_LOG_ERROR,
- "accept() failed: %s",
- strerror (errno));
- mem_pool_destroy (priv->request_ctx_pool);
- mem_pool_destroy (priv->ioq_pool);
- GF_FREE (this->private);
- GF_FREE (this);
- return -1;
+ rpc_transport_unref (this);
}
- priv->peer.trans = this;
- rpc_transport_ref (this);
-
- gf_rdma_get_transport_identifiers (this);
+ return ret;
+}
- priv->tcp_connected = 1;
- priv->handshake.incoming.state = RDMA_HANDSHAKE_START;
- priv->handshake.outgoing.state = RDMA_HANDSHAKE_START;
- priv->peer.send_count = options->send_count;
- priv->peer.recv_count = options->recv_count;
- priv->peer.send_size = options->send_size;
- priv->peer.recv_size = options->recv_size;
- INIT_LIST_HEAD (&priv->peer.ioq);
+static int32_t
+gf_rdma_listen (rpc_transport_t *this)
+{
+ union gf_sock_union sock_union = {{0, }, };
+ socklen_t sockaddr_len = 0;
+ gf_rdma_private_t *priv = NULL;
+ gf_rdma_peer_t *peer = NULL;
+ int ret = 0;
+ gf_rdma_ctx_t *rdma_ctx = NULL;
+ char service[NI_MAXSERV], host[NI_MAXHOST];
- if (rdma_create_qp (this) < 0) {
- gf_log (RDMA_LOG_NAME, GF_LOG_ERROR,
- "%s: could not create QP",
- this->name);
- rpc_transport_disconnect (this);
- return -1;
- }
+ priv = this->private;
+ peer = &priv->peer;
- priv->idx = event_register (this->ctx->event_pool, priv->sock,
- rdma_event_handler, this, 1, 1);
+ priv->entity = GF_RDMA_SERVER_LISTENER;
- pthread_mutex_init (&priv->read_mutex, NULL);
- pthread_mutex_init (&priv->write_mutex, NULL);
- pthread_mutex_init (&priv->recv_mutex, NULL);
- /* pthread_cond_init (&priv->recv_cond, NULL); */
- return 0;
-}
+ rdma_ctx = this->ctx->ib;
-static int32_t
-rdma_listen (rpc_transport_t *this)
-{
- struct sockaddr_storage sockaddr;
- socklen_t sockaddr_len;
- rdma_private_t *priv = this->private;
- int opt = 1, ret = 0;
- char service[NI_MAXSERV], host[NI_MAXHOST];
-
- memset (&sockaddr, 0, sizeof (sockaddr));
- ret = gf_rdma_server_get_local_sockaddr (this,
- (struct sockaddr *)&sockaddr,
+ ret = gf_rdma_server_get_local_sockaddr (this, &sock_union.sa,
&sockaddr_len);
if (ret != 0) {
- gf_log (this->name, GF_LOG_DEBUG,
+ gf_log (this->name, GF_LOG_WARNING,
"cannot find network address of server to bind to");
goto err;
}
- priv->sock = socket (((struct sockaddr *)&sockaddr)->sa_family,
- SOCK_STREAM, 0);
- if (priv->sock == -1) {
- gf_log ("rdma/server", GF_LOG_CRITICAL,
- "init: failed to create socket, error: %s",
+ ret = rdma_create_id (rdma_ctx->rdma_cm_event_channel,
+ &peer->cm_id, this, RDMA_PS_TCP);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "creation of rdma_cm_id failed (%s)",
strerror (errno));
- GF_FREE (this->private);
- ret = -1;
goto err;
}
- memcpy (&this->myinfo.sockaddr, &sockaddr, sockaddr_len);
+ memcpy (&this->myinfo.sockaddr, &sock_union.storage,
+ sockaddr_len);
this->myinfo.sockaddr_len = sockaddr_len;
- ret = getnameinfo ((struct sockaddr *)&this->myinfo.sockaddr,
- this->myinfo.sockaddr_len,
- host, sizeof (host),
+ ret = getnameinfo ((struct sockaddr *)&this->myinfo.sockaddr,
+ this->myinfo.sockaddr_len, host, sizeof (host),
service, sizeof (service),
NI_NUMERICHOST);
if (ret != 0) {
@@ -4687,57 +4429,58 @@ rdma_listen (rpc_transport_t *this)
"getnameinfo failed (%s)", gai_strerror (ret));
goto err;
}
+
sprintf (this->myinfo.identifier, "%s:%s", host, service);
-
- setsockopt (priv->sock, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof (opt));
- if (bind (priv->sock,
- (struct sockaddr *)&sockaddr,
- sockaddr_len) != 0) {
- ret = -1;
- gf_log ("rdma/server", GF_LOG_ERROR,
- "init: failed to bind to socket for %s (%s)",
- this->myinfo.identifier, strerror (errno));
+
+ ret = rdma_bind_addr (peer->cm_id, &sock_union.sa);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "rdma_bind_addr failed (%s)", strerror (errno));
goto err;
}
- if (listen (priv->sock, 10) != 0) {
- gf_log ("rdma/server", GF_LOG_ERROR,
- "init: listen () failed on socket for %s (%s)",
- this->myinfo.identifier, strerror (errno));
- ret = -1;
+ ret = rdma_listen (peer->cm_id, 10);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "rdma_listen failed (%s)", strerror (errno));
goto err;
}
- /* Register the main socket */
- priv->idx = event_register (this->ctx->event_pool, priv->sock,
- rdma_server_event_handler,
- rpc_transport_ref (this), 1, 0);
+ rpc_transport_ref (this);
+ ret = 0;
err:
+ if (ret < 0) {
+ if (peer->cm_id != NULL) {
+ rdma_destroy_id (peer->cm_id);
+ peer->cm_id = NULL;
+ }
+ }
+
return ret;
}
+
struct rpc_transport_ops tops = {
- .submit_request = rdma_submit_request,
- .submit_reply = rdma_submit_reply,
- .connect = rdma_connect,
- .disconnect = rdma_disconnect,
- .listen = rdma_listen,
+ .submit_request = gf_rdma_submit_request,
+ .submit_reply = gf_rdma_submit_reply,
+ .connect = gf_rdma_connect,
+ .disconnect = gf_rdma_disconnect,
+ .listen = gf_rdma_listen,
};
int32_t
init (rpc_transport_t *this)
{
- rdma_private_t *priv = NULL;
+ gf_rdma_private_t *priv = NULL;
priv = GF_CALLOC (1, sizeof (*priv), gf_common_mt_rdma_private_t);
if (!priv)
return -1;
this->private = priv;
- priv->sock = -1;
- if (rdma_init (this)) {
+ if (gf_rdma_init (this)) {
gf_log (this->name, GF_LOG_ERROR,
"Failed to initialize IB Device");
return -1;
@@ -4750,23 +4493,15 @@ void
fini (struct rpc_transport *this)
{
/* TODO: verify this function does graceful finish */
- rdma_private_t *priv = this->private;
+ gf_rdma_private_t *priv = NULL;
+
+ priv = this->private;
+
this->private = NULL;
if (priv) {
pthread_mutex_destroy (&priv->recv_mutex);
pthread_mutex_destroy (&priv->write_mutex);
- pthread_mutex_destroy (&priv->read_mutex);
-
- mem_pool_destroy (priv->request_ctx_pool);
- mem_pool_destroy (priv->ioq_pool);
- mem_pool_destroy (priv->reply_info_pool);
-
- /* pthread_cond_destroy (&priv->recv_cond); */
- if (priv->sock != -1) {
- event_unregister (this->ctx->event_pool,
- priv->sock, priv->idx);
- }
gf_log (this->name, GF_LOG_TRACE,
"called fini on transport: %p", this);
@@ -4778,51 +4513,63 @@ fini (struct rpc_transport *this)
/* TODO: expand each option */
struct volume_options options[] = {
{ .key = {"transport.rdma.port",
- "rdma-port"},
+ "rdma-port"},
.type = GF_OPTION_TYPE_INT,
.min = 1,
.max = 4,
.description = "check the option by 'ibv_devinfo'"
},
{ .key = {"transport.rdma.mtu",
- "rdma-mtu"},
+ "rdma-mtu"},
.type = GF_OPTION_TYPE_INT,
},
{ .key = {"transport.rdma.device-name",
- "rdma-device-name"},
+ "rdma-device-name"},
.type = GF_OPTION_TYPE_ANY,
.description = "check by 'ibv_devinfo'"
},
{ .key = {"transport.rdma.work-request-send-count",
- "rdma-work-request-send-count"},
+ "rdma-work-request-send-count"},
.type = GF_OPTION_TYPE_INT,
},
{ .key = {"transport.rdma.work-request-recv-count",
- "rdma-work-request-recv-count"},
+ "rdma-work-request-recv-count"},
.type = GF_OPTION_TYPE_INT,
},
- { .key = {"remote-port",
+ { .key = {"remote-port",
"transport.remote-port",
- "transport.rdma.remote-port"},
- .type = GF_OPTION_TYPE_INT
+ "transport.rdma.remote-port"},
+ .type = GF_OPTION_TYPE_INT
+ },
+ { .key = {"transport.rdma.attr-timeout",
+ "rdma-attr-timeout"},
+ .type = GF_OPTION_TYPE_INT
+ },
+ { .key = {"transport.rdma.attr-retry-cnt",
+ "rdma-attr-retry-cnt"},
+ .type = GF_OPTION_TYPE_INT
+ },
+ { .key = {"transport.rdma.attr-rnr-retry",
+ "rdma-attr-rnr-retry"},
+ .type = GF_OPTION_TYPE_INT
},
- { .key = {"transport.rdma.listen-port", "listen-port"},
- .type = GF_OPTION_TYPE_INT
+ { .key = {"transport.rdma.listen-port", "listen-port"},
+ .type = GF_OPTION_TYPE_INT
},
- { .key = {"transport.rdma.connect-path", "connect-path"},
- .type = GF_OPTION_TYPE_ANY
+ { .key = {"transport.rdma.connect-path", "connect-path"},
+ .type = GF_OPTION_TYPE_ANY
},
- { .key = {"transport.rdma.bind-path", "bind-path"},
- .type = GF_OPTION_TYPE_ANY
+ { .key = {"transport.rdma.bind-path", "bind-path"},
+ .type = GF_OPTION_TYPE_ANY
},
- { .key = {"transport.rdma.listen-path", "listen-path"},
- .type = GF_OPTION_TYPE_ANY
+ { .key = {"transport.rdma.listen-path", "listen-path"},
+ .type = GF_OPTION_TYPE_ANY
},
{ .key = {"transport.address-family",
- "address-family"},
+ "address-family"},
.value = {"inet", "inet6", "inet/inet6", "inet6/inet",
"unix", "inet-sdp" },
- .type = GF_OPTION_TYPE_STR
+ .type = GF_OPTION_TYPE_STR
},
{ .key = {"transport.socket.lowlat"},
.type = GF_OPTION_TYPE_BOOL
diff --git a/rpc/rpc-transport/rdma/src/rdma.h b/rpc/rpc-transport/rdma/src/rdma.h
index 97cfd046..7f76244f 100644
--- a/rpc/rpc-transport/rdma/src/rdma.h
+++ b/rpc/rpc-transport/rdma/src/rdma.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2006-2009 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _XPORT_RDMA_H
@@ -38,49 +29,59 @@
#include <list.h>
#include <arpa/inet.h>
#include <infiniband/verbs.h>
+#include <rdma/rdma_cma.h>
/* FIXME: give appropriate values to these macros */
-#define GF_DEFAULT_RDMA_LISTEN_PORT 6968
-#define RDMA_MAX_SEGMENTS 8
-#define RDMA_MAX_HEADER_SIZE (sizeof (rdma_header_t) \
- + RDMA_MAX_SEGMENTS \
- * sizeof (rdma_read_chunk_t))
-#define RDMA_INLINE_THRESHOLD (1024 * 128)
-#define RDMA_VERSION 1
-#define RDMA_POOL_SIZE 512
-
-typedef enum rdma_errcode {
+#define GF_DEFAULT_RDMA_LISTEN_PORT (GF_DEFAULT_BASE_PORT + 1)
+
+/* If you are changing GF_RDMA_MAX_SEGMENTS, please make sure to update
+ * GLUSTERFS_GF_RDMA_MAX_HEADER_SIZE defined in glusterfs.h .
+ */
+#define GF_RDMA_MAX_SEGMENTS 8
+
+#define GF_RDMA_VERSION 1
+#define GF_RDMA_POOL_SIZE 512
+
+/* Additional attributes */
+#define GF_RDMA_TIMEOUT 14
+#define GF_RDMA_RETRY_CNT 7
+#define GF_RDMA_RNR_RETRY 7
+
+typedef enum gf_rdma_errcode {
ERR_VERS = 1,
ERR_CHUNK = 2
-}rdma_errcode_t;
+}gf_rdma_errcode_t;
-struct rdma_err_vers {
- uint32_t rdma_vers_low; /* Version range supported by peer */
- uint32_t rdma_vers_high;
+struct gf_rdma_err_vers {
+ uint32_t gf_rdma_vers_low; /* Version range supported by peer */
+ uint32_t gf_rdma_vers_high;
}__attribute__ ((packed));
-typedef struct rdma_err_vers rdma_err_vers_t;
-
-typedef enum rdma_proc {
- RDMA_MSG = 0, /* An RPC call or reply msg */
- RDMA_NOMSG = 1, /* An RPC call or reply msg - separate body */
- RDMA_MSGP = 2, /* An RPC call or reply msg with padding */
- RDMA_DONE = 3, /* Client signals reply completion */
- RDMA_ERROR = 4 /* An RPC RDMA encoding error */
-}rdma_proc_t;
-
-typedef enum rdma_chunktype {
- rdma_noch = 0, /* no chunk */
- rdma_readch, /* some argument through rdma read */
- rdma_areadch, /* entire request through rdma read */
- rdma_writech, /* some result through rdma write */
- rdma_replych /* entire reply through rdma write */
-}rdma_chunktype_t;
-
-struct __rdma_header {
+typedef struct gf_rdma_err_vers gf_rdma_err_vers_t;
+
+typedef enum gf_rdma_proc {
+ GF_RDMA_MSG = 0, /* An RPC call or reply msg */
+ GF_RDMA_NOMSG = 1, /* An RPC call or reply msg - separate body */
+ GF_RDMA_MSGP = 2, /* An RPC call or reply msg with padding */
+ GF_RDMA_DONE = 3, /* Client signals reply completion */
+ GF_RDMA_ERROR = 4 /* An RPC RDMA encoding error */
+}gf_rdma_proc_t;
+
+typedef enum gf_rdma_chunktype {
+ gf_rdma_noch = 0, /* no chunk */
+ gf_rdma_readch, /* some argument through rdma read */
+ gf_rdma_areadch, /* entire request through rdma read */
+ gf_rdma_writech, /* some result through rdma write */
+ gf_rdma_replych /* entire reply through rdma write */
+}gf_rdma_chunktype_t;
+
+/* If you are modifying __gf_rdma_header, please make sure to change
+ * GLUSTERFS_GF_RDMA_MAX_HEADER_SIZE defined in glusterfs.h to reflect your changes
+ */
+struct __gf_rdma_header {
uint32_t rm_xid; /* Mirrors the RPC header xid */
uint32_t rm_vers; /* Version of this protocol */
uint32_t rm_credit; /* Buffers requested/granted */
- uint32_t rm_type; /* Type of message (enum rdma_proc) */
+ uint32_t rm_type; /* Type of message (enum gf_rdma_proc) */
union {
struct { /* no chunks */
uint32_t rm_empty[3]; /* 3 empty chunk lists */
@@ -94,45 +95,49 @@ struct __rdma_header {
struct {
uint32_t rm_type;
- rdma_err_vers_t rm_version;
+ gf_rdma_err_vers_t rm_version;
}__attribute__ ((packed)) rm_error;
uint32_t rm_chunks[0]; /* read, write and reply chunks */
}__attribute__ ((packed)) rm_body;
} __attribute__((packed));
-typedef struct __rdma_header rdma_header_t;
+typedef struct __gf_rdma_header gf_rdma_header_t;
-struct __rdma_segment {
+/* If you are modifying __gf_rdma_segment or __gf_rdma_read_chunk, please make sure
+ * to change GLUSTERFS_GF_RDMA_MAX_HEADER_SIZE defined in glusterfs.h to reflect
+ * your changes.
+ */
+struct __gf_rdma_segment {
uint32_t rs_handle; /* Registered memory handle */
uint32_t rs_length; /* Length of the chunk in bytes */
uint64_t rs_offset; /* Chunk virtual address or offset */
} __attribute__((packed));
-typedef struct __rdma_segment rdma_segment_t;
+typedef struct __gf_rdma_segment gf_rdma_segment_t;
/* read chunk(s), encoded as a linked list. */
-struct __rdma_read_chunk {
+struct __gf_rdma_read_chunk {
uint32_t rc_discrim; /* 1 indicates presence */
uint32_t rc_position; /* Position in XDR stream */
- rdma_segment_t rc_target;
+ gf_rdma_segment_t rc_target;
} __attribute__((packed));
-typedef struct __rdma_read_chunk rdma_read_chunk_t;
+typedef struct __gf_rdma_read_chunk gf_rdma_read_chunk_t;
/* write chunk, and reply chunk. */
-struct __rdma_write_chunk {
- rdma_segment_t wc_target;
+struct __gf_rdma_write_chunk {
+ gf_rdma_segment_t wc_target;
} __attribute__((packed));
-typedef struct __rdma_write_chunk rdma_write_chunk_t;
+typedef struct __gf_rdma_write_chunk gf_rdma_write_chunk_t;
/* write chunk(s), encoded as a counted array. */
-struct __rdma_write_array {
+struct __gf_rdma_write_array {
uint32_t wc_discrim; /* 1 indicates presence */
uint32_t wc_nchunks; /* Array count */
- struct __rdma_write_chunk wc_array[0];
+ struct __gf_rdma_write_chunk wc_array[0];
} __attribute__((packed));
-typedef struct __rdma_write_array rdma_write_array_t;
+typedef struct __gf_rdma_write_array gf_rdma_write_array_t;
/* options per transport end point */
-struct __rdma_options {
+struct __gf_rdma_options {
int32_t port;
char *device_name;
enum ibv_mtu mtu;
@@ -140,26 +145,29 @@ struct __rdma_options {
int32_t recv_count;
uint64_t recv_size;
uint64_t send_size;
+ uint8_t attr_timeout;
+ uint8_t attr_retry_cnt;
+ uint8_t attr_rnr_retry;
};
-typedef struct __rdma_options rdma_options_t;
+typedef struct __gf_rdma_options gf_rdma_options_t;
-struct __rdma_reply_info {
+struct __gf_rdma_reply_info {
uint32_t rm_xid; /* xid in network endian */
- rdma_chunktype_t type; /*
- * can be either rdma_replych
- * or rdma_writech.
+ gf_rdma_chunktype_t type; /*
+ * can be either gf_rdma_replych
+ * or gf_rdma_writech.
*/
- rdma_write_array_t *wc_array;
+ gf_rdma_write_array_t *wc_array;
struct mem_pool *pool;
};
-typedef struct __rdma_reply_info rdma_reply_info_t;
+typedef struct __gf_rdma_reply_info gf_rdma_reply_info_t;
-struct __rdma_ioq {
+struct __gf_rdma_ioq {
union {
struct list_head list;
struct {
- struct __rdma_ioq *next;
- struct __rdma_ioq *prev;
+ struct __gf_rdma_ioq *next;
+ struct __gf_rdma_ioq *prev;
};
};
@@ -174,8 +182,8 @@ struct __rdma_ioq {
struct iobref *iobref;
union {
- struct __rdma_ioq_request {
- /* used to build reply_chunk for RDMA_NOMSG type msgs */
+ struct __gf_rdma_ioq_request {
+ /* used to build reply_chunk for GF_RDMA_NOMSG type msgs */
struct iovec rsphdr_vec[MAX_IOVEC];
int rsphdr_count;
@@ -192,205 +200,183 @@ struct __rdma_ioq {
struct iobref *rsp_iobref;
}request;
- rdma_reply_info_t *reply_info;
+ gf_rdma_reply_info_t *reply_info;
}msg;
struct mem_pool *pool;
};
-typedef struct __rdma_ioq rdma_ioq_t;
+typedef struct __gf_rdma_ioq gf_rdma_ioq_t;
-typedef enum __rdma_send_post_type {
- RDMA_SEND_POST_NO_CHUNKLIST, /* post which is sent using rdma-send
+typedef enum __gf_rdma_send_post_type {
+ GF_RDMA_SEND_POST_NO_CHUNKLIST, /* post which is sent using rdma-send
* and the msg carries no
* chunklists.
*/
- RDMA_SEND_POST_READ_CHUNKLIST, /* post which is sent using rdma-send
+ GF_RDMA_SEND_POST_READ_CHUNKLIST, /* post which is sent using rdma-send
* and the msg carries only read
* chunklist.
*/
- RDMA_SEND_POST_WRITE_CHUNKLIST, /* post which is sent using
+ GF_RDMA_SEND_POST_WRITE_CHUNKLIST, /* post which is sent using
* rdma-send and the msg carries
* only write chunklist.
*/
- RDMA_SEND_POST_READ_WRITE_CHUNKLIST, /* post which is sent using
+ GF_RDMA_SEND_POST_READ_WRITE_CHUNKLIST, /* post which is sent using
* rdma-send and the msg
* carries both read and
* write chunklists.
*/
- RDMA_SEND_POST_RDMA_READ, /* RDMA read */
- RDMA_SEND_POST_RDMA_WRITE, /* RDMA write */
-}rdma_send_post_type_t;
-
+ GF_RDMA_SEND_POST_GF_RDMA_READ, /* RDMA read */
+ GF_RDMA_SEND_POST_GF_RDMA_WRITE, /* RDMA write */
+}gf_rdma_send_post_type_t;
+
/* represents one communication peer, two per transport_t */
-struct __rdma_peer {
- rpc_transport_t *trans;
- struct ibv_qp *qp;
+struct __gf_rdma_peer {
+ rpc_transport_t *trans;
+ struct rdma_cm_id *cm_id;
+ struct ibv_qp *qp;
+ pthread_t rdma_event_thread;
+ char quota_set;
int32_t recv_count;
int32_t send_count;
int32_t recv_size;
int32_t send_size;
- int32_t quota;
+ int32_t quota;
union {
- struct list_head ioq;
+ struct list_head ioq;
struct {
- rdma_ioq_t *ioq_next;
- rdma_ioq_t *ioq_prev;
+ gf_rdma_ioq_t *ioq_next;
+ gf_rdma_ioq_t *ioq_prev;
};
};
/* QP attributes, needed to connect with remote QP */
- int32_t local_lid;
- int32_t local_psn;
- int32_t local_qpn;
- int32_t remote_lid;
- int32_t remote_psn;
- int32_t remote_qpn;
+ int32_t local_lid;
+ int32_t local_psn;
+ int32_t local_qpn;
+ int32_t remote_lid;
+ int32_t remote_psn;
+ int32_t remote_qpn;
};
-typedef struct __rdma_peer rdma_peer_t;
+typedef struct __gf_rdma_peer gf_rdma_peer_t;
-struct __rdma_post_context {
- struct ibv_mr *mr[RDMA_MAX_SEGMENTS];
+struct __gf_rdma_post_context {
+ struct ibv_mr *mr[GF_RDMA_MAX_SEGMENTS];
int mr_count;
struct iovec vector[MAX_IOVEC];
int count;
struct iobref *iobref;
+ struct iobuf *hdr_iobuf;
char is_request;
- int rdma_reads;
- rdma_reply_info_t *reply_info;
+ int gf_rdma_reads;
+ gf_rdma_reply_info_t *reply_info;
};
-typedef struct __rdma_post_context rdma_post_context_t;
+typedef struct __gf_rdma_post_context gf_rdma_post_context_t;
typedef enum {
- RDMA_SEND_POST,
- RDMA_RECV_POST
-} rdma_post_type_t;
+ GF_RDMA_SEND_POST,
+ GF_RDMA_RECV_POST
+} gf_rdma_post_type_t;
-struct __rdma_post {
- struct __rdma_post *next, *prev;
+struct __gf_rdma_post {
+ struct __gf_rdma_post *next, *prev;
struct ibv_mr *mr;
char *buf;
int32_t buf_size;
char aux;
int32_t reused;
- struct __rdma_device *device;
- rdma_post_type_t type;
- rdma_post_context_t ctx;
+ struct __gf_rdma_device *device;
+ gf_rdma_post_type_t type;
+ gf_rdma_post_context_t ctx;
int refcount;
pthread_mutex_t lock;
};
-typedef struct __rdma_post rdma_post_t;
+typedef struct __gf_rdma_post gf_rdma_post_t;
-struct __rdma_queue {
- rdma_post_t active_posts, passive_posts;
+struct __gf_rdma_queue {
+ gf_rdma_post_t active_posts, passive_posts;
int32_t active_count, passive_count;
pthread_mutex_t lock;
};
-typedef struct __rdma_queue rdma_queue_t;
+typedef struct __gf_rdma_queue gf_rdma_queue_t;
-struct __rdma_qpreg {
+struct __gf_rdma_qpreg {
pthread_mutex_t lock;
int32_t count;
struct _qpent {
struct _qpent *next, *prev;
int32_t qp_num;
- rdma_peer_t *peer;
+ gf_rdma_peer_t *peer;
} ents[42];
};
-typedef struct __rdma_qpreg rdma_qpreg_t;
+typedef struct __gf_rdma_qpreg gf_rdma_qpreg_t;
/* context per device, stored in global glusterfs_ctx_t->ib */
-struct __rdma_device {
- struct __rdma_device *next;
+struct __gf_rdma_device {
+ struct __gf_rdma_device *next;
const char *device_name;
struct ibv_context *context;
int32_t port;
struct ibv_pd *pd;
struct ibv_srq *srq;
- rdma_qpreg_t qpreg;
+ gf_rdma_qpreg_t qpreg;
struct ibv_comp_channel *send_chan, *recv_chan;
struct ibv_cq *send_cq, *recv_cq;
- rdma_queue_t sendq, recvq;
- pthread_t send_thread, recv_thread;
+ gf_rdma_queue_t sendq, recvq;
+ pthread_t send_thread, recv_thread, async_event_thread;
+ struct mem_pool *request_ctx_pool;
+ struct mem_pool *ioq_pool;
+ struct mem_pool *reply_info_pool;
};
-typedef struct __rdma_device rdma_device_t;
+typedef struct __gf_rdma_device gf_rdma_device_t;
-typedef enum {
- RDMA_HANDSHAKE_START = 0,
- RDMA_HANDSHAKE_SENDING_DATA,
- RDMA_HANDSHAKE_RECEIVING_DATA,
- RDMA_HANDSHAKE_SENT_DATA,
- RDMA_HANDSHAKE_RECEIVED_DATA,
- RDMA_HANDSHAKE_SENDING_ACK,
- RDMA_HANDSHAKE_RECEIVING_ACK,
- RDMA_HANDSHAKE_RECEIVED_ACK,
- RDMA_HANDSHAKE_COMPLETE,
-} rdma_handshake_state_t;
-
-struct rdma_nbio {
- int state;
- char *buf;
- int count;
- struct iovec vector;
- struct iovec *pending_vector;
- int pending_count;
+struct __gf_rdma_ctx {
+ gf_rdma_device_t *device;
+ struct rdma_event_channel *rdma_cm_event_channel;
+ pthread_t rdma_cm_thread;
};
+typedef struct __gf_rdma_ctx gf_rdma_ctx_t;
-struct __rdma_request_context {
- struct ibv_mr *mr[RDMA_MAX_SEGMENTS];
+struct __gf_rdma_request_context {
+ struct ibv_mr *mr[GF_RDMA_MAX_SEGMENTS];
int mr_count;
struct mem_pool *pool;
- rdma_peer_t *peer;
+ gf_rdma_peer_t *peer;
struct iobref *iobref;
struct iobref *rsp_iobref;
};
-typedef struct __rdma_request_context rdma_request_context_t;
-
-struct __rdma_private {
- int32_t sock;
- int32_t idx;
- unsigned char connected;
- unsigned char tcp_connected;
- unsigned char ib_connected;
- in_addr_t addr;
+typedef struct __gf_rdma_request_context gf_rdma_request_context_t;
+
+typedef enum {
+ GF_RDMA_SERVER_LISTENER,
+ GF_RDMA_SERVER,
+ GF_RDMA_CLIENT,
+} gf_rdma_transport_entity_t;
+
+struct __gf_rdma_private {
+ int32_t idx;
+ unsigned char connected;
+ in_addr_t addr;
unsigned short port;
/* IB Verbs Driver specific variables, pointers */
- rdma_peer_t peer;
- struct __rdma_device *device;
- rdma_options_t options;
+ gf_rdma_peer_t peer;
+ struct __gf_rdma_device *device;
+ gf_rdma_options_t options;
/* Used by trans->op->receive */
- char *data_ptr;
- int32_t data_offset;
- int32_t data_len;
+ char *data_ptr;
+ int32_t data_offset;
+ int32_t data_len;
/* Mutex */
- pthread_mutex_t read_mutex;
- pthread_mutex_t write_mutex;
- pthread_barrier_t handshake_barrier;
- char handshake_ret;
- char is_server;
- rpc_transport_t *listener;
-
- pthread_mutex_t recv_mutex;
- pthread_cond_t recv_cond;
-
- struct mem_pool *request_ctx_pool;
- struct mem_pool *ioq_pool;
- struct mem_pool *reply_info_pool;
-
- /* used during rdma_handshake */
- struct {
- struct rdma_nbio incoming;
- struct rdma_nbio outgoing;
- int state;
- rdma_header_t header;
- char *buf;
- size_t size;
- } handshake;
+ pthread_mutex_t write_mutex;
+ rpc_transport_t *listener;
+ pthread_mutex_t recv_mutex;
+ pthread_cond_t recv_cond;
+ gf_rdma_transport_entity_t entity;
};
-typedef struct __rdma_private rdma_private_t;
+typedef struct __gf_rdma_private gf_rdma_private_t;
-#endif /* _XPORT_RDMA_H */
+#endif /* _XPORT_GF_RDMA_H */
diff --git a/rpc/rpc-transport/socket/src/Makefile.am b/rpc/rpc-transport/socket/src/Makefile.am
index 2c918c7e..71e6ed6f 100644
--- a/rpc/rpc-transport/socket/src/Makefile.am
+++ b/rpc/rpc-transport/socket/src/Makefile.am
@@ -3,13 +3,15 @@ noinst_HEADERS = socket.h name.h
rpctransport_LTLIBRARIES = socket.la
rpctransportdir = $(libdir)/glusterfs/$(PACKAGE_VERSION)/rpc-transport
-socket_la_LDFLAGS = -module -avoidversion
+socket_la_LDFLAGS = -module -avoid-version
socket_la_SOURCES = socket.c name.c
-socket_la_LIBADD = $(top_builddir)/libglusterfs/src/libglusterfs.la
+socket_la_LIBADD = $(top_builddir)/libglusterfs/src/libglusterfs.la -lssl
-AM_CFLAGS = -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Wall -D$(GF_HOST_OS)\
+AM_CPPFLAGS = $(GF_CPPFLAGS) \
-I$(top_srcdir)/libglusterfs/src -I$(top_srcdir)/rpc/rpc-lib/src/ \
- -I$(top_srcdir)/rpc/xdr/src/ -shared -nostartfiles $(GF_CFLAGS)
+ -I$(top_srcdir)/rpc/xdr/src/
+
+AM_CFLAGS = -Wall $(GF_CFLAGS)
CLEANFILES = *~
diff --git a/rpc/rpc-transport/socket/src/name.c b/rpc/rpc-transport/socket/src/name.c
index 8be29163..1647d5b6 100644
--- a/rpc/rpc-transport/socket/src/name.c
+++ b/rpc/rpc-transport/socket/src/name.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#include <sys/types.h>
@@ -24,18 +15,13 @@
#include <netdb.h>
#include <string.h>
-#ifdef CLIENT_PORT_CEILING
-#undef CLIENT_PORT_CEILING
-#endif
-
-#define CLIENT_PORT_CEILING 1024
-
#ifndef AF_INET_SDP
#define AF_INET_SDP 27
#endif
#include "rpc-transport.h"
#include "socket.h"
+#include "common-utils.h"
int32_t
gf_resolve_ip6 (const char *hostname,
@@ -48,9 +34,17 @@ static int32_t
af_inet_bind_to_port_lt_ceiling (int fd, struct sockaddr *sockaddr,
socklen_t sockaddr_len, int ceiling)
{
- int32_t ret = -1;
- /* struct sockaddr_in sin = {0, }; */
- uint16_t port = ceiling - 1;
+ int32_t ret = -1;
+ uint16_t port = ceiling - 1;
+ // by default assume none of the ports are blocked and all are available
+ gf_boolean_t ports[1024] = {_gf_false,};
+ int i = 0;
+
+ ret = gf_process_reserved_ports (ports);
+ if (ret != 0) {
+ for (i = 0; i < 1024; i++)
+ ports[i] = _gf_false;
+ }
while (port)
{
@@ -65,7 +59,11 @@ af_inet_bind_to_port_lt_ceiling (int fd, struct sockaddr *sockaddr,
((struct sockaddr_in *)sockaddr)->sin_port = htons (port);
break;
}
-
+ // ignore the reserved ports
+ if (ports[port] == _gf_true) {
+ port--;
+ continue;
+ }
ret = bind (fd, sockaddr, sockaddr_len);
if (ret == 0)
@@ -95,7 +93,7 @@ af_unix_client_bind (rpc_transport_t *this,
char *path = data_to_str (path_data);
if (!path || strlen (path) > UNIX_PATH_MAX) {
gf_log (this->name, GF_LOG_TRACE,
- "bind-path not specfied for unix socket, "
+ "bind-path not specified for unix socket, "
"letting connect to assign default value");
goto err;
}
@@ -111,7 +109,7 @@ af_unix_client_bind (rpc_transport_t *this,
}
} else {
gf_log (this->name, GF_LOG_TRACE,
- "bind-path not specfied for unix socket, "
+ "bind-path not specified for unix socket, "
"letting connect to assign default value");
}
@@ -126,6 +124,8 @@ client_fill_address_family (rpc_transport_t *this, sa_family_t *sa_family)
int32_t ret = -1;
if (sa_family == NULL) {
+ gf_log_callingfn ("", GF_LOG_WARNING,
+ "sa_family argument is NULL");
goto out;
}
@@ -140,24 +140,24 @@ client_fill_address_family (rpc_transport_t *this, sa_family_t *sa_family)
if (!(remote_host_data || connect_path_data) ||
(remote_host_data && connect_path_data)) {
gf_log (this->name, GF_LOG_ERROR,
- "transport.address-family not specified and "
- "not able to determine the "
- "same from other options (remote-host:%s and "
- "transport.unix.connect-path:%s)",
+ "transport.address-family not specified. "
+ "Could not guess default value from (remote-host:%s or "
+ "transport.unix.connect-path:%s) options",
data_to_str (remote_host_data),
data_to_str (connect_path_data));
+ *sa_family = AF_UNSPEC;
goto out;
}
if (remote_host_data) {
gf_log (this->name, GF_LOG_DEBUG,
"address-family not specified, guessing it "
- "to be inet/inet6");
- *sa_family = AF_UNSPEC;
+ "to be inet from (remote-host: %s)", data_to_str (remote_host_data));
+ *sa_family = AF_INET;
} else {
gf_log (this->name, GF_LOG_DEBUG,
"address-family not specified, guessing it "
- "to be unix");
+ "to be unix from (transport.unix.connect-path: %s)", data_to_str (connect_path_data));
*sa_family = AF_UNIX;
}
@@ -171,13 +171,11 @@ client_fill_address_family (rpc_transport_t *this, sa_family_t *sa_family)
*sa_family = AF_INET6;
} else if (!strcasecmp (address_family, "inet-sdp")) {
*sa_family = AF_INET_SDP;
- } else if (!strcasecmp (address_family, "inet/inet6")
- || !strcasecmp (address_family, "inet6/inet")) {
- *sa_family = AF_UNSPEC;
} else {
gf_log (this->name, GF_LOG_ERROR,
"unknown address-family (%s) specified",
address_family);
+ *sa_family = AF_UNSPEC;
goto out;
}
}
@@ -353,7 +351,7 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
struct sockaddr *addr,
socklen_t *addr_len)
{
- struct addrinfo hints, *res = 0;
+ struct addrinfo hints, *res = 0, *rp = NULL;
data_t *listen_port_data = NULL, *listen_host_data = NULL;
uint16_t listen_port = -1;
char service[NI_MAXSERV], *listen_host = NULL;
@@ -378,20 +376,20 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
{
listen_host = data_to_str (listen_host_data);
} else {
- if (addr->sa_family == AF_INET6) {
- struct sockaddr_in6 *in = (struct sockaddr_in6 *) addr;
- in->sin6_addr = in6addr_any;
- in->sin6_port = htons(listen_port);
- *addr_len = sizeof(struct sockaddr_in6);
+ if (addr->sa_family == AF_INET6) {
+ struct sockaddr_in6 *in = (struct sockaddr_in6 *) addr;
+ in->sin6_addr = in6addr_any;
+ in->sin6_port = htons(listen_port);
+ *addr_len = sizeof(struct sockaddr_in6);
goto out;
- } else if (addr->sa_family == AF_INET) {
- struct sockaddr_in *in = (struct sockaddr_in *) addr;
- in->sin_addr.s_addr = htonl(INADDR_ANY);
- in->sin_port = htons(listen_port);
- *addr_len = sizeof(struct sockaddr_in);
- goto out;
- }
- }
+ } else if (addr->sa_family == AF_INET) {
+ struct sockaddr_in *in = (struct sockaddr_in *) addr;
+ in->sin_addr.s_addr = htonl(INADDR_ANY);
+ in->sin_port = htons(listen_port);
+ *addr_len = sizeof(struct sockaddr_in);
+ goto out;
+ }
+ }
memset (service, 0, sizeof (service));
sprintf (service, "%d", listen_port);
@@ -399,7 +397,7 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
memset (&hints, 0, sizeof (hints));
hints.ai_family = addr->sa_family;
hints.ai_socktype = SOCK_STREAM;
- hints.ai_flags = AI_ADDRCONFIG | AI_PASSIVE;
+ hints.ai_flags = AI_PASSIVE | AI_ADDRCONFIG;
ret = getaddrinfo(listen_host, service, &hints, &res);
if (ret != 0) {
@@ -409,9 +407,20 @@ af_inet_server_get_local_sockaddr (rpc_transport_t *this,
ret = -1;
goto out;
}
+ /* IPV6 server can handle both ipv4 and ipv6 clients */
+ for (rp = res; rp != NULL; rp = rp->ai_next) {
+ if (rp->ai_addr == NULL)
+ continue;
+ if (rp->ai_family == AF_INET6) {
+ memcpy (addr, rp->ai_addr, rp->ai_addrlen);
+ *addr_len = rp->ai_addrlen;
+ }
+ }
- memcpy (addr, res->ai_addr, res->ai_addrlen);
- *addr_len = res->ai_addrlen;
+ if (!(*addr_len)) {
+ memcpy (addr, res->ai_addr, res->ai_addrlen);
+ *addr_len = res->ai_addrlen;
+ }
freeaddrinfo (res);
@@ -435,12 +444,14 @@ client_bind (rpc_transport_t *this,
*sockaddr_len = sizeof (struct sockaddr_in);
case AF_INET6:
- ret = af_inet_bind_to_port_lt_ceiling (sock, sockaddr,
- *sockaddr_len, CLIENT_PORT_CEILING);
+ if (!this->bind_insecure) {
+ ret = af_inet_bind_to_port_lt_ceiling (sock, sockaddr,
+ *sockaddr_len, GF_CLIENT_PORT_CEILING);
+ }
if (ret == -1) {
- gf_log (this->name, GF_LOG_WARNING,
+ gf_log (this->name, GF_LOG_DEBUG,
"cannot bind inet socket (%d) to port less than %d (%s)",
- sock, CLIENT_PORT_CEILING, strerror (errno));
+ sock, GF_CLIENT_PORT_CEILING, strerror (errno));
ret = 0;
}
break;
@@ -469,12 +480,9 @@ socket_client_get_remote_sockaddr (rpc_transport_t *this,
{
int32_t ret = 0;
- if ((sockaddr == NULL) || (sockaddr_len == NULL)
- || (sa_family == NULL)) {
- ret = -1;
- goto err;
- }
-
+ GF_VALIDATE_OR_GOTO ("socket", sockaddr, err);
+ GF_VALIDATE_OR_GOTO ("socket", sockaddr_len, err);
+ GF_VALIDATE_OR_GOTO ("socket", sa_family, err);
ret = client_fill_address_family (this, &sockaddr->sa_family);
if (ret) {
@@ -522,9 +530,7 @@ server_fill_address_family (rpc_transport_t *this, sa_family_t *sa_family)
data_t *address_family_data = NULL;
int32_t ret = -1;
- if (sa_family == NULL) {
- goto out;
- }
+ GF_VALIDATE_OR_GOTO ("socket", sa_family, out);
address_family_data = dict_get (this->options,
"transport.address-family");
@@ -540,18 +546,16 @@ server_fill_address_family (rpc_transport_t *this, sa_family_t *sa_family)
*sa_family = AF_INET_SDP;
} else if (!strcasecmp (address_family, "unix")) {
*sa_family = AF_UNIX;
- } else if (!strcasecmp (address_family, "inet/inet6")
- || !strcasecmp (address_family, "inet6/inet")) {
- *sa_family = AF_UNSPEC;
} else {
gf_log (this->name, GF_LOG_ERROR,
"unknown address family (%s) specified", address_family);
+ *sa_family = AF_UNSPEC;
goto out;
}
} else {
gf_log (this->name, GF_LOG_DEBUG,
- "option address-family not specified, defaulting to inet/inet6");
- *sa_family = AF_UNSPEC;
+ "option address-family not specified, defaulting to inet");
+ *sa_family = AF_INET;
}
ret = 0;
@@ -566,9 +570,9 @@ socket_server_get_local_sockaddr (rpc_transport_t *this, struct sockaddr *addr,
{
int32_t ret = -1;
- if ((addr == NULL) || (addr_len == NULL) || (sa_family == NULL)) {
- goto err;
- }
+ GF_VALIDATE_OR_GOTO ("socket", sa_family, err);
+ GF_VALIDATE_OR_GOTO ("socket", addr, err);
+ GF_VALIDATE_OR_GOTO ("socket", addr_len, err);
ret = server_fill_address_family (this, &addr->sa_family);
if (ret == -1) {
@@ -605,7 +609,7 @@ int32_t
fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *addr,
int32_t addr_len, char *identifier)
{
- struct sockaddr_storage tmpaddr;
+ union gf_sock_union sock_union;
char service[NI_MAXSERV] = {0,};
char host[NI_MAXHOST] = {0,};
@@ -617,26 +621,26 @@ fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *add
int16_t eight_to_ten = 0;
int16_t ten_to_twelve = 0;
- memset (&tmpaddr, 0, sizeof (tmpaddr));
- tmpaddr = *addr;
+ memset (&sock_union, 0, sizeof (sock_union));
+ sock_union.storage = *addr;
tmpaddr_len = addr_len;
- if (((struct sockaddr *) &tmpaddr)->sa_family == AF_INET6) {
- one_to_four = ((struct sockaddr_in6 *) &tmpaddr)->sin6_addr.s6_addr32[0];
- four_to_eight = ((struct sockaddr_in6 *) &tmpaddr)->sin6_addr.s6_addr32[1];
+ if (sock_union.sa.sa_family == AF_INET6) {
+ one_to_four = sock_union.sin6.sin6_addr.s6_addr32[0];
+ four_to_eight = sock_union.sin6.sin6_addr.s6_addr32[1];
#ifdef GF_SOLARIS_HOST_OS
- eight_to_ten = S6_ADDR16(((struct sockaddr_in6 *) &tmpaddr)->sin6_addr)[4];
+ eight_to_ten = S6_ADDR16(sock_union.sin6.sin6_addr)[4];
#else
- eight_to_ten = ((struct sockaddr_in6 *) &tmpaddr)->sin6_addr.s6_addr16[4];
+ eight_to_ten = sock_union.sin6.sin6_addr.s6_addr16[4];
#endif
#ifdef GF_SOLARIS_HOST_OS
- ten_to_twelve = S6_ADDR16(((struct sockaddr_in6 *) &tmpaddr)->sin6_addr)[5];
+ ten_to_twelve = S6_ADDR16(sock_union.sin6.sin6_addr)[5];
#else
- ten_to_twelve = ((struct sockaddr_in6 *) &tmpaddr)->sin6_addr.s6_addr16[5];
+ ten_to_twelve = sock_union.sin6.sin6_addr.s6_addr16[5];
#endif
- twelve_to_sixteen = ((struct sockaddr_in6 *) &tmpaddr)->sin6_addr.s6_addr32[3];
+ twelve_to_sixteen = sock_union.sin6.sin6_addr.s6_addr32[3];
/* ipv4 mapped ipv6 address has
bits 0-80: 0
@@ -648,8 +652,8 @@ fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *add
four_to_eight == 0 &&
eight_to_ten == 0 &&
ten_to_twelve == -1) {
- struct sockaddr_in *in_ptr = (struct sockaddr_in *)&tmpaddr;
- memset (&tmpaddr, 0, sizeof (tmpaddr));
+ struct sockaddr_in *in_ptr = &sock_union.sin;
+ memset (&sock_union, 0, sizeof (sock_union));
in_ptr->sin_family = AF_INET;
in_ptr->sin_port = ((struct sockaddr_in6 *)addr)->sin6_port;
@@ -658,7 +662,7 @@ fill_inet6_inet_identifiers (rpc_transport_t *this, struct sockaddr_storage *add
}
}
- ret = getnameinfo ((struct sockaddr *) &tmpaddr,
+ ret = getnameinfo (&sock_union.sa,
tmpaddr_len,
host, sizeof (host),
service, sizeof (service),
diff --git a/rpc/rpc-transport/socket/src/name.h b/rpc/rpc-transport/socket/src/name.h
index 66eccf8d..0a13d8a9 100644
--- a/rpc/rpc-transport/socket/src/name.h
+++ b/rpc/rpc-transport/socket/src/name.h
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
#ifndef _SOCKET_NAME_H
diff --git a/rpc/rpc-transport/socket/src/socket.c b/rpc/rpc-transport/socket/src/socket.c
index ae4bd56a..06b74b20 100644
--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1,20 +1,11 @@
/*
- Copyright (c) 2010 Gluster, Inc. <http://www.gluster.com>
+ Copyright (c) 2008-2012 Red Hat, Inc. <http://www.redhat.com>
This file is part of GlusterFS.
- GlusterFS is free software; you can redistribute it and/or modify
- it under the terms of the GNU Affero General Public License as published
- by the Free Software Foundation; either version 3 of the License,
- or (at your option) any later version.
-
- GlusterFS is distributed in the hope that it will be useful, but
- WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Affero General Public License for more details.
-
- You should have received a copy of the GNU Affero General Public License
- along with this program. If not, see
- <http://www.gnu.org/licenses/>.
+ This file is licensed to you under your choice of the GNU Lesser
+ General Public License, version 3 or any later version (LGPLv3 or
+ later), or the GNU General Public License, version 2 (GPLv2), in all
+ cases as published by the Free Software Foundation.
*/
@@ -37,104 +28,408 @@
/* ugly #includes below */
#include "protocol-common.h"
#include "glusterfs3-xdr.h"
-#include "glusterfs3.h"
+#include "xdr-nfs3.h"
+#include "rpcsvc.h"
#include <fcntl.h>
#include <errno.h>
#include <netinet/tcp.h>
#include <rpc/xdr.h>
-
+#include <sys/ioctl.h>
#define GF_LOG_ERRNO(errno) ((errno == ENOTCONN) ? GF_LOG_DEBUG : GF_LOG_ERROR)
#define SA(ptr) ((struct sockaddr *)ptr)
+#define SSL_ENABLED_OPT "transport.socket.ssl-enabled"
+#define SSL_OWN_CERT_OPT "transport.socket.ssl-own-cert"
+#define SSL_PRIVATE_KEY_OPT "transport.socket.ssl-private-key"
+#define SSL_CA_LIST_OPT "transport.socket.ssl-ca-list"
+#define OWN_THREAD_OPT "transport.socket.own-thread"
+
+/* TBD: do automake substitutions etc. (ick) to set these. */
+#if !defined(DEFAULT_CERT_PATH)
+#define DEFAULT_CERT_PATH "/etc/ssl/glusterfs.pem"
+#endif
+#if !defined(DEFAULT_KEY_PATH)
+#define DEFAULT_KEY_PATH "/etc/ssl/glusterfs.key"
+#endif
+#if !defined(DEFAULT_CA_PATH)
+#define DEFAULT_CA_PATH "/etc/ssl/glusterfs.ca"
+#endif
-#define __socket_proto_reset_pending(priv) do { \
- memset (&priv->incoming.frag.vector, 0, \
- sizeof (priv->incoming.frag.vector)); \
- priv->incoming.frag.pending_vector = \
- &priv->incoming.frag.vector; \
- priv->incoming.frag.pending_vector->iov_base = \
- priv->incoming.frag.fragcurrent; \
- priv->incoming.pending_vector = \
- priv->incoming.frag.pending_vector; \
- } while (0);
-
-
-#define __socket_proto_update_pending(priv) \
- do { \
- uint32_t remaining_fragsize = 0; \
- if (priv->incoming.frag.pending_vector->iov_len == 0) { \
- remaining_fragsize = RPC_FRAGSIZE (priv->incoming.fraghdr) \
- - priv->incoming.frag.bytes_read; \
- \
- priv->incoming.frag.pending_vector->iov_len = \
- remaining_fragsize > priv->incoming.frag.remaining_size \
- ? priv->incoming.frag.remaining_size : remaining_fragsize; \
- \
- priv->incoming.frag.remaining_size -= \
- priv->incoming.frag.pending_vector->iov_len; \
- } \
- } while (0);
-
-#define __socket_proto_update_priv_after_read(priv, ret, bytes_read) \
- { \
- priv->incoming.frag.fragcurrent += bytes_read; \
- priv->incoming.frag.bytes_read += bytes_read; \
- \
- if ((ret > 0) || (priv->incoming.frag.remaining_size != 0)) { \
- if (priv->incoming.frag.remaining_size != 0) { \
- __socket_proto_reset_pending (priv); \
- } \
- \
- gf_log (this->name, GF_LOG_TRACE, "partial read on non-blocking socket"); \
- \
- break; \
- } \
- }
-
-#define __socket_proto_init_pending(priv, size) \
- do { \
- uint32_t remaining_fragsize = 0; \
- remaining_fragsize = RPC_FRAGSIZE (priv->incoming.fraghdr) \
- - priv->incoming.frag.bytes_read; \
- \
- __socket_proto_reset_pending (priv); \
- \
- priv->incoming.frag.pending_vector->iov_len = \
- remaining_fragsize > size ? size : remaining_fragsize; \
- \
- priv->incoming.frag.remaining_size = \
- size - priv->incoming.frag.pending_vector->iov_len; \
- \
-} while (0);
+#define POLL_MASK_INPUT (POLLIN | POLLPRI)
+#define POLL_MASK_OUTPUT (POLLOUT)
+#define POLL_MASK_ERROR (POLLERR | POLLHUP | POLLNVAL)
+
+typedef int SSL_unary_func (SSL *);
+typedef int SSL_trinary_func (SSL *, void *, int);
+
+#define __socket_proto_reset_pending(priv) do { \
+ struct gf_sock_incoming_frag *frag; \
+ frag = &priv->incoming.frag; \
+ \
+ memset (&frag->vector, 0, sizeof (frag->vector)); \
+ frag->pending_vector = &frag->vector; \
+ frag->pending_vector->iov_base = frag->fragcurrent; \
+ priv->incoming.pending_vector = frag->pending_vector; \
+ } while (0)
+
+
+#define __socket_proto_update_pending(priv) \
+ do { \
+ uint32_t remaining; \
+ struct gf_sock_incoming_frag *frag; \
+ frag = &priv->incoming.frag; \
+ if (frag->pending_vector->iov_len == 0) { \
+ remaining = (RPC_FRAGSIZE (priv->incoming.fraghdr) \
+ - frag->bytes_read); \
+ \
+ frag->pending_vector->iov_len = \
+ (remaining > frag->remaining_size) \
+ ? frag->remaining_size : remaining; \
+ \
+ frag->remaining_size -= \
+ frag->pending_vector->iov_len; \
+ } \
+ } while (0)
+
+#define __socket_proto_update_priv_after_read(priv, ret, bytes_read) \
+ { \
+ struct gf_sock_incoming_frag *frag; \
+ frag = &priv->incoming.frag; \
+ \
+ frag->fragcurrent += bytes_read; \
+ frag->bytes_read += bytes_read; \
+ \
+ if ((ret > 0) || (frag->remaining_size != 0)) { \
+ if (frag->remaining_size != 0 && ret == 0) { \
+ __socket_proto_reset_pending (priv); \
+ } \
+ \
+ gf_log (this->name, GF_LOG_TRACE, \
+ "partial read on non-blocking socket"); \
+ \
+ break; \
+ } \
+ }
+
+#define __socket_proto_init_pending(priv,size) \
+ do { \
+ uint32_t remaining = 0; \
+ struct gf_sock_incoming_frag *frag; \
+ frag = &priv->incoming.frag; \
+ \
+ remaining = (RPC_FRAGSIZE (priv->incoming.fraghdr) \
+ - frag->bytes_read); \
+ \
+ __socket_proto_reset_pending (priv); \
+ \
+ frag->pending_vector->iov_len = \
+ (remaining > size) ? size : remaining; \
+ \
+ frag->remaining_size = (size - frag->pending_vector->iov_len); \
+ \
+ } while(0)
/* This will be used in a switch case and breaks from the switch case if all
* the pending data is not read.
*/
-#define __socket_proto_read(priv, ret) \
- { \
- size_t bytes_read = 0; \
- \
- __socket_proto_update_pending (priv); \
- \
- ret = __socket_readv (this, \
- priv->incoming.pending_vector, 1, \
- &priv->incoming.pending_vector, \
- &priv->incoming.pending_count, \
- &bytes_read); \
- if (ret == -1) { \
- gf_log (this->name, GF_LOG_TRACE, \
- "reading from socket failed. Error (%s), " \
- "peer (%s)", strerror (errno), \
- this->peerinfo.identifier); \
- break; \
- } \
+#define __socket_proto_read(priv, ret) \
+ { \
+ size_t bytes_read = 0; \
+ struct gf_sock_incoming *in; \
+ in = &priv->incoming; \
+ \
+ __socket_proto_update_pending (priv); \
+ \
+ ret = __socket_readv (this, \
+ in->pending_vector, 1, \
+ &in->pending_vector, \
+ &in->pending_count, \
+ &bytes_read); \
+ if (ret == -1) \
+ break; \
__socket_proto_update_priv_after_read (priv, ret, bytes_read); \
- }
+ }
+
+static int socket_init (rpc_transport_t *this);
+
+static void
+ssl_dump_error_stack (const char *caller)
+{
+ unsigned long errnum = 0;
+ char errbuf[120] = {0,};
+
+ /* OpenSSL docs explicitly give 120 as the error-string length. */
+
+ while ((errnum = ERR_get_error())) {
+ ERR_error_string(errnum,errbuf);
+ gf_log(caller,GF_LOG_ERROR," %s",errbuf);
+ }
+}
+
+static int
+ssl_do (rpc_transport_t *this, void *buf, size_t len, SSL_trinary_func *func)
+{
+ int r = (-1);
+ struct pollfd pfd = {-1,};
+ socket_private_t *priv = NULL;
+
+ GF_VALIDATE_OR_GOTO(this->name,this->private,out);
+ priv = this->private;
+
+ for (;;) {
+ if (buf) {
+ if (priv->connected == -1) {
+ /*
+ * Fields in the SSL structure (especially
+ * the BIO pointers) are not valid at this
+ * point, so we'll segfault if we pass them
+ * to SSL_read/SSL_write.
+ */
+ gf_log(this->name,GF_LOG_INFO,
+ "lost connection in %s", __func__);
+ break;
+ }
+ r = func(priv->ssl_ssl,buf,len);
+ }
+ else {
+ /*
+ * We actually need these functions to get to
+ * priv->connected == 1.
+ */
+ r = ((SSL_unary_func *)func)(priv->ssl_ssl);
+ }
+ switch (SSL_get_error(priv->ssl_ssl,r)) {
+ case SSL_ERROR_NONE:
+ return r;
+ case SSL_ERROR_WANT_READ:
+ pfd.fd = priv->sock;
+ pfd.events = POLLIN;
+ if (poll(&pfd,1,-1) < 0) {
+ gf_log(this->name,GF_LOG_ERROR,"poll error %d",
+ errno);
+ }
+ break;
+ case SSL_ERROR_WANT_WRITE:
+ pfd.fd = priv->sock;
+ pfd.events = POLLOUT;
+ if (poll(&pfd,1,-1) < 0) {
+ gf_log(this->name,GF_LOG_ERROR,"poll error %d",
+ errno);
+ }
+ break;
+ case SSL_ERROR_SYSCALL:
+ /* This is what we get when remote disconnects. */
+ gf_log(this->name,GF_LOG_DEBUG,
+ "syscall error (probably remote disconnect)");
+ errno = ENODATA;
+ goto out;
+ default:
+ errno = EIO;
+ goto out; /* "break" would just loop again */
+ }
+ }
+out:
+ return -1;
+}
+
+#define ssl_connect_one(t) ssl_do((t),NULL,0,(SSL_trinary_func *)SSL_connect)
+#define ssl_accept_one(t) ssl_do((t),NULL,0,(SSL_trinary_func *)SSL_accept)
+#define ssl_read_one(t,b,l) ssl_do((t),(b),(l),(SSL_trinary_func *)SSL_read)
+#define ssl_write_one(t,b,l) ssl_do((t),(b),(l),(SSL_trinary_func *)SSL_write)
+
+static int
+ssl_setup_connection (rpc_transport_t *this, int server)
+{
+ X509 *peer = NULL;
+ char peer_CN[256] = "";
+ int ret = -1;
+ socket_private_t *priv = NULL;
+
+ GF_VALIDATE_OR_GOTO(this->name,this->private,done);
+ priv = this->private;
+
+ priv->ssl_ssl = SSL_new(priv->ssl_ctx);
+ if (!priv->ssl_ssl) {
+ gf_log(this->name,GF_LOG_ERROR,"SSL_new failed");
+ ssl_dump_error_stack(this->name);
+ goto done;
+ }
+ priv->ssl_sbio = BIO_new_socket(priv->sock,BIO_NOCLOSE);
+ if (!priv->ssl_sbio) {
+ gf_log(this->name,GF_LOG_ERROR,"BIO_new_socket failed");
+ ssl_dump_error_stack(this->name);
+ goto free_ssl;
+ }
+ SSL_set_bio(priv->ssl_ssl,priv->ssl_sbio,priv->ssl_sbio);
+
+ if (server) {
+ ret = ssl_accept_one(this);
+ }
+ else {
+ ret = ssl_connect_one(this);
+ }
+
+ /* Make sure _the call_ succeeded. */
+ if (ret < 0) {
+ goto ssl_error;
+ }
+ /* Make sure _SSL verification_ succeeded, yielding an identity. */
+ if (SSL_get_verify_result(priv->ssl_ssl) != X509_V_OK) {
+ goto ssl_error;
+ }
+ peer = SSL_get_peer_certificate(priv->ssl_ssl);
+ if (!peer) {
+ goto ssl_error;
+ }
+
+ /* Finally, everything seems OK. */
+ X509_NAME_get_text_by_NID(X509_get_subject_name(peer),
+ NID_commonName, peer_CN, sizeof(peer_CN)-1);
+ peer_CN[sizeof(peer_CN)-1] = '\0';
+ gf_log(this->name,GF_LOG_INFO,"peer CN = %s", peer_CN);
+ return 0;
+
+ /* Error paths. */
+ssl_error:
+ gf_log(this->name,GF_LOG_ERROR,"SSL connect error");
+ ssl_dump_error_stack(this->name);
+free_ssl:
+ SSL_free(priv->ssl_ssl);
+ priv->ssl_ssl = NULL;
+done:
+ return ret;
+}
+
+
+static void
+ssl_teardown_connection (socket_private_t *priv)
+{
+ SSL_shutdown(priv->ssl_ssl);
+ SSL_clear(priv->ssl_ssl);
+ SSL_free(priv->ssl_ssl);
+ priv->ssl_ssl = NULL;
+}
+
+
+static ssize_t
+__socket_ssl_readv (rpc_transport_t *this, struct iovec *opvector, int opcount)
+{
+ socket_private_t *priv = NULL;
+ int sock = -1;
+ int ret = -1;
+
+ priv = this->private;
+ sock = priv->sock;
+
+ if (priv->use_ssl) {
+ ret = ssl_read_one (this, opvector->iov_base, opvector->iov_len);
+ } else {
+ ret = readv (sock, opvector, opcount);
+ }
+
+ return ret;
+}
+
+
+static ssize_t
+__socket_ssl_read (rpc_transport_t *this, void *buf, size_t count)
+{
+ struct iovec iov = {0, };
+ int ret = -1;
-int socket_init (rpc_transport_t *this);
+ iov.iov_base = buf;
+ iov.iov_len = count;
+
+ ret = __socket_ssl_readv (this, &iov, 1);
+
+ return ret;
+}
+
+
+static int
+__socket_cached_read (rpc_transport_t *this, struct iovec *opvector, int opcount)
+{
+ socket_private_t *priv = NULL;
+ int sock = -1;
+ struct gf_sock_incoming *in = NULL;
+ int req_len = -1;
+ int ret = -1;
+
+ priv = this->private;
+ sock = priv->sock;
+ in = &priv->incoming;
+ req_len = iov_length (opvector, opcount);
+
+ if (in->record_state == SP_STATE_READING_FRAGHDR) {
+ in->ra_read = 0;
+ in->ra_served = 0;
+ in->ra_max = 0;
+ in->ra_buf = NULL;
+ goto uncached;
+ }
+
+ if (!in->ra_max) {
+ /* first call after passing SP_STATE_READING_FRAGHDR */
+ in->ra_max = min (RPC_FRAGSIZE (in->fraghdr), GF_SOCKET_RA_MAX);
+ /* Note that the in->iobuf is the primary iobuf into which
+ headers are read into. By using this itself as our
+ read-ahead cache, we can avoid memory copies in iov_load
+ */
+ in->ra_buf = iobuf_ptr (in->iobuf);
+ }
+
+ /* fill read-ahead */
+ if (in->ra_read < in->ra_max) {
+ ret = __socket_ssl_read (this, &in->ra_buf[in->ra_read],
+ (in->ra_max - in->ra_read));
+ if (ret > 0)
+ in->ra_read += ret;
+
+ /* we proceed to test if there is still cached data to
+ be served even if readahead could not progress */
+ }
+
+ /* serve cached */
+ if (in->ra_served < in->ra_read) {
+ ret = iov_load (opvector, opcount, &in->ra_buf[in->ra_served],
+ min (req_len, (in->ra_read - in->ra_served)));
+
+ in->ra_served += ret;
+ /* Do not read uncached and cached in the same call */
+ goto out;
+ }
+
+ if (in->ra_read < in->ra_max)
+ /* If there was no cached data to be served, (and we are
+ guaranteed to have already performed an attempt to progress
+ readahead above), and we have not yet read out the full
+ readahead capacity, then bail out for now without doing
+ the uncached read below (as that will overtake future cached
+ read)
+ */
+ goto out;
+uncached:
+ ret = __socket_ssl_readv (this, opvector, opcount);
+out:
+ return ret;
+}
+
+static gf_boolean_t
+__does_socket_rwv_error_need_logging (socket_private_t *priv, int write)
+{
+ int read = !write;
+
+ if (priv->connected == -1) /* Didn't even connect, of course it fails */
+ return _gf_false;
+
+ if (read && (priv->read_fail_log == _gf_false))
+ return _gf_false;
+
+ return _gf_true;
+}
/*
* return value:
@@ -143,7 +438,7 @@ int socket_init (rpc_transport_t *this);
* > 0 = incomplete
*/
-int
+static int
__socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
struct iovec **pending_vector, int *pending_count, size_t *bytes,
int write)
@@ -155,22 +450,35 @@ __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
int opcount = 0;
int moved = 0;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
sock = priv->sock;
- opvector = vector;
- opcount = count;
+ opvector = vector;
+ opcount = count;
if (bytes != NULL) {
*bytes = 0;
}
- while (opcount) {
+ while (opcount > 0) {
+ if (opvector->iov_len == 0) {
+ gf_log(this->name,GF_LOG_DEBUG,
+ "would have passed zero length to read/write");
+ ++opvector;
+ --opcount;
+ continue;
+ }
if (write) {
- ret = writev (sock, opvector, opcount);
+ if (priv->use_ssl) {
+ ret = ssl_write_one(this,
+ opvector->iov_base, opvector->iov_len);
+ }
+ else {
+ ret = writev (sock, opvector, opcount);
+ }
if (ret == 0 || (ret == -1 && errno == EAGAIN)) {
/* done for now */
@@ -178,7 +486,13 @@ __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
}
this->total_bytes_write += ret;
} else {
- ret = readv (sock, opvector, opcount);
+ ret = __socket_cached_read (this, opvector, opcount);
+
+ if (ret == 0) {
+ gf_log(this->name,GF_LOG_DEBUG,"EOF on socket");
+ errno = ENODATA;
+ ret = -1;
+ }
if (ret == -1 && errno == EAGAIN) {
/* done for now */
break;
@@ -189,7 +503,7 @@ __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
if (ret == 0) {
/* Mostly due to 'umount' in client */
- gf_log (this->name, GF_LOG_TRACE,
+ gf_log (this->name, GF_LOG_DEBUG,
"EOF from peer %s", this->peerinfo.identifier);
opcount = -1;
errno = ENOTCONN;
@@ -199,9 +513,18 @@ __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
if (errno == EINTR)
continue;
- gf_log (this->name, GF_LOG_TRACE,
- "%s failed (%s)", write ? "writev" : "readv",
- strerror (errno));
+ if (__does_socket_rwv_error_need_logging (priv,
+ write)) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "%s on %s failed (%s)",
+ write ? "writev":"readv",
+ this->peerinfo.identifier,
+ strerror (errno));
+ }
+
+ if (priv->use_ssl) {
+ ssl_dump_error_stack(this->name);
+ }
opcount = -1;
break;
}
@@ -213,6 +536,17 @@ __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
moved = 0;
while (moved < ret) {
+ if (!opcount) {
+ gf_log(this->name,GF_LOG_DEBUG,
+ "ran out of iov, moved %d/%d",
+ moved, ret);
+ goto ran_out;
+ }
+ if (!opvector[0].iov_len) {
+ opvector++;
+ opcount--;
+ continue;
+ }
if ((ret - moved) >= opvector[0].iov_len) {
moved += opvector[0].iov_len;
opvector++;
@@ -222,13 +556,11 @@ __socket_rwv (rpc_transport_t *this, struct iovec *vector, int count,
opvector[0].iov_base += (ret - moved);
moved += (ret - moved);
}
- while (opcount && !opvector[0].iov_len) {
- opvector++;
- opcount--;
- }
}
}
+ran_out:
+
if (pending_vector)
*pending_vector = opvector;
@@ -240,7 +572,7 @@ out:
}
-int
+static int
__socket_readv (rpc_transport_t *this, struct iovec *vector, int count,
struct iovec **pending_vector, int *pending_count,
size_t *bytes)
@@ -248,43 +580,75 @@ __socket_readv (rpc_transport_t *this, struct iovec *vector, int count,
int ret = -1;
ret = __socket_rwv (this, vector, count,
- pending_vector, pending_count, bytes, 0);
+ pending_vector, pending_count, bytes, 0);
return ret;
}
-int
+static int
__socket_writev (rpc_transport_t *this, struct iovec *vector, int count,
struct iovec **pending_vector, int *pending_count)
{
int ret = -1;
ret = __socket_rwv (this, vector, count,
- pending_vector, pending_count, NULL, 1);
+ pending_vector, pending_count, NULL, 1);
return ret;
}
-int
-__socket_disconnect (rpc_transport_t *this)
+static int
+__socket_shutdown (rpc_transport_t *this)
{
- socket_private_t *priv = NULL;
int ret = -1;
+ socket_private_t *priv = this->private;
- if (!this || !this->private)
- goto out;
+ priv->connected = -1;
+ ret = shutdown (priv->sock, SHUT_RDWR);
+ if (ret) {
+ /* its already disconnected.. no need to understand
+ why it failed to shutdown in normal cases */
+ gf_log (this->name, GF_LOG_DEBUG,
+ "shutdown() returned %d. %s",
+ ret, strerror (errno));
+ }
+ return ret;
+}
+
+static int
+__socket_disconnect (rpc_transport_t *this)
+{
+ int ret = -1;
+ socket_private_t *priv = NULL;
+
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
+ gf_log (this->name, GF_LOG_TRACE,
+ "disconnecting %p, state=%u gen=%u sock=%d", this,
+ priv->ot_state, priv->ot_gen, priv->sock);
+
if (priv->sock != -1) {
- ret = shutdown (priv->sock, SHUT_RDWR);
- priv->connected = -1;
- gf_log (this->name, GF_LOG_TRACE,
- "shutdown() returned %d. set connection state to -1",
- ret);
+ ret = __socket_shutdown(this);
+ if (priv->own_thread) {
+ /*
+ * Without this, reconnect (= disconnect + connect)
+ * won't work except by accident.
+ */
+ close(priv->sock);
+ priv->sock = -1;
+ gf_log (this->name, GF_LOG_TRACE,
+ "OT_PLEASE_DIE on %p", this);
+ priv->ot_state = OT_PLEASE_DIE;
+ }
+ else if (priv->use_ssl) {
+ ssl_teardown_connection(priv);
+ }
}
out:
@@ -292,20 +656,22 @@ out:
}
-int
+static int
__socket_server_bind (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int ret = -1;
- int opt = 1;
+ int opt = 1;
+ int reuse_check_sock = -1;
+ struct sockaddr_storage unix_addr = {0};
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
- priv = this->private;
+ priv = this->private;
ret = setsockopt (priv->sock, SOL_SOCKET, SO_REUSEADDR,
- &opt, sizeof (opt));
+ &opt, sizeof (opt));
if (ret == -1) {
gf_log (this->name, GF_LOG_ERROR,
@@ -313,8 +679,23 @@ __socket_server_bind (rpc_transport_t *this)
strerror (errno));
}
+ /* reuse-address doesn't work for unix type sockets */
+ if (AF_UNIX == SA (&this->myinfo.sockaddr)->sa_family) {
+ memcpy (&unix_addr, SA (&this->myinfo.sockaddr),
+ this->myinfo.sockaddr_len);
+ reuse_check_sock = socket (AF_UNIX, SOCK_STREAM, 0);
+ if (reuse_check_sock >= 0) {
+ ret = connect (reuse_check_sock, SA (&unix_addr),
+ this->myinfo.sockaddr_len);
+ if ((ret == -1) && (ECONNREFUSED == errno)) {
+ unlink (((struct sockaddr_un*)&unix_addr)->sun_path);
+ }
+ close (reuse_check_sock);
+ }
+ }
+
ret = bind (priv->sock, (struct sockaddr *)&this->myinfo.sockaddr,
- this->myinfo.sockaddr_len);
+ this->myinfo.sockaddr_len);
if (ret == -1) {
gf_log (this->name, GF_LOG_ERROR,
@@ -331,7 +712,7 @@ out:
}
-int
+static int
__socket_nonblock (int fd)
{
int flags = 0;
@@ -345,17 +726,16 @@ __socket_nonblock (int fd)
return ret;
}
-
-int
+static int
__socket_nodelay (int fd)
{
int on = 1;
int ret = -1;
ret = setsockopt (fd, IPPROTO_TCP, TCP_NODELAY,
- &on, sizeof (on));
+ &on, sizeof (on));
if (!ret)
- gf_log ("", GF_LOG_TRACE,
+ gf_log (THIS->name, GF_LOG_TRACE,
"NODELAY enabled for socket %d", fd);
return ret;
@@ -363,45 +743,66 @@ __socket_nodelay (int fd)
static int
-__socket_keepalive (int fd, int keepalive_intvl)
+__socket_keepalive (int fd, int family, int keepalive_intvl, int keepalive_idle)
{
int on = 1;
int ret = -1;
ret = setsockopt (fd, SOL_SOCKET, SO_KEEPALIVE, &on, sizeof (on));
- if (ret == -1)
+ if (ret == -1) {
+ gf_log ("socket", GF_LOG_WARNING,
+ "failed to set keep alive option on socket %d", fd);
goto err;
+ }
if (keepalive_intvl == GF_USE_DEFAULT_KEEPALIVE)
goto done;
-#ifndef GF_LINUX_HOST_OS
+#if !defined(GF_LINUX_HOST_OS) && !defined(__NetBSD__)
+#ifdef GF_SOLARIS_HOST_OS
+ ret = setsockopt (fd, SOL_SOCKET, SO_KEEPALIVE, &keepalive_intvl,
+ sizeof (keepalive_intvl));
+#else
ret = setsockopt (fd, IPPROTO_TCP, TCP_KEEPALIVE, &keepalive_intvl,
sizeof (keepalive_intvl));
- if (ret == -1)
+#endif
+ if (ret == -1) {
+ gf_log ("socket", GF_LOG_WARNING,
+ "failed to set keep alive interval on socket %d", fd);
goto err;
+ }
#else
- ret = setsockopt (fd, IPPROTO_TCP, TCP_KEEPIDLE, &keepalive_intvl,
+ if (family != AF_INET)
+ goto done;
+
+ ret = setsockopt (fd, IPPROTO_TCP, TCP_KEEPIDLE, &keepalive_idle,
sizeof (keepalive_intvl));
- if (ret == -1)
+ if (ret == -1) {
+ gf_log ("socket", GF_LOG_WARNING,
+ "failed to set keep idle %d on socket %d, %s",
+ keepalive_idle, fd, strerror(errno));
goto err;
-
- ret = setsockopt (fd, IPPROTO_TCP, TCP_KEEPINTVL, &keepalive_intvl,
+ }
+ ret = setsockopt (fd, IPPROTO_TCP , TCP_KEEPINTVL, &keepalive_intvl,
sizeof (keepalive_intvl));
- if (ret == -1)
+ if (ret == -1) {
+ gf_log ("socket", GF_LOG_WARNING,
+ "failed to set keep interval %d on socket %d, %s",
+ keepalive_intvl, fd, strerror(errno));
goto err;
+ }
#endif
done:
- gf_log ("", GF_LOG_TRACE, "Keep-alive enabled for socket %d, interval "
- "%d", fd, keepalive_intvl);
+ gf_log (THIS->name, GF_LOG_TRACE, "Keep-alive enabled for socket %d, interval "
+ "%d, idle: %d", fd, keepalive_intvl, keepalive_idle);
err:
return ret;
}
-int
+static int
__socket_connect_finish (int fd)
{
int ret = -1;
@@ -419,13 +820,13 @@ __socket_connect_finish (int fd)
}
-void
+static void
__socket_reset (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
@@ -440,9 +841,7 @@ __socket_reset (rpc_transport_t *this)
iobuf_unref (priv->incoming.iobuf);
}
- if (priv->incoming.request_info != NULL) {
- GF_FREE (priv->incoming.request_info);
- }
+ GF_FREE (priv->incoming.request_info);
memset (&priv->incoming, 0, sizeof (priv->incoming));
@@ -458,13 +857,13 @@ out:
}
-void
+static void
socket_set_lastfrag (uint32_t *fragsize) {
(*fragsize) |= 0x80000000U;
}
-void
+static void
socket_set_frag_header_size (uint32_t size, char *haddr)
{
size = htonl (size);
@@ -472,22 +871,21 @@ socket_set_frag_header_size (uint32_t size, char *haddr)
}
-void
+static void
socket_set_last_frag_header_size (uint32_t size, char *haddr)
{
socket_set_lastfrag (&size);
socket_set_frag_header_size (size, haddr);
}
-struct ioq *
+static struct ioq *
__socket_ioq_new (rpc_transport_t *this, rpc_transport_msg_t *msg)
{
struct ioq *entry = NULL;
int count = 0;
uint32_t size = 0;
- if (!this)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
/* TODO: use mem-pool */
entry = GF_CALLOC (1, sizeof (*entry), gf_common_mt_ioq);
@@ -496,7 +894,7 @@ __socket_ioq_new (rpc_transport_t *this, rpc_transport_msg_t *msg)
count = msg->rpchdrcount + msg->proghdrcount + msg->progpayloadcount;
- assert (count <= (MAX_IOVEC - 1));
+ GF_ASSERT (count <= (MAX_IOVEC - 1));
size = iov_length (msg->rpchdr, msg->rpchdrcount)
+ iov_length (msg->proghdr, msg->proghdrcount)
@@ -547,11 +945,10 @@ out:
}
-void
+static void
__socket_ioq_entry_free (struct ioq *entry)
{
- if (!entry)
- return;
+ GF_VALIDATE_OR_GOTO ("socket", entry, out);
list_del_init (&entry->list);
if (entry->iobref)
@@ -559,17 +956,20 @@ __socket_ioq_entry_free (struct ioq *entry)
/* TODO: use mem-pool */
GF_FREE (entry);
+
+out:
+ return;
}
-void
+static void
__socket_ioq_flush (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
struct ioq *entry = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
@@ -583,35 +983,49 @@ out:
}
-int
-__socket_ioq_churn_entry (rpc_transport_t *this, struct ioq *entry)
+static int
+__socket_ioq_churn_entry (rpc_transport_t *this, struct ioq *entry, int direct)
{
- int ret = -1;
+ int ret = -1;
+ socket_private_t *priv = NULL;
+ char a_byte = 0;
ret = __socket_writev (this, entry->pending_vector,
- entry->pending_count,
+ entry->pending_count,
&entry->pending_vector,
- &entry->pending_count);
+ &entry->pending_count);
if (ret == 0) {
/* current entry was completely written */
- assert (entry->pending_count == 0);
+ GF_ASSERT (entry->pending_count == 0);
__socket_ioq_entry_free (entry);
+ priv = this->private;
+ if (priv->own_thread) {
+ /*
+ * The pipe should only remain readable if there are
+ * more entries after this, so drain the byte
+ * representing this entry.
+ */
+ if (!direct && read(priv->pipe[0],&a_byte,1) < 1) {
+ gf_log(this->name,GF_LOG_WARNING,
+ "read error on pipe");
+ }
+ }
}
return ret;
}
-int
+static int
__socket_ioq_churn (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int ret = 0;
struct ioq *entry = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
@@ -619,16 +1033,16 @@ __socket_ioq_churn (rpc_transport_t *this)
/* pick next entry */
entry = priv->ioq_next;
- ret = __socket_ioq_churn_entry (this, entry);
+ ret = __socket_ioq_churn_entry (this, entry, 0);
if (ret != 0)
break;
}
- if (list_empty (&priv->ioq)) {
+ if (!priv->own_thread && list_empty (&priv->ioq)) {
/* all pending writes done, not interested in POLLOUT */
priv->idx = event_select_on (this->ctx->event_pool,
- priv->sock, priv->idx, -1, 0);
+ priv->sock, priv->idx, -1, 0);
}
out:
@@ -636,14 +1050,14 @@ out:
}
-int
+static int
socket_event_poll_err (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int ret = -1;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
@@ -661,14 +1075,14 @@ out:
}
-int
+static int
socket_event_poll_out (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int ret = -1;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
@@ -691,56 +1105,58 @@ out:
}
-inline int
+static inline int
__socket_read_simple_msg (rpc_transport_t *this)
{
- socket_private_t *priv = NULL;
- int ret = 0;
- uint32_t remaining_size = 0;
- size_t bytes_read = 0;
+ int ret = 0;
+ uint32_t remaining_size = 0;
+ size_t bytes_read = 0;
+ socket_private_t *priv = NULL;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
- switch (priv->incoming.frag.simple_state) {
+ in = &priv->incoming;
+ frag = &in->frag;
+
+ switch (frag->simple_state) {
case SP_STATE_SIMPLE_MSG_INIT:
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
__socket_proto_init_pending (priv, remaining_size);
- priv->incoming.frag.simple_state =
- SP_STATE_READING_SIMPLE_MSG;
+ frag->simple_state = SP_STATE_READING_SIMPLE_MSG;
/* fall through */
case SP_STATE_READING_SIMPLE_MSG:
ret = 0;
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
if (remaining_size > 0) {
ret = __socket_readv (this,
- priv->incoming.pending_vector, 1,
- &priv->incoming.pending_vector,
- &priv->incoming.pending_count,
+ in->pending_vector, 1,
+ &in->pending_vector,
+ &in->pending_count,
&bytes_read);
}
if (ret == -1) {
- gf_log (this->name, GF_LOG_TRACE,
+ gf_log (this->name, GF_LOG_WARNING,
"reading from socket failed. Error (%s), "
"peer (%s)", strerror (errno),
this->peerinfo.identifier);
break;
}
- priv->incoming.frag.bytes_read += bytes_read;
- priv->incoming.frag.fragcurrent += bytes_read;
+ frag->bytes_read += bytes_read;
+ frag->fragcurrent += bytes_read;
if (ret > 0) {
gf_log (this->name, GF_LOG_TRACE,
@@ -749,8 +1165,7 @@ __socket_read_simple_msg (rpc_transport_t *this)
}
if (ret == 0) {
- priv->incoming.frag.simple_state
- = SP_STATE_SIMPLE_MSG_INIT;
+ frag->simple_state = SP_STATE_SIMPLE_MSG_INIT;
}
}
@@ -759,7 +1174,7 @@ out:
}
-inline int
+static inline int
__socket_read_simple_request (rpc_transport_t *this)
{
return __socket_read_simple_msg (this);
@@ -770,9 +1185,14 @@ __socket_read_simple_request (rpc_transport_t *this)
#define rpc_verf_addr(fragcurrent) (fragcurrent - 4)
+#define rpc_msgtype_addr(buf) (buf + 4)
+
+#define rpc_prognum_addr(buf) (buf + RPC_MSGTYPE_SIZE + 4)
+#define rpc_progver_addr(buf) (buf + RPC_MSGTYPE_SIZE + 8)
+#define rpc_procnum_addr(buf) (buf + RPC_MSGTYPE_SIZE + 12)
-inline int
-__socket_read_vectored_request (rpc_transport_t *this)
+static inline int
+__socket_read_vectored_request (rpc_transport_t *this, rpcsvc_vector_sizer vector_sizer)
{
socket_private_t *priv = NULL;
int ret = 0;
@@ -780,17 +1200,27 @@ __socket_read_vectored_request (rpc_transport_t *this)
char *addr = NULL;
struct iobuf *iobuf = NULL;
uint32_t remaining_size = 0;
- uint32_t gluster_write_proc_len = 0;
- gfs3_write_req write_req = {{0,},};
+ ssize_t readsize = 0;
+ size_t size = 0;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
+ sp_rpcfrag_request_state_t *request = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
- switch (priv->incoming.frag.call_body.request.vector_state) {
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+ frag = &in->frag;
+ request = &frag->call_body.request;
+
+ switch (request->vector_state) {
case SP_STATE_VECTORED_REQUEST_INIT:
- addr = rpc_cred_addr (iobuf_ptr (priv->incoming.iobuf));
+ request->vector_sizer_state = 0;
+
+ addr = rpc_cred_addr (iobuf_ptr (in->iobuf));
/* also read verf flavour and verflen */
credlen = ntoh32 (*((uint32_t *)addr))
@@ -798,88 +1228,114 @@ __socket_read_vectored_request (rpc_transport_t *this)
__socket_proto_init_pending (priv, credlen);
- priv->incoming.frag.call_body.request.vector_state =
- SP_STATE_READING_CREDBYTES;
+ request->vector_state = SP_STATE_READING_CREDBYTES;
/* fall through */
case SP_STATE_READING_CREDBYTES:
__socket_proto_read (priv, ret);
- priv->incoming.frag.call_body.request.vector_state =
- SP_STATE_READ_CREDBYTES;
+ request->vector_state = SP_STATE_READ_CREDBYTES;
/* fall through */
case SP_STATE_READ_CREDBYTES:
- addr = rpc_verf_addr (priv->incoming.frag.fragcurrent);
-
- /* FIXME: Also handle procedures other than glusterfs-write
- * here
- */
- /* also read proc-header */
- gluster_write_proc_len = xdr_sizeof ((xdrproc_t) xdr_gfs3_write_req,
- &write_req);
+ addr = rpc_verf_addr (frag->fragcurrent);
+ verflen = ntoh32 (*((uint32_t *)addr));
- if (gluster_write_proc_len == 0) {
- gf_log (this->name, GF_LOG_DEBUG,
- "xdr_sizeof on gfs3_write_req failed");
- ret = -1;
- goto out;
+ if (verflen == 0) {
+ request->vector_state = SP_STATE_READ_VERFBYTES;
+ goto sp_state_read_verfbytes;
}
-
- verflen = ntoh32 (*((uint32_t *)addr))
- + gluster_write_proc_len;
-
__socket_proto_init_pending (priv, verflen);
- priv->incoming.frag.call_body.request.vector_state
- = SP_STATE_READING_VERFBYTES;
+ request->vector_state = SP_STATE_READING_VERFBYTES;
/* fall through */
case SP_STATE_READING_VERFBYTES:
__socket_proto_read (priv, ret);
- priv->incoming.frag.call_body.request.vector_state =
- SP_STATE_READ_VERFBYTES;
+ request->vector_state = SP_STATE_READ_VERFBYTES;
/* fall through */
case SP_STATE_READ_VERFBYTES:
- if (priv->incoming.payload_vector.iov_base == NULL) {
- iobuf = iobuf_get (this->ctx->iobuf_pool);
+sp_state_read_verfbytes:
+ /* set the base_addr 'persistently' across multiple calls
+ into the state machine */
+ in->proghdr_base_addr = frag->fragcurrent;
+
+ request->vector_sizer_state =
+ vector_sizer (request->vector_sizer_state,
+ &readsize, in->proghdr_base_addr,
+ frag->fragcurrent);
+ __socket_proto_init_pending (priv, readsize);
+
+ request->vector_state = SP_STATE_READING_PROGHDR;
+
+ /* fall through */
+
+ case SP_STATE_READING_PROGHDR:
+ __socket_proto_read (priv, ret);
+
+ request->vector_state = SP_STATE_READ_PROGHDR;
+
+ /* fall through */
+
+ case SP_STATE_READ_PROGHDR:
+sp_state_read_proghdr:
+ request->vector_sizer_state =
+ vector_sizer (request->vector_sizer_state,
+ &readsize, in->proghdr_base_addr,
+ frag->fragcurrent);
+ if (readsize == 0) {
+ request->vector_state = SP_STATE_READ_PROGHDR_XDATA;
+ goto sp_state_read_proghdr_xdata;
+ }
+
+ __socket_proto_init_pending (priv, readsize);
+
+ request->vector_state = SP_STATE_READING_PROGHDR_XDATA;
+
+ /* fall through */
+
+ case SP_STATE_READING_PROGHDR_XDATA:
+ __socket_proto_read (priv, ret);
+
+ request->vector_state = SP_STATE_READ_PROGHDR;
+ /* check if the vector_sizer() has more to say */
+ goto sp_state_read_proghdr;
+
+ case SP_STATE_READ_PROGHDR_XDATA:
+sp_state_read_proghdr_xdata:
+ if (in->payload_vector.iov_base == NULL) {
+
+ size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
+ iobuf = iobuf_get2 (this->ctx->iobuf_pool, size);
if (!iobuf) {
- gf_log (this->name, GF_LOG_ERROR,
- "unable to allocate IO buffer "
- "for peer %s",
- this->peerinfo.identifier);
ret = -1;
break;
}
- if (priv->incoming.iobref == NULL) {
- priv->incoming.iobref = iobref_new ();
- if (priv->incoming.iobref == NULL) {
- gf_log (this->name, GF_LOG_ERROR,
- "out of memory");
+ if (in->iobref == NULL) {
+ in->iobref = iobref_new ();
+ if (in->iobref == NULL) {
ret = -1;
iobuf_unref (iobuf);
break;
}
}
- iobref_add (priv->incoming.iobref, iobuf);
+ iobref_add (in->iobref, iobuf);
iobuf_unref (iobuf);
- priv->incoming.payload_vector.iov_base
- = iobuf_ptr (iobuf);
+ in->payload_vector.iov_base = iobuf_ptr (iobuf);
- priv->incoming.frag.fragcurrent = iobuf_ptr (iobuf);
+ frag->fragcurrent = iobuf_ptr (iobuf);
}
- priv->incoming.frag.call_body.request.vector_state =
- SP_STATE_READING_PROG;
+ request->vector_state = SP_STATE_READING_PROG;
/* fall through */
@@ -890,19 +1346,15 @@ __socket_read_vectored_request (rpc_transport_t *this)
ret = __socket_read_simple_msg (this);
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
- if ((ret == -1)
- || ((ret == 0)
- && (remaining_size == 0)
- && RPC_LASTFRAG (priv->incoming.fraghdr))) {
- priv->incoming.frag.call_body.request.vector_state
- = SP_STATE_VECTORED_REQUEST_INIT;
- priv->incoming.payload_vector.iov_len
- = (unsigned long)priv->incoming.frag.fragcurrent
- - (unsigned long)
- priv->incoming.payload_vector.iov_base;
+ if ((ret == -1) ||
+ ((ret == 0) && (remaining_size == 0)
+ && RPC_LASTFRAG (in->fraghdr))) {
+ request->vector_state = SP_STATE_VECTORED_REQUEST_INIT;
+ in->payload_vector.iov_len
+ = ((unsigned long)frag->fragcurrent
+ - (unsigned long)in->payload_vector.iov_base);
}
break;
}
@@ -911,70 +1363,77 @@ out:
return ret;
}
-
-#define rpc_msgtype_addr(buf) (buf + 4)
-
-#define rpc_prognum_addr(buf) (buf + RPC_MSGTYPE_SIZE + 4)
-
-#define rpc_procnum_addr(buf) (buf + RPC_MSGTYPE_SIZE + 12)
-
-
-inline int
+static inline int
__socket_read_request (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
- uint32_t prognum = 0, procnum = 0;
+ uint32_t prognum = 0, procnum = 0, progver = 0;
uint32_t remaining_size = 0;
int ret = -1;
char *buf = NULL;
+ rpcsvc_vector_sizer vector_sizer = NULL;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
+ sp_rpcfrag_request_state_t *request = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
- switch (priv->incoming.frag.call_body.request.header_state) {
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+ frag = &in->frag;
+ request = &frag->call_body.request;
+
+ switch (request->header_state) {
case SP_STATE_REQUEST_HEADER_INIT:
__socket_proto_init_pending (priv, RPC_CALL_BODY_SIZE);
- priv->incoming.frag.call_body.request.header_state
- = SP_STATE_READING_RPCHDR1;
+ request->header_state = SP_STATE_READING_RPCHDR1;
/* fall through */
case SP_STATE_READING_RPCHDR1:
__socket_proto_read (priv, ret);
- priv->incoming.frag.call_body.request.header_state =
- SP_STATE_READ_RPCHDR1;
+ request->header_state = SP_STATE_READ_RPCHDR1;
/* fall through */
case SP_STATE_READ_RPCHDR1:
- buf = rpc_prognum_addr (iobuf_ptr (priv->incoming.iobuf));
+ buf = rpc_prognum_addr (iobuf_ptr (in->iobuf));
prognum = ntoh32 (*((uint32_t *)buf));
- buf = rpc_procnum_addr (iobuf_ptr (priv->incoming.iobuf));
+ buf = rpc_progver_addr (iobuf_ptr (in->iobuf));
+ progver = ntoh32 (*((uint32_t *)buf));
+
+ buf = rpc_procnum_addr (iobuf_ptr (in->iobuf));
procnum = ntoh32 (*((uint32_t *)buf));
- if ((prognum == GLUSTER3_1_FOP_PROGRAM)
- && (procnum == GF_FOP_WRITE)) {
- ret = __socket_read_vectored_request (this);
+ if (priv->is_server) {
+ /* this check is needed as rpcsvc and rpc-clnt
+ * actor structures are not same */
+ vector_sizer =
+ rpcsvc_get_program_vector_sizer ((rpcsvc_t *)this->mydata,
+ prognum, progver, procnum);
+ }
+
+ if (vector_sizer) {
+ ret = __socket_read_vectored_request (this, vector_sizer);
} else {
ret = __socket_read_simple_request (this);
}
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
if ((ret == -1)
|| ((ret == 0)
&& (remaining_size == 0)
- && (RPC_LASTFRAG (priv->incoming.fraghdr)))) {
- priv->incoming.frag.call_body.request.header_state =
- SP_STATE_REQUEST_HEADER_INIT;
+ && (RPC_LASTFRAG (in->fraghdr)))) {
+ request->header_state = SP_STATE_REQUEST_HEADER_INIT;
}
break;
@@ -985,35 +1444,40 @@ out:
}
-inline int
+static inline int
__socket_read_accepted_successful_reply (rpc_transport_t *this)
{
- socket_private_t *priv = NULL;
- int ret = 0;
- struct iobuf *iobuf = NULL;
- uint32_t gluster_read_rsp_hdr_len = 0;
- gfs3_read_rsp read_rsp = {0, };
-
- if (!this || !this->private)
- goto out;
+ socket_private_t *priv = NULL;
+ int ret = 0;
+ struct iobuf *iobuf = NULL;
+ gfs3_read_rsp read_rsp = {0, };
+ ssize_t size = 0;
+ ssize_t default_read_size = 0;
+ char *proghdr_buf = NULL;
+ XDR xdr;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
+
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
- switch (priv->incoming.frag.call_body.reply.accepted_success_state) {
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+ frag = &in->frag;
+
+ switch (frag->call_body.reply.accepted_success_state) {
case SP_STATE_ACCEPTED_SUCCESS_REPLY_INIT:
- gluster_read_rsp_hdr_len = xdr_sizeof ((xdrproc_t) xdr_gfs3_read_rsp,
- &read_rsp);
+ default_read_size = xdr_sizeof ((xdrproc_t) xdr_gfs3_read_rsp,
+ &read_rsp);
- if (gluster_read_rsp_hdr_len == 0) {
- gf_log (this->name, GF_LOG_DEBUG,
- "xdr_sizeof on gfs3_read_rsp failed");
- ret = -1;
- goto out;
- }
- __socket_proto_init_pending (priv, gluster_read_rsp_hdr_len);
+ proghdr_buf = frag->fragcurrent;
+
+ __socket_proto_init_pending (priv, default_read_size);
- priv->incoming.frag.call_body.reply.accepted_success_state
+ frag->call_body.reply.accepted_success_state
= SP_STATE_READING_PROC_HEADER;
/* fall through */
@@ -1021,44 +1485,79 @@ __socket_read_accepted_successful_reply (rpc_transport_t *this)
case SP_STATE_READING_PROC_HEADER:
__socket_proto_read (priv, ret);
- priv->incoming.frag.call_body.reply.accepted_success_state
- = SP_STATE_READ_PROC_HEADER;
+ /* there can be 'xdata' in read response, figure it out */
+ xdrmem_create (&xdr, proghdr_buf, default_read_size,
+ XDR_DECODE);
- /* fall through */
+ /* This will fail if there is xdata sent from server, if not,
+ well and good, we don't need to worry about */
+ xdr_gfs3_read_rsp (&xdr, &read_rsp);
- case SP_STATE_READ_PROC_HEADER:
- if (priv->incoming.payload_vector.iov_base == NULL) {
- iobuf = iobuf_get (this->ctx->iobuf_pool);
+ free (read_rsp.xdata.xdata_val);
+
+ /* need to round off to proper roof (%4), as XDR packing pads
+ the end of opaque object with '0' */
+ size = roof (read_rsp.xdata.xdata_len, 4);
+
+ if (!size) {
+ frag->call_body.reply.accepted_success_state
+ = SP_STATE_READ_PROC_OPAQUE;
+ goto read_proc_opaque;
+ }
+
+ __socket_proto_init_pending (priv, size);
+
+ frag->call_body.reply.accepted_success_state
+ = SP_STATE_READING_PROC_OPAQUE;
+
+ case SP_STATE_READING_PROC_OPAQUE:
+ __socket_proto_read (priv, ret);
+
+ frag->call_body.reply.accepted_success_state
+ = SP_STATE_READ_PROC_OPAQUE;
+
+ case SP_STATE_READ_PROC_OPAQUE:
+ read_proc_opaque:
+ if (in->payload_vector.iov_base == NULL) {
+
+ size = (RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read);
+
+ iobuf = iobuf_get2 (this->ctx->iobuf_pool, size);
if (iobuf == NULL) {
ret = -1;
goto out;
}
- if (priv->incoming.iobref == NULL) {
- priv->incoming.iobref = iobref_new ();
- if (priv->incoming.iobref == NULL) {
+ if (in->iobref == NULL) {
+ in->iobref = iobref_new ();
+ if (in->iobref == NULL) {
ret = -1;
iobuf_unref (iobuf);
goto out;
}
}
- iobref_add (priv->incoming.iobref, iobuf);
+ iobref_add (in->iobref, iobuf);
iobuf_unref (iobuf);
- priv->incoming.payload_vector.iov_base
- = iobuf_ptr (iobuf);
+ in->payload_vector.iov_base = iobuf_ptr (iobuf);
+
+ in->payload_vector.iov_len = size;
}
- priv->incoming.frag.fragcurrent
- = priv->incoming.payload_vector.iov_base;
+ frag->fragcurrent = in->payload_vector.iov_base;
+ frag->call_body.reply.accepted_success_state
+ = SP_STATE_READ_PROC_HEADER;
+
+ /* fall through */
+
+ case SP_STATE_READ_PROC_HEADER:
/* now read the entire remaining msg into new iobuf */
ret = __socket_read_simple_msg (this);
if ((ret == -1)
- || ((ret == 0)
- && RPC_LASTFRAG (priv->incoming.fraghdr))) {
- priv->incoming.frag.call_body.reply.accepted_success_state
+ || ((ret == 0) && RPC_LASTFRAG (in->fraghdr))) {
+ frag->call_body.reply.accepted_success_state
= SP_STATE_ACCEPTED_SUCCESS_REPLY_INIT;
}
@@ -1072,7 +1571,7 @@ out:
#define rpc_reply_verflen_addr(fragcurrent) ((char *)fragcurrent - 4)
#define rpc_reply_accept_status_addr(fragcurrent) ((char *)fragcurrent - 4)
-inline int
+static inline int
__socket_read_accepted_reply (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
@@ -1080,19 +1579,24 @@ __socket_read_accepted_reply (rpc_transport_t *this)
char *buf = NULL;
uint32_t verflen = 0, len = 0;
uint32_t remaining_size = 0;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+ frag = &in->frag;
- switch (priv->incoming.frag.call_body.reply.accepted_state) {
+ switch (frag->call_body.reply.accepted_state) {
case SP_STATE_ACCEPTED_REPLY_INIT:
__socket_proto_init_pending (priv,
RPC_AUTH_FLAVOUR_N_LENGTH_SIZE);
- priv->incoming.frag.call_body.reply.accepted_state
+ frag->call_body.reply.accepted_state
= SP_STATE_READING_REPLY_VERFLEN;
/* fall through */
@@ -1100,13 +1604,13 @@ __socket_read_accepted_reply (rpc_transport_t *this)
case SP_STATE_READING_REPLY_VERFLEN:
__socket_proto_read (priv, ret);
- priv->incoming.frag.call_body.reply.accepted_state
+ frag->call_body.reply.accepted_state
= SP_STATE_READ_REPLY_VERFLEN;
/* fall through */
case SP_STATE_READ_REPLY_VERFLEN:
- buf = rpc_reply_verflen_addr (priv->incoming.frag.fragcurrent);
+ buf = rpc_reply_verflen_addr (frag->fragcurrent);
verflen = ntoh32 (*((uint32_t *) buf));
@@ -1115,7 +1619,7 @@ __socket_read_accepted_reply (rpc_transport_t *this)
__socket_proto_init_pending (priv, len);
- priv->incoming.frag.call_body.reply.accepted_state
+ frag->call_body.reply.accepted_state
= SP_STATE_READING_REPLY_VERFBYTES;
/* fall through */
@@ -1123,19 +1627,19 @@ __socket_read_accepted_reply (rpc_transport_t *this)
case SP_STATE_READING_REPLY_VERFBYTES:
__socket_proto_read (priv, ret);
- priv->incoming.frag.call_body.reply.accepted_state
+ frag->call_body.reply.accepted_state
= SP_STATE_READ_REPLY_VERFBYTES;
- buf = rpc_reply_accept_status_addr (priv->incoming.frag.fragcurrent);
+ buf = rpc_reply_accept_status_addr (frag->fragcurrent);
- priv->incoming.frag.call_body.reply.accept_status
+ frag->call_body.reply.accept_status
= ntoh32 (*(uint32_t *) buf);
/* fall through */
case SP_STATE_READ_REPLY_VERFBYTES:
- if (priv->incoming.frag.call_body.reply.accept_status
+ if (frag->call_body.reply.accept_status
== SUCCESS) {
ret = __socket_read_accepted_successful_reply (this);
} else {
@@ -1145,14 +1649,13 @@ __socket_read_accepted_reply (rpc_transport_t *this)
ret = __socket_read_simple_msg (this);
}
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr)
+ - frag->bytes_read;
if ((ret == -1)
- || ((ret == 0)
- && (remaining_size == 0)
- && (RPC_LASTFRAG (priv->incoming.fraghdr)))) {
- priv->incoming.frag.call_body.reply.accepted_state
+ || ((ret == 0) && (remaining_size == 0)
+ && (RPC_LASTFRAG (in->fraghdr)))) {
+ frag->call_body.reply.accepted_state
= SP_STATE_ACCEPTED_REPLY_INIT;
}
@@ -1164,7 +1667,7 @@ out:
}
-inline int
+static inline int
__socket_read_denied_reply (rpc_transport_t *this)
{
return __socket_read_simple_msg (this);
@@ -1174,25 +1677,29 @@ __socket_read_denied_reply (rpc_transport_t *this)
#define rpc_reply_status_addr(fragcurrent) ((char *)fragcurrent - 4)
-inline int
+static inline int
__socket_read_vectored_reply (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int ret = 0;
char *buf = NULL;
uint32_t remaining_size = 0;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
+ in = &priv->incoming;
+ frag = &in->frag;
- switch (priv->incoming.frag.call_body.reply.status_state) {
+ switch (frag->call_body.reply.status_state) {
case SP_STATE_ACCEPTED_REPLY_INIT:
__socket_proto_init_pending (priv, RPC_REPLY_STATUS_SIZE);
- priv->incoming.frag.call_body.reply.status_state
+ frag->call_body.reply.status_state
= SP_STATE_READING_REPLY_STATUS;
/* fall through */
@@ -1200,37 +1707,33 @@ __socket_read_vectored_reply (rpc_transport_t *this)
case SP_STATE_READING_REPLY_STATUS:
__socket_proto_read (priv, ret);
- buf = rpc_reply_status_addr (priv->incoming.frag.fragcurrent);
+ buf = rpc_reply_status_addr (frag->fragcurrent);
- priv->incoming.frag.call_body.reply.accept_status
+ frag->call_body.reply.accept_status
= ntoh32 (*((uint32_t *) buf));
- priv->incoming.frag.call_body.reply.status_state
+ frag->call_body.reply.status_state
= SP_STATE_READ_REPLY_STATUS;
/* fall through */
case SP_STATE_READ_REPLY_STATUS:
- if (priv->incoming.frag.call_body.reply.accept_status
- == MSG_ACCEPTED) {
+ if (frag->call_body.reply.accept_status == MSG_ACCEPTED) {
ret = __socket_read_accepted_reply (this);
} else {
ret = __socket_read_denied_reply (this);
}
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
if ((ret == -1)
- || ((ret == 0)
- && (remaining_size == 0)
- && (RPC_LASTFRAG (priv->incoming.fraghdr)))) {
- priv->incoming.frag.call_body.reply.status_state
+ || ((ret == 0) && (remaining_size == 0)
+ && (RPC_LASTFRAG (in->fraghdr)))) {
+ frag->call_body.reply.status_state
= SP_STATE_ACCEPTED_REPLY_INIT;
- priv->incoming.payload_vector.iov_len
- = (unsigned long)priv->incoming.frag.fragcurrent
- - (unsigned long)
- priv->incoming.payload_vector.iov_base;
+ in->payload_vector.iov_len
+ = (unsigned long)frag->fragcurrent
+ - (unsigned long)in->payload_vector.iov_base;
}
break;
}
@@ -1240,7 +1743,7 @@ out:
}
-inline int
+static inline int
__socket_read_simple_reply (rpc_transport_t *this)
{
return __socket_read_simple_msg (this);
@@ -1248,7 +1751,7 @@ __socket_read_simple_reply (rpc_transport_t *this)
#define rpc_xid_addr(buf) (buf)
-inline int
+static inline int
__socket_read_reply (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
@@ -1256,27 +1759,29 @@ __socket_read_reply (rpc_transport_t *this)
int32_t ret = -1;
rpc_request_info_t *request_info = NULL;
char map_xid = 0;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
+ in = &priv->incoming;
+ frag = &in->frag;
- buf = rpc_xid_addr (iobuf_ptr (priv->incoming.iobuf));
+ buf = rpc_xid_addr (iobuf_ptr (in->iobuf));
- if (priv->incoming.request_info == NULL) {
- priv->incoming.request_info = GF_CALLOC (1,
- sizeof (*request_info),
- gf_common_mt_rpc_trans_reqinfo_t);
- if (priv->incoming.request_info == NULL) {
- gf_log (this->name, GF_LOG_ERROR, "out of memory");
+ if (in->request_info == NULL) {
+ in->request_info = GF_CALLOC (1, sizeof (*request_info),
+ gf_common_mt_rpc_trans_reqinfo_t);
+ if (in->request_info == NULL) {
goto out;
}
map_xid = 1;
}
- request_info = priv->incoming.request_info;
+ request_info = in->request_info;
if (map_xid) {
request_info->xid = ntoh32 (*((uint32_t *) buf));
@@ -1288,22 +1793,22 @@ __socket_read_reply (rpc_transport_t *this)
{
ret = rpc_transport_notify (this,
RPC_TRANSPORT_MAP_XID_REQUEST,
- priv->incoming.request_info);
+ in->request_info);
}
pthread_mutex_lock (&priv->lock);
if (ret == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "notify for event MAP_XID failed");
goto out;
}
}
- if ((request_info->prognum == GLUSTER3_1_FOP_PROGRAM)
+ if ((request_info->prognum == GLUSTER_FOP_PROGRAM)
&& (request_info->procnum == GF_FOP_READ)) {
if (map_xid && request_info->rsp.rsp_payload_count != 0) {
- priv->incoming.iobref
- = iobref_ref (request_info->rsp.rsp_iobref);
- priv->incoming.payload_vector
- = *request_info->rsp.rsp_payload;
+ in->iobref = iobref_ref (request_info->rsp.rsp_iobref);
+ in->payload_vector = *request_info->rsp.rsp_payload;
}
ret = __socket_read_vectored_reply (this);
@@ -1316,62 +1821,65 @@ out:
/* returns the number of bytes yet to be read in a fragment */
-inline int
+static inline int
__socket_read_frag (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int32_t ret = 0;
char *buf = NULL;
uint32_t remaining_size = 0;
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+ frag = &in->frag;
- switch (priv->incoming.frag.state) {
+ switch (frag->state) {
case SP_STATE_NADA:
__socket_proto_init_pending (priv, RPC_MSGTYPE_SIZE);
- priv->incoming.frag.state = SP_STATE_READING_MSGTYPE;
+ frag->state = SP_STATE_READING_MSGTYPE;
/* fall through */
case SP_STATE_READING_MSGTYPE:
__socket_proto_read (priv, ret);
- priv->incoming.frag.state = SP_STATE_READ_MSGTYPE;
+ frag->state = SP_STATE_READ_MSGTYPE;
/* fall through */
case SP_STATE_READ_MSGTYPE:
- buf = rpc_msgtype_addr (iobuf_ptr (priv->incoming.iobuf));
- priv->incoming.msg_type = ntoh32 (*((uint32_t *)buf));
+ buf = rpc_msgtype_addr (iobuf_ptr (in->iobuf));
+ in->msg_type = ntoh32 (*((uint32_t *)buf));
- if (priv->incoming.msg_type == CALL) {
+ if (in->msg_type == CALL) {
ret = __socket_read_request (this);
- } else if (priv->incoming.msg_type == REPLY) {
+ } else if (in->msg_type == REPLY) {
ret = __socket_read_reply (this);
- } else if (priv->incoming.msg_type == GF_UNIVERSAL_ANSWER) {
+ } else if (in->msg_type == GF_UNIVERSAL_ANSWER) {
gf_log ("rpc", GF_LOG_ERROR,
"older version of protocol/process trying to "
- "connect from %s. use newer verison on that node",
+ "connect from %s. use newer version on that node",
this->peerinfo.identifier);
} else {
gf_log ("rpc", GF_LOG_ERROR,
"wrong MSG-TYPE (%d) received from %s",
- priv->incoming.msg_type,
+ in->msg_type,
this->peerinfo.identifier);
ret = -1;
}
- remaining_size = RPC_FRAGSIZE (priv->incoming.fraghdr)
- - priv->incoming.frag.bytes_read;
+ remaining_size = RPC_FRAGSIZE (in->fraghdr) - frag->bytes_read;
if ((ret == -1)
- || ((ret == 0)
- && (remaining_size == 0)
- && (RPC_LASTFRAG (priv->incoming.fraghdr)))) {
- priv->incoming.frag.state = SP_STATE_NADA;
+ || ((ret == 0) && (remaining_size == 0)
+ && (RPC_LASTFRAG (in->fraghdr)))) {
+ frag->state = SP_STATE_NADA;
}
break;
@@ -1382,31 +1890,36 @@ out:
}
-inline
+static inline
void __socket_reset_priv (socket_private_t *priv)
{
- if (priv->incoming.iobref) {
- iobref_unref (priv->incoming.iobref);
- priv->incoming.iobref = NULL;
+ struct gf_sock_incoming *in = NULL;
+
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+
+ if (in->iobref) {
+ iobref_unref (in->iobref);
+ in->iobref = NULL;
}
- if (priv->incoming.iobuf) {
- iobuf_unref (priv->incoming.iobuf);
+ if (in->iobuf) {
+ iobuf_unref (in->iobuf);
}
- if (priv->incoming.request_info != NULL) {
- GF_FREE (priv->incoming.request_info);
- priv->incoming.request_info = NULL;
+ if (in->request_info != NULL) {
+ GF_FREE (in->request_info);
+ in->request_info = NULL;
}
- memset (&priv->incoming.payload_vector, 0,
- sizeof (priv->incoming.payload_vector));
+ memset (&in->payload_vector, 0,
+ sizeof (in->payload_vector));
- priv->incoming.iobuf = NULL;
+ in->iobuf = NULL;
}
-int
+static int
__socket_proto_state_machine (rpc_transport_t *this,
rpc_transport_pollin_t **pollin)
{
@@ -1415,55 +1928,40 @@ __socket_proto_state_machine (rpc_transport_t *this,
struct iobuf *iobuf = NULL;
struct iobref *iobref = NULL;
struct iovec vector[2];
+ struct gf_sock_incoming *in = NULL;
+ struct gf_sock_incoming_frag *frag = NULL;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
- while (priv->incoming.record_state != SP_STATE_COMPLETE) {
- switch (priv->incoming.record_state) {
+ /* used to reduce the indirection */
+ in = &priv->incoming;
+ frag = &in->frag;
- case SP_STATE_NADA:
- iobuf = iobuf_get (this->ctx->iobuf_pool);
- if (!iobuf) {
- gf_log (this->name, GF_LOG_ERROR,
- "unable to allocate IO buffer "
- "for peer %s",
- this->peerinfo.identifier);
- ret = -ENOMEM;
- goto out;
- }
+ while (in->record_state != SP_STATE_COMPLETE) {
+ switch (in->record_state) {
- priv->incoming.iobuf = iobuf;
- priv->incoming.iobuf_size = 0;
- priv->incoming.total_bytes_read = 0;
- priv->incoming.payload_vector.iov_len = 0;
+ case SP_STATE_NADA:
+ in->total_bytes_read = 0;
+ in->payload_vector.iov_len = 0;
- priv->incoming.pending_vector = priv->incoming.vector;
- priv->incoming.pending_vector->iov_base =
- &priv->incoming.fraghdr;
+ in->pending_vector = in->vector;
+ in->pending_vector->iov_base = &in->fraghdr;
- priv->incoming.frag.fragcurrent = iobuf_ptr (iobuf);
- priv->incoming.pending_vector->iov_len =
- sizeof (priv->incoming.fraghdr);
+ in->pending_vector->iov_len = sizeof (in->fraghdr);
- priv->incoming.record_state = SP_STATE_READING_FRAGHDR;
+ in->record_state = SP_STATE_READING_FRAGHDR;
/* fall through */
case SP_STATE_READING_FRAGHDR:
- ret = __socket_readv (this,
- priv->incoming.pending_vector, 1,
- &priv->incoming.pending_vector,
- &priv->incoming.pending_count,
+ ret = __socket_readv (this, in->pending_vector, 1,
+ &in->pending_vector,
+ &in->pending_count,
NULL);
- if (ret == -1) {
- gf_log (this->name, GF_LOG_TRACE,
- "reading from socket failed. Error (%s), "
- "peer (%s)", strerror (errno),
- this->peerinfo.identifier);
+ if (ret == -1)
goto out;
- }
if (ret > 0) {
gf_log (this->name, GF_LOG_TRACE, "partial "
@@ -1472,33 +1970,40 @@ __socket_proto_state_machine (rpc_transport_t *this,
}
if (ret == 0) {
- priv->incoming.record_state =
- SP_STATE_READ_FRAGHDR;
+ in->record_state = SP_STATE_READ_FRAGHDR;
}
/* fall through */
case SP_STATE_READ_FRAGHDR:
- priv->incoming.fraghdr = ntoh32 (priv->incoming.fraghdr);
- priv->incoming.record_state = SP_STATE_READING_FRAG;
- priv->incoming.total_bytes_read
- += RPC_FRAGSIZE(priv->incoming.fraghdr);
+ in->fraghdr = ntoh32 (in->fraghdr);
+ in->total_bytes_read += RPC_FRAGSIZE(in->fraghdr);
+ iobuf = iobuf_get2 (this->ctx->iobuf_pool,
+ (in->total_bytes_read +
+ sizeof (in->fraghdr)));
+ if (!iobuf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ in->iobuf = iobuf;
+ in->iobuf_size = 0;
+ frag->fragcurrent = iobuf_ptr (iobuf);
+ in->record_state = SP_STATE_READING_FRAG;
/* fall through */
case SP_STATE_READING_FRAG:
ret = __socket_read_frag (this);
- if ((ret == -1)
- || (priv->incoming.frag.bytes_read !=
- RPC_FRAGSIZE (priv->incoming.fraghdr))) {
+ if ((ret == -1) ||
+ (frag->bytes_read != RPC_FRAGSIZE (in->fraghdr))) {
goto out;
}
- priv->incoming.frag.bytes_read = 0;
+ frag->bytes_read = 0;
- if (!RPC_LASTFRAG (priv->incoming.fraghdr)) {
- priv->incoming.record_state =
- SP_STATE_READING_FRAGHDR;
+ if (!RPC_LASTFRAG (in->fraghdr)) {
+ in->record_state = SP_STATE_READING_FRAGHDR;
break;
}
@@ -1507,72 +2012,65 @@ __socket_proto_state_machine (rpc_transport_t *this,
*/
if (pollin != NULL) {
int count = 0;
- priv->incoming.iobuf_size
- = priv->incoming.total_bytes_read
- - priv->incoming.payload_vector.iov_len;
+ in->iobuf_size = (in->total_bytes_read -
+ in->payload_vector.iov_len);
memset (vector, 0, sizeof (vector));
- if (priv->incoming.iobref == NULL) {
- priv->incoming.iobref = iobref_new ();
- if (priv->incoming.iobref == NULL) {
- gf_log (this->name,
- GF_LOG_ERROR,
- "out of memory");
+ if (in->iobref == NULL) {
+ in->iobref = iobref_new ();
+ if (in->iobref == NULL) {
ret = -1;
goto out;
}
}
- vector[count].iov_base
- = iobuf_ptr (priv->incoming.iobuf);
- vector[count].iov_len
- = priv->incoming.iobuf_size;
-
- iobref = priv->incoming.iobref;
+ vector[count].iov_base = iobuf_ptr (in->iobuf);
+ vector[count].iov_len = in->iobuf_size;
- iobref_add (iobref,
- priv->incoming.iobuf);
- iobuf_unref (priv->incoming.iobuf);
- priv->incoming.iobuf = NULL;
+ iobref = in->iobref;
count++;
- if (priv->incoming.payload_vector.iov_base
- != NULL) {
- vector[count]
- = priv->incoming.payload_vector;
+ if (in->payload_vector.iov_base != NULL) {
+ vector[count] = in->payload_vector;
count++;
}
*pollin = rpc_transport_pollin_alloc (this,
vector,
count,
+ in->iobuf,
iobref,
- priv->incoming.request_info);
+ in->request_info);
+ iobuf_unref (in->iobuf);
+ in->iobuf = NULL;
+
if (*pollin == NULL) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "transport pollin allocation failed");
ret = -1;
goto out;
}
- if (priv->incoming.msg_type == REPLY)
+ if (in->msg_type == REPLY)
(*pollin)->is_reply = 1;
- priv->incoming.request_info = NULL;
+ in->request_info = NULL;
}
- priv->incoming.record_state = SP_STATE_COMPLETE;
+ in->record_state = SP_STATE_COMPLETE;
break;
case SP_STATE_COMPLETE:
/* control should not reach here */
- gf_log (this->name, GF_LOG_DEBUG, "control reached to "
+ gf_log (this->name, GF_LOG_WARNING, "control reached to "
"SP_STATE_COMPLETE, which should not have "
"happened");
break;
}
}
- if (priv->incoming.record_state == SP_STATE_COMPLETE) {
- priv->incoming.record_state = SP_STATE_NADA;
+ if (in->record_state == SP_STATE_COMPLETE) {
+ in->record_state = SP_STATE_NADA;
__socket_reset_priv (priv);
}
@@ -1584,41 +2082,45 @@ out:
}
-int
+static int
socket_proto_state_machine (rpc_transport_t *this,
rpc_transport_pollin_t **pollin)
{
socket_private_t *priv = NULL;
- int ret = 0;
+ int ret = 0;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
- priv = this->private;
+ priv = this->private;
- pthread_mutex_lock (&priv->lock);
- {
- ret = __socket_proto_state_machine (this, pollin);
- }
+ pthread_mutex_lock (&priv->lock);
+ {
+ ret = __socket_proto_state_machine (this, pollin);
+ }
pthread_mutex_unlock (&priv->lock);
out:
- return ret;
+ return ret;
}
-int
+static int
socket_event_poll_in (rpc_transport_t *this)
{
int ret = -1;
rpc_transport_pollin_t *pollin = NULL;
+ socket_private_t *priv = this->private;
ret = socket_proto_state_machine (this, &pollin);
if (pollin != NULL) {
+ priv->ot_state = OT_CALLBACK;
ret = rpc_transport_notify (this, RPC_TRANSPORT_MSG_RECEIVED,
pollin);
-
+ if (priv->ot_state == OT_CALLBACK) {
+ priv->ot_state = OT_RUNNING;
+ }
rpc_transport_pollin_destroy (pollin);
}
@@ -1626,7 +2128,7 @@ socket_event_poll_in (rpc_transport_t *this)
}
-int
+static int
socket_connect_finish (rpc_transport_t *this)
{
int ret = -1;
@@ -1634,58 +2136,57 @@ socket_connect_finish (rpc_transport_t *this)
rpc_transport_event_t event = 0;
char notify_rpc = 0;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
pthread_mutex_lock (&priv->lock);
{
- if (priv->connected)
- goto unlock;
+ if (priv->connected != 0)
+ goto unlock;
+
+ get_transport_identifiers (this);
- ret = __socket_connect_finish (priv->sock);
+ ret = __socket_connect_finish (priv->sock);
- if (ret == -1 && errno == EINPROGRESS)
- ret = 1;
+ if (ret == -1 && errno == EINPROGRESS)
+ ret = 1;
- if (ret == -1 && errno != EINPROGRESS) {
- if (!priv->connect_finish_log) {
- gf_log (this->name, GF_LOG_ERROR,
- "connection to %s failed (%s)",
+ if (ret == -1 && errno != EINPROGRESS) {
+ if (!priv->connect_finish_log) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "connection to %s failed (%s)",
this->peerinfo.identifier,
- strerror (errno));
- priv->connect_finish_log = 1;
- }
- __socket_disconnect (this);
- notify_rpc = 1;
- event = RPC_TRANSPORT_DISCONNECT;
- goto unlock;
- }
+ strerror (errno));
+ priv->connect_finish_log = 1;
+ }
+ __socket_disconnect (this);
+ goto unlock;
+ }
- if (ret == 0) {
- notify_rpc = 1;
-
- this->myinfo.sockaddr_len =
- sizeof (this->myinfo.sockaddr);
-
- ret = getsockname (priv->sock,
- SA (&this->myinfo.sockaddr),
- &this->myinfo.sockaddr_len);
- if (ret == -1) {
- gf_log (this->name, GF_LOG_DEBUG,
- "getsockname on (%d) failed (%s)",
- priv->sock, strerror (errno));
- __socket_disconnect (this);
- event = GF_EVENT_POLLERR;
- goto unlock;
- }
+ if (ret == 0) {
+ notify_rpc = 1;
- priv->connected = 1;
- priv->connect_finish_log = 0;
- event = RPC_TRANSPORT_CONNECT;
- get_transport_identifiers (this);
- }
+ this->myinfo.sockaddr_len =
+ sizeof (this->myinfo.sockaddr);
+
+ ret = getsockname (priv->sock,
+ SA (&this->myinfo.sockaddr),
+ &this->myinfo.sockaddr_len);
+ if (ret == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "getsockname on (%d) failed (%s)",
+ priv->sock, strerror (errno));
+ __socket_disconnect (this);
+ event = GF_EVENT_POLLERR;
+ goto unlock;
+ }
+
+ priv->connected = 1;
+ priv->connect_finish_log = 0;
+ event = RPC_TRANSPORT_CONNECT;
+ }
}
unlock:
pthread_mutex_unlock (&priv->lock);
@@ -1699,31 +2200,29 @@ out:
/* reads rpc_requests during pollin */
-int
+static int
socket_event_handler (int fd, int idx, void *data,
int poll_in, int poll_out, int poll_err)
{
- rpc_transport_t *this = NULL;
+ rpc_transport_t *this = NULL;
socket_private_t *priv = NULL;
- int ret = 0;
+ int ret = -1;
this = data;
- if (!this || !this->private || !this->xl)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->xl, out);
THIS = this->xl;
priv = this->private;
-
pthread_mutex_lock (&priv->lock);
{
priv->idx = idx;
}
pthread_mutex_unlock (&priv->lock);
- if (!priv->connected) {
- ret = socket_connect_finish (this);
- }
+ ret = (priv->connected == 1) ? 0 : socket_connect_finish(this);
if (!ret && poll_out) {
ret = socket_event_poll_out (this);
@@ -1734,17 +2233,203 @@ socket_event_handler (int fd, int idx, void *data,
}
if ((ret < 0) || poll_err) {
- gf_log ("transport", GF_LOG_TRACE, "disconnecting now");
+ /* Logging has happened already in earlier cases */
+ gf_log ("transport", ((ret >= 0) ? GF_LOG_INFO : GF_LOG_DEBUG),
+ "disconnecting now");
socket_event_poll_err (this);
- //rpc_transport_unref (this);
- }
+ rpc_transport_unref (this);
+ }
out:
- return 0;
+ return ret;
}
-int
+static void *
+socket_poller (void *ctx)
+{
+ rpc_transport_t *this = ctx;
+ socket_private_t *priv = this->private;
+ struct pollfd pfd[2] = {{0,},};
+ gf_boolean_t to_write = _gf_false;
+ int ret = 0;
+ uint32_t gen = 0;
+
+ priv->ot_state = OT_RUNNING;
+
+ if (priv->use_ssl) {
+ if (ssl_setup_connection(this,priv->connected) < 0) {
+ gf_log (this->name,GF_LOG_ERROR, "%s setup failed",
+ priv->connected ? "server" : "client");
+ goto err;
+ }
+ }
+
+ if (!priv->bio) {
+ ret = __socket_nonblock (priv->sock);
+ if (ret == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "NBIO on %d failed (%s)",
+ priv->sock, strerror (errno));
+ goto err;
+ }
+ }
+
+ if (priv->connected == 0) {
+ THIS = this->xl;
+ ret = socket_connect_finish (this);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "asynchronous socket_connect_finish failed");
+ }
+ }
+
+ ret = rpc_transport_notify (this->listener,
+ RPC_TRANSPORT_ACCEPT, this);
+ if (ret != 0) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "asynchronous rpc_transport_notify failed");
+ }
+
+ gen = priv->ot_gen;
+ for (;;) {
+ pthread_mutex_lock(&priv->lock);
+ to_write = !list_empty(&priv->ioq);
+ pthread_mutex_unlock(&priv->lock);
+ pfd[0].fd = priv->pipe[0];
+ pfd[0].events = POLL_MASK_ERROR;
+ pfd[0].revents = 0;
+ pfd[1].fd = priv->sock;
+ pfd[1].events = POLL_MASK_INPUT | POLL_MASK_ERROR;
+ pfd[1].revents = 0;
+ if (to_write) {
+ pfd[1].events |= POLL_MASK_OUTPUT;
+ }
+ else {
+ pfd[0].events |= POLL_MASK_INPUT;
+ }
+ if (poll(pfd,2,-1) < 0) {
+ gf_log(this->name,GF_LOG_ERROR,"poll failed");
+ break;
+ }
+ if (pfd[0].revents & POLL_MASK_ERROR) {
+ gf_log(this->name,GF_LOG_ERROR,
+ "poll error on pipe");
+ break;
+ }
+ /* Only glusterd actually seems to need this. */
+ THIS = this->xl;
+ if (pfd[1].revents & POLL_MASK_INPUT) {
+ ret = socket_event_poll_in(this);
+ if (ret >= 0) {
+ /* Suppress errors while making progress. */
+ pfd[1].revents &= ~POLL_MASK_ERROR;
+ }
+ else if (errno == ENOTCONN) {
+ ret = 0;
+ }
+ if (priv->ot_state == OT_PLEASE_DIE) {
+ gf_log (this->name, GF_LOG_TRACE,
+ "OT_IDLE on %p (input request)",
+ this);
+ priv->ot_state = OT_IDLE;
+ break;
+ }
+ }
+ else if (pfd[1].revents & POLL_MASK_OUTPUT) {
+ ret = socket_event_poll_out(this);
+ if (ret >= 0) {
+ /* Suppress errors while making progress. */
+ pfd[1].revents &= ~POLL_MASK_ERROR;
+ }
+ else if (errno == ENOTCONN) {
+ ret = 0;
+ }
+ if (priv->ot_state == OT_PLEASE_DIE) {
+ gf_log (this->name, GF_LOG_TRACE,
+ "OT_IDLE on %p (output request)",
+ this);
+ priv->ot_state = OT_IDLE;
+ break;
+ }
+ }
+ else {
+ /*
+ * This usually means that we left poll() because
+ * somebody pushed a byte onto our pipe. That wakeup
+ * is why the pipe is there, but once awake we can do
+ * all the checking we need on the next iteration.
+ */
+ ret = 0;
+ }
+ if (pfd[1].revents & POLL_MASK_ERROR) {
+ gf_log(this->name,GF_LOG_ERROR,
+ "poll error on socket");
+ break;
+ }
+ if (ret < 0) {
+ gf_log(this->name,GF_LOG_ERROR,
+ "error in polling loop");
+ break;
+ }
+ if (priv->ot_gen != gen) {
+ gf_log (this->name, GF_LOG_TRACE,
+ "generation mismatch, my %u != %u",
+ gen, priv->ot_gen);
+ return NULL;
+ }
+ }
+
+err:
+ /* All (and only) I/O errors should come here. */
+ pthread_mutex_lock(&priv->lock);
+ if (priv->ssl_ssl) {
+ /*
+ * We're always responsible for this part, but only actually
+ * have to do it if we got far enough for ssl_ssl to be valid
+ * (i.e. errors in ssl_setup_connection don't count).
+ */
+ ssl_teardown_connection(priv);
+ }
+ __socket_shutdown(this);
+ close(priv->sock);
+ priv->sock = -1;
+ priv->ot_state = OT_IDLE;
+ pthread_mutex_unlock(&priv->lock);
+ rpc_transport_notify (this->listener, RPC_TRANSPORT_DISCONNECT,
+ this);
+ rpc_transport_unref (this);
+ return NULL;
+}
+
+
+static void
+socket_spawn (rpc_transport_t *this)
+{
+ socket_private_t *priv = this->private;
+
+ switch (priv->ot_state) {
+ case OT_IDLE:
+ case OT_PLEASE_DIE:
+ break;
+ default:
+ gf_log (this->name, GF_LOG_WARNING,
+ "refusing to start redundant poller");
+ return;
+ }
+
+ priv->ot_gen += 7;
+ priv->ot_state = OT_SPAWNING;
+ gf_log (this->name, GF_LOG_TRACE,
+ "spawning %p with gen %u", this, priv->ot_gen);
+
+ if (pthread_create(&priv->thread,NULL,socket_poller,this) != 0) {
+ gf_log (this->name, GF_LOG_ERROR,
+ "could not create poll thread");
+ }
+}
+
+static int
socket_server_event_handler (int fd, int idx, void *data,
int poll_in, int poll_out, int poll_err)
{
@@ -1756,15 +2441,16 @@ socket_server_event_handler (int fd, int idx, void *data,
struct sockaddr_storage new_sockaddr = {0, };
socklen_t addrlen = sizeof (new_sockaddr);
socket_private_t *new_priv = NULL;
- glusterfs_ctx_t *ctx = NULL;
+ glusterfs_ctx_t *ctx = NULL;
this = data;
- if (!this || !this->private || !this->xl)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->xl, out);
THIS = this->xl;
priv = this->private;
- ctx = this->ctx;
+ ctx = this->ctx;
pthread_mutex_lock (&priv->lock);
{
@@ -1772,39 +2458,33 @@ socket_server_event_handler (int fd, int idx, void *data,
if (poll_in) {
new_sock = accept (priv->sock, SA (&new_sockaddr),
- &addrlen);
+ &addrlen);
- if (new_sock == -1)
+ if (new_sock == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "accept on %d failed (%s)",
+ priv->sock, strerror (errno));
goto unlock;
-
- if (!priv->bio) {
- ret = __socket_nonblock (new_sock);
-
- if (ret == -1) {
- gf_log (this->name, GF_LOG_DEBUG,
- "NBIO on %d failed (%s)",
- new_sock, strerror (errno));
-
- close (new_sock);
- goto unlock;
- }
}
if (priv->nodelay) {
ret = __socket_nodelay (new_sock);
if (ret == -1) {
- gf_log (this->name, GF_LOG_ERROR,
+ gf_log (this->name, GF_LOG_WARNING,
"setsockopt() failed for "
"NODELAY (%s)",
strerror (errno));
}
}
- if (priv->keepalive) {
+ if (priv->keepalive &&
+ new_sockaddr.ss_family != AF_UNIX) {
ret = __socket_keepalive (new_sock,
- priv->keepaliveintvl);
+ new_sockaddr.ss_family,
+ priv->keepaliveintvl,
+ priv->keepaliveidle);
if (ret == -1)
- gf_log (this->name, GF_LOG_ERROR,
+ gf_log (this->name, GF_LOG_WARNING,
"Failed to set keep-alive: %s",
strerror (errno));
}
@@ -1814,20 +2494,29 @@ socket_server_event_handler (int fd, int idx, void *data,
if (!new_trans)
goto unlock;
+ ret = pthread_mutex_init(&new_trans->lock, NULL);
+ if (ret == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "pthread_mutex_init() failed: %s",
+ strerror (errno));
+ close (new_sock);
+ goto unlock;
+ }
+
new_trans->name = gf_strdup (this->name);
memcpy (&new_trans->peerinfo.sockaddr, &new_sockaddr,
- addrlen);
+ addrlen);
new_trans->peerinfo.sockaddr_len = addrlen;
new_trans->myinfo.sockaddr_len =
- sizeof (new_trans->myinfo.sockaddr);
+ sizeof (new_trans->myinfo.sockaddr);
ret = getsockname (new_sock,
SA (&new_trans->myinfo.sockaddr),
&new_trans->myinfo.sockaddr_len);
if (ret == -1) {
- gf_log (this->name, GF_LOG_DEBUG,
+ gf_log (this->name, GF_LOG_WARNING,
"getsockname on %d failed (%s)",
new_sock, strerror (errno));
close (new_sock);
@@ -1835,7 +2524,11 @@ socket_server_event_handler (int fd, int idx, void *data,
}
get_transport_identifiers (new_trans);
- socket_init (new_trans);
+ ret = socket_init(new_trans);
+ if (ret != 0) {
+ close(new_sock);
+ goto unlock;
+ }
new_trans->ops = this->ops;
new_trans->init = this->init;
new_trans->fini = this->fini;
@@ -1846,27 +2539,74 @@ socket_server_event_handler (int fd, int idx, void *data,
new_trans->listener = this;
new_priv = new_trans->private;
+ new_priv->use_ssl = priv->use_ssl;
+ new_priv->sock = new_sock;
+ new_priv->own_thread = priv->own_thread;
+
+ new_priv->ssl_ctx = priv->ssl_ctx;
+ if (priv->use_ssl && !priv->own_thread) {
+ if (ssl_setup_connection(new_trans,1) < 0) {
+ gf_log(this->name,GF_LOG_ERROR,
+ "server setup failed");
+ close(new_sock);
+ goto unlock;
+ }
+ }
+
+ if (!priv->bio && !priv->own_thread) {
+ ret = __socket_nonblock (new_sock);
+
+ if (ret == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "NBIO on %d failed (%s)",
+ new_sock, strerror (errno));
+
+ close (new_sock);
+ goto unlock;
+ }
+ }
+
pthread_mutex_lock (&new_priv->lock);
{
- new_priv->sock = new_sock;
+ /*
+ * In the own_thread case, this is used to
+ * indicate that we're initializing a server
+ * connection.
+ */
new_priv->connected = 1;
+ new_priv->is_server = _gf_true;
rpc_transport_ref (new_trans);
- new_priv->idx =
- event_register (ctx->event_pool,
- new_sock,
- socket_event_handler,
- new_trans, 1, 0);
+ if (new_priv->own_thread) {
+ if (pipe(new_priv->pipe) < 0) {
+ gf_log(this->name,GF_LOG_ERROR,
+ "could not create pipe");
+ }
+ socket_spawn(new_trans);
+ }
+ else {
+ new_priv->idx =
+ event_register (ctx->event_pool,
+ new_sock,
+ socket_event_handler,
+ new_trans,
+ 1, 0);
+ if (new_priv->idx == -1)
+ ret = -1;
+ }
- if (new_priv->idx == -1)
- ret = -1;
}
pthread_mutex_unlock (&new_priv->lock);
- if (ret == -1)
+ if (ret == -1) {
+ gf_log (this->name, GF_LOG_WARNING,
+ "failed to register the socket with event");
goto unlock;
+ }
- ret = rpc_transport_notify (this, RPC_TRANSPORT_ACCEPT,
- new_trans);
+ if (!priv->own_thread) {
+ ret = rpc_transport_notify (this,
+ RPC_TRANSPORT_ACCEPT, new_trans);
+ }
}
}
unlock:
@@ -1877,14 +2617,14 @@ out:
}
-int
+static int
socket_disconnect (rpc_transport_t *this)
{
socket_private_t *priv = NULL;
int ret = -1;
- if (!this || !this->private)
- goto out;
+ GF_VALIDATE_OR_GOTO ("socket", this, out);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, out);
priv = this->private;
@@ -1899,25 +2639,27 @@ out:
}
-int
+static int
socket_connect (rpc_transport_t *this, int port)
{
int ret = -1;
- int sock = -1;
+ int sock = -1;
socket_private_t *priv = NULL;
- struct sockaddr_storage sockaddr = {0, };
socklen_t sockaddr_len = 0;
- glusterfs_ctx_t *ctx = NULL;
+ glusterfs_ctx_t *ctx = NULL;
sa_family_t sa_family = {0, };
+ char *local_addr = NULL;
+ union gf_sock_union sock_union;
+ struct sockaddr_in *addr = NULL;
- if (!this || !this->private)
- goto err;
+ GF_VALIDATE_OR_GOTO ("socket", this, err);
+ GF_VALIDATE_OR_GOTO ("socket", this->private, err);
priv = this->private;
- ctx = this->ctx;
+ ctx = this->ctx;
if (!priv) {
- gf_log (this->name, GF_LOG_DEBUG,
+ gf_log_callingfn (this->name, GF_LOG_WARNING,
"connect() called on uninitialized transport");
goto err;
}
@@ -1929,22 +2671,41 @@ socket_connect (rpc_transport_t *this, int port)
pthread_mutex_unlock (&priv->lock);
if (sock != -1) {
- gf_log (this->name, GF_LOG_TRACE,
- "connect () called on transport already connected");
- ret = 0;
+ gf_log_callingfn (this->name, GF_LOG_TRACE,
+ "connect () called on transport already connected");
+ errno = EINPROGRESS;
+ ret = -1;
goto err;
}
- ret = socket_client_get_remote_sockaddr (this, SA (&sockaddr),
+ gf_log (this->name, GF_LOG_TRACE,
+ "connecting %p, state=%u gen=%u sock=%d", this,
+ priv->ot_state, priv->ot_gen, priv->sock);
+
+ ret = socket_client_get_remote_sockaddr (this, &sock_union.sa,
&sockaddr_len, &sa_family);
if (ret == -1) {
/* logged inside client_get_remote_sockaddr */
goto err;
}
- if (port > 0)
- ((struct sockaddr_in *) (&sockaddr))->sin_port = htons (port);
-
+ if (port > 0) {
+ sock_union.sin.sin_port = htons (port);
+ }
+ if (ntohs(sock_union.sin.sin_port) == GF_DEFAULT_SOCKET_LISTEN_PORT) {
+ if (priv->use_ssl) {
+ gf_log(this->name,GF_LOG_DEBUG,
+ "disabling SSL for portmapper connection");
+ priv->use_ssl = _gf_false;
+ }
+ }
+ else {
+ if (priv->ssl_enabled && !priv->use_ssl) {
+ gf_log(this->name,GF_LOG_DEBUG,
+ "re-enabling SSL for I/O connection");
+ priv->use_ssl = _gf_true;
+ }