glusterd: auth allow enhancements

* PROBLEM:

  When address-based authentication is enabled on a volume,
  the gNfs server, self-heal daemon (shd), and other operations
  such as quota, rebalance, replace-brick and geo-replication
  either stop working or the services are not started if all
  the peers' ipv{4,6} addresses or hostnames are not added in
  the "set auth.allow" operation, breaking the functionality
  of several operations.

  E.g:
    volume vol in a cluster of two peers:
    /mnt/brick1 in 192.168.1.4
    /mnt/brick2 in 192.168.1.5

    option auth.allow 192.168.1.6
    (allow connection requests only from 192.168.1.6)

    This will disrupt the nfs servers on 192.168.1.{4,5}.
    brick server processes reject connection requests from both
    nfs servers (on 4,5), because the peer addresses are not in
    the auth.allow list.

    Same holds true for local mounts (on peer machines),
    self-heal daemon, and other operations which perform
    a glusterfs mount on one of the peers.

* SOLUTION:

  Login-based authentication (username/password pairs,
  henceforth referred to as "keys") for gluster services and
  operations.

  These *per-volume* keys can be used to by-pass the addr-based
  authentication, provided none of the peers' addresses are put
  in the auth.reject list, to enable gluster services like gNfs,
  self-heal daemon and internal operations on volumes when
  auth.allow option is exercised.

* IMPLEMENTATION:

  1. Glusterd generates keys for each volume and stores it in
     memory as well as in respective volfiles.
     A new TRUSTED-FUSE volfile is generated which is
     fuse volfile + keys in protocol/client,
     and is named trusted-<volname>-fuse.vol.
     This is used by all local mounts. ANY local mount (on any peer)
     is granted the trusted-fuse volfile instead of fuse volfile
     via getspec. non-local mounts are NOT granted the trusted fuse
     volfile.

  2. The keys generated for the volume is written to each server
     volfile telling servers to allow users with these keys.

  3. NFS, self-heal daemon and replace-brick volfiles are updated
     with the volume's authentication keys.

  4. The keys are NOT written to fuse volfiles for obvious reasons.

  5. The ownership of volfiles and logfiles is restricted to root users.

  6. Merging two identical definitions of peer_info_t in auth/addr
     and rpc-lib, throwing away the one in auth/addr.

  7. Code cleanup in numerous places as appropriate.

* IMPORTANT NOTES:

  1. One SHOULD NOT put any of the peer addresses in the auth.reject
     list if one wants any of the glusterd services and features
     such as gNfs, self-heal, rebalance, geo-rep and quota.

  2. If one wants to use username/password based authentication
     to volumes, one shall append to the server, nfs and shd volfiles,
     the keys one wants to use for authentication, *while_retaining
     those_generated_by_glusterd*.
     See doc/authentication.txt file for details.

Change-Id: Ie0331d625ad000d63090e2d622fe1728fbfcc453
BUG: 789942
Signed-off-by: Rajesh Amaravathi <rajesh@redhat.com>
Reviewed-on: http://review.gluster.com/2733
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vijay@gluster.com>

14 files changed, 538 insertions, 144 deletions

diff --git a/xlators/mgmt/glusterd/src/glusterd-geo-rep.c b/xlators/mgmt/glusterd/src/glusterd-geo-rep.c
index db43ecb4d31..3a62899f8b3 100644
--- a/xlators/mgmt/glusterd/src/glusterd-geo-rep.c
+++ b/xlators/mgmt/glusterd/src/glusterd-geo-rep.c
@@ -1719,14 +1719,15 @@ glusterd_do_gsync_log_rotation_mst_slv (glusterd_volinfo_t *volinfo, char *slave
         uuid_t           uuid = {0, };
         glusterd_conf_t *priv = NULL;
         int              ret  = 0;
-        char errmsg[1024] = {0,};
+        char             errmsg[1024] = {0,};
+        xlator_t        *this    = NULL;
 
         GF_ASSERT (volinfo);
         GF_ASSERT (slave);
         GF_ASSERT (THIS);
-        GF_ASSERT (THIS->private);
-
-        priv = THIS->private;
+        this = THIS;
+        GF_ASSERT (this->private);
+        priv = this->private;
 
         ret = glusterd_gsync_get_uuid (slave, volinfo, uuid);
         if ((ret == 0) && (uuid_compare (priv->uuid, uuid) != 0))
@@ -1735,7 +1736,7 @@ glusterd_do_gsync_log_rotation_mst_slv (glusterd_volinfo_t *volinfo, char *slave
         if (ret) {
                 snprintf(errmsg, sizeof(errmsg), "geo-replication session b/w %s %s not active",
                          volinfo->volname, slave);
-                gf_log ("", GF_LOG_WARNING, errmsg);
+                gf_log (this->name, GF_LOG_WARNING, "%s", errmsg);
                 if (op_errstr)
                         *op_errstr = gf_strdup(errmsg);
                 goto out;
@@ -1744,7 +1745,7 @@ glusterd_do_gsync_log_rotation_mst_slv (glusterd_volinfo_t *volinfo, char *slave
         ret = glusterd_do_gsync_log_rotate (volinfo->volname, slave, &uuid, op_errstr);
 
  out:
-        gf_log ("", GF_LOG_DEBUG, "Returning with %d", ret);
+        gf_log (this->name, GF_LOG_DEBUG, "Returning with %d", ret);
         return ret;
 }
 
@@ -1827,7 +1828,7 @@ glusterd_rotate_gsync_logs (dict_t *dict, char **op_errstr, dict_t *rsp_dict)
         if ((ret) || (!exists)) {
                 snprintf (errmsg, sizeof(errmsg), "Volume %s does not"
                           " exist", volname);
-                gf_log ("", GF_LOG_WARNING, errmsg);
+                gf_log ("", GF_LOG_WARNING, "%s", errmsg);
                 *op_errstr = gf_strdup (errmsg);
                 ret = -1;
                 goto out;
diff --git a/xlators/mgmt/glusterd/src/glusterd-handshake.c b/xlators/mgmt/glusterd/src/glusterd-handshake.c
index 04e5fae7bd5..dd41330b628 100644
--- a/xlators/mgmt/glusterd/src/glusterd-handshake.c
+++ b/xlators/mgmt/glusterd/src/glusterd-handshake.c
@@ -40,11 +40,13 @@
 extern struct rpc_clnt_program gd_peer_prog;
 extern struct rpc_clnt_program gd_mgmt_prog;
 
+#define TRUSTED_PREFIX         "trusted-"
+
 typedef ssize_t (*gfs_serialize_t) (struct iovec outmsg, void *data);
 
 static size_t
 build_volfile_path (const char *volname, char *path,
-                    size_t path_len)
+                    size_t path_len, char *trusted_str)
 {
         struct stat         stbuf       = {0,};
         int32_t             ret         = -1;
@@ -61,7 +63,7 @@ build_volfile_path (const char *volname, char *path,
         if (strstr (volname, "gluster/")) {
                 server = strchr (volname, '/') + 1;
                 glusterd_get_nodesvc_volfile (server, priv->workdir,
-                                                    path, path_len);
+                                              path, path_len);
                 ret = 1;
                 goto out;
         } else if (volname[0] != '/') {
@@ -86,20 +88,24 @@ build_volfile_path (const char *volname, char *path,
                 if (ret)
                         goto out;
         }
+
         ret = snprintf (path, path_len, "%s/vols/%s/%s.vol",
                         priv->workdir, volinfo->volname, volname);
         if (ret == -1)
                 goto out;
 
         ret = stat (path, &stbuf);
+
         if ((ret == -1) && (errno == ENOENT)) {
-                ret = snprintf (path, path_len, "%s/vols/%s/%s-fuse.vol",
-                                priv->workdir, volinfo->volname, volname);
+                snprintf (path, path_len, "%s/vols/%s/%s%s-fuse.vol",
+                          priv->workdir, volinfo->volname,
+                          (trusted_str ? trusted_str : ""), dup_volname);
                 ret = stat (path, &stbuf);
         }
+
         if ((ret == -1) && (errno == ENOENT)) {
-                ret = snprintf (path, path_len, "%s/vols/%s/%s-tcp.vol",
-                                priv->workdir, volinfo->volname, volname);
+                snprintf (path, path_len, "%s/vols/%s/%s-tcp.vol",
+                          priv->workdir, volinfo->volname, volname);
         }
 
         ret = 1;
@@ -112,20 +118,23 @@ out:
 int
 server_getspec (rpcsvc_request_t *req)
 {
-        int32_t               ret = -1;
-        int32_t               op_errno = 0;
-        int32_t               spec_fd = -1;
-        size_t                file_len = 0;
-        char                  filename[ZR_PATH_MAX] = {0,};
-        struct stat           stbuf = {0,};
-        char                 *volume = NULL;
-        int                   cookie = 0;
-
-        gf_getspec_req    args = {0,};
-        gf_getspec_rsp    rsp  = {0,};
-
-
-        if (!xdr_to_generic (req->msg[0], &args, (xdrproc_t)xdr_gf_getspec_req)) {
+        int32_t               ret                    = -1;
+        int32_t               op_errno               = 0;
+        int32_t               spec_fd                = -1;
+        size_t                file_len               = 0;
+        char                  filename[ZR_PATH_MAX]  = {0,};
+        struct stat           stbuf                  = {0,};
+        char                 *volume                 = NULL;
+        char                 *tmp                    = NULL;
+        int                   cookie                 = 0;
+        rpc_transport_t      *trans                  = NULL;
+        gf_getspec_req        args                   = {0,};
+        gf_getspec_rsp        rsp                    = {0,};
+        char                  addrstr[RPCSVC_PEER_STRLEN] = {0};
+
+
+        if (!xdr_to_generic (req->msg[0], &args,
+                             (xdrproc_t)xdr_gf_getspec_req)) {
                 //failed to decode msg;
                 req->rpc_err = GARBAGE_ARGS;
                 goto fail;
@@ -133,7 +142,25 @@ server_getspec (rpcsvc_request_t *req)
 
         volume = args.key;
 
-        ret = build_volfile_path (volume, filename, sizeof (filename));
+        trans = req->trans;
+        ret = rpcsvc_transport_peername (trans, (char *)&addrstr,
+                                         sizeof (addrstr));
+        if (ret)
+                goto fail;
+
+        tmp = strrchr (addrstr, ':');
+        *tmp = '\0';
+
+        /* we trust the local admin */
+        if (!glusterd_is_local_addr (addrstr)) {
+
+                ret = build_volfile_path (volume, filename,
+                                          sizeof (filename),
+                                          TRUSTED_PREFIX);
+        } else {
+                ret = build_volfile_path (volume, filename,
+                                          sizeof (filename), NULL);
+        }
 
         if (ret > 0) {
                 /* to allocate the proper buffer to hold the file data */
diff --git a/xlators/mgmt/glusterd/src/glusterd-op-sm.c b/xlators/mgmt/glusterd/src/glusterd-op-sm.c
index 2a4bf82ee69..77ed83f8bbf 100644
--- a/xlators/mgmt/glusterd/src/glusterd-op-sm.c
+++ b/xlators/mgmt/glusterd/src/glusterd-op-sm.c
@@ -918,39 +918,38 @@ glusterd_op_set_volume (dict_t *dict)
 
         ret = dict_get_int32 (dict, "count", &dict_count);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Count(dict),not set in Volume-Set");
+                gf_log (this->name, GF_LOG_ERROR, "Count(dict),not set in Volume-Set");
                 goto out;
         }
 
-        if ( dict_count == 0 ) {
+        if (dict_count == 0) {
                 ret = glusterd_volset_help (dict);
                 if (ret)
-                        gf_log ("glusterd", GF_LOG_ERROR, "Volume set help"
-                                                        "internal error");
+                        gf_log (this->name, GF_LOG_ERROR, "Volume set"
+                                " help internal error");
                 goto out;
         }
 
         ret = dict_get_str (dict, "volname", &volname);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get volume name");
+                gf_log (this->name, GF_LOG_ERROR, "Unable to get volume name");
                 goto out;
         }
 
         ret = glusterd_volinfo_find (volname, &volinfo);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to allocate memory");
+                gf_log (this->name, GF_LOG_ERROR, "Unable to allocate memory");
                 goto out;
         }
 
-        for ( count = 1; ret != -1 ; count++ ) {
+        for (count = 1; ret != -1 ; count++) {
 
                 global_opt = _gf_false;
                 sprintf (str, "key%d", count);
                 ret = dict_get_str (dict, str, &key);
 
-                if (ret) {
+                if (ret)
                         break;
-                }
 
                 if (!ret) {
                         ret = glusterd_check_option_exists (key, &key_fixed);
@@ -969,7 +968,7 @@ glusterd_op_set_volume (dict_t *dict)
                 sprintf (str, "value%d", count);
                 ret = dict_get_str (dict, str, &value);
                 if (ret) {
-                        gf_log ("", GF_LOG_ERROR,
+                        gf_log (this->name, GF_LOG_ERROR,
                                 "invalid key,value pair in 'volume set'");
                         ret = -1;
                         goto out;
@@ -979,7 +978,7 @@ glusterd_op_set_volume (dict_t *dict)
                         value = gf_strdup (value);
 
                 if (!value) {
-                        gf_log ("", GF_LOG_ERROR,
+                        gf_log (this->name, GF_LOG_ERROR,
                                 "Unable to set the options in 'volume set'");
                         ret = -1;
                         goto out;
@@ -995,8 +994,7 @@ glusterd_op_set_volume (dict_t *dict)
                                if (ret)
                                        goto out;
                        }
-                }
-                else {
+                } else {
                         ret = dict_set_dynstr (volinfo->dict, key, value);
                         if (ret)
                                 goto out;
@@ -1004,13 +1002,12 @@ glusterd_op_set_volume (dict_t *dict)
 
                 if (key_fixed) {
                         GF_FREE (key_fixed);
-
                         key_fixed = NULL;
                 }
         }
 
         if ( count == 1 ) {
-                gf_log ("", GF_LOG_ERROR, "No options received ");
+                gf_log (this->name, GF_LOG_ERROR, "No options received ");
                 ret = -1;
                 goto out;
         }
@@ -1018,7 +1015,8 @@ glusterd_op_set_volume (dict_t *dict)
         if (!global_opt) {
                 ret = glusterd_create_volfiles_and_notify_services (volinfo);
                 if (ret) {
-                        gf_log ("", GF_LOG_ERROR, "Unable to create volfile for"
+                        gf_log (this->name, GF_LOG_ERROR,
+                                "Unable to create volfile for"
                                 " 'volume set'");
                         ret = -1;
                         goto out;
@@ -1031,19 +1029,19 @@ glusterd_op_set_volume (dict_t *dict)
                 if (GLUSTERD_STATUS_STARTED == volinfo->status) {
                         ret = glusterd_nodesvcs_handle_reconfigure (volinfo);
                         if (ret) {
-                                gf_log ("", GF_LOG_WARNING,
+                                gf_log (this->name, GF_LOG_WARNING,
                                          "Unable to restart NFS-Server");
                                 goto out;
                         }
                 }
 
-        }
-        else {
+        } else {
                 list_for_each_entry (voliter, &priv->volumes, vol_list) {
                         volinfo = voliter;
                         ret = glusterd_create_volfiles_and_notify_services (volinfo);
                         if (ret) {
-                                gf_log ("", GF_LOG_ERROR, "Unable to create volfile for"
+                                gf_log (this->name, GF_LOG_ERROR,
+                                        "Unable to create volfile for"
                                         " 'volume set'");
                                 ret = -1;
                                 goto out;
@@ -1057,7 +1055,7 @@ glusterd_op_set_volume (dict_t *dict)
                         if (GLUSTERD_STATUS_STARTED == volinfo->status) {
                                 ret = glusterd_nodesvcs_handle_reconfigure (volinfo);
                                 if (ret) {
-                                        gf_log ("", GF_LOG_WARNING,
+                                        gf_log (this->name, GF_LOG_WARNING,
                                                 "Unable to restart NFS-Server");
                                         goto out;
                                 }
@@ -1069,7 +1067,7 @@ glusterd_op_set_volume (dict_t *dict)
  out:
         if (key_fixed)
                 GF_FREE (key_fixed);
-        gf_log ("", GF_LOG_DEBUG, "returning %d", ret);
+        gf_log (this->name, GF_LOG_DEBUG, "returning %d", ret);
         return ret;
 }
 
diff --git a/xlators/mgmt/glusterd/src/glusterd-quota.c b/xlators/mgmt/glusterd/src/glusterd-quota.c
index 32264b7b6e7..b063421ec1a 100644
--- a/xlators/mgmt/glusterd/src/glusterd-quota.c
+++ b/xlators/mgmt/glusterd/src/glusterd-quota.c
@@ -246,11 +246,11 @@ out:
 int32_t
 glusterd_quota_initiate_fs_crawl (glusterd_conf_t *priv, char *volname)
 {
-        int32_t   ret = 0;
-        pid_t     pid;
-        char      mountdir [] = "/tmp/mntXXXXXX";
-        runner_t  runner = {0,};
-        int       status = 0;
+        pid_t                      pid;
+        int32_t                    ret              = 0;
+        int                        status           = 0;
+        char                       mountdir[]       = "/tmp/mntXXXXXX";
+        runner_t                   runner           = {0};
 
         if (mkdtemp (mountdir) == NULL) {
                 gf_log ("glusterd", GF_LOG_DEBUG,
@@ -260,9 +260,10 @@ glusterd_quota_initiate_fs_crawl (glusterd_conf_t *priv, char *volname)
         }
 
         runinit (&runner);
-        runner_add_args (&runner, SBIN_DIR"/glusterfs", "-s",
-                         "localhost", "--volfile-id", volname, "-l",
-                         DEFAULT_LOG_FILE_DIRECTORY"/quota-crawl.log",
+        runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                         "-s", "localhost",
+                         "--volfile-id", volname,
+                         "-l", DEFAULT_LOG_FILE_DIRECTORY"/quota-crawl.log",
                          mountdir, NULL);
 
         ret = runner_run_reuse (&runner);
diff --git a/xlators/mgmt/glusterd/src/glusterd-replace-brick.c b/xlators/mgmt/glusterd/src/glusterd-replace-brick.c
index 40202dd82ca..01bc974b75f 100644
--- a/xlators/mgmt/glusterd/src/glusterd-replace-brick.c
+++ b/xlators/mgmt/glusterd/src/glusterd-replace-brick.c
@@ -711,6 +711,8 @@ static const char *client_volfile_str =  "volume mnt-client\n"
         " option remote-subvolume %s\n"
         " option remote-port %d\n"
         " option transport-type %s\n"
+        " option username %s\n"
+        " option password %s\n"
         "end-volume\n"
         "volume mnt-wb\n"
         " type performance/write-behind\n"
@@ -722,23 +724,33 @@ rb_generate_client_volfile (glusterd_volinfo_t *volinfo,
                             glusterd_brickinfo_t *src_brickinfo)
 {
         glusterd_conf_t  *priv                  = NULL;
+        xlator_t         *this                  = NULL;
         FILE             *file                  = NULL;
         char              filename[PATH_MAX]    = {0, };
         int               ret                   = -1;
+        int               fd                    = -1;
         char             *ttype                 = NULL;
 
-        priv = THIS->private;
+        this = THIS;
+        priv = this->private;
 
-        gf_log ("", GF_LOG_DEBUG,
-                "Creating volfile");
+        gf_log (this->name, GF_LOG_DEBUG, "Creating volfile");
 
         snprintf (filename, PATH_MAX, "%s/vols/%s/%s",
                   priv->workdir, volinfo->volname,
                   RB_CLIENTVOL_FILENAME);
 
+        fd = open (filename, O_CREAT | O_RDONLY, S_IRUSR | S_IWUSR);
+        if (fd < 0) {
+                gf_log (this->name, GF_LOG_ERROR,
+                        "%s", strerror (errno));
+                goto out;
+        }
+        close (fd);
+
         file = fopen (filename, "w+");
         if (!file) {
-                gf_log ("", GF_LOG_DEBUG,
+                gf_log (this->name, GF_LOG_DEBUG,
                         "Open of volfile failed");
                 ret = -1;
                 goto out;
@@ -753,7 +765,10 @@ rb_generate_client_volfile (glusterd_volinfo_t *volinfo,
 	}
 
         fprintf (file, client_volfile_str, src_brickinfo->hostname,
-                 src_brickinfo->path, src_brickinfo->port, ttype);
+                 src_brickinfo->path,
+                 src_brickinfo->port, ttype,
+                 glusterd_auth_get_username (volinfo),
+                 glusterd_auth_get_password (volinfo));
 
         fclose (file);
         GF_FREE (ttype);
@@ -775,6 +790,8 @@ static const char *dst_brick_volfile_str = "volume src-posix\n"
         "end-volume\n"
         "volume src-server\n"
         " type protocol/server\n"
+        " option auth.login.%s.allow %s\n"
+        " option auth.login.%s.password %s\n"
         " option auth.addr.%s.allow *\n"
         " option transport-type %s\n"
         " subvolumes %s\n"
@@ -785,23 +802,34 @@ rb_generate_dst_brick_volfile (glusterd_volinfo_t *volinfo,
                                glusterd_brickinfo_t *dst_brickinfo)
 {
         glusterd_conf_t    *priv                = NULL;
+        xlator_t           *this                = NULL;
         FILE               *file                = NULL;
         char                filename[PATH_MAX]  = {0, };
         int                 ret                 = -1;
+        int                 fd                  = -1;
         char               *trans_type          = NULL;
 
-        priv = THIS->private;
+        this = THIS;
+        priv = this->private;
 
-        gf_log ("", GF_LOG_DEBUG,
+        gf_log (this->name, GF_LOG_DEBUG,
                 "Creating volfile");
 
         snprintf (filename, PATH_MAX, "%s/vols/%s/%s",
                   priv->workdir, volinfo->volname,
                   RB_DSTBRICKVOL_FILENAME);
 
+        fd = creat (filename, S_IRUSR | S_IWUSR);
+        if (fd < 0) {
+                gf_log (this->name, GF_LOG_ERROR,
+                        "%s", strerror (errno));
+                goto out;
+        }
+        close (fd);
+
         file = fopen (filename, "w+");
         if (!file) {
-                gf_log ("", GF_LOG_DEBUG,
+                gf_log (this->name, GF_LOG_DEBUG,
                         "Open of volfile failed");
                 ret = -1;
                 goto out;
@@ -813,10 +841,17 @@ rb_generate_dst_brick_volfile (glusterd_volinfo_t *volinfo,
 		goto out;
 	}
 
-        fprintf (file, dst_brick_volfile_str, dst_brickinfo->path,
+        fprintf (file, dst_brick_volfile_str,
+                 dst_brickinfo->path,
                  uuid_utoa (volinfo->volume_id),
-                 dst_brickinfo->path, dst_brickinfo->path,
-                 trans_type, dst_brickinfo->path);
+                 dst_brickinfo->path,
+                 dst_brickinfo->path,
+                 glusterd_auth_get_username (volinfo),
+                 glusterd_auth_get_username (volinfo),
+                 glusterd_auth_get_password (volinfo),
+                 dst_brickinfo->path,
+                 trans_type,
+                 dst_brickinfo->path);
 
 	GF_FREE (trans_type);
 
diff --git a/xlators/mgmt/glusterd/src/glusterd-store.c b/xlators/mgmt/glusterd/src/glusterd-store.c
index 18d60d0a428..900c12a82da 100644
--- a/xlators/mgmt/glusterd/src/glusterd-store.c
+++ b/xlators/mgmt/glusterd/src/glusterd-store.c
@@ -96,7 +96,7 @@ glusterd_store_mkstemp (glusterd_store_handle_t *shandle)
         GF_ASSERT (shandle->path);
 
         snprintf (tmppath, sizeof (tmppath), "%s.tmp", shandle->path);
-        fd = open (tmppath, O_RDWR | O_CREAT | O_TRUNC, 0644);
+        fd = open (tmppath, O_RDWR | O_CREAT | O_TRUNC, 0600);
         if (fd <= 0) {
                 gf_log ("glusterd", GF_LOG_ERROR, "Failed to open %s, "
                         "error: %s", tmppath, strerror (errno));
@@ -631,6 +631,16 @@ glusterd_volume_exclude_options_write (int fd, glusterd_volinfo_t *volinfo)
         if (ret)
                 goto out;
 
+        ret = glusterd_store_save_value (fd, GLUSTERD_STORE_KEY_USERNAME,
+                                         glusterd_auth_get_username (volinfo));
+        if (ret)
+                goto out;
+
+        ret = glusterd_store_save_value (fd, GLUSTERD_STORE_KEY_PASSWORD,
+                                         glusterd_auth_get_password (volinfo));
+        if (ret)
+                goto out;
+
 out:
         if (ret)
                 gf_log ("", GF_LOG_ERROR, "Unable to write volume values"
@@ -1174,7 +1184,7 @@ glusterd_store_handle_new (char *path, glusterd_store_handle_t **handle)
         if (!spath)
                 goto out;
 
-        fd = open (path, O_RDWR | O_CREAT | O_APPEND, 0644);
+        fd = open (path, O_RDWR | O_CREAT | O_APPEND, 0600);
         if (fd <= 0) {
                 gf_log ("glusterd", GF_LOG_ERROR, "Failed to open file: %s, "
                         "error: %s", path, strerror (errno));
@@ -1269,7 +1279,7 @@ glusterd_store_uuid ()
                 handle = priv->handle;
         }
 
-        handle->fd = open (handle->path, O_RDWR | O_CREAT | O_TRUNC, 0644);
+        handle->fd = open (handle->path, O_RDWR | O_CREAT | O_TRUNC, 0600);
         if (handle->fd <= 0) {
                 ret = -1;
                 goto out;
@@ -1859,6 +1869,16 @@ glusterd_store_retrieve_volume (char    *volname)
                                 gf_log ("", GF_LOG_WARNING,
                                         "failed to parse uuid");
 
+                } else if (!strncmp (key, GLUSTERD_STORE_KEY_USERNAME,
+                                     strlen (GLUSTERD_STORE_KEY_USERNAME))) {
+
+                        glusterd_auth_set_username (volinfo, value);
+
+                } else if (!strncmp (key, GLUSTERD_STORE_KEY_PASSWORD,
+                                     strlen (GLUSTERD_STORE_KEY_PASSWORD))) {
+
+                        glusterd_auth_set_password (volinfo, value);
+
                 } else if (strstr (key, "slave")) {
                         ret = dict_set_dynstr (volinfo->gsync_slaves, key,
                                                 gf_strdup (value));
diff --git a/xlators/mgmt/glusterd/src/glusterd-store.h b/xlators/mgmt/glusterd/src/glusterd-store.h
index f55fb8c2e48..b381e5a0cce 100644
--- a/xlators/mgmt/glusterd/src/glusterd-store.h
+++ b/xlators/mgmt/glusterd/src/glusterd-store.h
@@ -60,6 +60,8 @@ typedef enum glusterd_store_ver_ac_{
 #define GLUSTERD_STORE_KEY_RB_SRC_BRICK   "rb_src"
 #define GLUSTERD_STORE_KEY_RB_DST_BRICK   "rb_dst"
 #define GLUSTERD_STORE_KEY_VOL_DEFRAG     "rebalance_status"
+#define GLUSTERD_STORE_KEY_USERNAME       "username"
+#define GLUSTERD_STORE_KEY_PASSWORD       "password"
 
 #define GLUSTERD_STORE_KEY_BRICK_HOSTNAME "hostname"
 #define GLUSTERD_STORE_KEY_BRICK_PATH     "path"
diff --git a/xlators/mgmt/glusterd/src/glusterd-utils.c b/xlators/mgmt/glusterd/src/glusterd-utils.c
index 9ec9e16f18d..4ec8ae5dc2a 100644
--- a/xlators/mgmt/glusterd/src/glusterd-utils.c
+++ b/xlators/mgmt/glusterd/src/glusterd-utils.c
@@ -194,11 +194,13 @@ glusterd_is_local_addr (char *hostname)
         int32_t         found = 0;
         int             sd = -1;
         char            *ip = NULL;
+        xlator_t        *this = NULL;
 
+        this = THIS;
         ret = getaddrinfo (hostname, NULL, NULL, &result);
 
         if (ret != 0) {
-                gf_log ("", GF_LOG_ERROR, "error in getaddrinfo: %s\n",
+                gf_log (this->name, GF_LOG_ERROR, "error in getaddrinfo: %s\n",
                         gai_strerror(ret));
                 goto out;
         }
@@ -210,7 +212,8 @@ glusterd_is_local_addr (char *hostname)
         }
 
         for (res = result; res != NULL; res = res->ai_next) {
-                gf_log ("glusterd", GF_LOG_DEBUG, "%s ", get_ip_from_addrinfo (res, &ip));
+                gf_log (this->name, GF_LOG_DEBUG, "%s ",
+                        get_ip_from_addrinfo (res, &ip));
                 sd = socket (res->ai_family, SOCK_DGRAM, 0);
                 if (sd == -1)
                         goto out;
@@ -218,7 +221,8 @@ glusterd_is_local_addr (char *hostname)
                 ret = bind (sd, res->ai_addr, res->ai_addrlen);
                 if (ret == 0) {
                         found = _gf_true;
-                        gf_log ("glusterd", GF_LOG_INFO, "%s is local", get_ip_from_addrinfo (res, &ip));
+                        gf_log (this->name, GF_LOG_DEBUG, "%s is local",
+                                get_ip_from_addrinfo (res, &ip));
                         close (sd);
                         break;
                 }
@@ -229,10 +233,8 @@ out:
         if (result)
                 freeaddrinfo (result);
 
-        if (found)
-                gf_log ("glusterd", GF_LOG_DEBUG, "%s is local", hostname);
-        else
-                gf_log ("glusterd", GF_LOG_DEBUG, "%s is not local", hostname);
+        if (!found)
+                gf_log (this->name, GF_LOG_DEBUG, "%s is not local", hostname);
 
         return !found;
 }
@@ -566,6 +568,56 @@ out:
         return ret;
 }
 
+void
+glusterd_auth_cleanup (glusterd_volinfo_t *volinfo) {
+
+        GF_ASSERT (volinfo);
+
+        if (volinfo->auth.username)
+                GF_FREE (volinfo->auth.username);
+
+        if (volinfo->auth.password)
+                GF_FREE (volinfo->auth.password);
+}
+
+char *
+glusterd_auth_get_username (glusterd_volinfo_t *volinfo) {
+
+        GF_ASSERT (volinfo);
+        GF_ASSERT (volinfo->auth.username);
+
+        return volinfo->auth.username;
+}
+
+char *
+glusterd_auth_get_password (glusterd_volinfo_t *volinfo) {
+
+        GF_ASSERT (volinfo);
+        GF_ASSERT (volinfo->auth.password);
+
+        return volinfo->auth.password;
+}
+
+int32_t
+glusterd_auth_set_username (glusterd_volinfo_t *volinfo, char *username) {
+
+        GF_ASSERT (volinfo);
+        GF_ASSERT (username);
+
+        volinfo->auth.username = gf_strdup (username);
+        return 0;
+}
+
+int32_t
+glusterd_auth_set_password (glusterd_volinfo_t *volinfo, char *password) {
+
+        GF_ASSERT (volinfo);
+        GF_ASSERT (password);
+
+        volinfo->auth.password = gf_strdup (password);
+        return 0;
+}
+
 int32_t
 glusterd_brickinfo_delete (glusterd_brickinfo_t *brickinfo)
 {
@@ -624,6 +676,8 @@ glusterd_volinfo_delete (glusterd_volinfo_t *volinfo)
         if (volinfo->logdir)
                 GF_FREE (volinfo->logdir);
 
+        glusterd_auth_cleanup (volinfo);
+
         GF_FREE (volinfo);
         ret = 0;
 
@@ -1328,7 +1382,7 @@ glusterd_volume_compute_cksum (glusterd_volinfo_t  *volinfo)
         snprintf (cksum_path, sizeof (cksum_path), "%s/%s",
                   path, GLUSTERD_CKSUM_FILE);
 
-        fd = open (cksum_path, O_RDWR | O_APPEND | O_CREAT| O_TRUNC, 0644);
+        fd = open (cksum_path, O_RDWR | O_APPEND | O_CREAT| O_TRUNC, 0600);
 
         if (-1 == fd) {
                 gf_log (THIS->name, GF_LOG_ERROR, "Unable to open %s, errno: %d",
@@ -1432,6 +1486,7 @@ glusterd_add_volume_to_dict (glusterd_volinfo_t *volinfo,
         char                    *volume_id_str  = NULL;
         char                    *src_brick      = NULL;
         char                    *dst_brick      = NULL;
+        char                    *str            = NULL;
         glusterd_voldict_ctx_t   ctx            = {0};
 
         GF_ASSERT (dict);
@@ -1513,6 +1568,28 @@ glusterd_add_volume_to_dict (glusterd_volinfo_t *volinfo,
                 goto out;
 
         memset (key, 0, sizeof (key));
+        snprintf (key, sizeof (key), "volume%d.username", count);
+        str = glusterd_auth_get_username (volinfo);
+        if (!str) {
+                ret = -1;
+                goto out;
+        }
+        ret = dict_set_dynstr (dict, key, gf_strdup (str));
+        if (ret)
+                goto out;
+
+        memset (key, 0, sizeof (key));
+        snprintf (key, sizeof (key), "volume%d.password", count);
+        str = glusterd_auth_get_password (volinfo);
+        if (!str) {
+                ret = -1;
+                goto out;
+        }
+        ret = dict_set_dynstr (dict, key, gf_strdup (str));
+        if (ret)
+                goto out;
+
+        memset (key, 0, sizeof (key));
         snprintf (key, 256, "volume%d."GLUSTERD_STORE_KEY_RB_STATUS, count);
         ret = dict_set_int32 (dict, key, volinfo->rb_status);
         if (ret)
@@ -1899,6 +1976,7 @@ glusterd_import_volinfo (dict_t *vols, int count,
         char               msg[2048]         = {0};
         char               *src_brick        = NULL;
         char               *dst_brick        = NULL;
+        char               *str              = NULL;
         int                rb_status         = 0;
 
         GF_ASSERT (vols);
@@ -2008,6 +2086,32 @@ glusterd_import_volinfo (dict_t *vols, int count,
         }
 
         memset (key, 0, sizeof (key));
+        snprintf (key, sizeof (key), "volume%d.username", count);
+        ret = dict_get_str (vols, key, &str);
+        if (ret) {
+                snprintf (msg, sizeof (msg),
+                          "%s missing in payload for %s",
+                          key, volname);
+                goto out;
+        }
+        ret = glusterd_auth_set_username (new_volinfo, str);
+        if (ret)
+                goto out;
+
+        memset (key, 0, sizeof (key));
+        snprintf (key, sizeof (key), "volume%d.password", count);
+        ret = dict_get_str (vols, key, &str);
+        if (ret) {
+                snprintf (msg, sizeof (msg),
+                          "%s missing in payload for %s",
+                          key, volname);
+                goto out;
+        }
+        ret = glusterd_auth_set_password (new_volinfo, str);
+        if (ret)
+                goto out;
+
+        memset (key, 0, sizeof (key));
         snprintf (key, sizeof (key), "volume%d.transport_type", count);
         ret = dict_get_uint32 (vols, key, &new_volinfo->transport_type);
         if (ret) {
@@ -2524,16 +2628,16 @@ out:
 int32_t
 glusterd_nodesvc_start (char *server, gf_boolean_t pmap_signin)
 {
-        int32_t                 ret = -1;
-        xlator_t                *this = NULL;
-        glusterd_conf_t         *priv = NULL;
-        runner_t                runner = {0,};
-        char                    pidfile[PATH_MAX] = {0,};
-        char                    logfile[PATH_MAX] = {0,};
-        char                    volfile[PATH_MAX] = {0,};
-        char                    rundir[PATH_MAX] = {0,};
-        char                    shd_sockfpath[PATH_MAX] = {0,};
-        char                    volfileid[256]   = {0};
+        int32_t                 ret                        = -1;
+        xlator_t               *this                       = NULL;
+        glusterd_conf_t        *priv                       = NULL;
+        runner_t                runner                     = {0,};
+        char                    pidfile[PATH_MAX]          = {0,};
+        char                    logfile[PATH_MAX]          = {0,};
+        char                    volfile[PATH_MAX]          = {0,};
+        char                    rundir[PATH_MAX]           = {0,};
+        char                    shd_sockfpath[PATH_MAX]    = {0,};
+        char                    volfileid[256]             = {0};
 #ifdef DEBUG
         char                    valgrind_logfile[PATH_MAX] = {0};
 #endif
@@ -2544,7 +2648,7 @@ glusterd_nodesvc_start (char *server, gf_boolean_t pmap_signin)
         priv = this->private;
 
         glusterd_get_nodesvc_rundir (server, priv->workdir,
-                                           rundir, sizeof (rundir));
+                                     rundir, sizeof (rundir));
         ret = mkdir (rundir, 0777);
 
         if ((ret == -1) && (EEXIST != errno)) {
@@ -2554,9 +2658,9 @@ glusterd_nodesvc_start (char *server, gf_boolean_t pmap_signin)
         }
 
         glusterd_get_nodesvc_pidfile (server, priv->workdir,
-                                            pidfile, sizeof (pidfile));
+                                      pidfile, sizeof (pidfile));
         glusterd_get_nodesvc_volfile (server, priv->workdir,
-                                            volfile, sizeof (volfile));
+                                      volfile, sizeof (volfile));
         ret = access (volfile, F_OK);
         if (ret) {
                 gf_log ("", GF_LOG_ERROR, "%s Volfile %s is not present",
@@ -2592,17 +2696,21 @@ glusterd_nodesvc_start (char *server, gf_boolean_t pmap_signin)
 #endif
 
         if (pmap_signin) {
-                runner_add_args (&runner, SBIN_DIR"/glusterfs", "-s",
-                                 "localhost", "--volfile-id", volfileid,
-                                 "-p", pidfile, "-l", logfile,
+                runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                                 "-s", "localhost",
+                                 "--volfile-id", volfileid,
+                                 "-p", pidfile,
+                                 "-l", logfile,
                                  "-S", shd_sockfpath, NULL);
         } else {
-                runner_add_args (&runner, SBIN_DIR"/glusterfs", "-f", volfile,
-                                 "-p", pidfile, "-l", logfile, NULL);
+                runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                                 "-f", volfile,
+                                 "-p", pidfile,
+                                 "-l", logfile, NULL);
         }
 
-        runner_log (&runner, "", GF_LOG_DEBUG, "Starting the nfs/glustershd "
-                    "services");
+        runner_log (&runner, "", GF_LOG_DEBUG,
+                    "Starting the nfs/glustershd services");
 
         ret = runner_run (&runner);
         if (ret == 0) {
@@ -2816,8 +2924,8 @@ int
 glusterd_nodesvcs_start (glusterd_volinfo_t *volinfo)
 {
         return glusterd_nodesvcs_batch_op (volinfo,
-                                            glusterd_nfs_server_start,
-                                            glusterd_shd_start);
+                                           glusterd_nfs_server_start,
+                                           glusterd_shd_start);
 }
 
 int
@@ -3603,12 +3711,15 @@ glusterd_friend_find_by_hostname (const char *hoststr,
         struct sockaddr_in      *s4 = NULL;
         struct in_addr          *in_addr = NULL;
         char                    hname[1024] = {0,};
+        xlator_t                *this  = NULL;
+
 
+        this = THIS;
         GF_ASSERT (hoststr);
         GF_ASSERT (peerinfo);
 
         *peerinfo = NULL;
-        priv    = THIS->private;
+        priv    = this->private;
 
         GF_ASSERT (priv);
 
@@ -3616,7 +3727,7 @@ glusterd_friend_find_by_hostname (const char *hoststr,
                 if (!strncasecmp (entry->hostname, hoststr,
                                   1024)) {
 
-                        gf_log ("glusterd", GF_LOG_DEBUG,
+                        gf_log (this->name, GF_LOG_DEBUG,
                                  "Friend %s found.. state: %d", hoststr,
                                   entry->state.state);
                         *peerinfo = entry;
@@ -3624,9 +3735,10 @@ glusterd_friend_find_by_hostname (const char *hoststr,
                 }
         }
 
-        ret = getaddrinfo(hoststr, NULL, NULL, &addr);
+        ret = getaddrinfo (hoststr, NULL, NULL, &addr);
         if (ret != 0) {
-                gf_log ("", GF_LOG_ERROR, "error in getaddrinfo: %s\n",
+                gf_log (this->name, GF_LOG_ERROR,
+                        "error in getaddrinfo: %s\n",
                         gai_strerror(ret));
                 goto out;
         }
@@ -3655,7 +3767,7 @@ glusterd_friend_find_by_hostname (const char *hoststr,
                         if (!strncasecmp (entry->hostname, host,
                             1024) || !strncasecmp (entry->hostname,hname,
                             1024)) {
-                                gf_log ("glusterd", GF_LOG_DEBUG,
+                                gf_log (this->name, GF_LOG_DEBUG,
                                         "Friend %s found.. state: %d",
                                         hoststr, entry->state.state);
                                 *peerinfo = entry;
@@ -3666,7 +3778,7 @@ glusterd_friend_find_by_hostname (const char *hoststr,
         }
 
 out:
-        gf_log ("glusterd", GF_LOG_DEBUG, "Unable to find friend: %s", hoststr);
+        gf_log (this->name, GF_LOG_DEBUG, "Unable to find friend: %s", hoststr);
         if (addr)
                 freeaddrinfo (addr);
         return -1;
@@ -4827,6 +4939,29 @@ glusterd_get_client_filepath (char *filepath, glusterd_volinfo_t *volinfo,
                           path, volinfo->volname);
 }
 
+void
+glusterd_get_trusted_client_filepath (char *filepath,
+                                      glusterd_volinfo_t *volinfo,
+                                      gf_transport_type type)
+{
+        char  path[PATH_MAX] = {0,};
+        glusterd_conf_t *priv = NULL;
+
+        priv = THIS->private;
+
+        GLUSTERD_GET_VOLUME_DIR (path, volinfo, priv);
+
+        if ((volinfo->transport_type == GF_TRANSPORT_BOTH_TCP_RDMA) &&
+            (type == GF_TRANSPORT_RDMA))
+                snprintf (filepath, PATH_MAX,
+                          "%s/trusted-%s.rdma-fuse.vol",
+                          path, volinfo->volname);
+        else
+                snprintf (filepath, PATH_MAX,
+                          "%s/trusted-%s-fuse.vol",
+                          path, volinfo->volname);
+}
+
 int
 glusterd_volume_defrag_restart (glusterd_volinfo_t *volinfo, char *op_errstr,
                               size_t len, int cmd, defrag_cbk_fn_t cbk)
diff --git a/xlators/mgmt/glusterd/src/glusterd-utils.h b/xlators/mgmt/glusterd/src/glusterd-utils.h
index e52b25e3199..f71ecc404b6 100644
--- a/xlators/mgmt/glusterd/src/glusterd-utils.h
+++ b/xlators/mgmt/glusterd/src/glusterd-utils.h
@@ -87,6 +87,21 @@ glusterd_submit_request (struct rpc_clnt *rpc, void *req,
 int32_t
 glusterd_volinfo_new (glusterd_volinfo_t **volinfo);
 
+char *
+glusterd_auth_get_username (glusterd_volinfo_t *volinfo);
+
+char *
+glusterd_auth_get_password (glusterd_volinfo_t *volinfo);
+
+int32_t
+glusterd_auth_set_username (glusterd_volinfo_t *volinfo, char *username);
+
+int32_t
+glusterd_auth_set_password (glusterd_volinfo_t *volinfo, char *password);
+
+void
+glusterd_auth_cleanup (glusterd_volinfo_t *volinfo);
+
 gf_boolean_t
 glusterd_check_volume_exists (char *volname);
 
@@ -274,6 +289,7 @@ glusterd_new_brick_validate (char *brick, glusterd_brickinfo_t *brickinfo,
                              char *op_errstr, size_t len);
 int32_t
 glusterd_volume_brickinfos_delete (glusterd_volinfo_t *volinfo);
+
 int32_t
 glusterd_volume_brickinfo_get (uuid_t uuid, char *hostname, char *path,
                                glusterd_volinfo_t *volinfo,
@@ -383,9 +399,15 @@ glusterd_friend_remove_cleanup_vols (uuid_t uuid);
 
 gf_boolean_t
 glusterd_chk_peers_connected_befriended (uuid_t skip_uuid);
+
 void
-glusterd_get_client_filepath (char *filepath, glusterd_volinfo_t *volinfo,
+glusterd_get_client_filepath (char *filepath,
+                              glusterd_volinfo_t *volinfo,
                               gf_transport_type type);
+void
+glusterd_get_trusted_client_filepath (char *filepath,
+                                      glusterd_volinfo_t *volinfo,
+                                      gf_transport_type type);
 int
 glusterd_restart_rebalance (glusterd_conf_t *conf);
 #endif
diff --git a/xlators/mgmt/glusterd/src/glusterd-volgen.c b/xlators/mgmt/glusterd/src/glusterd-volgen.c
index fe79ea487b3..919994b1572 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volgen.c
+++ b/xlators/mgmt/glusterd/src/glusterd-volgen.c
@@ -1117,8 +1117,12 @@ free_fp:
 static int
 volgen_write_volfile (volgen_graph_t *graph, char *filename)
 {
-        char *ftmp = NULL;
-        FILE *f = NULL;
+        char        *ftmp = NULL;
+        FILE        *f = NULL;
+        int          fd   = 0;
+        xlator_t    *this = NULL;
+
+        this = THIS;
 
         if (gf_asprintf (&ftmp, "%s.tmp", filename) == -1) {
                 ftmp = NULL;
@@ -1126,6 +1130,15 @@ volgen_write_volfile (volgen_graph_t *graph, char *filename)
                 goto error;
         }
 
+        fd = creat (ftmp, S_IRUSR | S_IWUSR);
+        if (fd < 0) {
+                gf_log (this->name, GF_LOG_ERROR, "%s",
+                        strerror (errno));
+                goto error;
+        }
+
+        close (fd);
+
         f = fopen (ftmp, "w");
         if (!f)
                 goto error;
@@ -1153,7 +1166,8 @@ volgen_write_volfile (volgen_graph_t *graph, char *filename)
         if (f)
                 fclose (f);
 
-        gf_log ("", GF_LOG_ERROR, "failed to create volfile %s", filename);
+        gf_log (this->name, GF_LOG_ERROR,
+                "failed to create volfile %s", filename);
 
         return -1;
 }
@@ -1497,6 +1511,7 @@ server_graph_builder (volgen_graph_t *graph, glusterd_volinfo_t *volinfo,
         int       ret                   = 0;
         char     *xlator                = NULL;
         char     *loglevel              = NULL;
+        char      key[1024]             = {0};
 
         path = param;
         volname = volinfo->volname;
@@ -1556,6 +1571,16 @@ server_graph_builder (volgen_graph_t *graph, glusterd_volinfo_t *volinfo,
 		if (NULL == ptranst)
 			return -1;
 
+                ret = xlator_set_option (rbxl, "username",
+                                         glusterd_auth_get_username (volinfo));
+                if (ret)
+                        return -1;
+
+                ret = xlator_set_option (rbxl, "password",
+                                         glusterd_auth_get_password (volinfo));
+                if (ret)
+                        return -1;
+
                 ret = xlator_set_option (rbxl, "transport-type", ptranst);
                 GF_FREE (ptranst);
                 if (ret)
@@ -1596,6 +1621,22 @@ server_graph_builder (volgen_graph_t *graph, glusterd_volinfo_t *volinfo,
         if (ret)
                 return -1;
 
+        memset (key, 0, sizeof (key));
+        snprintf (key, sizeof (key), "auth.login.%s.allow", path);
+        ret = xlator_set_option (xl, key,
+                                 glusterd_auth_get_username (volinfo));
+        if (ret)
+                return -1;
+
+        memset (key, 0, sizeof (key));
+        snprintf (key, sizeof (key), "auth.login.%s.password",
+                  glusterd_auth_get_username (volinfo));
+
+        ret = xlator_set_option (xl, key,
+                                 glusterd_auth_get_password (volinfo));
+        if (ret)
+                return -1;
+
         ret = volgen_graph_set_options_generic (graph, set_dict,
                                                 (xlator && loglevel) ? (void *)set_dict : volinfo,
                                                 (xlator && loglevel) ?  &server_spec_extended_option_handler :
@@ -1922,10 +1963,12 @@ volgen_graph_build_clients (volgen_graph_t *graph, glusterd_volinfo_t *volinfo,
 {
         int                      i                  = 0;
         int                      ret                = -1;
+        uint32_t                 client_type        = GF_CLIENT_OTHER;
         char                     transt[16]         = {0,};
         char                    *volname            = NULL;
-        glusterd_brickinfo_t    *brick = NULL;
-        xlator_t                *xl                = NULL;
+        char                    *str                = NULL;
+        glusterd_brickinfo_t    *brick              = NULL;
+        xlator_t                *xl                 = NULL;
 
         volname = volinfo->volname;
 
@@ -1968,8 +2011,26 @@ volgen_graph_build_clients (volgen_graph_t *graph, glusterd_volinfo_t *volinfo,
                 ret = xlator_set_option (xl, "transport-type", transt);
                 if (ret)
                         goto out;
+
+                ret = dict_get_uint32 (set_dict, "trusted-client",
+                                       &client_type);
+
+                if (!ret && client_type == GF_CLIENT_TRUSTED) {
+
+                        str = glusterd_auth_get_username (volinfo);
+                        ret = xlator_set_option (xl, "username", str);
+                        if (ret)
+                                goto out;
+
+                        str = glusterd_auth_get_password (volinfo);
+                        ret = xlator_set_option (xl, "password", str);
+                        if (ret)
+                                goto out;
+                }
+
                 i++;
         }
+
         if (i != volinfo->brick_count) {
                 gf_log ("", GF_LOG_ERROR,
                         "volume inconsistency: actual number of bricks (%d) "
@@ -2501,7 +2562,7 @@ build_shd_graph (volgen_graph_t *graph, dict_t *mod_dict)
         int                ret            = 0;
         gf_boolean_t       valid_config   = _gf_false;
         xlator_t           *iostxl        = NULL;
-        int                rclusters       = 0;
+        int                rclusters      = 0;
         int                replica_count  = 0;
 
         this = THIS;
@@ -2534,6 +2595,11 @@ build_shd_graph (volgen_graph_t *graph, dict_t *mod_dict)
                 if (ret)
                         goto out;
 
+                ret = dict_set_uint32 (set_dict, "trusted-client",
+                                       GF_CLIENT_TRUSTED);
+                if (ret)
+                        goto out;
+
                 dict_copy (voliter->dict, set_dict);
                 if (mod_dict)
                         dict_copy (mod_dict, set_dict);
@@ -2650,12 +2716,21 @@ build_nfs_graph (volgen_graph_t *graph, dict_t *mod_dict)
                 if (ret)
                         goto out;
 
-                ret = dict_set_str (set_dict, "performance.client-io-threads", "off");
+                ret = dict_set_str (set_dict, "performance.client-io-threads",
+                                    "off");
                 if (ret)
                         goto out;
 
                 ret = dict_set_str (set_dict, "client-transport-type",
                                     nfs_xprt);
+                if (ret)
+                        goto out;
+
+                ret = dict_set_uint32 (set_dict, "trusted-client",
+                                       GF_CLIENT_TRUSTED);
+                if (ret)
+                        goto out;
+
                 ret = build_client_graph (&cgraph, voliter, set_dict);
                 if (ret)
                         goto out;
@@ -2814,7 +2889,7 @@ generate_brick_volfiles (glusterd_volinfo_t *volinfo)
         get_vol_tstamp_file (tstamp_file, volinfo);
 
         if (ret) {
-                ret = open (tstamp_file, O_WRONLY|O_CREAT|O_EXCL, 0644);
+                ret = open (tstamp_file, O_WRONLY|O_CREAT|O_EXCL, 0600);
                 if (ret == -1 && errno == EEXIST) {
                         gf_log ("", GF_LOG_DEBUG, "timestamp file exist");
                         ret = -2;
@@ -2889,7 +2964,8 @@ enumerate_transport_reqs (gf_transport_type type, char **types)
 }
 
 static int
-generate_client_volfiles (glusterd_volinfo_t *volinfo)
+generate_client_volfiles (glusterd_volinfo_t *volinfo,
+                          glusterd_client_type_t client_type)
 {
         char               filepath[PATH_MAX] = {0,};
         int                ret = -1;
@@ -2908,7 +2984,21 @@ generate_client_volfiles (glusterd_volinfo_t *volinfo)
                 if (ret)
                         goto out;
                 type = transport_str_to_type (types[i]);
-                glusterd_get_client_filepath (filepath, volinfo, type);
+
+                ret = dict_set_uint32 (dict, "trusted-client", client_type);
+                if (ret)
+                        goto out;
+
+                if (client_type == GF_CLIENT_TRUSTED) {
+                        glusterd_get_trusted_client_filepath (filepath,
+                                                              volinfo,
+                                                              type);
+                } else {
+                        glusterd_get_client_filepath (filepath,
+                                                      volinfo,
+                                                      type);
+                }
+
                 ret = generate_single_transport_client_volfile (volinfo,
                                                                 filepath,
                                                                 dict);
@@ -2929,7 +3019,7 @@ glusterd_create_rb_volfiles (glusterd_volinfo_t *volinfo,
 
         ret = glusterd_generate_brick_volfile (volinfo, brickinfo);
         if (!ret)
-                ret = generate_client_volfiles (volinfo);
+                ret = generate_client_volfiles (volinfo, GF_CLIENT_TRUSTED);
         if (!ret)
                 ret = glusterd_fetchspec_notify (THIS);
 
@@ -2939,23 +3029,33 @@ glusterd_create_rb_volfiles (glusterd_volinfo_t *volinfo,
 int
 glusterd_create_volfiles_and_notify_services (glusterd_volinfo_t *volinfo)
 {
-        int ret = -1;
+        int        ret  = -1;
+        xlator_t  *this = NULL;
+
+        this = THIS;
 
         ret = generate_brick_volfiles (volinfo);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR,
+                gf_log (this->name, GF_LOG_ERROR,
                         "Could not generate volfiles for bricks");
                 goto out;
         }
 
-        ret = generate_client_volfiles (volinfo);
+        ret = generate_client_volfiles (volinfo, GF_CLIENT_TRUSTED);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR,
-                        "Could not generate volfile for client");
+                gf_log (this->name, GF_LOG_ERROR,
+                        "Could not generate trusted client volfiles");
                 goto out;
         }
 
-        ret = glusterd_fetchspec_notify (THIS);
+        ret = generate_client_volfiles (volinfo, GF_CLIENT_OTHER);
+        if (ret) {
+                gf_log (this->name, GF_LOG_ERROR,
+                        "Could not generate client volfiles");
+                goto out;
+        }
+
+        ret = glusterd_fetchspec_notify (this);
 
 out:
         return ret;
@@ -3085,11 +3185,11 @@ out:
 
 int
 validate_clientopts (glusterd_volinfo_t *volinfo,
-                    dict_t *val_dict,
-                    char **op_errstr)
+                     dict_t *val_dict,
+                     char **op_errstr)
 {
         volgen_graph_t graph = {0,};
-        int     ret = -1;
+        int            ret   = -1;
 
         GF_ASSERT (volinfo);
 
@@ -3112,7 +3212,7 @@ validate_brickopts (glusterd_volinfo_t *volinfo,
                     char **op_errstr)
 {
         volgen_graph_t graph = {0,};
-        int     ret = -1;
+        int            ret   = -1;
 
         GF_ASSERT (volinfo);
 
diff --git a/xlators/mgmt/glusterd/src/glusterd-volgen.h b/xlators/mgmt/glusterd/src/glusterd-volgen.h
index 974aed934ba..feb1796c9bb 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volgen.h
+++ b/xlators/mgmt/glusterd/src/glusterd-volgen.h
@@ -36,6 +36,11 @@
 #define VKEY_FEATURES_QUOTA       "features.quota"
 #define VKEY_PERF_STAT_PREFETCH   "performance.stat-prefetch"
 
+typedef enum {
+        GF_CLIENT_TRUSTED,
+        GF_CLIENT_OTHER
+} glusterd_client_type_t;
+
 #define COMPLETE_OPTION(key, completion, ret)                           \
         do {                                                            \
                 if (!strchr (key, '.')) {                               \
diff --git a/xlators/mgmt/glusterd/src/glusterd-volume-ops.c b/xlators/mgmt/glusterd/src/glusterd-volume-ops.c
index 1ce0faf17ee..9df9d4219a7 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volume-ops.c
+++ b/xlators/mgmt/glusterd/src/glusterd-volume-ops.c
@@ -46,9 +46,9 @@ glusterd_handle_create_volume (rpcsvc_request_t *req)
         char                   *brick       = NULL;
         char                   *bricks      = NULL;
         char                   *volname     = NULL;
-        int                    brick_count = 0;
+        int                    brick_count  = 0;
         char                   *tmpptr      = NULL;
-        int                    i           = 0;
+        int                    i            = 0;
         char                   *brick_list  = NULL;
         void                   *cli_rsp     = NULL;
         char                    err_str[2048] = {0,};
@@ -57,9 +57,12 @@ glusterd_handle_create_volume (rpcsvc_request_t *req)
         char                   *free_ptr    = NULL;
         char                   *trans_type  = NULL;
         uuid_t                  volume_id   = {0,};
+        uuid_t                  tmp_uuid    = {0};
         glusterd_brickinfo_t    *tmpbrkinfo = NULL;
-        glusterd_volinfo_t      tmpvolinfo = {{0},};
-        int32_t                 type       = 0;
+        glusterd_volinfo_t      tmpvolinfo  = {{0},};
+        int32_t                 type        = 0;
+        char                   *username    = NULL;
+        char                   *password    = NULL;
 
         GF_ASSERT (req);
 
@@ -197,6 +200,20 @@ glusterd_handle_create_volume (rpcsvc_request_t *req)
                 brickinfo = NULL;
         }
 
+        /* generate internal username and password */
+
+        uuid_generate (tmp_uuid);
+        username = gf_strdup (uuid_utoa (tmp_uuid));
+        ret = dict_set_dynstr (dict, "internal-username", username);
+        if (ret)
+                goto out;
+
+        uuid_generate (tmp_uuid);
+        password = gf_strdup (uuid_utoa (tmp_uuid));
+        ret = dict_set_dynstr (dict, "internal-password", password);
+        if (ret)
+                goto out;
+
         ret = glusterd_op_begin (req, GD_OP_CREATE_VOLUME, dict);
         gf_cmd_log ("Volume create", "on volname: %s %s", volname,
                     (ret != 0) ? "FAILED": "SUCCESS");
@@ -1184,6 +1201,8 @@ glusterd_op_create_volume (dict_t *dict, char **op_errstr)
         char                 *saveptr    = NULL;
         char                 *trans_type = NULL;
         char                 *str        = NULL;
+        char                 *username   = NULL;
+        char                 *password   = NULL;
 
         this = THIS;
         GF_ASSERT (this);
@@ -1194,14 +1213,16 @@ glusterd_op_create_volume (dict_t *dict, char **op_errstr)
         ret = glusterd_volinfo_new (&volinfo);
 
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to allocate memory");
+                gf_log (this->name, GF_LOG_ERROR,
+                        "Unable to allocate memory");
                 goto out;
         }
 
         ret = dict_get_str (dict, "volname", &volname);
 
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get volume name");
+                gf_log (this->name, GF_LOG_ERROR,
+                        "Unable to get volume name");
                 goto out;
         }
 
@@ -1210,19 +1231,19 @@ glusterd_op_create_volume (dict_t *dict, char **op_errstr)
 
         ret = dict_get_int32 (dict, "type", &volinfo->type);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get type");
+                gf_log (this->name, GF_LOG_ERROR, "Unable to get type");
                 goto out;
         }
 
         ret = dict_get_int32 (dict, "count", &volinfo->brick_count);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get count");
+                gf_log (this->name, GF_LOG_ERROR, "Unable to get count");
                 goto out;
         }
 
         ret = dict_get_int32 (dict, "port", &volinfo->port);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get port");
+                gf_log (this->name, GF_LOG_ERROR, "Unable to get port");
                 goto out;
         }
 
@@ -1230,7 +1251,7 @@ glusterd_op_create_volume (dict_t *dict, char **op_errstr)
 
         ret = dict_get_str (dict, "bricks", &bricks);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get bricks");
+                gf_log (this->name, GF_LOG_ERROR, "Unable to get bricks");
                 goto out;
         }
 
@@ -1272,20 +1293,39 @@ glusterd_op_create_volume (dict_t *dict, char **op_errstr)
 
         ret = dict_get_str (dict, "transport", &trans_type);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get transport");
+                gf_log (this->name, GF_LOG_ERROR,
+                        "Unable to get transport");
                 goto out;
         }
 
         ret = dict_get_str (dict, "volume-id", &str);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "Unable to get volume-id");
+                gf_log (this->name, GF_LOG_ERROR,
+                        "Unable to get volume-id");
                 goto out;
         }
         ret = uuid_parse (str, volinfo->volume_id);
         if (ret) {
-                gf_log ("", GF_LOG_ERROR, "unable to parse uuid %s", str);
+                gf_log (this->name, GF_LOG_ERROR,
+                        "unable to parse uuid %s", str);
+                goto out;
+        }
+
+        ret = dict_get_str (dict, "internal-username", &username);
+        if (ret) {
+                gf_log (this->name, GF_LOG_ERROR,
+                        "unable to get internal username");
+                goto out;
+        }
+        glusterd_auth_set_username (volinfo, username);
+
+        ret = dict_get_str (dict, "internal-password", &password);
+        if (ret) {
+                gf_log (this->name, GF_LOG_ERROR,
+                        "unable to get internal password");
                 goto out;
         }
+        glusterd_auth_set_password (volinfo, password);
 
         if (strcasecmp (trans_type, "rdma") == 0) {
                 volinfo->transport_type = GF_TRANSPORT_RDMA;
diff --git a/xlators/mgmt/glusterd/src/glusterd.c b/xlators/mgmt/glusterd/src/glusterd.c
index 28e80310ec2..ce044cf35ff 100644
--- a/xlators/mgmt/glusterd/src/glusterd.c
+++ b/xlators/mgmt/glusterd/src/glusterd.c
@@ -243,7 +243,7 @@ out:
         return ret;
 }
 
-/* defined in usterd-utils.c -- no
+/* defined in glusterd-utils.c -- no
  * glusterd header where it would be
  * appropriate to put to, and too
  * accidental routine to place in
diff --git a/xlators/mgmt/glusterd/src/glusterd.h b/xlators/mgmt/glusterd/src/glusterd.h
index e200f49d1db..e8193bba24f 100644
--- a/xlators/mgmt/glusterd/src/glusterd.h
+++ b/xlators/mgmt/glusterd/src/glusterd.h
@@ -192,6 +192,13 @@ typedef enum gf_rb_status_ {
         GF_RB_STATUS_PAUSED,
 } gf_rb_status_t;
 
+struct _auth {
+        char       *username;
+        char       *password;
+};
+
+typedef struct _auth auth_t;
+
 struct glusterd_volinfo_ {
         char                    volname[GLUSTERD_MAX_VOLUME_NAME];
         int                     type;
@@ -229,6 +236,7 @@ struct glusterd_volinfo_ {
         dict_t                  *dict;
 
         uuid_t                  volume_id;
+        auth_t                  auth;
         char                    *logdir;
 
         dict_t                  *gsync_slaves;