diff options
author | Varun Shastry <vshastry@redhat.com> | 2013-04-19 12:34:51 +0530 |
---|---|---|
committer | Krishnan Parthasarathi <kparthas@redhat.com> | 2013-08-12 00:48:14 +0530 |
commit | 184e88bfc8f9c6b180c56adfd029e2aaece1297f (patch) | |
tree | 154de47dd6fe5b3e91201ffde27e7e732381aaa9 /cli | |
parent | a1fe3d040a8c9b032cbcb5e831383628cddfa39a (diff) |
features/quota: Improvements to quota
Old implementation
* Client side implementation of quota
- Not secure
- Increased traffic in updating the ctx
New Implementation
* 2 stages of quota implementation is done: Soft and hard quota
Upon reaching soft quota limit on the directory it logs/alerts in the quota
daemon log (ie DEFAULT_LOG_DIR/quotad.log) and no more writes allowed after
hard quota limit. After reaching the soft-limit the daemon alerts the
user/admin repeatively for every 'alert-time', which is configurable.
* Quota is moved to server-side.
There will be 2 quota xlators
i. Quota Server
It takes care of the enforcing the quota and maintains the context
specific to the brick. Since this doesn't have the complete picture of
the cluster, cluster wide usage is updated from the quota daemon. This
updated context is saved and used for the enforcement.
It updates its context by searching the QUOTA_UPDATE_KEY from the dict
in the setxattr call, and is updated from nowhere else.
The quota is always loaded in the server graph and is by passed if the
feature is not enabled.
Options specific to quota-server:
server-quota - Specifies whether the features is on/off. It is used
to by pass the quota if turned off.
deem-statfs - If set to on, it takes quota limits into
consideration while estimating fs size. (df command)
ii. Quota Daemon
This is the new xlator introduced with this patch. Its the
*gluster client* process with no mount point, started upon enabling
quota or restarting the volume. This is a single process for all the
volumes in the cluster. Its volfile stored in
GLUSTERD_DEFAULT_WORKI_DIR/quotad/quotad.vol.
It queries for the sizes on all the bricks, aggregates the size and
sends back the updated size, periodically. The timeout between
successive updation is configurable and typically/by default more for
below-soft-quota usage and less for above-soft-quota usage. It
maintains the timeout inside the limit structure based on the usage;
below soft limit and above soft limit.
There will be thread running per volume which iterates through the list
and decides whether the size to be queried in the current iteration
based on its timeout. It takes the next iteration time taking the least
of the timeouts in the list of entries.
Maintains a separate inode table for each volume in the quotad. In the
first iteration it builds the table for quota-dirs (dirs on which limit
is set) and its components.
Options specific to quotad:
hard-timeout - Timeout for updation of usage to the quota-server
when the usage is crosses the soft-limit.
soft-timeout - Timeout for the updation of usage to the
quota-server when the usage is below soft-limit.
alert-time - Frequency of logging after the usage reached
soft limit.
Options common to both:
default-soft-limit - This is used when individual paths are not
configured with soft-limit and default value of
this option is 90% of the hard-limit.
limit-set - String containing all the limits.
Thus in the current implementation we'll have 2 quota xlators: one in server
graph and one in trusted client (quota daemon) of which the sole
purpose will be to aggregate the quota size xattrs from all the bricks and
send the same to server quota xlator.
* Changes in glusterd and CLI
A single volfile is created for all the volumes, similar to nfs volfile.
All files related to quota client (volfile, pid etc) are stored in
GLUSTERD_DEFAULT_WORK_DIR/quotad/.
The new pattern of the quota limit stores in
limit-set = <single-dir-limit>[,<single-dir-limit>]
single-dir-limit = <abs-path>:<hard-limit>[:<soft-limit-in-percent>]
It also introduces new options:
volume quota <VOLNAME> {enable|disable|list [<path> ...]|remove <path>| default-soft-limit <percent>} |
volume quota <VOLNAME> {limit-usage <path> <size> |soft-limit <path> <percent>} |
volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>}
Credit:
Raghavendra Bhat <rabhat@redhat.com>
Varun Shastry <vshastry@redhat.com>
Shishir Gowda <sgowda@redhat.com>
Kruthika Dhananjay <kdhananj@redhat.com>
Brian Foster <bfoster@redhat.com>
Krishnan Parthasarathi <kparthas@redhat.com>
Change-Id: I16ec5be0c2faaf42b14034b9ccaf17796adef082
BUG: 969461
Signed-off-by: Varun Shastry <vshastry@redhat.com>
Diffstat (limited to 'cli')
-rw-r--r-- | cli/src/cli-cmd-parser.c | 132 | ||||
-rw-r--r-- | cli/src/cli-cmd-volume.c | 6 | ||||
-rw-r--r-- | cli/src/cli-rpc-ops.c | 378 |
3 files changed, 417 insertions, 99 deletions
diff --git a/cli/src/cli-cmd-parser.c b/cli/src/cli-cmd-parser.c index 26ac6f38..12feb75b 100644 --- a/cli/src/cli-cmd-parser.c +++ b/cli/src/cli-cmd-parser.c @@ -532,8 +532,13 @@ cli_cmd_quota_parse (const char **words, int wordcount, dict_t **options) uint64_t value = 0; gf_quota_type type = GF_QUOTA_OPTION_TYPE_NONE; char *opwords[] = { "enable", "disable", "limit-usage", - "remove", "list", "version", NULL }; + "remove", "list", "soft-limit", + "alert-time", "soft-timeout", + "hard-timeout", "default-soft-limit", + NULL}; char *w = NULL; + uint32_t time = 0; + double percent = 0; GF_ASSERT (words); GF_ASSERT (options); @@ -614,8 +619,8 @@ cli_cmd_quota_parse (const char **words, int wordcount, dict_t **options) if (words[4][0] != '/') { cli_err ("Please enter absolute path"); - - return -2; + ret = -1; + goto out; } ret = dict_set_str (dict, "path", (char *) words[4]); if (ret) @@ -623,14 +628,14 @@ cli_cmd_quota_parse (const char **words, int wordcount, dict_t **options) if (!words[5]) { cli_err ("Please enter the limit value to be set"); - - return -2; + ret = -1; + goto out; } ret = gf_string2bytesize (words[5], &value); if (ret != 0) { cli_err ("Please enter a correct value"); - return -1; + goto out; } ret = dict_set_str (dict, "limit", (char *) words[5]); @@ -649,8 +654,8 @@ cli_cmd_quota_parse (const char **words, int wordcount, dict_t **options) if (words[4][0] != '/') { cli_err ("Please enter absolute path"); - - return -2; + ret = -1; + goto out; } ret = dict_set_str (dict, "path", (char *) words[4]); @@ -683,8 +688,109 @@ cli_cmd_quota_parse (const char **words, int wordcount, dict_t **options) goto set_type; } - if (strcmp (w, "version") == 0) { - type = GF_QUOTA_OPTION_TYPE_VERSION; + if (strcmp (w, "soft-limit") == 0) { + if (wordcount != 6) { + ret = -1; + goto out; + } + + type = GF_QUOTA_OPTION_TYPE_SOFT_LIMIT; + if (words[4][0] != '/') { + cli_err ("Please enter absolute path"); + ret = -1; + goto out; + } + ret = dict_set_str (dict, "path", (char *) words[4]); + if (ret) + goto out; + + if (!words[5]) { + cli_err ("Please enter the limit value to be set"); + ret = -1; + goto out; + } + + ret = gf_string2percent (words[5], &percent); + if (ret != 0) { + cli_err ("Please enter a correct value"); + goto out; + } + + ret = dict_set_str (dict, "limit", (char *) words[5]); + if (ret < 0) + goto out; + + goto set_type; + + } + if (strcmp (w, "alert-time") == 0) { + if (wordcount != 5) { + ret = -1; + goto out; + } + type = GF_QUOTA_OPTION_TYPE_ALERT_TIME; + + ret = gf_string2time (words[4], &time); + if (ret) { + cli_err ("Invalid argument %s. Please enter a valid " + "string", words[4]); + goto out; + } + + ret = dict_set_str (dict, "value", (char *)words[4]); + if (ret < 0) + goto out; + goto set_type; + } + if (strcmp (w, "soft-timeout") == 0) { + if (wordcount != 5) { + ret = -1; + goto out; + } + type = GF_QUOTA_OPTION_TYPE_SOFT_TIMEOUT; + + ret = gf_string2time (words[4], &time); + if (ret) { + cli_err ("Invalid argument %s. Please enter a valid " + "string", words[4]); + goto out; + } + + ret = dict_set_str (dict, "value", (char *)words[4]); + if (ret < 0) + goto out; + goto set_type; + } + if (strcmp (w, "hard-timeout") == 0) { + if(wordcount != 5) { + ret = -1; + goto out; + } + type = GF_QUOTA_OPTION_TYPE_HARD_TIMEOUT; + + ret = gf_string2time (words[4], &time); + if (ret) { + cli_err ("Invalid argument %s. Please enter a valid " + "string", words[4]); + goto out; + } + + ret = dict_set_str (dict, "value", (char *)words[4]); + if (ret < 0) + goto out; + goto set_type; + } + if (strcmp (w, "default-soft-limit") == 0) { + if(wordcount != 5) { + ret = -1; + goto out; + } + type = GF_QUOTA_OPTION_TYPE_DEFAULT_SOFT_LIMIT; + + ret = dict_set_str (dict, "value", (char *)words[4]); + if (ret < 0) + goto out; + goto set_type; } else { GF_ASSERT (!"opword mismatch"); } @@ -2333,7 +2439,7 @@ cli_cmd_validate_dumpoption (const char *arg, char **option) { char *opwords[] = {"all", "nfs", "mem", "iobuf", "callpool", "priv", "fd", "inode", "history", "inodectx", "fdctx", - NULL}; + "quotad", NULL}; char *w = NULL; w = str_getunamb (arg, opwords); @@ -2365,6 +2471,10 @@ cli_cmd_volume_statedump_options_parse (const char **words, int wordcount, strncat (option_str, option, strlen (option)); strncat (option_str, " ", 1); } + if((strstr (option_str, "nfs")) && strstr (option_str, "quotad")) { + ret = -1; + goto out; + } dict = dict_new (); if (!dict) diff --git a/cli/src/cli-cmd-volume.c b/cli/src/cli-cmd-volume.c index 8d508158..101759ba 100644 --- a/cli/src/cli-cmd-volume.c +++ b/cli/src/cli-cmd-volume.c @@ -1904,7 +1904,9 @@ struct cli_cmd volume_cmds[] = { cli_cmd_volume_profile_cbk, "volume profile operations"}, - { "volume quota <VOLNAME> <enable|disable|limit-usage|list|remove> [path] [value]", + { "volume quota <VOLNAME> {enable|disable|list [<path> ...]|remove <path>| default-soft-limit <percent>} |\n" + "volume quota <VOLNAME> {limit-usage <path> <size> |soft-limit <path> <percent>} |\n" + "volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>}", cli_cmd_quota_cbk, "quota translator specific operations"}, @@ -1922,7 +1924,7 @@ struct cli_cmd volume_cmds[] = { cli_cmd_volume_heal_cbk, "self-heal commands on volume specified by <VOLNAME>"}, - {"volume statedump <VOLNAME> [nfs] [all|mem|iobuf|callpool|priv|fd|" + {"volume statedump <VOLNAME> [nfs|quotad] [all|mem|iobuf|callpool|priv|fd|" "inode|history]...", cli_cmd_volume_statedump_cbk, "perform statedump on bricks"}, diff --git a/cli/src/cli-rpc-ops.c b/cli/src/cli-rpc-ops.c index ea80f936..388805af 100644 --- a/cli/src/cli-rpc-ops.c +++ b/cli/src/cli-rpc-ops.c @@ -2334,101 +2334,288 @@ out: return ret; } -int32_t -gf_cli_print_limit_list (char *volname, char *limit_list, - char *op_errstr) +static void +print_quota_output_for_version_1 (char *path, char *hl, char *mountdir) { - int64_t size = 0; - int64_t limit_value = 0; - int32_t i, j; - int32_t len = 0, ret = -1; - char *size_str = NULL; - char path [PATH_MAX] = {0, }; - char ret_str [1024] = {0, }; - char value [1024] = {0, }; - char mountdir [] = "/tmp/mntXXXXXX"; - char abspath [PATH_MAX] = {0, }; - char *colon_ptr = NULL; - runner_t runner = {0,}; + uint64_t used_space = 0; + uint64_t limit_value = 0; + char *used_str = NULL; + char abspath[PATH_MAX] = {0,}; + char ret_str[1024] = {0,}; + int ret = -1; - GF_VALIDATE_OR_GOTO ("cli", volname, out); - GF_VALIDATE_OR_GOTO ("cli", limit_list, out); + snprintf (abspath, sizeof (abspath) - 1, "%s/%s", mountdir, path); - if (!connected) - goto out; + ret = sys_lgetxattr (abspath, "trusted.limit.list", (void *) ret_str, + 4096); + if (ret < 0) { + cli_out ("%-20s %10s", path, hl); + } else { + sscanf (ret_str, "%"PRIu64",%"PRIu64, &used_space, + &limit_value); - len = strlen (limit_list); - if (len == 0) { - cli_err ("%s", op_errstr?op_errstr:"quota limit not set "); - goto out; + used_str = gf_uint64_2human_readable (used_space); + + if (used_str == NULL) + cli_out ("%-20s %10s %20"PRIu64, path, hl, + used_space); + else + cli_out ("%-20s %10s %20s", path, hl, used_str); } - if (mkdtemp (mountdir) == NULL) { - gf_log ("cli", GF_LOG_WARNING, "failed to create a temporary " - "mount directory"); - ret = -1; + GF_FREE (used_str); + return; +} + +static void +print_quota_output_for_version_2 (char *path, char *hl, char *sl, + char *mountdir) +{ + uint64_t used_space = 0; + uint64_t limit_value = 0; + uint64_t hard_limit = 0; + uint64_t avail = 0; + char *used_str = NULL; + char *avail_str = NULL; + char abspath[PATH_MAX] = {0,}; + char ret_str[1024] = {0,}; + int ret = -1; + + + snprintf (abspath, sizeof (abspath) - 1, "%s/%s", mountdir, path); + + ret = sys_lgetxattr (abspath, "trusted.limit.list", (void *) ret_str, + 4096); + + if (ret < 0) { + cli_out ("%-40s %7s %9s %11s %7s", path, hl, sl, "N/A", "N/A"); + } else { + sscanf (ret_str, "%"PRIu64",%"PRIu64, &used_space, + &limit_value); + + used_str = gf_uint64_2human_readable (used_space); + + ret = gf_string2bytesize (hl, &hard_limit); + + if (hard_limit > used_space) + avail = hard_limit - used_space; + else + avail = 0; + + avail_str = gf_uint64_2human_readable (avail); + if (used_str == NULL) + cli_out ("%-40s %7s %9s %11"PRIu64 + "%9"PRIu64, path, hl, + sl, used_space, avail); + else + cli_out ("%-40s %7s %9s %11s %7s", path, hl, sl, + used_str, avail_str); + } + + GF_FREE (used_str); + GF_FREE (avail_str); + return; +} + +static void +gf_quota_print_limit_list (char *path, char *hl, char *sl, char *mountdir, + uint32_t op_version) +{ + if (op_version == 1) + print_quota_output_for_version_1 (path, hl, mountdir); + else + print_quota_output_for_version_2 (path, hl, sl, mountdir); +} + +void +gf_cli_quota_print_first_row (uint32_t op_version) +{ + if (op_version == 1) { + cli_out ("\tpath\t\t limit_set \t\t size"); + cli_out ("-----------------------------------------------------" + "-----------------------------"); + } else { + cli_out (" Path Hard-limit " + "Soft-limit Used Available"); + cli_out ("-----------------------------------------------------" + "---------------------------"); + } +} + +int +gf_cli_print_limit_list_from_dict (char *volname, uint32_t op_version, + dict_t *dict, char *default_sl, int count, + char *mountdir, int entry_count, + char *op_errstr) +{ + int ret = -1; + int i = 0; + int j = 0; + char key[1024] = {0,}; + char *limit = NULL; + char path[PATH_MAX] = {0,}; + char *hl = NULL; + char *sl = NULL; + char *sl_final = NULL; + + if (!dict|| count <= 0) + goto out; + + if (entry_count == 0) { + if (strlen (op_errstr) == 0) + cli_err ("Limit not set on specified directories"); goto out; } - /* Mount a temporary client to fetch the disk usage - * of the directory on which the limit is set. - */ - ret = runcmd (SBIN_DIR"/glusterfs", "-s", - "localhost", "--volfile-id", volname, "-l", - DEFAULT_LOG_FILE_DIRECTORY"/quota-list.log", - mountdir, NULL); - if (ret) { - gf_log ("cli", GF_LOG_WARNING, "failed to mount glusterfs client"); - ret = -1; - goto rm_dir; + gf_cli_quota_print_first_row (op_version); + + while (count--) { + j = 0; + sl = NULL; + hl = NULL; + snprintf (key, sizeof (key), "path%d", i++); + + ret = dict_get_str (dict, key, &limit); + if (ret < 0) { + gf_log ("cli", GF_LOG_DEBUG, "Path not present in limit" + " list"); + continue; + } + if (!strcmp (limit, "Not set")) + continue; + + ret = gf_get_hard_limit (limit, &hl); + + ret = gf_get_soft_limit (limit, &sl); + if (ret == 1) + sl_final = sl; + else + sl_final = default_sl; + + while (limit[j] != ':') { + path[j] = limit[j]; + j = j + 1; + } + + path[j] = '\0'; + + ret = gf_canonicalize_path (path); + if (ret) + goto out; + + gf_quota_print_limit_list (path, hl, sl_final, mountdir, + op_version); + + GF_FREE (hl); + GF_FREE (sl); } +out: + return ret; +} + +int32_t +gf_cli_print_limit_list_from_string (char *volname, char *limit_list, + uint32_t op_version, char *default_sl, + char *mountdir) +{ + int32_t i = 0; + int32_t j = 0; + int32_t len = 0; + int ret = -1; + char path[PATH_MAX] = {0,}; + char *colon_ptr = NULL; + char *sl_final = NULL; + char *hl = NULL; + char *sl = NULL; len = strlen (limit_list); if (len == 0) { - cli_err ("quota limit not set "); - goto unmount; + ret = -1; + goto out; } - i = 0; + gf_cli_quota_print_first_row (op_version); - cli_out ("\tpath\t\t limit_set\t size"); - cli_out ("-----------------------------------------------------------" - "-----------------------"); while (i < len) { j = 0; + sl = hl = NULL; while (limit_list [i] != ',' && limit_list [i] != '\0') { path [j++] = limit_list[i++]; } path [j] = '\0'; - //here path[] contains both path and limit value + //here path[] contains both path and limit value - colon_ptr = strrchr (path, ':'); + ret = gf_get_hard_limit (path, &hl); + ret = gf_get_soft_limit (path, &sl); + + if (ret == 1) + sl_final = sl; + else + sl_final = default_sl; + + colon_ptr = strchr (path, ':'); *colon_ptr = '\0'; - strcpy (value, ++colon_ptr); - snprintf (abspath, sizeof (abspath), "%s/%s", mountdir, path); + gf_quota_print_limit_list (path, hl, sl_final, mountdir, + op_version); - ret = sys_lgetxattr (abspath, "trusted.limit.list", (void *) ret_str, 4096); - if (ret < 0) { - cli_out ("%-20s %10s", path, value); - } else { - sscanf (ret_str, "%"PRId64",%"PRId64, &size, - &limit_value); - size_str = gf_uint64_2human_readable ((uint64_t) size); - if (size_str == NULL) { - cli_out ("%-20s %10s %20"PRId64, path, - value, size); - } else { - cli_out ("%-20s %10s %20s", path, - value, size_str); - GF_FREE (size_str); - } - } i++; + + GF_FREE (hl); + GF_FREE (sl); } -unmount: +out: + return ret; +} + +int +gf_cli_quota_list_run_crawler (char *volname, char *limit_list, dict_t *dict, + int count, char *op_errstr, uint32_t op_version, + char *default_sl, int entry_count) +{ + int ret = -1; + runner_t runner = {0,}; + char mountdir [] = "/tmp/mntXXXXXX"; + + GF_VALIDATE_OR_GOTO ("cli", volname, out); + + if (!connected) + goto out; + + if ((!limit_list) && (count <= 0)) + goto out; + + if (mkdtemp (mountdir) == NULL) { + gf_log ("cli", GF_LOG_WARNING, "failed to create a temporary " + "mount directory"); + ret = -1; + goto out; + } + + /* Mount a temporary client to fetch the disk usage + * of the directory on which the limit is set. + */ + ret = runcmd (SBIN_DIR"/glusterfs", "-s", + "localhost", "--volfile-id", volname, "-l", + DEFAULT_LOG_FILE_DIRECTORY"/quota-list.log", + mountdir, NULL); + if (ret) { + gf_log ("cli", GF_LOG_WARNING, "failed to mount glusterfs " + "client"); + ret = -1; + goto rm_dir; + } + + if (limit_list) + gf_cli_print_limit_list_from_string (volname, limit_list, + op_version, default_sl, + mountdir); + else if (count > 0) + gf_cli_print_limit_list_from_dict (volname, op_version, dict, + default_sl, count, mountdir, + entry_count, op_errstr); runinit (&runner); runner_add_args (&runner, "umount", @@ -2445,20 +2632,24 @@ rm_dir: rmdir (mountdir); out: return ret; + } int gf_cli_quota_cbk (struct rpc_req *req, struct iovec *iov, int count, void *myframe) { - gf_cli_rsp rsp = {0,}; - int ret = -1; - dict_t *dict = NULL; - char *volname = NULL; - char *limit_list = NULL; - int32_t type = 0; - char msg[1024] = {0,}; - call_frame_t *frame = NULL; + gf_cli_rsp rsp = {0,}; + int ret = -1; + dict_t *dict = NULL; + char *volname = NULL; + char *limit_list = NULL; + int32_t type = 0; + char msg[5120] = {0,}; + call_frame_t *frame = NULL; + uint32_t op_version = 1; + char *default_sl = NULL; + int entry_count = 0; if (-1 == req->rpc_status) { goto out; @@ -2491,7 +2682,7 @@ gf_cli_quota_cbk (struct rpc_req *req, struct iovec *iov, rsp.dict.dict_len, &dict); if (ret < 0) { - gf_log ("glusterd", GF_LOG_ERROR, + gf_log ("cli", GF_LOG_ERROR, "failed to " "unserialize req-buffer to dictionary"); goto out; @@ -2508,11 +2699,30 @@ gf_cli_quota_cbk (struct rpc_req *req, struct iovec *iov, gf_log (frame->this->name, GF_LOG_TRACE, "failed to get limit_list"); + ret = dict_get_uint32 (dict, "op-version", &op_version); + if (ret) + gf_log (frame->this->name, GF_LOG_TRACE, "failed to get " + "op-version"); + + ret = dict_get_str (dict, "default-soft-limit", &default_sl); + if (ret) + gf_log (frame->this->name, GF_LOG_TRACE, "failed to get " + "default soft limit"); + ret = dict_get_int32 (dict, "type", &type); if (ret) gf_log (frame->this->name, GF_LOG_TRACE, "failed to get type"); + ret = dict_get_int32 (dict, "count", &count); + if (ret) + gf_log (frame->this->name, GF_LOG_TRACE, "failed to get count"); + + ret = dict_get_int32 (dict, "entry-count", &entry_count); + if (ret) + gf_log (frame->this->name, GF_LOG_TRACE, "failed to get entry " + "count"); + if (type == GF_QUOTA_OPTION_TYPE_LIST) { if (global_state->mode & GLUSTER_MODE_XML) { ret = cli_xml_output_vol_quota_limit_list @@ -2525,19 +2735,15 @@ gf_cli_quota_cbk (struct rpc_req *req, struct iovec *iov, } - if (limit_list) { - gf_cli_print_limit_list (volname, - limit_list, - rsp.op_errstr); - } else { - gf_log ("cli", GF_LOG_INFO, "Received resp to quota " - "command "); - if (rsp.op_errstr) - snprintf (msg, sizeof (msg), "%s", - rsp.op_errstr); - } + gf_log ("cli", GF_LOG_INFO, "Received resp to quota command"); + + gf_cli_quota_list_run_crawler (volname, limit_list, dict, count, + rsp.op_errstr, op_version, + default_sl, entry_count); + if (rsp.op_errstr) + snprintf (msg, sizeof (msg), "%s", rsp.op_errstr); } else { - gf_log ("cli", GF_LOG_INFO, "Received resp to quota command "); + gf_log ("cli", GF_LOG_INFO, "Received resp to quota command"); if (rsp.op_errstr) snprintf (msg, sizeof (msg), "%s", rsp.op_errstr); else |