glusterfs.git/xlators/performance, branch release-3.9

ec: Invalidations in disperse volume should not update the stat

2017-01-17T14:55:46+00:00

Backport of http://review.gluster.org/16329

Issue:
In disperse volume, the file is present across bricks, hence the stat
from one brick doesn't carry the valid size of the file. Therefore
the upcall from one brick updating the md-cache results in wrong size
being updated.

Fix:
If the notification is cache invalidation then, indicate md-cache that
the attributes is invalid.

>Reviewed-on: http://review.gluster.org/16329
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Xavier Hernandez 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Pranith Kumar Karampuri 
(cherry picked from commit 95d07a3d2d68805d93d36a447436e27c48777939)

BUG: 1410688
Change-Id: Id89d2283478e70b62b435a8891fffc86d2be8cb2
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/16341
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

dht/md-cache: Filter invalidate if the file is made a linkto file

2017-01-05T06:10:12+00:00

Backport of http://review.gluster.org/15789

Upcall as a part of setattr, sends an invalidation and the
invalidation carries the resulting stat value. When a file
is converted to linkto files, even then an invalidation
is set and as a result the mountpoint shows the sticky
bit in the stat of the file.
eg: ---------T. 945 root root 0 Nov  8 10:14 hardlink.999

Fix:
When dht recieves a notification of sticky bit change, it updates
the flag, to indicate md-cache to send the subsequent lookup.

>Reviewed-on: http://review.gluster.org/15789
>Smoke: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Reviewed-by: Niels de Vos 
>CentOS-regression: Gluster Build System 
>Reviewed-by: Susant Palai 
>Reviewed-by: Rajesh Joseph 
>(cherry picked from commit 4536f7bdf16f8286d67598eda9a46c029f0c0bf4)

Change-Id: Ic2fd7a5b196db0754f9b97072e644e6bf69da606
BUG: 1401376
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/16022
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G

afr,dht,ec: Replace GF_EVENT_CHILD_MODIFIED with event SOME_DESCENDENT_DOWN/UP

2016-12-13T09:46:14+00:00

Backport of http://review.gluster.org/#/c/15764/

Currently these are few events related to child_up/down:
GF_EVENT_CHILD_UP :  Issued when any of the protocol client
connects.
GF_EVENT_CHILD_MODIFIED : Issued by afr/dht/ec
GF_EVENT_CHILD_DOWN : Issued when any of the protocol client
disconnects.
These events get modified at the dht/afr/ec layers. Here is a
brief on the same.

DHT:
- All the subvolumes reported once, and atleast one child came
  up, then GF_EVENT_CHILD_UP is issued
- connect GF_EVENT_CHILD_UP is issued
- disconnect GF_EVENT_CHILD_MODIFIED is issued
- All the subvolumes disconnected, GF_EVENT_CHILD_DOWN is issued

AFR:
- First subvolume came up, then GF_EVENT_CHILD_UP is issued
- Subsequent subvolumes coming up, results in GF_EVENT_CHILD_MODIFIED
- Any of the subvolumes go down, then GF_EVENT_SOME_CHILD_DOWN is issued
- Last up subvolume goes down, then GF_EVENT_CHILD_DOWN is issued

Until the patch [1] introduced GF_EVENT_SOME_CHILD_UP,
GF_EVENT_CHILD_MODIFIED was issued by afr/dht when any of the subvolumes
go up or down.

Now with md-cache changes, there is a necessity to differentiate between
child up and down. Hence, introducing GF_EVENT_SOME_DESCENDENT_DOWN/UP and
getting rid of GF_EVENT_CHILD_MODIFIED.

[1] http://review.gluster.org/12573

>Reviewed-on: http://review.gluster.org/15764
>CentOS-regression: Gluster Build System 
>NetBSD-regression: NetBSD Build System 
>Smoke: Gluster Build System 
>Reviewed-by: N Balachandran 
>Reviewed-by: Pranith Kumar Karampuri  
>Reviewed-by: Rajesh Joseph 
(cherry picked from commit f7ab6c45963fa0da68acedfb14281cd2456abc68)

Change-Id: I704140b6598f7ec705493251d2dbc4191c965a58
BUG: 1396880
Signed-off-by: Poornima G 
Reviewed-on: http://review.gluster.org/15890
NetBSD-regression: NetBSD Build System 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

performance/io-threads: Exit threads in fini() as well

2016-11-25T06:27:47+00:00

Problem:
io-threads starts the thread in 'init()' but doesn't clean them up
on 'fini()'. It relies on PARENT_DOWN to exit threads but there can
be cases where event before PARENT_UP the graph init code can think
of issuing fini(). This code path is hit when glfs_init() is called
on a volume that is in 'stopped' state. It leads to a crash in ganesha
process, because the io-thread tries to access freed memory.

Fix:
Ideal fix would be to wait for all fops in io-thread list to be completed on
PARENT_DOWN, and have fini() do cleanup of threads. Because there is no proper
documentation about how PARENT_DOWN/fini are supposed to be used,
we are getting different kinds of sequences in different higher level protocols.
So for now cleaning up in both PARENT_DOWN and fini(). Fuse doesn't call fini()
gfapi is not calling PARENT_DOWN in some cases, so for now I don't see
another way out.

 >BUG: 1396793
 >Change-Id: I9c9154e7d57198dbaff0f30d3ffc25f6d8088aec
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15888
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Raghavendra G 
 >(cherry picked from commit 25817a8c868b6c1b8149117f13e4216a99e453aa)

Change-Id: Id55e7c2f3e90c013d40e59bfbfb3f1583b8c4061
BUG: 1397381
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15921
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Raghavendra G

performance/open-behind: Avoid deadlock in statedump

2016-11-24T05:44:25+00:00

Problem:
open-behind is taking fd->lock then inode->lock where as statedump is taking
inode->lock then fd->lock, so it is leading to deadlock

In open-behind, following code exists:
void
ob_fd_free (ob_fd_t *ob_fd)
{
        loc_wipe (&ob_fd->loc); <<--- this takes (inode->lock)
.......
}

int
ob_wake_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
             int op_ret, int op_errno, fd_t *fd_ret, dict_t *xdata)
{
	.......
        LOCK (&fd->lock); <<---- fd->lock
        {
	.......
                __fd_ctx_del (fd, this, NULL);
                ob_fd_free (ob_fd); <<<---------------
        }
        UNLOCK (&fd->lock);
.......
}
=================================================================
In statedump this code exists:
inode_dump (inode_t *inode, char *prefix)
{
.......
	ret = TRY_LOCK(&inode->lock); <<---- inode->lock
.......
	fd_ctx_dump (fd, prefix); <<<-----
.......
}
fd_ctx_dump (fd_t *fd, char *prefix)
{
.......
        LOCK (&fd->lock); <<<------------------ this takes fd-lock
        {
.......
}

Fix:
Make sure open-behind doesn't call ob_fd_free() inside fd->lock

 >BUG: 1393259
 >Change-Id: I4abdcfc5216270fa1e2b43f7b73445f49e6d6e6e
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15808
 >Smoke: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Poornima G 
 >Reviewed-by: Raghavendra G 

BUG: 1393677
Change-Id: I45a0fbed683ef6acb7900df87534927f332fdaaa
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15817
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Poornima G

md-cache: Invalidate cache entry for open() with O_TRUNC

2016-11-04T03:19:55+00:00

When a file is opened with O_TRUNC flag set, its size gets
set to '0'. This case needs to be handled in md-cache to
avoid sending incorrect cached stat.

This is backport of below mainline patch -
http://review.gluster.org/#/c/15618/

> Change-Id: I95d1f8a6634734898883ede010c3e7b0b7eb97d9
> BUG: 1382266
> Signed-off-by: Soumya Koduri 
> Reviewed-on: http://review.gluster.org/15618
> Smoke: Gluster Build System 
> Reviewed-by: jiffin tony Thottan 
> Tested-by: jiffin tony Thottan 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Kaleb KEITHLEY 
> (cherry picked from commit 6ca5d6382f03685b31b12accb095093cf1486603)

Change-Id: I9780a4051adaa9030bd33da68dbc0a31212de054
BUG: 1391448
Signed-off-by: Soumya Koduri 
Reviewed-on: http://review.gluster.org/15770
Reviewed-by: jiffin tony Thottan 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Kaleb KEITHLEY

performance/write-behind: fix flush stuck by former failed writes

2016-11-03T09:28:20+00:00

the issue is happened in this case:
assume a file is opened with fd1 and fd2.
1. some WRITE opto fd1 got error, they were add back to 'todo' queue
   because of those error.
2. fd2 closed, a FLUSH op is send to write-behind.
3. FLUSH can not be unwind because it's not a legal waiter for those
   failed write(as func __wb_request_waiting_on() say). and those failed
   WRITE also can not be ended if fd1 is not closed. fd2 stuck in close
   syscall.

to resolve this issue, we can change the way we determine 2 requests is
'conflict': flush/fsync is not conflict with those write that is not
belonged to them. so __wb_pick_winds() can wind the FLUSH op.

below is some information when the stuck issue happen:
glusterdump logs:
[xlator.performance.write-behind.wb_inode]
path=/ltp-F9eG0ZSOME/rw-buffered-16436
inode=0x7fdbe8039b9c
window_conf=1048576
window_current=249856
transit-size=0
dontsync=0

[.WRITE]
request-ptr=0x7fdbe8020200
refcount=1
wound=no
generation-number=4
req->op_ret=-1
req->op_errno=116
sync-attempts=3
sync-in-progress=no
size=131072
offset=1220608
lied=-1
append=0
fulfilled=0
go=0

[.WRITE]
request-ptr=0x7fdbe8068c30
refcount=1
wound=no
generation-number=5
req->op_ret=-1
req->op_errno=116
sync-attempts=2
sync-in-progress=no
size=118784
offset=1351680
lied=-1
append=0
fulfilled=0
go=0

[.FLUSH]
request-ptr=0x7fdbe8021cd0
refcount=1
wound=no
generation-number=6
req->op_ret=0
req->op_errno=0
sync-attempts=0

gdb detail about above 3 requests:
(gdb) print *((wb_request_t *)0x7fdbe8021cd0)
$2 = {all = {next = 0x7fdbe803a608, prev = 0x7fdbe8068c30}, todo = {next
= 0x7fdbe803a618, prev = 0x7fdbe8068c40}, lie = {next = 0x7fdbe8021cf0,
    prev = 0x7fdbe8021cf0}, winds = {next = 0x7fdbe8021d00, prev =
0x7fdbe8021d00}, unwinds = {next = 0x7fdbe8021d10, prev =
0x7fdbe8021d10}, wip = {
    next = 0x7fdbe8021d20, prev = 0x7fdbe8021d20}, stub =
0x7fdbe80224dc, write_size = 0, orig_size = 0, total_size = 0, op_ret =
0, op_errno = 0,
  refcount = 1, wb_inode = 0x7fdbe803a5f0, fop = GF_FOP_FLUSH, lk_owner
= {len = 8, data = "W\322T\f\271\367y$", '\000' },
  iobref = 0x0, gen = 6, fd = 0x7fdbe800f0dc, wind_count = 0, ordering =
{size = 0, off = 0, append = 0, tempted = 0, lied = 0, fulfilled = 0,
    go = 0}}
(gdb) print *((wb_request_t *)0x7fdbe8020200)
$3 = {all = {next = 0x7fdbe8068c30, prev = 0x7fdbe803a608}, todo = {next
= 0x7fdbe8068c40, prev = 0x7fdbe803a618}, lie = {next = 0x7fdbe8068c50,
    prev = 0x7fdbe803a628}, winds = {next = 0x7fdbe8020230, prev =
0x7fdbe8020230}, unwinds = {next = 0x7fdbe8020240, prev =
0x7fdbe8020240}, wip = {
    next = 0x7fdbe8020250, prev = 0x7fdbe8020250}, stub =
0x7fdbe8062c3c, write_size = 131072, orig_size = 4096, total_size = 0,
op_ret = -1,
  op_errno = 116, refcount = 1, wb_inode = 0x7fdbe803a5f0, fop =
GF_FOP_WRITE, lk_owner = {len = 8, data = '\000' },
  iobref = 0x7fdbe80311a0, gen = 4, fd = 0x7fdbe805c89c, wind_count = 3,
ordering = {size = 131072, off = 1220608, append = 0, tempted = -1,
    lied = -1, fulfilled = 0, go = 0}}
(gdb) print *((wb_request_t *)0x7fdbe8068c30)
$4 = {all = {next = 0x7fdbe8021cd0, prev = 0x7fdbe8020200}, todo = {next
= 0x7fdbe8021ce0, prev = 0x7fdbe8020210}, lie = {next = 0x7fdbe803a628,
    prev = 0x7fdbe8020220}, winds = {next = 0x7fdbe8068c60, prev =
0x7fdbe8068c60}, unwinds = {next = 0x7fdbe8068c70, prev =
0x7fdbe8068c70}, wip = {
    next = 0x7fdbe8068c80, prev = 0x7fdbe8068c80}, stub =
0x7fdbe806746c, write_size = 118784, orig_size = 4096, total_size = 0,
op_ret = -1,
  op_errno = 116, refcount = 1, wb_inode = 0x7fdbe803a5f0, fop =
GF_FOP_WRITE, lk_owner = {len = 8, data = '\000' },
  iobref = 0x7fdbe8052b10, gen = 5, fd = 0x7fdbe805c89c, wind_count = 2,
ordering = {size = 118784, off = 1351680, append = 0, tempted = -1,
    lied = -1, fulfilled = 0, go = 0}}

you can see they are all on 'todo' queue, and FLUSH op fd is not the
same WRITE op fd.

> Change-Id: Id687f9cd3b9f281e1a97c83f1ce981ede272b8ab
> BUG: 1372211
> Signed-off-by: Ryan Ding 

Change-Id: Id687f9cd3b9f281e1a97c83f1ce981ede272b8ab
BUG: 1390837
Signed-off-by: Ryan Ding 
Reviewed-on: http://review.gluster.org/15761
Tested-by: Raghavendra G 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

performance/io-threads: Exit all threads on PARENT_DOWN

2016-10-25T06:52:45+00:00

Problem:
When glfs_fini() is called on a volume where client.io-threads is enabled,
fini() will free up iothread xl's private structure but there would be some
threads that are sleeping which would wakeup after the timedwait completes
leading to accessing already free'd memory.

Fix:
As part of parent-down, exit all sleeping threads.

 >BUG: 1381830
 >Change-Id: I0bb8d90241112c355fb22ee3fbfd7307f475b339
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: http://review.gluster.org/15620
 >Smoke: Gluster Build System 
 >CentOS-regression: Gluster Build System 
 >NetBSD-regression: NetBSD Build System 
 >Reviewed-by: Raghavendra G 
 >(cherry picked from commit d7a5ca16911caca03cec1112d4be56a9cda2ee30)

Change-Id: I450e5db84c83fae3237aaa7915c08cd3ee9bde2c
BUG: 1387894
Signed-off-by: Pranith Kumar K 
Reviewed-on: http://review.gluster.org/15701
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

performance/open-behind: Pass O_DIRECT flags for anon fd reads when required

2016-09-23T14:46:09+00:00

        Backport of: http://review.gluster.org/15537
        cherry-picked from a412a4f50d8ca2ae68dbfa93b80757889150ce99

Writes are already passing the correct flags at the time of open().

Also, make io-cache honor direct-io for anon-fds with
O_DIRECT flag during reads.

Change-Id: Iba1a5c2c9e5fe334ba199cdbffd5b08764218460
BUG: 1378814
Signed-off-by: Krutika Dhananjay 
Reviewed-on: http://review.gluster.org/15553
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Pranith Kumar Karampuri

compound fops: Some fixes to compound fops framework

2016-08-31T06:50:40+00:00

Change-Id: I808fd5f9f002a35bff94d310c5d61a781e49570b
BUG: 1360169
Signed-off-by: Anuradha Talur 
Reviewed-on: http://review.gluster.org/15010
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra G