glusterfs.git/xlators/cluster/dht/src/tier.c, branch v3.8.12

cluster/tier: dont promote if estimated block consumption > hi watermark

2016-08-05T08:12:11+00:00

Add test to fail promotion if estimated block consumption grows
beyond hi watermark.

Skip file migrations until next cycle if tier_get_fs_stat() fails
in tier_migrate_using_query_file()

> Reviewed-on: http://review.gluster.org/14780
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> Reviewed-by: mohammed rafi  kc 
> Reviewed-by: Dan Lambright 

(cherry picked from commit 1f4e41e8c2f5f4af4564caba0a08996853f089f4)

Change-Id: Ice04572fa739c09109c4433e65965197482a7beb
BUG: 1362198
Signed-off-by: Milind Changire 
Reviewed-on: http://review.gluster.org/15065
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

cluster/tier: downgrade max-cycle-time log message to INFO

2016-05-21T15:50:23+00:00

The "max cycle time" log message was incorrectly logged as
an error. Downgrade it to INFO.

This is a backport of 14361

> Change-Id: Ia7d074423019fa79443bc6ea694148b7b8da455d
> BUG: 1335973
> Signed-off-by: Dan Lambright 

Change-Id: I91cbd0165fa6c72d3fa5e373bc12578ef0bde9da
BUG: 1336472
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/14362
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: N Balachandran

tier/migrator: Fetch the next query file for the next cycle

2016-04-30T14:30:49+00:00

Problem: When we spawn promote and demote thread, query files
are build. And only query file with index 0 is picked for migration
as the first query file. This may not be suitable for scenarios,
where the file in the query are too big to move in the first cycle,
as a result file in the other query files always get missed. We need to
shuffle so that other query files also get a chance.

Fix: Remember the previous first query file and shift it by one index,
before the migration starts.

Change-Id: I704947bcf4bab6b20b1179a6d9ae4a15a3d51bd9
BUG: 1330353
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/14068
Tested-by: Joseph Fernandes
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright 
NetBSD-regression: NetBSD Build System

cluster/tier: add tunable to migrate files based on size

2016-03-17T06:12:04+00:00

This fix adds a paramater "tier-max_promote_size" to control wether
a file is migrated or not based on its size. By default the value
is 0, meaning all files are migrated. If set to a non-zero
value, files larger than the parameter won't be moved
in tiered volumes.

Change-Id: Ia6b88e9b2508935bef500d956f9192e59670fe00
BUG: 1313495
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/13570
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Joseph Fernandes

TIER: stopping the tierd when the volume goes down

2016-03-15T06:06:01+00:00

If there are large number of files to be migrated
and by this time if the volume goes down, then the tierd
has to be stopped. But on a huge query file list it keeps
checking for each file before stopping. If the volume comes
up before the old tierd dies then due to the
presence of old tierd new one won't be created. After
the old one completes the task, it dies and the status
ends up as failed.

This patch will check if the status is still running and then let
it continue its work. Else it will stop running the tierd.

Change-Id: I6522a4e2919e84bf502b99b13873795b9274f3cd
BUG: 1315659
Signed-off-by: hari gowtham 
Reviewed-on: http://review.gluster.org/13646
Tested-by: Dan Lambright 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright

Tier: Avoiding stale entries from causing demotion to stop

2016-03-14T06:05:42+00:00

When the parent GFID is a stale entry, the lookup on this parent
fails and this in turn fails the demotion process.

This patch will make the stale entry error to be skipped.

Situation for pargfid to be stale:
Consider a folder from a tar file. Once the tar file is untared
the files in the tar-file will start to demote.
when the demotion is under progress, if we tend to delete
the actual folder, then the files under it which are
undergoing demotion will do a lookup on the parent which was
deleted and become stale entry. This stale entry fails the
Lookup and this will fail the demotion of the other files(not from
tar) that are supposed to be demoted.

Change-Id: I3d47c32c4077526d477a25912b0135bab98b23fc
BUG: 1311178
Signed-off-by: hari gowtham 
Reviewed-on: http://review.gluster.org/13501
Tested-by: hari gowtham 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright 
NetBSD-regression: NetBSD Build System

cluster/tier: make promotion and demotion independant

2016-02-29T01:32:22+00:00

Currently a main loop in tiering spawns promotion and demotion threads,
and does a join to wait for them to complete. When one of the two
threads takes a long time, the main thread waits for it
before exiting the join. It may wait so long the scheduled time
for the other thread is skipped. In the case of demotion, it may be
a long time before another attempt. This patch fixes that by
making the promotion and demotion activities independant. A side
effect of this change is the logic is significantly simplified.

Change-Id: I1196bd4bbfc95e8aa326a9bd4ebf395032369d1c
BUG: 1306852
Signed-off-by: Dan Lambright 
Reviewed-on: http://review.gluster.org/13433
Smoke: Gluster Build System 
Reviewed-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/tier : Fixed wrong variable comparison

2016-02-11T06:25:42+00:00

The wrong variable was being checked to determine
the watermark value.

Change-Id: If4c97fa70b772187f1fcbdf5193e077cb356a8b1
BUG: 1303895
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/13357
Smoke: Gluster Build System 
Reviewed-by: Dan Lambright 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System

cluster/tier: Create linkfiles to hardlinks correctly

2016-02-07T17:13:35+00:00

There is a bug in the way hardlinks are handled in tiered volumes. Ideally, the tier linkto files on the cold tier to files that are hardlinks to each other on the hot tier, should themselves be hardlinks of each other. As they are not, they end up being files with the same gfid but different names for the cold tier dht, and end up overwriting the cached-subvol information stored in the dht inode-ctx.

Change-Id: Ic658a316836e6a1729cfea848b7d212674b0edd2
BUG: 1305277
Signed-off-by: N Balachandran 
Reviewed-on: http://review.gluster.org/13391
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Dan Lambright 
Tested-by: Dan Lambright

tier/gfdb : Round-Robin read of query files

2016-02-03T18:04:40+00:00

1. Each brick on a host will get a separate query file.
2. While reading query record from these query files we
   read them in a Round-Robin manner.
3. When an error occurs during migration we rename it to
   query file with an time stamp and .err extension for
   better debugging.

Change-Id: I27c4285d24fd695d2d5cbd9fd7db3879d277ecc8
BUG: 1302772
Signed-off-by: Joseph Fernandes 
Reviewed-on: http://review.gluster.org/13293
Smoke: Gluster Build System 
Tested-by: N Balachandran 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Dan Lambright 
CentOS-regression: Gluster Build System