From 48c834e5a99cb779dcc8c6a4b3d7cdd924d9fa26 Mon Sep 17 00:00:00 2001 From: Joseph Fernandes Date: Mon, 29 Feb 2016 10:45:42 +0530 Subject: WORM-Retention : Design specs for WORM/Retention feature in GlusterFS Adding high level design specs for WORM/Retention feature in GlusterFS Change-Id: I3b113c37de1227fb6fca37371ad0117b3f1eda53 Signed-off-by: Joseph Fernandes Reviewed-on: http://review.gluster.org/13538 Reviewed-by: Niels de Vos Tested-by: Niels de Vos --- under_review/image/WORM-Life.png | Bin 0 -> 63286 bytes under_review/image/WORM-Tier-Life.png | Bin 0 -> 50205 bytes under_review/image/WORM-Tiering2.png | Bin 0 -> 108420 bytes under_review/worm-compliance.md | 158 ++++++++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+) create mode 100644 under_review/image/WORM-Life.png create mode 100644 under_review/image/WORM-Tier-Life.png create mode 100644 under_review/image/WORM-Tiering2.png create mode 100644 under_review/worm-compliance.md diff --git a/under_review/image/WORM-Life.png b/under_review/image/WORM-Life.png new file mode 100644 index 0000000..935787f Binary files /dev/null and b/under_review/image/WORM-Life.png differ diff --git a/under_review/image/WORM-Tier-Life.png b/under_review/image/WORM-Tier-Life.png new file mode 100644 index 0000000..6fb71a9 Binary files /dev/null and b/under_review/image/WORM-Tier-Life.png differ diff --git a/under_review/image/WORM-Tiering2.png b/under_review/image/WORM-Tiering2.png new file mode 100644 index 0000000..c44888f Binary files /dev/null and b/under_review/image/WORM-Tiering2.png differ diff --git a/under_review/worm-compliance.md b/under_review/worm-compliance.md new file mode 100644 index 0000000..85f2360 --- /dev/null +++ b/under_review/worm-compliance.md @@ -0,0 +1,158 @@ +Feature +------- +WORM/Retention Semantics + +Owners +------ +- Joseph Fernandes +- Karthik Subrahmanya +- Vijaikumar Mallikarjuna +- Raghavendra Talur + +Summary +------- + +This feature is about having WORM-based compliance/archiving solution in glusterfs. It mainly focuses on +- **WORM/Retention**: + Store data in a tamper-proof and secure way & Data accessibility policies + + To address the above we are proposing the following in glusterfs + - Write Once Read Multiple (WORM) File System + +Detailed Description +-------------------- +#### Write Once Read Multiple (WORM) File System + - ##### WORM/Retention FS: + A file-system that supports immutable(read-only) and undeletable files. Each file have its own worm/retention attribute. + + - ##### WORM/Retention transition: + There are 3 states in WORM/Retention FS: + 1. WORM/Retained (WR): where a file is both immutable and undeletable for a specific amount of time + 2. WORM (W) : where a file is only immutable + 3. Legal Hold (LH) : where a file is immutable and is non-delete-able for indefinite time + + A file makes transition between these 3 states. + Methods of making WORM/Retention transition: + - **Manual POSIX command**: + Using POSIX command chmod -w or 444 or equivalent mode. + - **Automatic Retention Transition (Auto-commit)**: + Automatic transition between the different states of a file. + Dormant Files (files which are untouched for a long time) will be converted into WORM files based on Auto-commit Period (Time interval at which the namespace scan has to take place to make the state transition) + - **Lazy Auto-commit**: + IO Triggered Using timeouts for untouched files. The next IO will cause the transition. + - **Scheduled Auto-commit**: + Scan Triggered Using timeouts for untouched files. The next scheduled namespace scan will cause the transition. + CTR DB via **libgfdb** can be used to find files that have not changed. This can be verified with stat of the file. + + **Retention Profiles/Policies**: + Configurable Policies that guide the WORM/Retention behavior. Contains, + - Default Retention Period: default time period till which a WORM file should be retained(undeletable) + - Auto-commit Period: Time interval at which the namespace scan has to take place to check for the dormant files and make the state ransition. + - Mode of Retention: + 1. Relax : Where the Retention period can be increased or decreased + 2. Enterprise: Where we can only increase the retention time but can not decrease it + 3. Compliance: Where we can not change the retention time + - Auto-Deletion + + It can be on a Volume Level or Directory/Share Level + + ##### Life cycle of a file in a WORM/Retained filesystem + + ![WORM-LifeCycle](image/WORM-Life.png) + + +Current status +------ +Proposal phase : POC + +Benefit to GlusterFS +------ +* Glusterfs will have more realistic Compliance & Archival solution +* Glusterfs HSM (Hierarchical Storage Management) + +Scope +------ +1. Implement WORM-Retention Semantics in GlusterFS using the WORM Xlator +2. Integrate WORM-Retention with Bitrot with WORM-Retention Feature for Data Validation +3. Help in implementing Control of atime,mtime,ctime [3], as its a requirement for us +4. Tiering based on Compliance + +Nature of proposed change +------------- +Change to the existing WORM Translator. The existing WORM implementations works at volume level as a switch. It makes all the files in the volume read-only if it is switched on. + +In this design we are changing it from volume level to file level, in which it saves the WORM/Retention attribute for each file in the volume. According to the WORM/Retention value it makes the file immutable/undeletable or both, so that the other files in the volume will not get immutable/undeletable. The files once WORMed, can not revert back to the normal state. + +You can always decide on which type of WORM you want to make use of and switch between either volume level WORM or File level WORM. + +Implications on manageability +------------- + + +Few vol set commands for setting retention profiles. + +Implications on presentation layer +------------- + + + +TBD + +Implications on persistence layer +------------- + +None + +Implications on 'GlusterFS' backend +------------- +None that we can see + +Modification to GlusterFS metadata +------------- +1. Per file worm retention state + retention period xattr +2. on the root of the vol the retention profile as xattr. + +Implications on 'glusterd' +------------- + + + +1. Few vol set commands for setting retention profiles. + +How To Test +------------- + + + +TBD As the design gets in detail. + +User Experience +------------- + + + +TBD As the design gets in detail. + +Dependencies +------------- + + + + +Documentation +------------- + + + +TBD + +Status +------------- + +POC + +Comments and Discussion +------------- + + + -- cgit