blob: 2894fa71302c542f79d0f142027526769f8724bb (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
|
Managing Hadoop Compatible Storage
==================================
GlusterFS provides compatibility for Apache Hadoop and it uses the
standard file system APIs available in Hadoop to provide a new storage
option for Hadoop deployments. Existing MapReduce based applications can
use GlusterFS seamlessly. This new functionality opens up data within
Hadoop deployments to any file-based or object-based application.
Architecture Overview
=====================
The following diagram illustrates Hadoop integration with GlusterFS:
Advantages
==========
The following are the advantages of Hadoop Compatible Storage with
GlusterFS:
- Provides simultaneous file-based and object-based access within
Hadoop.
- Eliminates the centralized metadata server.
- Provides compatibility with MapReduce applications and rewrite is
not required.
- Provides a fault tolerant file system.
Preparing to Install Hadoop Compatible Storage
==============================================
This section provides information on pre-requisites and list of
dependencies that will be installed during installation of Hadoop
compatible storage.
Pre-requisites
--------------
The following are the pre-requisites to install Hadoop Compatible
Storage :
- Hadoop 0.20.2 is installed, configured, and is running on all the
machines in the cluster.
- Java Runtime Environment
- Maven (mandatory only if you are building the plugin from the
source)
- JDK (mandatory only if you are building the plugin from the source)
- getfattr - command line utility
Installing, and Configuring Hadoop Compatible Storage
=====================================================
This section describes how to install and configure Hadoop Compatible
Storage in your storage environment and verify that it is functioning
correctly.
1. Download `glusterfs-hadoop-0.20.2-0.1.x86_64.rpm` file to each
server on your cluster. You can download the file from [][].
2. To install Hadoop Compatible Storage on all servers in your cluster,
run the following command:
`# rpm –ivh --nodeps glusterfs-hadoop-0.20.2-0.1.x86_64.rpm`
The following files will be extracted:
- /usr/local/lib/glusterfs-Hadoop-version-gluster\_plugin\_version.jar
- /usr/local/lib/conf/core-site.xml
3. (Optional) To install Hadoop Compatible Storage in a different
location, run the following command:
`# rpm –ivh --nodeps –prefix /usr/local/glusterfs/hadoop glusterfs-hadoop- 0.20.2-0.1.x86_64.rpm`
4. Edit the `conf/core-site.xml` file. The following is the sample
`conf/core-site.xml` file:
<configuration>
<property>
<name>fs.glusterfs.impl</name>
<value>org.apache.hadoop.fs.glusterfs.Gluster FileSystem</value>
</property>
<property>
<name>fs.default.name</name>
<value>glusterfs://fedora1:9000</value>
</property>
<property>
<name>fs.glusterfs.volname</name>
<value>hadoopvol</value>
</property>
<property>
<name>fs.glusterfs.mount</name>
<value>/mnt/glusterfs</value>
</property>
<property>
<name>fs.glusterfs.server</name>
<value>fedora2</value>
</property>
<property>
<name>quick.slave.io</name>
<value>Off</value>
</property>
</configuration>
The following are the configurable fields:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Property Name Default Value Description
---------------------- -------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
fs.default.name glusterfs://fedora1:9000 Any hostname in the cluster as the server and any port number.
fs.glusterfs.volname hadoopvol GlusterFS volume to mount.
fs.glusterfs.mount /mnt/glusterfs The directory used to fuse mount the volume.
fs.glusterfs.server fedora2 Any hostname or IP address on the cluster except the client/master.
quick.slave.io Off Performance tunable option. If this option is set to On, the plugin will try to perform I/O directly from the disk file system (like ext3 or ext4) the file resides on. Hence read performance will improve and job would run faster.
> **Note**
>
> This option is not tested widely
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
5. Create a soft link in Hadoop’s library and configuration directory
for the downloaded files (in Step 3) using the following commands:
`# ln -s >`
For example,
`# ln –s /usr/local/lib/glusterfs-0.20.2-0.1.jar /lib/glusterfs-0.20.2-0.1.jar`
`# ln –s /usr/local/lib/conf/core-site.xml /conf/core-site.xml `
6. (Optional) You can run the following command on Hadoop master to
build the plugin and deploy it along with core-site.xml file,
instead of repeating the above steps:
`# build-deploy-jar.py -d -c `
Starting and Stopping the Hadoop MapReduce Daemon
=================================================
To start and stop MapReduce daemon
- To start MapReduce daemon manually, enter the following command:
`# /bin/start-mapred.sh`
- To stop MapReduce daemon manually, enter the following command:
`# /bin/stop-mapred.sh `
> **Note**
>
> You must start Hadoop MapReduce daemon on all servers.
[]: http://download.gluster.com/pub/gluster/glusterfs/qa-releases/3.3-beta-2/glusterfs-hadoop-0.20.2-0.1.x86_64.rpm
|
