Hadoop Configuration Files

Hadoop Configuration Files

1. hadoop-env.sh

This file specifies environment variables that affect the JDK used by Hadoop Daemon (bin/hadoop).
As Hadoop framework is written in Java and uses Java Runtime environment, one of the important environment variables for Hadoop daemon is $JAVA_HOME in hadoop-env.sh. This variable directs Hadoop daemon to the Java path in the system.

This file is also used for setting another Hadoop daemon execution environment such as heap size (HADOOP_HEAP), hadoop home (HADOOP_HOME), log file location (HADOOP_LOG_DIR), etc.


Note: For the simplicity of understanding the cluster setup, we have configured only necessary parameters to start a cluster.

2. core-site.sh

This file informs Hadoop daemon where NameNode runs in the cluster. It contains the configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce.

Name node, Hadoop daemon, Configuration settings,Hadoop Core

Where hostname and port are the machine and port on which NameNode daemon runs and listens. It also informs the Name Node as to which IP and port it should bind. The commonly used port is 8020 and you can also specify IP address rather than hostname.

3. hdfs-site.sh

This file contains the configuration settings for HDFS daemons; the Name Node, the Secondary Name Node, and the data nodes.

You can also configure hdfs-site.xml to specify default block replication and permission checking on HDFS. The actual number of replications can also be specified when the file is created. The default is used if replication is not specified in create time.

The value “true” for property ‘dfs.permissions’ enables permission checking in HDFS and the value “false” turns off the permission checking. Switching from one parameter value to the other does not change the mode, owner or group of files or directories.

HDFS daemons; the Name Node, the Secondary Name Node, and the data nodes.

4.mapred-site.sh

This file contains the configuration settings for MapReduce daemons; the job tracker and the task-trackers. The mapred.job.tracker parameter is a hostname (or IP address) and port pair on which the Job Tracker listens for RPC communication. This parameter specify the location of the Job Tracker to Task Trackers and MapReduce clients.

MapReduce daemons; the job tracker and the task-trackers

You can replicate all of the four files explained above to all the Data Nodes and Secondary Namenode. These files can then be configured for any node specific configuration e.g. in case of a different JAVA HOME on one of the Datanodes.

5.Masters

This file informs about the Secondary Namenode location to hadoop daemon. The ‘masters’ file at Master server contains a hostname Secondary Name Node servers.

Secondary Namenode location, hadoop daemon

6.Slaves

The ‘slaves’ file at Master node contains a list of hosts, one per line, that are to host Data Node and Task Tracker servers.

Slaves file, Master node, Hadoop

The ‘slaves’ file on Slave server contains the IP address of the slave node. Notice that the ‘slaves’ file at Slave node contains only its own IP address and not of any other Data Nodes in the cluster.

Installation of Hadoop 1.x

Installation of Hadoop 1.x

1. Configure the hostname
     vi /etc/hostname
2. Configure reverse DNS
     vi /etc/hosts
3. Create directory

mkdir /usr/java
mkdir /usr/hadoop
mkdir /usr/hadoop/data
mkdir /usr/hadoop/namenode
mkdir /usr/hadoop/tmp

4. Configure Java

5. Add group
groupadd hadoop
useradd -g hadoop hduser
passwd hduser

6. Extract Hadoop file

7. Give permission to Hadoop folder of hduser

chown -R hduser:hadoop /usr/hadoop

8. Password less SSH

ssh-keygen
cd .ssh/
cat id_rsa.pub >> authorized_keys

9. Configure .bashrc file

export HADOOP_HOME=/usr/hadoop/hadoop-1.2.1/
export PATH=$PATH:$HADOOP_HOME/bin


10. Configure Hadoop Configuration file.

hdfs-site.xml

<configuration>
<property>
    <name>dfs.data.dir</name>
    <value>/usr/hadoop/data</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
    <value>/usr/hadoop/namenode</value>
<final>true</final>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
</description>
    </property>
</configuration>


core-site.xml

<property>
        <name>fs.default.name</name>
        <value>hdfs://node1.hadoop.com:54310</value>
    </property>
        <property>
        <name>dfs.permission</name>
        <value>false</value>
    </property>
        <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/hadoop/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>

mapred-site.xml

<property>
<name>mapred.job.tracker</name>
<value>node1.hadoop.com:54311</value>
<description>The host and port that the MapReduce job tracker runs at. Tf "local", then jobs are in-process as a single map and reduce task.</description>
</property>

Slave and master file : add the hostname into it.

11. Format the namenode

hadoop namenode -format -force

12. run the start-all.sh file.