Hadoop MultiNode Configuration

     Hadoop is a framework written in Java for running applications on large clusters of commodity hardware and incorporates features similar to those of the Google File System (GFS) and of the MapReduce computing paradigm. Hadoop’s HDFS is a highly fault-tolerant distributed file system and, like Hadoop in general, designed to be deployed on low-cost hardware. It provides high throughput access.

In this procedure i am using 

• Ubuntu 14.04
• Hadoop 1.2.1

STEP 1
 First you need to configure the single node .To configure Hadoop single node just follow the below link
Single Node Configuration

STEP 2

In multi-node we need more than one system.One system act as Master(in this procedure bimal is Master) and more than one slave systems.When the number of system increases the efficiency also increases .

Now we are getting into Multi-Node configuration of Hadoop.First we need to edit a file in /etc/hosts by using the following command.

• bimal@bimal:~$ sudo gedit /etc/hosts

In multi-node we have more than one systems.Then you need to add their IP address and name of the systems to the file /etc/hosts.
Dynamic and Static IP address are possible to use but Static IP address is prferable.

SSH ACCESS (Only for Master)

Then you need to copy the masters system public key to all slave systems .Run the following command as hduser.

• hduser@bimal:~$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub hduser@thomas-hp-pavilion-g6-notebook-pc

If you have more than one slave systems then you should  copy the masters public key to each of the slave systems, by using the above command.

So connecting master to master(bimal).

• hduser@bimal:~$ ssh bimal

So connecting Master to Slave(thomas-hp-pavilion-g6-notebook-pc).

• hduser@bimal:~$ ssh thomas-hp-pavilion-g6-notebook-pc

EDIT .xml files(For both Master and Slaves)

    Change directory to /usr/local/hadoop/conf/

• hduser@bimal:/usr/local/hadoop/conf$ cd /usr/local/hadoop/conf/
• hduser@bimal:/usr/local/hadoop/conf$ sudo gedit core-site.xml

Then change the localhost to masters name
Edit core-site.xml

Edit mapred-site.xml

• hduser@bimal:/usr/local/hadoop/conf$ sudo gedit mapred-site.xml

Edit hdfs-site.xml

• hduser@bimal:/usr/local/hadoop/conf$ sudo gedit hdfs-site.xml

According to the number of systems and replication you can change the value.

Edit masters and slaves file(Both Master and Slaves)


Edit masters file

• hduser@bimal:/usr/local/hadoop/conf$ sudo gedit masters

Both master and slaves must edit the file conf/masters and add your master name to the file.

Edit slaves file

•  hduser@bimal:/usr/local/hadoop/conf$ sudo gedit slaves

Both master and slave must edit slaves file and add all slaves host name to the slaves file

Formatting the HDFS file system using namenode

By using the following command to format the HDFS file system.

• hduser@bimal:/usr/local/hadoop/bin$ hadoop namenode -format

After formating the file system we start the cluster using the below command

•  hduser@bimal:/usr/local/hadoop/bin$ start-dfs.sh

After executing the above command execute the following command

• hduser@bimal:/usr/local/hadoop/bin$ start-mapred.sh
• hduser@bimal:/usr/local/hadoop/bin$ jps

 Some times DataNode may not started then you need to clear /app/hadoop/tmp/

• hduser@bimal:/usr/local/hadoop/bin$ rm -r /app/hadoop/tmp/*

Then re-format the HDFS file system before reformatting you need to stop all nodes.

• hduser@bimal:/usr/local/hadoop/bin$ stop-all.sh

• hduser@bimal:/usr/local/hadoop/bin$ hadoop namenode -format
• Then start nodes using the above commands

Reference

• http://www.michael-noll.com/