2 machines VM : Master et Slave

Master : training@training
Slave : training@training2

1) Install Java 8
add-apt-repository ppa:webupd8team/java
apt update
apt install -y oracle-java8-set-default

2) Install Hadoop
wget http://apache.claz.org/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz
tar -xzvf hadoop-3.1.1.tar.gz
mv hadoop-3.1.1 /usr/local/hadoop

3) Ajoute des variables d’environements

update-alternatives --display java
sudo nano ~/.bashrc

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/
local/games:/
usr/local/hadoop/bin:/usr/local/hadoop/sbin"
export JAVA_HOME="/usr/lib/jvm/java-8-oracle/jre"
export PATH=$PATH:JAVA_HOME
export HADOOP_HOME="/usr/local/hadoop"
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME

sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

export JAVA_HOME="/usr/lib/jvm/java-8-oracle/jre"

4) les droits d’acces à utilisateur training

usermod -aG training training
chown training:root -R /usr/local/hadoop
chmod g+rwx -R /usr/local/hadoop

5) Cloner la machine VM
6) Géneration de clé RSA

Dans le Master
ssh-keygen -t rsa
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh-copy-id training@training2

7) Configuration hadoop dans Master(training)

core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop1.admintome.lab:9000</value>
 </property>
</configuration>

hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/data/nameNode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/data/dataNode</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>

8) Copier les fichiers de configuration dans le slave

scp /usr/local/hadoop/etc/hadoop/* training2:/usr/local/hadoop/etc/hadoop/

9) Configuration hadoop dans le Slave (training2)

Yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
</property>

10) Démarer Hadoop et Yarn

Dans le Master :
hdfs namenode -format
start-dfs.sh
start-yarn.sh

11) Screenshots

Hadoop Version

HDFS & YARN

hdfs dfsadmin -report

Web UI : http://192.168.1.5:9870