apache spark on multinode cluster

Steps to install Apache Spark on multi-node cluster

2.1. Recommended Platform
·         OS – Linux is supported as a development and deployment platform. You can use Ubuntu 14.04 / 16.04 or later .
·         Spark – Apache Spark 2.x

2.2. Install Spark on Master

I. Prerequisites

a. Add Entries in hosts file
Edit hosts file
1
sudo nano /etc/hosts
Now add entries of master and slaves
1
2
3
MASTER-IP master
SLAVE01-IP slave01
SLAVE02-IP slave02
(NOTE: In place of MASTER-IP, SLAVE01-IP, SLAVE02-IP put the value of the corresponding IP)
b. Install Java 7 (Recommended Oracle Java)
1
2
3
4
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
c. Install Scala
1
sudo apt-get install scala
d. Configure SSH
i. Install Open SSH Server-Client
1
sudo apt-get install openssh-server openssh-client
ii. Generate Key Pairs
1
ssh-keygen -t rsa
iii. Configure passwordless SSH
Copy the content of .ssh/id_rsa.pub (of master) to .ssh/authorized_keys (of all the slaves as well as master)

Example :-
My master machine is master01(1.2.3.4) ,I want to acess slave01(2.3.4.5) without password ,So
master01>ssh slave01@2.3.4.5 mkdir -p .ssh

>cat .ssh/id_rsa.pub | ssh  slave01@2.3.4.5 'cat >> .ssh/authorized_keys'
>ssh  slave01@2.3.4.5 "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
ssh  slave01@2.3.4.5
iv. Check by SSH to all the Slaves
1
2
ssh slave01
ssh slave02

II. Install Spark

a. Download Spark
You can download the latest version of spark from http://spark.apache.org/downloads.html.
b. Untar Tarball
tar xzf spark-2.0.0-bin-hadoop2.6.tgz
Create files spark-env.sh  and slaves in conf folder .
è Goto spark-env.sh in conf folder .Add according to your java amd spark home path .
export JAVA_HOME=/usr/lib/jvm/java-8-oracle

export SPARK_CLASSPATH=/home/master01/instances/spark-2.2.0-bin-hadoop2.7/jars
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=1.2.3.4:2181,2.3.4.5:2181,IpAdrress3:2181 -Dspark.deploy.zookeeper.dir=/jars"
export SPARK_WORKER_CORES=4

here ,I am running 3 zookeeper in cluster mode .

è In slaves file add your slave machine ip address or host name ,

Create a file in SPARK_HOME folder  called as ha.conf
And put according to your home path in all machines .
spark.deploy.recoveryMode=ZOOKEEPER
spark.deploy.zookeeper.url=IPAdress1:2181,IPAdress2:2181,IPAdress3:2181
spark.deploy.zookeeper.dir=/home/slave01/instances/spark-2.2.0-bin-hadoop2.7/jars
Do the same thing in both slaves ,only no need to add slaves ip in slaves file .

Start master node :-

./sbin/start-master.sh -h <IpAddress/Hostname> -p 7077 --webui-port 8080 --properties-file ha.conf

Start slave node :-

./sbin/start-slave.sh <IpAddress/Hostname>:7077



Comments