apache spark on multinode cluster
Steps to install Apache Spark
on multi-node cluster
2.1.
Recommended Platform
·
OS – Linux is supported
as a development and deployment platform. You can use Ubuntu 14.04 / 16.04
or later .
·
Spark – Apache Spark 2.x
2.2. Install Spark on Master
I. Prerequisites
a. Add Entries in hosts file
Edit hosts file
1
|
sudo
nano /etc/hosts |
Now add entries of master and slaves
1
2
3
|
MASTER-IP
masterSLAVE01-IP
slave01SLAVE02-IP
slave02 |
(NOTE: In place of MASTER-IP, SLAVE01-IP, SLAVE02-IP put the
value of the corresponding IP)
b. Install Java 7 (Recommended
Oracle Java)
1
2
3
4
|
sudo
apt-get install python-software-propertiessudo
add-apt-repository ppa:webupd8team/javasudo
apt-get updatesudo
apt-get install oracle-java7-installer |
c. Install Scala
1
|
sudo
apt-get install scala |
d. Configure SSH
i. Install Open SSH
Server-Client
1
|
sudo
apt-get install openssh-server openssh-client |
ii. Generate Key Pairs
1
|
ssh-keygen
-t rsa |
iii. Configure passwordless SSH
Copy the content of .ssh/id_rsa.pub (of master) to
.ssh/authorized_keys (of all the slaves as well as master)
My master machine is master01(1.2.3.4) ,I want to acess slave01(2.3.4.5) without
password ,So
master01>ssh slave01@2.3.4.5 mkdir -p .ssh
>ssh slave01@2.3.4.5 "chmod 700 .ssh; chmod 640
.ssh/authorized_keys"
ssh slave01@2.3.4.5
iv. Check by SSH to all the Slaves
1
2
|
ssh
slave01ssh
slave02 |
II. Install Spark
a. Download Spark
b. Untar Tarball
tar xzf spark-2.0.0-bin-hadoop2.6.tgz
Create files spark-env.sh
and slaves in conf folder .
è
Goto spark-env.sh in conf folder .Add according
to your java amd spark home path .
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER
-Dspark.deploy.zookeeper.url=1.2.3.4:2181,2.3.4.5:2181,IpAdrress3:2181
-Dspark.deploy.zookeeper.dir=/jars"
export SPARK_WORKER_CORES=4
è
In slaves file add your slave machine ip address
or host name ,
And put according to your home path in all machines .
spark.deploy.recoveryMode=ZOOKEEPER
spark.deploy.zookeeper.url=IPAdress1:2181,IPAdress2:2181,IPAdress3:2181
spark.deploy.zookeeper.dir=/home/slave01/instances/spark-2.2.0-bin-hadoop2.7/jars
Do the
same thing in both slaves ,only no need to add slaves ip in slaves file .
Start master node :-
./sbin/start-master.sh -h <IpAddress/Hostname> -p 7077 --webui-port 8080 --properties-file ha.conf
Start slave node :-
./sbin/start-slave.sh <IpAddress/Hostname>:7077
Comments
Post a Comment