Posts

apache spark on multinode cluster

Steps to install Apache Spark on multi-node cluster 2.1. Recommended Platform ·          OS –  Linux is supported as a development and deployment platform. You can use Ubuntu 14.04 / 16.04 or later . ·          Spark –  Apache Spark 2.x 2.2. Install Spark on Master I. Prerequisites a. Add Entries in hosts file Edit hosts file 1 sudo nano /etc/hosts Now add entries of master and slaves 1 2 3 MASTER-IP master SLAVE01-IP slave01 SLAVE02-IP slave02 (NOTE: In place of MASTER-IP, SLAVE01-IP, SLAVE02-IP put the value of the corresponding IP) b. Install Java 7 (Recommended Oracle Java) 1 2 3 4 sudo apt-get install python-software-properties sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java7-installer c. Install Scala 1 sudo apt-...