Installation of Spark on Google Compute Engine

Spark On Google Compute Engine

GCE stands for Google Compute Engine. It is an infrastructure as a service which lets you execute your large scale computing workloads on the virtual machines which have been hosted at the infrastructure for Google. It is also possible for you to create your own and new machine and this will take you a very short time.

The following are the installation steps:

1. Begin by creating a CentOS image on the GCE. The space used should be at least 3.8 GB, otherwise the system will not compile.

2. Once you are done, you can perform an ssh into the new machine. An example of how this can be done is shown below:

gcloud compute –service_version=”xyx1″ –project=”spark-project-1″ ssh –
-zone=”europe-west1-a” “spark-box-3g”

3. The next step should be the installation of Java. The following command can be used for this purpose:

sudo yum install java-1.7.0-openjdk-devel

To see the packages which are available for your download, just use the following command:

yum search java | grep ‘java-’

You should also ensure that Python, Java, and Scala have been installed.

                                       Checkout Apache Spark Interview Questions

4. The next step should involve the installation of Git. This can be done by executing the command given below:

yum install git

5. You can then use the “wget” command so as to obtain the necessary packages online.

6. You can then run the assembly sbt/sbt.

Once you have completed the above steps, you will be done and set to go.