If you're looking for Apache Spark Interview Questions for Experienced or Freshers, you are at the right place. There are a lot of opportunities from many reputed companies in the world. According to research Apache Spark has a market share of about 4.9%. So, You still have an opportunity to move ahead in your career in Apache Spark Development. Mindmajix offers Advanced Apache Spark Interview Questions 2022 that helps you in cracking your interview & acquire your dream career as Apache Spark Developer.
|Interested in mastering Apache Spark Course? Enroll now for a FREE demo on Apache Spark Training.|
|Spark Vs Hadoop|
|Data processing||Part of Hadoop, hence batch processing||Batch Processing even for high volumes|
|Streaming Engine||Apache spark streaming - micro-batches||Map-Reduce|
|Data Flow||Direct Acyclic Graph-DAG||Map-Reduce|
|Computation Model||Collect and process||Map-Reduce batch-oriented model|
|Performance||Slow due to batch processing||Slow due to batch processing|
|Memory Management||Automatic memory management in the latest release||Dynamic and static - Configurable|
|Fault Tolerance||Recovery available without extra code||Highly fault-tolerant due to Map-Reduce|
|Scalability||Highly scalable - spark Cluster(8000 Nodes)||Highly scalable - Produces a large number of nodes|
Learn Spark vs Hadoop What's Better to Learn First.
Spark is a parallel data processing framework. It allows to develop of fast, unified big data applications combine batch, streaming, and interactive analytics.
Spark is the third-generation distributed data processing platform. It’s the unified big data solution for all big data processing problems such as batch, interacting, streaming processing. So it can ease many big data problems.
Spark’s primary core abstraction is called Resilient Distributed Datasets. RDD is a collection of partitioned data that satisfies these properties. Immutable, distributed, lazily evaluated, catchable are common RDD properties.
|Do you know what is Apache Airflow?|
Once created and assign a value, it’s not possible to change, this property is called Immutability. Spark is by default immutable, it does not allow updates and modifications. Please note data collection is not immutable, but data value is immutable.
RDD can automatically the data is distributed across different parallel computing nodes.
If you execute a bunch of programs, it’s not mandatory to evaluate immediately. Especially in Transformations, this Laziness is a trigger.
Keep all the data in memory for computation, rather than going to the disk. So Spark can catch the data 100 times faster than Hadoop.
Spark is responsible for scheduling, distributing, and monitoring the application across the cluster.
Partition is a logical division of the data, this idea is derived from Map-reduce (split). Logical data is specifically derived to process the data. Small chunks of data also it can support scalability and speed up the process. Input data, intermediate data, and output data everything is Partitioned RDD.
Spark uses map-reduce API to do the partition the data. In Input format, we can create a number of partitions. By default, HDFS block size is partition size (for best performance), but it’s possible to change partition size like Split.
Spark is a processing engine, there is no storage engine. It can retrieve data from any storage engine like HDFS, S3, and other data resources.
No not mandatory, but there is no separate storage in Spark, so it uses a local file system to store the data. You can load data from the local system and process it, Hadoop or HDFS is not mandatory to run spark application.
When a programmer creates an RDDs, SparkContext connects to the Spark cluster to create a new SparkContext object. SparkContext tells spark how to access the cluster. SparkConf is a key factor to create a programmer application.
SparkCore is a base engine of the apache spark framework. Memory management, fault tolerance, scheduling, and monitoring jobs, interacting with store systems are primary functionalities of Spark.
SparkSQL is a special component on the spark core engine that supports SQL and HiveQueryLanguage without changing any syntax. It’s possible to join the SQL table and HQL table.
Spark Streaming is the real-time processing of streaming data API. Spark streaming gathers streaming data from different resources like web server log files, social media data, stock market data, or Hadoop ecosystems like Flume, and Kafka.
The programmer sets a specific time in the configuration, within this time how much data gets into the Spark, that data separates as a batch. The input stream (DStream) goes into spark streaming. The framework breaks up into small chunks called batches, then feeds into the spark engine for processing. Spark Streaming API passes those batches to the core engine. The core engine can generate the final results in the form of streaming batches. The output also in the form of batches. It can allow streaming data and batch data for processing.
Mahout is a machine learning library for Hadoop, similarly, MLlib is a Spark library. MetLib provides different algorithms, that algorithms scale-out on the cluster for data processing. Most of the data scientists use this MLlib library.
|Leave an Inquiry to Learn: Apache Spark Course in Bangalore|
GraphX is a Spark API for manipulating Graphs and collections. It unifies ETL, other analysis, and iterative graph computation. It's the fastest graph system, provides fault tolerance and ease of use without special skills.
FS API can read data from different storage devices like HDFS, S3, or Local FileSystem. Spark uses FS API to read data from different storage engines.
Every transformation generates a new partition. Partitions use HDFS API so that partition is immutable, distributed, and fault-tolerant. Partition also aware of data locality.
Spark provides two special operations on RDDs called transformations and Actions. Transformation follows lazy operation and temporarily holds the data until unless called the Action. Each transformation generates/returns a new RDD. Example of transformations: Map, flatMap, groupByKey, reduceByKey, filter, co-group, join, sortByKey, Union, distinct, sample are common spark transformations.
Actions are RDD’s operation, that value returns back to the spar driver programs, which kick off a job to execute on a cluster. Transformation’s output is an input of Actions. reduce, collect, take a sample, take, first, saveAsTextfile, saveAsSequenceFile, countByKey, for each is common actions in Apache spark.
Lineage is an RDD process to reconstruct lost partitions. Spark not replicate the data in memory, if data lost, RDD uses lineage to rebuild lost data. Each RDD remembers how the RDD build from other datasets.
The map is a specific line or row to process that data. In FlatMap each input item can be mapped to multiple output items (so the function should return a Seq rather than a single item). So most frequently used to return Array elements.
Broadcast variables let the programmer keep a read-only variable cached on each machine, rather than shipping a copy of it with tasks. Spark supports 2 types of shared variables called broadcast variables (like Hadoop distributed cache) and accumulators (like Hadoop counters). Broadcast variables are stored as Array Buffers, which sends read-only values to work nodes.
Spark of-line debuggers are called accumulators. Spark accumulators are similar to Hadoop counters, to count the number of events and what’s happening during the job you can use accumulators. Only the driver program can read an accumulator value, not the tasks.
There are two methods to persist the data, such as persist() to persist permanently and cache() to persist temporarily in the memory. Different storage level options there such as MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY, and many more. Both persist() and cache() uses different options depends on the task.
Yes. For the following reason:
Since spark runs on top of Yarn, it utilizes yarn for the execution of its commands over the cluster’s nodes.
So, you just have to install Spark on one node.
Spark utilizes memory. The developer has to be careful. A casual developer might make the following mistakes:
The first problem is well tackled by Hadoop Map reduce paradigm as it ensures that the data your code is churning is fairly small at a point in time thus you can make the mistake of trying to handle whole data on a single node.
The second mistake is possible in Map-Reduce too. While writing Map-Reduce, the user may hit a service from inside of map() or reduce() too many times. This overloading of service is also possible while using Spark.
The full form of RDD is a resilient distributed dataset. It is a representation of data located on a network that is
RDD provides two kinds of operations: Transformations and Actions.
The transformations are the functions that are applied on an RDD (resilient distributed data set). The transformation results in another RDD. A transformation is not executed until an action follows.
The example of transformations are:
An action brings back the data from the RDD to the local machine. Execution of action results in all the previously created transformations. The example of actions are:
def myAvg(x, y):
avg = myrdd.reduce(myAvg);
The average function is not commutative and associative;
I would simply sum it and then divide it by count.
def sum(x, y): return x+y; total = myrdd.reduce(sum); avg = total / myrdd.count();
The only problem with the above code is that the total might become very big thus overflow. So, I would rather divide each number by count and then sum in the following way.
cnt = myrdd.count(); def devideByCnd(x): return x/cnt; myrdd1 = myrdd.map(devideByCnd); avg = myrdd.reduce(sum);
# We would first load the file as RDD from HDFS on a spark
numsAsText = sc.textFile(“hdfs://hadoop1.knowbigdata.com/user/student/sgiri/mynumbersfile.txt”);
# Define the function to compute the squaresdef toSqInt(str):
v = int(str); return v*v;
#Run the function on spark rdd as transformation
nums = numsAsText.map(toSqInt);
#Run the summation as reduce action
total = nums.reduce(sum)
#finally compute the square root. For which we need to import math.
import math; print math.sqrt(total);
numsAsText =sc.textFile(“hdfs://hadoop1.knowbigdata.com/user/student/sgiri/mynumbersfile.txt”); def toInt(str): return int(str); nums = numsAsText.map(toInt); def sqrtOfSumOfSq(x, y): return math.sqrt(x*x+y*y); total = nums.reduce(sum) import math; print math.sqrt(total);
Yes. The approach is correct and sqrtOfSumOfSq is a valid reducer.
You are doing the square and square root as part of the reducing action while I am squaring in the map() and summing in reduce in my approach.
My approach will be faster because in your case the reducer code is heavy as it is calling math.sqrt() and reducer code is generally executed approximately n-1 times the spark RDD.
The only downside of my approach is that there is a huge chance of integer overflow because I am computing the sum of squares as part of the map.
#This will load the bigtextfile.txt as RDD in the sparklines =
#define a function that can break each line into words
def toWords(line): return line.split();
# Run the towards function on each element of RDD on spark as flatMap transformation.
# We are going to flatMap instead of the map because our function is returning multiple values.
words = lines.flatMap(toWords);
# Convert each word into (key, value) pair. Her key will be the word itself and her value will be 1.
def toTuple(word): return (word, 1); wordsTuple = words.map(toTuple);
# Now we can easily do the reduceByKey() action.
def sum(x, y): return x+y; counts = wordsTuple.reduceByKey(sum)
# Now, print
lines = sc.textFile(“hdfs://hadoop1.knowbigdata.com/user/student/sgiri/bigtextfile.txt”); def isFound(line): if line.find(“mykeyword”) > -1: return 1; return 0; foundBits = lines.map(isFound); sum = foundBits.reduce(sum); if sum > 0: print “FOUND”; else: print “NOT FOUND”;
Yes. The search is not stopping even after the word we are looking for has been found. Our map code would keep executing on all the nodes which are very inefficient.
We could utilize accumulators to report whether the word has been found or not and then stop the job. Something on these lines:
import thread, threading
from time import sleep
result = “Not Set” lock = threading.Lock() accum = sc.accumulator(0) def map_func(line): #introduce delay to emulate the slowness sleep(1); if line.find(“Adventures”) > -1: accum.add(1); return 1; return 0; def start_job(): global result try: sc.setJobGroup(“job_to_cancel”, “some description”) lines = sc.textFile(“hdfs://hadoop1.knowbigdata.com/user/student/sgiri/wordcount/input/big.txt”); result = lines.map(map_func); result.take(1); except Exception as e: result = “Cancelled” lock.release() def stop_job(): while accum.value < 3 : sleep(1); sc.cancelJobGroup(“job_to_cancel”) supress = lock.acquire() supress = thread.start_new_thread(start_job, tuple()) supress = thread.start_new_thread(stop_job, tuple()) supress = lock.acquire() [/tab]
Facing technical problems in your current IT job, let us help you. MindMajix has highly technical people who can assist you in solving technical problems in your project.
We have come across many developers in the USA, Australia, and other countries who have recently got the job but they are struggling to survive in the job because of less technical knowledge, exposure, and the kind of work given to them.
We are here to help you.
Let us know your profile and the kind of help you are looking for and we shall do our best to help you out. The job support is provided by Mindmajix Technical experts who have more than 10 years of work experience in the IT technologies landscape.
|Explore Apache Spark Sample Resumes! Download & Edit, Get Noticed by Top Employers!.. Download Now!|
Stay updated with our newsletter, packed with Tutorials, Interview Questions, How-to's, Tips & Tricks, Latest Trends & Updates, and more ➤ Straight to your inbox!
|Apache Spark Training||May 23 to Jun 07|
|Apache Spark Training||May 28 to Jun 12|
|Apache Spark Training||May 30 to Jun 14|
|Apache Spark Training||Jun 04 to Jun 19|
Vinod M is a Big data expert writer at Mindmajix and contributes in-depth articles on various Big Data Technologies. He also has experience in writing for Docker, Hadoop, Microservices, Commvault, and few BI tools. You can be in touch with him via LinkedIn and Twitter.
Copyright © 2013 - 2022 MindMajix Technologies