Home  >  Blog  >   Hadoop

Apache Hadoop YARN

YARN stands for Yet Another Resource Negotiator. It’s a cluster management technology that was introduced in Hadoop 2.0. This blog walks you through all the aspects of Apache Hadoop YARN like introduction, architecture, features, components, and more.

Rating: 4
  
 
6202

Apache Hadoop Yet Another Resource Negotiator popularly known as Apache Hadoop YARN.

Hadoop is a data-processing ecosystem that provides a framework for processing any type of data. YARN is one of the key features in the second-generation Hadoop 2 version of the Apache Software Foundation's open source distributed processing framework. YARN is the architectural center of Hadoop that allows multiple data processing engines like real-time streaming, interactive SQL, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics.

Hadoop Yarn

With YARN, Hadoop has been improved wherein practically huge volumes of raw unstructured data can now be stored for analysis. The introduction of new YARN Resource Manager to the Hadoop ecosystem brings a powerful and flexible platform for data analysis and growth, making Hadoop an open computing and robust environment that can scale into the future.

Looking forward to becoming a Hadoop Developer? Check out the Big Data Hadoop Certification Training and get certified today.

What is Apache Hadoop YARN 

The fundamental concept of YARN is to divide the capabilities of Resource Management and Job Monitoring/Scheduling into separate wizards. The notion behind this division is to have a global RM (ResourceManager) and per-application AM(ApplicationMaster). An application is either a single job or a DAG of jobs. The ResourceManager and the NodeManager constitutes the data-computation framework. The ResourceManager is the ultimate authority that adjudicates resources among all the applications in the system. The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (memory,cpu,network, disk) and reporting the same to the Scheduler/ResourceManager. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

Related Blog: Apache NiFi Tutorial

Resource manager

Components of ResourceManager

The ResourceManager has two main components: Scheduler and ApplicationsManager.

Scheduler

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees about restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource Container which incorporates elements such as memory, cpu, disk, network etc.

Checkout Hadoop Interview Questions

ApplicationsManager

The Scheduler has a pluggable policy which is responsible for partitioning the cluster resources among the various queues, applications etc. The current schedulers such as the CapacityScheduler and the FairScheduler would be some examples of plug-ins.

Apache Hadoop 2.0 and YARN

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.

Checkout Hadoop Tutorial

Apache Hadoop YARN Features

YARN is the prerequisite for Enterprise Hadoop, providing resource management and a central platform to deliver security, consistent operations, and data governance tools across Hadoop clusters.

YARN also extends the power of Hadoop to incumbent and new technologies found within the data center so that they can take advantage of cost effective, linear-scale storage and processing. It provides ISVs and developers a consistent framework for writing data access applications that run IN Hadoop.

MindMajix YouTube Channel

Multi-tenancy

YARN allows multiple access engines (either open-source or proprietary) to use Hadoop as the common standard for batch, interactive and real-time engines that can simultaneously access the same data set.

Multi-tenant data processing improves an enterprise’s return on its Hadoop investments.

Cluster utilization

YARN’s dynamic allocation of cluster resources improves utilization over more static MapReduce rules used in early versions of Hadoop

Scalability

Data center processing power continues to rapidly expand. YARN’s ResourceManager focuses exclusively on scheduling and keeps pace as clusters expand to thousands of nodes managing petabytes of data.

Compatibility

Existing MapReduce applications developed for Hadoop 1 can run YARN without any disruption to existing processes that already work.

Explore Hadoop Sample Resumes! Download & Edit, Get Noticed by Top Employers!Download Now!

Key Notes of Apache Hadoop YARN

Several important points about Apache Hadoop YARN are worth noting:

Hadoop is a data-processing ecosystem that provides a framework for processing any type of data.

Hadoop is no longer a single MapReduce application engine.

The ability of Hadoop to perform Extract, Transform, and Load (ETL) at runtime enables the developer to use a growing array of possible analysis engines, including MapReduce, graph processing, in-memory, custom, high-performance computing, and others.

The concept of a "data lake" is now possible, where practically unlimited amounts of raw unstructured data can be stored for later or real-time analysis.

The addition of YARN to the Hadoop ecosystem offers a flexible and powerful platform for data analysis and growth. Hadoop is no longer a "one-trick pony"; it now provides a very robust and open computing environment that can scale into the future.

Apache Hadoop YARN Overview

List of Big Data Courses:

 Hadoop Adminstartion MapReduce
 Big Data On AWS Informatica Big Data Integration
 Bigdata Greenplum DBA Informatica Big Data Edition
 Hadoop Hive Impala
 Hadoop Testing Apache Mahout
Join our newsletter
inbox

Stay updated with our newsletter, packed with Tutorials, Interview Questions, How-to's, Tips & Tricks, Latest Trends & Updates, and more ➤ Straight to your inbox!

Course Schedule
NameDates
Hadoop TrainingMar 23 to Apr 07View Details
Hadoop TrainingMar 26 to Apr 10View Details
Hadoop TrainingMar 30 to Apr 14View Details
Hadoop TrainingApr 02 to Apr 17View Details
Last updated: 03 Apr 2023
About Author

 

Technical Content Writer

read more
Recommended Courses

1 / 15