Apache Hadoop Yet Another Resource Negotiator popularly known as Apache Hadoop YARN.
Hadoop is a data-processing ecosystem that provides a framework for processing any type of data.YARN is one of the key features in the second-generation Hadoop 2 version of the Apache Software Foundation's open source distributed processing framework. YARN is the architectural center of Hadoop that allows multiple data processing engines like real-time streaming, interactive SQL, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics.
With YARN, Hadoop has been improved wherein practically huge volumes of raw unstructured data can now be stored for analysis. The introduction of new YARN Resource Manager to the Hadoop ecosystem brings a powerful and flexible platform for data analysis and growth, making Hadoop an open computing and robust environment that can scale into the future.
Apache Hadoop YARN Overview
The fundamental concept of YARN is to divide the capabilities of Resource Management and Job Monitoring/Scheduling into separate wizards. The notion behind this division is to have a global RM (ResourceManager) and per-application AM(ApplicationMaster). An application is either a single job or a DAG of jobs. The ResourceManager and the NodeManager constitutes the data-computation framework. The ResourceManager is the ultimate authority that adjudicates resources among all the applications in the system. The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (memory,cpu,network, disk) and reporting the same to the Scheduler/ResourceManager. The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.
Related Blog: Apache NiFi Tutorial
The ResourceManager has two main components: Scheduler and ApplicationsManager.
The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees about restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource Container which incorporates elements such as memory, cpu, disk, network etc.
Subscribe to our youtube channel to get new updates..!
The Scheduler has a pluggable policy which is responsible for partitioning the cluster resources among the various queues, applications etc. The current schedulers such as the CapacityScheduler and the FairScheduler would be some examples of plug-ins.
The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.
Apache Hadoop YARN Features
YARN is the prerequisite for Enterprise Hadoop, providing resource management and a central platform to deliver security, consistent operations, and data governance tools across Hadoop clusters.
YARN also extends the power of Hadoop to incumbent and new technologies found within the data center so that they can take advantage of cost effective, linear-scale storage and processing. It provides ISVs and developers a consistent framework for writing data access applications that run IN Hadoop.
YARN allows multiple access engines (either open-source or proprietary) to use Hadoop as the common standard for batch, interactive and real-time engines that can simultaneously access the same data set.
Multi-tenant data processing improves an enterprise’s return on its Hadoop investments.
- Cluster utilization
YARN’s dynamic allocation of cluster resources improves utilization over more static MapReduce rules used in early versions of Hadoop
Data center processing power continues to rapidly expand. YARN’s ResourceManager focuses exclusively on scheduling and keeps pace as clusters expand to thousands of nodes managing petabytes of data.
Existing MapReduce applications developed for Hadoop 1 can run YARN without any disruption to existing processes that already work.
Key Notes of Apache Hadoop YARN
Several important points about Apache Hadoop YARN are worth noting:
Hadoop is a data-processing ecosystem that provides a framework for processing any type of data.
Hadoop is no longer a single MapReduce application engine.
The ability of Hadoop to perform Extract, Transform, and Load (ETL) at runtime enables the developer to use a growing array of possible analysis engines, including MapReduce, graph processing, in-memory, custom, high-performance computing, and others.
The concept of a "data lake" is now possible, where practically unlimited amounts of raw unstructured data can be stored for later or real-time analysis.
The addition of YARN to the Hadoop ecosystem offers a flexible and powerful platform for data analysis and growth. Hadoop is no longer a "one-trick pony"; it now provides a very robust and open computing environment that can scale into the future.