Apache Hadoop YARN

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.

Checkout Hadoop Tutorial

Apache Hadoop YARN Features

YARN is the prerequisite for Enterprise Hadoop, providing resource management and a central platform to deliver security, consistent operations, and data governance tools across Hadoop clusters.

YARN also extends the power of Hadoop to incumbent and new technologies found within the data center so that they can take advantage of cost effective, linear-scale storage and processing. It provides ISVs and developers a consistent framework for writing data access applications that run IN Hadoop.

Multi-tenancy

YARN allows multiple access engines (either open-source or proprietary) to use Hadoop as the common standard for batch, interactive and real-time engines that can simultaneously access the same data set.

Multi-tenant data processing improves an enterprise’s return on its Hadoop investments.

Cluster utilization

YARN’s dynamic allocation of cluster resources improves utilization over more static MapReduce rules used in early versions of Hadoop

Scalability

Data center processing power continues to rapidly expand. YARN’s ResourceManager focuses exclusively on scheduling and keeps pace as clusters expand to thousands of nodes managing petabytes of data.

Compatibility

Existing MapReduce applications developed for Hadoop 1 can run YARN without any disruption to existing processes that already work.

Key Notes of Apache Hadoop YARN

Several important points about Apache Hadoop YARN are worth noting:

Hadoop is a data-processing ecosystem that provides a framework for processing any type of data.

Hadoop is no longer a single MapReduce application engine.

The ability of Hadoop to perform Extract, Transform, and Load (ETL) at runtime enables the developer to use a growing array of possible analysis engines, including MapReduce, graph processing, in-memory, custom, high-performance computing, and others.

The concept of a "data lake" is now possible, where practically unlimited amounts of raw unstructured data can be stored for later or real-time analysis.

The addition of YARN to the Hadoop ecosystem offers a flexible and powerful platform for data analysis and growth. Hadoop is no longer a "one-trick pony"; it now provides a very robust and open computing environment that can scale into the future.