Azure Data Factory - Data Processing Services

Introduction to Azure Data Factory (ADF)

Azure Data Factory Tutorial gives you in-depth information about how to utilize it efficiently and effectively. Microsoft Azure is another offering in terms of cloud computing. It is one of the growing collections of cloud services. Developers and IT professionals can utilize this platform to build, deploy, and manage applications from any part of the global network of data centers. Using the Azure cloud platform, you will get enough freedom to build and deploy applications. You can build them from wherever you want to, and practice using the tools that are available in Microsoft Azure.

What is Azure Data Factory

Let us understand what an Azure data factory is. And how it is helping organizations and individuals in terms of accomplishing their day to day operational tasks. Let's say - A gaming company is storing a lot of log information. So that later on they can take collective decisions on certain parameters. And they utilize this log information. Usually, some of the information is stored in on-premise data storage and the rest of the information is stored in the cloud.

So to analyze the data, we need to have an intermediary job. The one which consolidates all the information into one place. Then analyzes the data by using Hadoop in the cloud (Azure HDInsight) and SQL server on data storage premises. Let's say this process runs once a week.

This is a platform where the organizations can create a workflow and can ingest the data from on-premise data stores and also from the cloud stores.

If you want to learn azure and build your career in this domain visit Mindmajix a global online training platform: "Azure course" This course will help you to become an expert in this domain.

including the data from both these stores, the job can transform or process data by using Hadoop. It can be then used for BI applications if necessary.

Related Page: Azure Service Bus

Azure Data Factory Following Activities

1. First of all, it is a cloud-based solution where it can integrate with different types of data stores to gather information or data.

2. It helps you to create data-driven workflows to execute the same

3. All the data-driven workflows are called “pipelines”.

4. Once the data is gathered, processing tools like Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics can be used where the data can be transformed. And can be passed to the BI professionals where they can analyze the data.

In a sense, it is an Extract and Load (EL) tool where it will then Transform and Load (TL) platform. Rather than our traditional methods of Extract, Transform, and Load (ETL) tool.

Related Page: Azure Site Recovery

As of now, in Azure Data Factory, the data is consumed and produced by the defined workflows where it is time-based data. (i.e. it can be defined for hourly, daily, weekly, etc). So based on these parameters, the workflow would execute and do the job. For instance, it happens on an hourly basis or on a daily basis. It is all based on the setting.

Microsoft Azure Interview Questions

How does Azure Data factory work?

Azure Data Factory processes the data from the pipeline. It basically works in the three stages:

Connect and Collect: 

Connects to various SaaS services, or FTP or File sharing servers. Once the Azure Data Factory secures the connection, it starts collecting the Data from therein. There are on-premise sources and cloud storage. Azure Data Factory collects the information from all the sources available to make it available to one centralized source.

Transform and Enrich:

Once the collection of Data is done, all the data is transformed. The transformation could be done using various methods like HDInsight Hadoop, Spark, Data Lake Analytics, and Machine Learning. 

Publish:

The transformed data is then available on the local storage or local cloud space in the form of SQL. The data is in centralized storage, to be accessed and processed by BI and analytical teams. 

Related Page: Azure Stack

Data migration activities with Data Factory

Data Migration takes place in one of the two forms: From one data storage to another. From on-premise to cloud storage or vise versa.  

Azure Data Migration Copy Activities:

The Copy Activities in Azure Data Migration are responsible for copying the source data to the sink data store.

The data stores - both source and sink, that Azure supports are - Azure Blob storage, Azure Cosmos DB (DocumentDB API), Azure Data Lake Store, Oracle, Cassandra, and a few others. 

Azure Data Migration Transformation Activities:

The transformation Activities in Azure Data Migration are responsible for transforming the data into pipelines. Azure supports Hive, MapReduce, Spark, to name a few, for transformation activities. 

Microsoft Azure Tutorial

How the Data Factory components work together?

The aforementioned key components work hand in hand. For them to work in sync, geographical areas not the bar. 

• Globally: The Azure Data Factory components could be accessed by global users. 

• Regionally: The components are placed together locally. Still, the access of the same does not limit the regional boundaries.  

Related Page: Azure Logic Apps

Azure Data Migration

To perform the data migration task, you need to create all these key components. Initially, create a data factory on Azure. Deploy the editable templates of your components. After creating your Data factory on the Azure portal, go either with the default editor or visual studio or PowerShell for further data factory template edits.

Azure Data Migration with DataCopy Wizard on Azure

The easiest way to migrate Azure data is by using the DataCopy Wizard. You simply need to give a source location, destination location, and Priorities for the actions. And your pipeline is set. Once your pipeline setup is ready, you will be seeing the recap of your data migration process. After confirmation, you are good to proceed.

What are Custom DataCopy Activities

Custom DataCopy is creating your key components all by yourself. You can set your own priorities and functioning. The customization of DataCopy Activities requires some additional steps depending on your requirements. 

How to Monitor and manage Azure Data Factory pipelines

Azure DataFactory also enables you to monitor and manage your pipelines in a customized manner. Here are the steps to monitor and manage your Azure Data Factory Pipelines:

Step 1: Click on Monitor & Manage on the Data Factory tab.

Step 2: Click on - Resource Explorer. 

Step 3: You will find - pipelines, datasets, linked services in a tree format.

And voila! You can monitor and manage your pipeline setup with ease. 

Explain the Microsoft Workflow In-Depth:

As we have discussed, a pipeline is nothing but a data-driven workflow wherein Azure Data Factory is executed in three simple steps, they are:

1. Connect and Collect

2. Transform and Enrich

3. Publish

Workflow In Depth

Connect and Collect:

When it comes to data storage, especially in enterprises, a variety of data stores are utilized to store the data. The first and foremost step in building an Information production system is to connect all the required sources of the data. Such as - Saas services, file shares, FTP, web services. So that the data can be pushed to a centralized location for data processing.

Without a proper data factor, the organizations have to build or develop a custom data movement component. So, the data sources can be integrated. This is an expensive affair without the use of Data Factory. Even though these data movement controls are custom-built, it lacks the industry standards. Where the monitoring and alerting mechanism aren’t that effective, compared to the industry standard. So the data factor makes it comfortable for the enterprises where the pipelines would take care of the data consolidation point. For example, if you want to collect the data at a single point then you can do that in Azure Data Lake Store.

Further, if you want to transform or analyze the data then the cloud source data can be the source, and analysis can be done by using Azure Data Lake Analytics, etc.

Transform and Enrich:

As completing the connect and collect phase, the next phase is to transform the data and massage it to a level where the reporting layer can be utilized and harvest the data and generate respective analyzed reports. Tools like Data Lake Analytics and Machine learning can be achieved at this stage. Within this process, it is considered to be reliable because the produced transformed data is well maintained and controlled.

Publish:

Once the above two stages are completed, the data will be transformed into a stage where the BI team can actually consume the data and start with their analysis. The transformed data from the cloud will be pushed to on-premises sources like SQL Server.

What are the Key Components of Azure Data Factory?

For an Azure subscription, Azure data factory instances can be more than one and it is not necessary to have one Azure data factory instance for one Azure subscription. The Azure data factor is defined with four key components that work hand in hand where it provides the platform to effectively execute the workflow.

Pipeline:

A data factory can have one too many pipelines associated with it and it is not mandatory to have only one pipeline per data factory. Further, a pipeline can be defined as a group of activities.

Activity:

As defined above, a group of activities is called together as a Pipeline. So activities are defined as a specific set of activities to perform on the data. For example, A copy activity will only copy data from one datastore to another data store.

Data Factory Supports 2 Types Of Activities:

1. Data movement activities

2. Data transformation activities

If you are interested to learn Azure and become an expert in it, then check out our  Azure training Course at your near Cities

Microsoft Azure Course Bangalore, Microsoft Azure Course Hyderabad, Microsoft Azure Course Pune, Microsoft Azure Course Delhi, Microsoft Azure Course Chennai, Microsoft Azure Course Newyork, Microsoft Azure Course Washington, Microsoft Azure Course Dallas, Microsoft Azure Course Maryland

These courses are incorporated with Live instructor-led training, Industry Use cases, and hands-on live projects. This training program will make you an expert in AWS and help you to achieve your dream job.

We hope you have enjoyed reading about Azure Data Factory and the steps involved of consolidating the data and transforming the data altogether. If you have any valuable suggestions that are worth reading then please do advise in the comments section below.