{"id":72171,"date":"2017-08-11T17:10:41","date_gmt":"2017-08-11T17:10:41","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=72171"},"modified":"2021-02-23T19:24:57","modified_gmt":"2021-02-23T19:24:57","slug":"uploading-files-azure-data-lake-using-net-app","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/cloud\/azure\/uploading-files-azure-data-lake-using-net-app\/","title":{"rendered":"Uploading Files to Azure Data Lake Using a .NET App"},"content":{"rendered":"

Azure Data Lake allows you to rapidly store and ingest data simply whatever its form, and do any type of processing and analysis on it across platforms and languages. It is ideal for batch, streaming and interactive analytics because it provides a unified repository for diverse enterprise data requirements.<\/p>\n

There are many ways of uploading the files to Data Lake store, using Azure Data Factory Services, using Azure Data Lake Store .NET SDK, Java SDK, Node, REST API and so on. In this article, we will look into how to use the Azure Data Lake Store .NET SDK for uploading files from local system to Azure Data Lake store. To do this, you will need Visual studio 2015\/2017 and an Azure Subscription<\/p>\n

If you haven\u2019t already done so, the first step is to create a Data Lake Store in Azure to get started.<\/p>\n

Register Azure App<\/h2>\n

All external application interactions are controlled or secured using an Azure Active Directory App. We will start off by creating a Azure Active Directory App for our Data Lake solution. Go to https:\/\/portal.azure.com<\/a> and select \u2018Azure Active Directory\u2019<\/em> from the left-side navigation bar.<\/p>\n

Select the \u2018App registration\u2019<\/em> option from the Azure Active Directory blade. This will open a blade with list of registered apps and the option to register a new application. Select New application registration<\/strong> button on top of the blade.<\/p>\n

<\/p>\n

This will open the \u2018Create\u2019<\/strong> blade to create a new app. Specify the name and Sign-on URL. Select the application type as \u2018Web App\/API\u2019<\/em> or \u2018Native<\/em>\u2019.The \u2018Native<\/em>\u2019 term indicates that the application will be installed in the user device or computer. \u2018Web App\/API\u2019<\/em> means the authentication is from a web-based application.<\/p>\n

Because we are planning to use a console application for uploading the files, we will add a dummy Sign-on URL such as http:\/\/appname.<\/p>\n

<\/p>\n

Grant Permission in Data Lake Store<\/h2>\n

This AD app will be used to authenticate the users to upload files from local system to Azure Data Lake store. This application will need appropriate permission to perform activities in the target resource, that is, the Data Lake store. We can provide the necessary permission to the App by adding the app under Access<\/strong> section of Data Lake Store.<\/p>\n

Navigate to the Data Lake Store and select the \u2018Data Explorer\u2019<\/em> option.<\/p>\n

<\/p>\n

Select the \u2018Access<\/em>\u2019 option from the Data Lake Store blade. You can select the specific folder and specify access at the folder level, if required. In case you receive a \u201cForbidden<\/strong>\u201d exception at the time of file copy, that indicates that your app don\u2019t have enough permission to do the operation. Verify the app permissions at root level to fix the exception.<\/p>\n

Note the Examples folder in the Store list. We will be using this folder for uploading files from .NET code. <\/em><\/p>\n

The \u2018Access\u2019<\/em> blade will display the allowed users and apps list along with the type of permission that each of the users or apps have for this particular folder. Click on the \u2018Add\u2019<\/em> option at the top to grant permission for a new user or app.<\/p>\n

<\/p>\n

Select the previously created app name from the \u2018Select User or Group\u2019<\/em> blade and click \u2019OK<\/em>\u2019<\/strong>. You can search for a particular user or app by entering a few characters in the \u2018Select\u2019<\/em> box; a filtered list of users or apps will be listed down.<\/p>\n

<\/p>\n

Select the required permissions for specific apps from the Permissions list. Also, we can restrict the apps\u2019 access to a specific folder or sub folders. Access permission entry is for files and folders that control the access to that object. Default permission entry is for a folder that controls the access permissions associated with the child elements.<\/p>\n

<\/p>\n

File Upload Solution<\/h2>\n

Now we can create the solution to upload files from local system to Azure Data Lake Store. Open VS 2015\/2017 and create a new console application.<\/p>\n

Add the following two Nuget packages to the project.<\/p>\n