{"id":93713,"date":"2022-04-11T17:00:13","date_gmt":"2022-04-11T17:00:13","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=93713"},"modified":"2022-05-05T20:57:36","modified_gmt":"2022-05-05T20:57:36","slug":"migrate-all-your-azure-storage-gen-1-to-gen-2","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/migrate-all-your-azure-storage-gen-1-to-gen-2\/","title":{"rendered":"Migrate all your Azure Storage Gen 1 to Gen 2"},"content":{"rendered":"<p>Microsoft already announced the retirement date of <strong>Azure Data Lake Storage Gen 1<\/strong>: February 29, 2024. You can check details about the deprecation on\u00a0<a href=\"https:\/\/github.com\/azure-deprecation\/dashboard\/issues\/137\">https:\/\/github.com\/azure-deprecation\/dashboard\/issues\/137<\/a><\/p>\n<p>You may think you still have time available, but why should you wait? <strong>ADLS Gen 2<\/strong> has many more features than Gen 1. If this were not enough, it also has the life-cycle management feature. If used correctly, the life-cycle management feature can save you a lot of money.<\/p>\n<p>We need to convert all our <strong>ADLS Gen 1<\/strong> to <strong>ADLS Gen2<\/strong>. Let&#8217;s first find the bad guys on our environment and then make the conversion to Gen2.<\/p>\n<p>Of course, if you would like so, you can skip the entire explanation, copy the policy and the code and execute. However, understanding how this works will prove to your co-workers you are &#8220;The guy&#8221;.<\/p>\n<h2>Using Azure Policies to find the ADLS Gen 1<\/h2>\n<p>I will use Azure policies to find any storage account which isn&#8217;t already an <strong>ADLS Gen 2<\/strong>. I already wrote about policies before, so I leave links with the details about how to use them. Let&#8217;s focus here on the code.<\/p>\n<p>The code below has the following features:<\/p>\n<ul>\n<li>We are looking for storage accounts<\/li>\n<li>Only storage accounts which are not Gen 2<\/li>\n<li>We are only auditing, pointing which are the storage accounts not compliant with this rule<\/li>\n<\/ul>\n<p><!-- HTML generated using hilite.me --><\/p>\n<div style=\"background: #ffffff; overflow: auto; width: auto; border: solid gray; border-width: .1em .1em .1em .8em; padding: .2em .6em;\">\n<pre style=\"margin: 0; line-height: 125%;\" class=\"crayon:false\">{\r\n   <span style=\"color: #007700;\">\"policyRule\"<\/span>:{\r\n      <span style=\"color: #007700;\">\"if\"<\/span>:{\r\n         <span style=\"color: #007700;\">\"allOf\"<\/span>:[\r\n            {\r\n               <span style=\"color: #007700;\">\"field\"<\/span>:<span style=\"background-color: #fff0f0;\">\"type\"<\/span>,\r\n               <span style=\"color: #007700;\">\"equals\"<\/span>:<span style=\"background-color: #fff0f0;\">\"Microsoft.Storage\/storageAccounts\"<\/span>\r\n            },\r\n            {\r\n               <span style=\"color: #007700;\">\"field\"<\/span>:<span style=\"background-color: #fff0f0;\">\"kind\"<\/span>,\r\n               <span style=\"color: #007700;\">\"notEquals\"<\/span>:<span style=\"background-color: #fff0f0;\">\"StorageV2\"<\/span>\r\n            }\r\n         ]\r\n      },\r\n      <span style=\"color: #007700;\">\"then\"<\/span>:{\r\n         <span style=\"color: #007700;\">\"effect\"<\/span>:<span style=\"background-color: #fff0f0;\">\"audit\"<\/span>\r\n      }\r\n   }\r\n}\r\n<\/pre>\n<\/div>\n<p>I wrote about tenant wide policy evaluation on this blog <a href=\"https:\/\/www.red-gate.com\/simple-talk\/blogs\/evaluation-policies-in-a-tenant-wide-level-and-more-azure-tricks\/\">https:\/\/www.red-gate.com\/simple-talk\/blogs\/evaluation-policies-in-a-tenant-wide-level-and-more-azure-tricks\/<\/a><\/p>\n<p>Once the policy definition has been created, assigned to the root management group and executed, we can proceed to the next step. Now we will need <strong>Powershell.<\/strong><\/p>\n<h2>Finding the bad Storage Accounts<\/h2>\n<p>The core feature is the upgrade. Yes, we can upgrade an <strong>ADLS Gen1<\/strong> to <strong>ADLS Gen2<\/strong> using a single statement.<\/p>\n<p>The challenge is how to find all the <strong>ADLS Gen1<\/strong> storage accounts in our environment. <strong>KQL<\/strong> to the rescue.<\/p>\n<p>Azure Resource Graph allows you to use <strong>KQL (Kusto Query Language)<\/strong> to find the resources in your Azure Tenant. We already have the policy. We can use <strong>KQL<\/strong> to find all the objects which are not compliant with the policy.<\/p>\n<p>First, we need to locate our policy Id to use in our query. The following query will help with this:<\/p>\n<div><span style=\"font-family: Courier New; font-size: 10pt;\"> policyresources <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">type<\/span>\u00a0<span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span>\u00a0<span style=\"color: maroon;\">&#8220;microsoft.authorization\/policyassignments&#8221;<\/span> <br \/>\n<span style=\"color: silver;\">|<\/span><span style=\"color: maroon;\">project<\/span>\u00a0<span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">displayName<\/span><span style=\"color: silver;\">,<\/span>\u00a0<span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">policyDefinitionId<\/span> <\/span><\/div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-93715\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2022\/04\/Migrate01.png\" alt=\"\" width=\"1280\" height=\"520\" \/><\/p>\n<p>How could we discover this is the query we need? Simple: On the left side of Azure resource graph we can navigate across the object structure to find the objects we need. Clicking on the objects, <strong>Azure Resource Graph<\/strong> will build pieces of the query for us, we just need to complete.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-93716\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2022\/04\/Migrate02.png\" alt=\"\" width=\"540\" height=\"390\" \/><\/p>\n<p>Now we know more about our policy, we need to find <strong>policyState<\/strong> objects. All objects which were not compliant with our policy.<\/p>\n<p>This is a good start:<\/p>\n<div><span style=\"font-family: Courier New; font-size: 10pt;\"> policyresources <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">type<\/span>\u00a0<span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span>\u00a0<span style=\"color: maroon;\">&#8220;microsoft.policyinsights\/policystates&#8221;<\/span> <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">policyDefinitionName<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: maroon;\">&#8220;2df4ec72-1da3-4445-8053-95177a143ae2&#8221;<\/span> <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">complianceState<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: maroon;\">&#8220;NonCompliant&#8221;<\/span> <\/span><\/div>\n<p>As the result of the query, we need the resource group, the storage account name and the subscription.<\/p>\n<p>The storage account name and the subscription are inside the Properties <strong>JSON<\/strong> value. We will need to use some string functions to extract both of them. The final query will be like this:<\/p>\n<div><span style=\"font-family: Courier New; font-size: 10pt;\"> policyresources <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">type<\/span>\u00a0<span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span>\u00a0<span style=\"color: maroon;\">&#8220;microsoft.policyinsights\/policystates&#8221;<\/span> <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">policyDefinitionName<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: maroon;\">&#8220;2df4ec72-1da3-4445-8053-95177a143ae2&#8221;<\/span> <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: blue;\">where<\/span>\u00a0<span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">complianceState<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: maroon;\">&#8220;NonCompliant&#8221;<\/span> <br \/>\n<span style=\"color: silver;\">|<\/span>\u00a0<span style=\"color: maroon;\">project<\/span>\u00a0<span style=\"color: maroon;\">resourceGroup<\/span><span style=\"color: silver;\">,<\/span><span style=\"color: maroon;\">resource<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: maroon;\">split<\/span><span style=\"color: maroon;\">(<\/span><span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">resourceId<\/span><span style=\"color: silver;\">,<\/span><span style=\"color: maroon;\">&#8220;\/&#8221;<\/span><span style=\"color: maroon;\">)<\/span><span style=\"color: maroon;\">[array_length(split(properties.resourceId,&#8221;\/&#8221;))-1]<\/span><span style=\"color: silver;\">,<\/span> <br \/>\n<span style=\"color: maroon;\">subscription<\/span><span style=\"color: silver;\">=<\/span><span style=\"color: maroon;\">split<\/span><span style=\"color: maroon;\">(<\/span><span style=\"color: maroon;\">properties<\/span><span style=\"color: silver;\">.<\/span><span style=\"color: maroon;\">resourceId<\/span><span style=\"color: silver;\">,<\/span><span style=\"color: maroon;\">&#8220;\/&#8221;<\/span><span style=\"color: maroon;\">)<\/span><span style=\"color: maroon;\">[2]<\/span> <\/span><\/div>\n<h2>Building the Powershell Code<\/h2>\n<p>Once we have the query to list the bad storage accounts for us, it&#8217;s time to build the <strong>Powershell<\/strong> code.<\/p>\n<p>The code will be simple:<\/p>\n<ul>\n<li>We need to execute the query. The\u00a0<strong>Search-AzGraph<\/strong>\u00a0cmdlet can execute a resource graph query.<\/li>\n<li>We execute a loop over the result, to process each storage account.<\/li>\n<li>Before processing the storage account, we set the context to the subscription of the storage account. Let&#8217;s limit the number of context changes by using a variable to identify what&#8217;s the subscription on the current context.<\/li>\n<li>Finally, we convert the storage account to ADLS Gen 2.<\/li>\n<\/ul>\n<p>This is the final <strong>Powershell<\/strong> code:<\/p>\n<p><!-- HTML generated using hilite.me --><\/p>\n<div style=\"background: #ffffff; overflow: auto; width: auto; border: solid gray; border-width: .1em .1em .1em .8em; padding: .2em .6em;\">\n<pre style=\"margin: 0; line-height: 125%;\" class=\"crayon:false\"><span style=\"color: #888888;\"># Run Azure Resource Graph query with `order by` first, then with `limit`<\/span>\r\n<span style=\"color: #007020;\">Write-Host<\/span> <span style=\"background-color: #fff0f0;\">\"Startting\"<\/span>\r\n<span style=\"color: #996633;\">$result<\/span>=(Search-AzGraph -Query <span style=\"background-color: #fff0f0;\">'policyresources<\/span>\r\n<span style=\"background-color: #fff0f0;\">| where type == \"microsoft.policyinsights\/policystates\"<\/span>\r\n<span style=\"background-color: #fff0f0;\">| where properties.policyDefinitionName==\"2df4ec72-1da3-4445-8053-95177a143ae2\"<\/span>\r\n<span style=\"background-color: #fff0f0;\">| where properties.complianceState==\"NonCompliant\"<\/span>\r\n<span style=\"background-color: #fff0f0;\">| project resourceGroup,resource=split(properties.resourceId,\"\/\")[array_length(split(properties.resourceId,\"\/\"))-1],<\/span>\r\n<span style=\"background-color: #fff0f0;\">subscription=split(properties.resourceId,\"\/\")[2]'<\/span>)\r\n\r\n<span style=\"color: #996633;\">$subscription<\/span>=<span style=\"background-color: #fff0f0;\">\"\"<\/span>\r\n<span style=\"color: #007020;\">write-host<\/span> <span style=\"color: #996633;\">$result<\/span>\r\n<span style=\"color: #996633;\">$result<\/span> | <span style=\"color: #008800; font-weight: bold;\">foreach<\/span> {\r\n\r\n<span style=\"color: #008800; font-weight: bold;\">if<\/span> (<span style=\"color: #996633;\">$subscription<\/span> <span style=\"color: #333333;\">-ne<\/span> <span style=\"color: #996633;\">$_<\/span>.subscription)\r\n{\r\n<span style=\"color: #996633;\">$subscription<\/span>=<span style=\"color: #996633;\">$_<\/span>.subscription\r\n\r\n<span style=\"color: #007020;\">Write-Host<\/span> <span style=\"background-color: #fff0f0;\">\"changing subscription to <\/span><span style=\"background-color: #eeeeee;\">$(<\/span><span style=\"color: #ff0000; background-color: #ffaaaa;\">$<\/span><span style=\"background-color: #eeeeee;\">_.subscription)<\/span><span style=\"background-color: #fff0f0;\">\"<\/span>\r\n<span style=\"color: #007020;\">Set-AzContext<\/span> -Subscription <span style=\"color: #996633;\">$subscription<\/span>\r\n}\r\n\r\n<span style=\"color: #007020;\">Write-Host<\/span> <span style=\"background-color: #fff0f0;\">\"Upgrading <\/span><span style=\"background-color: #eeeeee;\">$(<\/span><span style=\"color: #ff0000; background-color: #ffaaaa;\">$<\/span><span style=\"background-color: #eeeeee;\">_.resource)<\/span><span style=\"background-color: #fff0f0;\">\"<\/span>\r\n<span style=\"color: #007020;\">Set-AzStorageAccount<\/span> -ResourceGroupName <span style=\"color: #996633;\">$_<\/span>.resourceGroup -Name <span style=\"color: #996633;\">$_<\/span>.resource -UpgradeToStorageV2 -AccessTier Hot -Confirm<span style=\"color: #ff0000; background-color: #ffaaaa;\">:<\/span><span style=\"color: #996633;\">$false<\/span>\r\n}<span style=\"color: #888888;\"># Run Azure Resource Graph query with `order by` first, then with `limit`<\/span>\r\n<span style=\"color: #007020;\">Write-Host<\/span> <span style=\"background-color: #fff0f0;\">\"Startting\"<\/span>\r\n<span style=\"color: #996633;\">$result<\/span>=(Search-AzGraph -Query <span style=\"background-color: #fff0f0;\">'policyresources<\/span>\r\n<span style=\"background-color: #fff0f0;\">| where type == \"microsoft.policyinsights\/policystates\"<\/span>\r\n<span style=\"background-color: #fff0f0;\">| where properties.policyDefinitionName==\"2df4ec72-1da3-4445-8053-95177a143ae2\"<\/span>\r\n<span style=\"background-color: #fff0f0;\">| where properties.complianceState==\"NonCompliant\"<\/span>\r\n<span style=\"background-color: #fff0f0;\">| project resourceGroup,resource=split(properties.resourceId,\"\/\")[array_length(split(properties.resourceId,\"\/\"))-1],<\/span>\r\n<span style=\"background-color: #fff0f0;\">subscription=split(properties.resourceId,\"\/\")[2]'<\/span>)\r\n\r\n<span style=\"color: #996633;\">$subscription<\/span>=<span style=\"background-color: #fff0f0;\">\"\"<\/span>\r\n<span style=\"color: #007020;\">write-host<\/span> <span style=\"color: #996633;\">$result<\/span>\r\n<span style=\"color: #996633;\">$result<\/span> | <span style=\"color: #008800; font-weight: bold;\">foreach<\/span> {\r\n\r\n<span style=\"color: #008800; font-weight: bold;\">if<\/span> (<span style=\"color: #996633;\">$subscription<\/span> <span style=\"color: #333333;\">-ne<\/span> <span style=\"color: #996633;\">$_<\/span>.subscription)\r\n{\r\n<span style=\"color: #996633;\">$subscription<\/span>=<span style=\"color: #996633;\">$_<\/span>.subscription\r\n\r\n<span style=\"color: #007020;\">Write-Host<\/span> <span style=\"background-color: #fff0f0;\">\"changing subscription to <\/span><span style=\"background-color: #eeeeee;\">$(<\/span><span style=\"color: #ff0000; background-color: #ffaaaa;\">$<\/span><span style=\"background-color: #eeeeee;\">_.subscription)<\/span><span style=\"background-color: #fff0f0;\">\"<\/span>\r\n<span style=\"color: #007020;\">Set-AzContext<\/span> -Subscription <span style=\"color: #996633;\">$subscription<\/span>\r\n}\r\n\r\n<span style=\"color: #007020;\">Write-Host<\/span> <span style=\"background-color: #fff0f0;\">\"Upgrading <\/span><span style=\"background-color: #eeeeee;\">$(<\/span><span style=\"color: #ff0000; background-color: #ffaaaa;\">$<\/span><span style=\"background-color: #eeeeee;\">_.resource)<\/span><span style=\"background-color: #fff0f0;\">\"<\/span>\r\n<span style=\"color: #007020;\">Set-AzStorageAccount<\/span> -ResourceGroupName <span style=\"color: #996633;\">$_<\/span>.resourceGroup -Name <span style=\"color: #996633;\">$_<\/span>.resource -UpgradeToStorageV2 -AccessTier Hot -Confirm<span style=\"color: #ff0000; background-color: #ffaaaa;\">:<\/span><span style=\"color: #996633;\">$false<\/span>\r\n}\r\n<\/pre>\n<\/div>\n<h2>\u00a0<\/h2>\n<h2>Summary<\/h2>\n<p>This is more than only a simple code, this is the code I executed on my own environment to make the migration of all my storage accounts to ADLS Gen 2<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Need to migrate your Azure Data Lake Storage Gen1 before Microsoft retires the service? This post from Dennes Torres explains how to automate the migration.&hellip;<\/p>\n","protected":false},"author":50808,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[145946,145491,126250],"coauthors":[6810],"class_list":["post-93713","post","type-post","status-publish","format-standard","hentry","category-blogs","tag-azure-resource-graph","tag-azure-storage","tag-kql"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/93713","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/50808"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=93713"}],"version-history":[{"count":9,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/93713\/revisions"}],"predecessor-version":[{"id":94186,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/93713\/revisions\/94186"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=93713"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=93713"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=93713"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=93713"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}