{"id":88351,"date":"2020-10-05T17:00:13","date_gmt":"2020-10-05T17:00:13","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=88351"},"modified":"2020-10-02T10:57:04","modified_gmt":"2020-10-02T10:57:04","slug":"data-azure-links-dont-want-miss","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/blogs\/data-azure-links-dont-want-miss\/","title":{"rendered":"Data and Azure links you don&#8217;t want to miss"},"content":{"rendered":"<p>Browser tabs hold some similarities with tribbles in the way they multiply. Before you can notice, you have hundreds of them. However, in the same way tibbles are fluffy, browser tabs sometimes hold fluffy information as well, the kind of information you know you need but usually you are not sure where to hold it and may not be able to find it again.<\/p>\n<p>An interesting solution is share with all of you the best links open when my desktop ship get too floaded with them and transport the other ones to a klingon ship. Let&#8217;s see what I have today:<\/p>\n<h2><sub>Synapse Analytics<\/sub><\/h2>\n<h3>Many CTAS Sintaxes with the table configuration<\/h3>\n<p style=\"padding-left: 30px\">This link has many CTAS (Create Table as Select) syntax examples, not only with simple queries but aggregations and more, besides also many different distribution examples.<\/p>\n<p><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql-data-warehouse\/sql-data-warehouse-develop-ctas?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql-data-warehouse\/sql-data-warehouse-develop-ctas?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nParquet to SQL Type Mapping<\/h3>\n<p style=\"padding-left: 30px\">Synapse SQL Pool is very demanding about parquet data types. This link has details about the parquet data type mapping to SQL. However, there is something even better on this link: The address and query samples for public blob storages with data about the pandemic and the NYC Taxi sample<\/p>\n<p><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql\/query-parquet-files?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql\/query-parquet-files?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p>&nbsp;<\/p>\n<h3>Parquet complex types<\/h3>\n<p style=\"padding-left: 30px\">This link is a blog post with interesting\u00a0 examples of parquet complex type, using a public blob storage for that<\/p>\n<p><a href=\"https:\/\/techcommunity.microsoft.com\/t5\/azure-synapse-analytics\/easily-load-complex-data-types-with-azure-synapse-analytics\/ba-p\/1479676?WT.mc_id=DP-MVP-4014132\">https:\/\/techcommunity.microsoft.com\/t5\/azure-synapse-analytics\/easily-load-complex-data-types-with-azure-synapse-analytics\/ba-p\/1479676?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>COPY INTO and public example of PARQUET complex types<\/h3>\n<p style=\"padding-left: 30px\">This is the documentation of the COPY INTO statement. Unlinke CTAS, in this case we can import data into existing table. COPY INTO also is extremely flexible in relation to the loading data options<\/p>\n<p><a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/t-sql\/statements\/copy-into-transact-sql?view=azure-sqldw-latest&amp;WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/sql\/t-sql\/statements\/copy-into-transact-sql?view=azure-sqldw-latest&amp;WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nExternal tables and URI format reference<\/h3>\n<p style=\"padding-left: 30px\">There are many links about creating external tables, but this one is special: on this link you will find a very interesting explanation about the LOCATION setting, including what prefix you should use with each kind of blob storage<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql\/develop-tables-external-tables?tabs=sql-pool&amp;WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql\/develop-tables-external-tables?tabs=sql-pool&amp;WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nMore URI format reference for external tables<\/h3>\n<p style=\"padding-left: 30px\">This other link contains reference for many kinds for external sources, Oracle, Teradata and more<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/t-sql\/statements\/create-external-data-source-transact-sql?view=sql-server-ver15&amp;WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/sql\/t-sql\/statements\/create-external-data-source-transact-sql?view=sql-server-ver15&amp;WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nCreate external table as select<\/h3>\n<p style=\"padding-left: 30px\">This is the reverse of the most famouse cousing, CTAS. Instead of loading data, we can use this syntax to export data.<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/t-sql\/statements\/create-external-table-as-select-transact-sql?view=aps-pdw-2016-au7&amp;WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/sql\/t-sql\/statements\/create-external-table-as-select-transact-sql?view=aps-pdw-2016-au7&amp;WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nStorage access with credentials<\/h3>\n<p style=\"padding-left: 30px\">This link contains details about how to access a private blog using SQL On demand and many different credentials: SAS, user identity and managed identity<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql\/develop-storage-files-storage-access-control?tabs=shared-access-signature&amp;WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/synapse-analytics\/sql\/develop-storage-files-storage-access-control?tabs=shared-access-signature&amp;WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p>&nbsp;<\/p>\n<h2>Azure Monitor, Log Analytics and more<\/h2>\n<h3>\nResource Manager Templates for Log Analytics<\/h3>\n<p style=\"padding-left: 30px\">This link explain in details how to build an ARM template to create and configure a log analytics workspace. Each piece of configuration is explained, I used this link to help in automate the export of log analytics workspace to ARM templates. I will write about this later.<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/azure-monitor\/samples\/resource-manager-workspace?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/azure-monitor\/samples\/resource-manager-workspace?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>Application Insight Workbooks<\/h3>\n<p style=\"padding-left: 30px\">This link is a github repo full of workbooks built to provide management information about many azure services<\/p>\n<p><a href=\"https:\/\/github.com\/microsoft\/Application-Insights-Workbooks\">https:\/\/github.com\/microsoft\/Application-Insights-Workbooks<\/a><\/p>\n<h3>Resource Manager Templates for Workbooks<\/h3>\n<p style=\"padding-left: 30px\">This link explains how to create an ARM template to install an Azure Monitor Workbook. Specially the type property is not so easy to find in documentations<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/azure-monitor\/samples\/resource-manager-workbooks?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/azure-monitor\/samples\/resource-manager-workbooks?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>Programatically Manage Workbooks<\/h3>\n<p style=\"padding-left: 30px\">This link explains how to create an ARM template to install an Azure Monitor Workbook template. The template will be available for the creation of new workbooks from it.<\/p>\n<p><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/azure-monitor\/platform\/workbooks-automate?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/azure-monitor\/platform\/workbooks-automate?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p>&nbsp;<\/p>\n<h2>Powershell<\/h2>\n<p style=\"padding-left: 30px\">Creating ARM templates involves manipulating JSON files. While building some script to manipulate and create ARM templates, I had the challenge to manipulate custom objects, creating or removing properties on the fly. These are references I found about it.<\/p>\n<p style=\"padding-left: 30px\"><strong>Remove a Powershell object property<\/strong><br \/>\n<a href=\"https:\/\/lazywinadmin.com\/2017\/06\/Remove_PSObject_Property.html\">https:\/\/lazywinadmin.com\/2017\/06\/Remove_PSObject_Property.html<\/a><\/p>\n<p style=\"padding-left: 30px\"><strong>Add-Member<\/strong><br \/>\n<a href=\"https:\/\/ss64.com\/ps\/add-member.html\">https:\/\/ss64.com\/ps\/add-member.html<\/a><br \/>\n<a href=\"https:\/\/community.idera.com\/database-tools\/powershell\/ask_the_experts\/f\/learn_powershell_from_don_jones-24\/13052\/adding-new-values-to-a-psobject\">https:\/\/community.idera.com\/database-tools\/powershell\/ask_the_experts\/f\/learn_powershell_from_don_jones-24\/13052\/adding-new-values-to-a-psobject<\/a><br \/>\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/powershell\/module\/microsoft.powershell.utility\/add-member?view=powershell-7&amp;WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/powershell\/module\/microsoft.powershell.utility\/add-member?view=powershell-7&amp;WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p style=\"padding-left: 30px\"><strong>Array of Custom Objects<\/strong><br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/17353797\/how-to-initialize-an-array-of-custom-objects\/17354800\">https:\/\/stackoverflow.com\/questions\/17353797\/how-to-initialize-an-array-of-custom-objects\/17354800<\/a><\/p>\n<p style=\"padding-left: 30px\">Add Many properties<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/51620654\/can-i-create-a-custom-powershell-object-with-an-array-property\">https:\/\/stackoverflow.com\/questions\/51620654\/can-i-create-a-custom-powershell-object-with-an-array-property<\/a><br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/36200749\/how-do-you-add-more-property-values-to-a-custom-object\">https:\/\/stackoverflow.com\/questions\/36200749\/how-do-you-add-more-property-values-to-a-custom-object<\/a><\/p>\n<p style=\"padding-left: 30px\"><strong>Real-Time Twitter sentiment Analysis in Azure Stream Analytics<\/strong><br \/>\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/stream-analytics\/stream-analytics-twitter-sentiment-analysis-trends?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/stream-analytics\/stream-analytics-twitter-sentiment-analysis-trends?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p>&nbsp;<\/p>\n<h2>Billing<\/h2>\n<h3>\nBilling Permissions on Azure<\/h3>\n<p style=\"padding-left: 30px\">You can delegate the task to manage billing information and maybe you should (I hate billing). These two links help with this, explaining how to give access to another user to manage billing and what are the existing roles in a customer agreement scenario<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/costs\/assign-access-acm-data?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/costs\/assign-access-acm-data?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/manage\/understand-mca-roles?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/manage\/understand-mca-roles?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nAzure Comsumption Insights<\/h3>\n<p style=\"padding-left: 30px\">This link explains how to user Power BI to access Azure Comsumption Insights API to manage Enterprise Agreement azure subscriptions<\/p>\n<p><a href=\"https:\/\/docs.microsoft.com\/en-us\/power-bi\/connect-data\/desktop-connect-azure-consumption-insights?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/power-bi\/connect-data\/desktop-connect-azure-consumption-insights?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>Azure Cost Management App<\/h3>\n<p style=\"padding-left: 30px\">This link explains how to use Power BI Azure Cost Management Application to manage Azure billing. This application is exclusive for enterprise agreement subscriptions<\/p>\n<p style=\"padding-left: 30px\"><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/costs\/analyze-cost-data-azure-cost-management-power-bi-template-app?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/costs\/analyze-cost-data-azure-cost-management-power-bi-template-app?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<p>&nbsp;<\/p>\n<h3>Azure Enterprise Portal<\/h3>\n<p style=\"padding-left: 30px\">The enterprise portal is used by companies with Enterprise Agreement contract. On this link you can learn details about the enterprise portal.<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/manage\/ea-portal-get-started#create-another-enterprise-administrator#?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/manage\/ea-portal-get-started#create-another-enterprise-administrator#?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n<h3>\nAzure Enterprise Roles<\/h3>\n<p style=\"padding-left: 30px\">The roles used by companies with Enterprise Agreement contract are different than the roles used by companies with customer agreement. The link below explain these roles<\/p>\n<p style=\"padding-left: 30px\">\n<a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/manage\/understand-ea-roles?WT.mc_id=DP-MVP-4014132\">https:\/\/docs.microsoft.com\/en-us\/azure\/cost-management-billing\/manage\/understand-ea-roles?WT.mc_id=DP-MVP-4014132<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Browser tabs hold some similarities with tribbles in the way they multiply. Before you can notice, you have hundreds of them. However, in the same way tibbles are fluffy, browser tabs sometimes hold fluffy information as well, the kind of information you know you need but usually you are not sure where to hold it&#8230;&hellip;<\/p>\n","protected":false},"author":50808,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[124956,124957,124955,124954,4635,124953],"coauthors":[6810],"class_list":["post-88351","post","type-post","status-publish","format-standard","hentry","category-blogs","tag-bigdata","tag-datalake","tag-dataplatform","tag-loganalytics","tag-powershell","tag-synapse"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/88351","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/50808"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=88351"}],"version-history":[{"count":4,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/88351\/revisions"}],"predecessor-version":[{"id":88587,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/88351\/revisions\/88587"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=88351"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=88351"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=88351"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=88351"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}