2187-just_azure.svg

In my last article I covered the basic principles involved in configuration of diagnostics for Azure Cloud Services (web/worker roles). We took a look at both imperative and declarative models for configuring the diagnostic related data to collect and save, the order of precedence and why it is important to always use declarative configuration, and also some practical tips for configuring your Azure diagnostics setup.

Now that we understand what is needed to enable our web/worker role(s) to collect and save important diagnostic data we can explore a few advanced ways to configure what data is collected and how it is saved. There are two scenarios we will explore in this article – how to remotely change diagnostic configuration at runtime, and how to instruct the Azure diagnostic agent to save key data points immediately instead of waiting for a scheduled transfer time. Let’s go!

Remote Configuration

You might be asking yourself a very good question right about now – “self, why do we even need to remotely configure diagnostics?” Good question! Consider the scenario where you have a web role deployed, but it suddenly starts to act a little “funny” (you know . . . bit rot). You need to figure out what is going on with that role. There are two options – redeploy the cloud service with an updated configuration to enable more verbose logging, or change the diagnostic configuration. Redeploying is a perfectly acceptable solution, but it isn’t exactly a fast operation. You’ll need to go through the build and deployment process as the diagnostic.wadcfg file is bundled in the deployment package (.cspkg file).That all takes time and maybe your code isn’t in a good place – although one could rightful argue that a quality source code control system and deployment process would allow you to get previously deployed code from a label or tag. It’s all extra ceremony to deal with, when all you really want is to add a few performance counters or turn up the verbosity of your logging. That’s where remote configuration comes in to the picture. With the ability to remotely change the diagnostic configuration, we can quickly and easily make such changes without going through the cloud service deployment process.

From our previous articles (if you haven’t read those yet, shame on you and do it now . . . . I’ll wait . . . . ok) we know the diagnostic configuration is saved in Azure blob storage, and that the diagnostic agent periodically polls for changes to that configuration. If there is a change, the agent will load the new configuration.

2219-Diag-3-Fig-1-Diagnostic-Polling1-62

Figure 1 – Diagnostic Agent Process

What we need to do now is to change that configuration in blob storage. There are a few ways to accomplish this goal:

  1. Write some code
  2. Use Visual Studio
  3. Use a 3rd party tool

Update Diagnostic Configuration via Code

Changing the diagnostic configuration can be invoked from any number of applications. It could be an application hosted in Azure, or an application hosted externally – perhaps a management application inside an on-premises datacenter.

In order to be sure we are working with the right role, we need to get the deployment ID and role name. The role name is in the ServiceConfiguration.csdef file, and the deployment ID can be found in the Azure management portal when looking at the properties of the deployed service (under the “quick glance” area of the “Dashboard” page).

Heeding my dad’s advice to follow the KISS principle, let’s get started by creating a simple console application. We’ll need to add a reference to the Microsoft.WindowsAzure.Diagnostics.dll assembly to the project. For this example, I’m using version 2.4.0.0 from the Azure SDK.

2219-Diag-3-Fig-2-Add-Microsoft.WindowsA

Figure 2 – Add Reference to Microsoft.WindowsAzure.Diagnostics

Now we can write some code:

After running the code, have a look at the diagnostic configuration files in blob storage. Notice the timestamps on the files are the same. Previously the timestamps were likely a little different – as each was created when the role instance came online, and each may come online at a slightly different time according to the update domain.

2219-Diag-3-Fig-3-After-Update-Code-620.

Figure 3 – Diagnostic Configuration Files in Blob Storage

Sure, writing code is cool and all, but this code is basic. Wouldn’t it be great if there was tooling to do this work for us? Yes, yes, there is such tooling – Visual Studio and Cerebrata Azure Management Studio can both set the diagnostic configuration remotely.

Visual Studio

We can use Visual Studio to update the diagnostic configuration data for the entire role, or for select instances. To get started, open Server Explorer, navigate to the desired Cloud Service, and then right-click on the role for which to update the diagnostic data.

2219-Diag-3-Fig-4-VS-Update-Diagnostic-1

Figure 4 – Update Diagnostics with Visual Studio

Next, click on the Update Diagnostic Settings… option. At this point Visual Studio will retrieve the necessary diagnostic configuration information from Azure blob storage. The diagnostic configuration wizard should look familiar – it is the same as we saw when configuring diagnostics for the first time. This time the settings previously configured are already filled in. Now we just have to set the new options.

2219-Diag-3-Fig-5-VS-Update-Perf-Counter

Figure 5 – Diagnostic Configuration Wizard

Clicking OK will then save the updated configuration back to blob storage, where the diagnostic agent will pick up and apply to the instance(s) on the next polling cycle (default is 1 minute).

Third Party Tools

As we’ve seen so far, in order to force an on-demand transfer, the diagnostic configuration file in blob storage needs to be updated. Instead of writing code to do this, a third-party tool can be used to simplify the process. For the purposes of this article’s examples, let’s take a look at how Cerebrata Azure Management Studio (AMS) can help.

2219-Diag-3-Fig-6-Subscriptions-in-AMS.p

Figure 6 – Hosted Services in Cerebrata Azure Management Studio

AMS makes it really quick and easy to view the current diagnostic configuration, and selectively update that configuration. To get started, expand the Hosted Services section beneath your subscription, and then double-click on the desired service. This will open a details view of the service.

From the details view, you can now select the deployment and then the Diagnostics button. You’ll want to select the Remote Diagnostics option. We’ll come back to the On-Demand Transfer option later. I know . . . I’m such a tease.

2219-Diag-3-Fig-7-Remote-Diagnostics-1-6

Figure 7 – Detailed Management View in Cerebrata Azure Management Studio

You’ll then need to select a storage account – remember this is the storage account for your diagnostic data. You will next select the role instances you want to work with – all the instances or just select instances. Make the desired changes to the diagnostics configuration and save the new configuration. AMS will take care of updating the corresponding configuration file in blob storage.

I will point out though that one feature I like is how performance counters are set. I don’t know about you, but I really hate typing in the performance counter definition. With AMS, I can quickly select the counter I want, even saving it for future use (like a favorite).

2219-Diag-3-Fig-8-AMS-Set-Performance-Co

Figure 8 – Set Performance Counters in Cerebrata Azure Management Studio

Unintended Consequence

We’ve seen how to update the Azure diagnostic configuration for existing roles. But what happens when a new instance comes online? For example, if there is a scaling action to increase the number of role instances. When a new instance comes online, Azure uses the most recent deployment as the base for the new instance. That means the code and the diagnostic configuration. The new instance will have the diagnostic configuration that was in place when the role was originally deployed. It will not have any configuration changes made after the deployment. The consequence here is that if changes were made to the configuration, the instances would have different configuration – they’re out of sync.

Remote configuration of diagnostic data can be a good and valuable feature. Nevertheless, be aware of the consequences. In general, I often prefer to do a deployment update to ensure all my instances are in sync. However, there are times when a quick fix is certainly necessary. Just be sure to do a deployment to get that latest “golden image” in place with the desired state.

On-Demand Transfer

So far we have seen how we can remotely configure the diagnostics configuration for already deployed Azure Cloud Services (web/worker roles). Doing so can be immensely helpful when trying to learn more about what’s going on with your solution. In most cases, there is still going to be a delay between the time the data is collected and the time it is persisted in Azure table or blob storage. What if we need that data sooner . . . . like right freaking now?! Perhaps something bad has happened in our application and we need to collect and send that data right away. Or, maybe the diagnostic configuration is set to only collect data and not to perform a scheduled transfer – like how the default configuration works. This is where an on-demand transfer can come in handy.

At a high level, an on-demand transfer works as follows:

  1. Obtain information about the deployment and role instance(s).
  2. Configure options for the transfer, such as the timespan of data you wish to transfer.
  3. Set the data to be transferred (performance counters, event logs, etc.)
  4. Update the diagnostic configuration file
  5. Clean up
    1. Optionally remove/End the on-demand transfer configuration
    2. Optionally check for a message on an Azure storage queue to know when the transfer is complete

I bet that last point caught your attention. In order to know when an on-demand transfer is complete, you must check for a message on an Azure storage queue (which you configure). We’ll see more of that when looking at the code. Speaking of which, let’s get on with it already!

To initiate the on-demand transfer, we’ll need code such as the following:

Note: In the above code, make sure your project is referencing the Microsoft.WindowsAzure.Diagnostics.dll assembly (version 2.4).

After running this code, take a look at the diagnostics configuration file in blob storage. You’ll notice a new element, OnDemandTransfers, added to the XML file. This is instructing the diagnostic agent to execute the on-demand transfer. Note that the BeginOnDemandTransfer isn’t really an async operation as the name might suggest. It actually isn’t automatically kicking off that transfer, it is simply updating the configuration file to start doing the on-demand transfer.

When the on-demand transfer completes, a message is placed on the Azure storage queue we specified in our code. We can retrieve this message to learn more about the transfer. This could be helpful, for example, if you have a process that was monitoring this queue for those messages and then updating some other system or alerting a human.

Note: In the above code, make sure your project is referencing the WindowsAzure.Storage package (version 4.2.1) from NuGet.

If you were to execute this code, you should notice multiple messages on the queue. A message is placed on the queue for each diagnostic data buffer that is transferred.

Note the reference to EndOnDemandTransfer in the code above. This method is actually updating the configuration file in blob storage to remove the data buffer transfer element(s) from the OnDemandTransfers section. Failure to do so will prevent future on-demand transfers from occurring (as only one can be in process at any given time.

On-Demand Transfer with a Tool

Much like how Cerebrata Azure Management Studio (AMS) can help us with remote configuration of diagnostic data, it too can help us with performing an on-demand transfer. Remember that “On-Demand Transfer” option I teased you with earlier – let’s go there. The process for configuring/initiating an on-demand transfer is pretty similar.

AMS will invoke the necessary API calls to update the configuration file in storage to start the on-demand transfer process. AMS will also a uniquely named Azure storage queue for the completion message and monitor that queue. If you opt-in to allowing AMS to “refresh” to transfer status, it’ll handle getting that message from the queue and subsequently ending the corresponding transfer – thus effectively updating the configuration file in storage.

Pretty easy, right? If you do need to invoke an on-demand transfer manually, a tool like Cerebrata AMS can be very helpful since it takes care of many of those important details on our behalf.

Needle in the Haystack

When we perform an on-demand transfer, it is possible to get duplicate records if the diagnostic agent has already persisted data, and then the on-demand transfer window overlaps. So how do you tell if the record is from a scheduled transfer or an on-demand transfer? I’m so glad you asked! Look at the row key. The row key will indicate “OnDemandQuery” at the end.

2219-Diag-3-Fig-9-AMS-View-OnDemandQuery

Figure 9 – On-Demand Transfer Records

Let’s Wrap It Up Already

So far in this series we have reviewed some of the basic features of Azure diagnostics, including what data can be collected, how it is collected, and where that data is stored. We’ve also seen a few different ways to configure the collection of data, and some advice on how to get the most out of your diagnostics.

In this article we reviewed a few advanced features in Azure diagnostics – remote configuration and on-demand transfers. With remote configuration we learned how to update the configuration for a running Cloud Service without the need to do a full redeploy. On-demand transfers allow us to send important data to persistent storage right away, without waiting on a scheduled transfer time.

Next time we will review how to configure a common logging framework, NLog, to work with Azure Diagnostics. We’ll also take an introductory look at the new changes coming in Azure Diagnostics 1.2. So far, we have been using version 1.0. There are some interesting new features in version 1.2 that are worth exploring. Until then, happy coding!