Azure BizTalk Services: Monitoring and Managing BizTalk Services Solutions

Once you've built your BizTalk solution, you will need to deploy and monitor it. Matt Milner shows that, while there are several ways to monitor your solution, and manual deployment via Visual Studio, the automated deployment of BizTalk solutions is probably best served by Azure IaaS with BizTalk Server virtual machines.

Previous articles in this series have focused on building solutions using BizTalk Services and BizTalk Server running in Azure Virtual Machines. However, there is more to the story than that: Solutions have to be deployed and monitored once they have been developed. Azure BizTalk Services provides mechanisms for deploying solutions to the cloud infrastructure in Azure and monitoring them once they have been deployed.

Monitoring BizTalk Services

There are several different aspects to monitoring an Azure BizTalk Service. This includes

  • performance of the provisioned service
  • diagnostics
  • business information tracking

Azure BizTalk Services can deal with all these.

When a BizTalk Service is provisioned, virtual machines are created to handle the runtime processing. At the same time, a user must identify both an Azure SQL Database and an Azure Storage Account to be used by the BizTalk Service. These storage resources are used to store the monitoring and tracking information produced by the runtime operations. The Azure SQL Database holds the key information used to support querying tracking information while Azure Storage Tables and Blobs are used to store things like debug logs and diagnostic information.

Metrics

When a BizTalk Service is provisioned, the scale of the service is decided. After provisioning the scale can of course be changed to meet increasing or decreasing demand, but those decisions are generally based on the current performance characteristics of the service. The dashboard page for each Azure BizTalk Service contains a monitoring view showing relevant performance characteristics.

2306-BizTalkMonitorDashboard-cd5ff06f-8a

Figure : Azure BizTalk Service Dashboard

By default there are several relevant performance metrics here, such as the CPU Usage and the failures occurring. These measures can help to assess more easily the health of a service and check that it has the processing resources required to handle the work being assigned to it. In addition, there are messaging-specific metrics that are focused more on the application layer than the physical hardware. By default, the counters for Messages Processed, Messages Received, and Message Processing Latency are present on the dashboard view in order to provide insight to the core characteristics of the message processing in the service.

By switching to the Monitor tab in the Azure Portal, you get the same basic view of the metrics in the chart. However, the monitor view also provides range information about each of the values, which can be helpful when looking at performance over time. A currently high CPU is different from a consistently high CPU. Likewise, a high latency now versus an average over time may lead to different decisions about how to respond.

2306-BizTalkMonitorMonitoring-a355a063-f

Figure : Azure BizTalk Monitor

Unlike the dashboard view, the monitor view enables the addition of several other metrics around messaging that can be helpful in certain solutions. Those messaging indicators include currently processing messages, sent messages and round trip latency, which are appropriate measures for spotting problems in certain situations.

2306-BizTalkMonitorAddMetric-1b798441-fd

Figure : Additional Metrics for Monitor View

The monitor view can provide an overall health-check of an Azure BizTalk Service, to understand whether it is performing as expected and as required. When problems do arise, or if more specific information is required, it is time to turn to tracking or logs.

Tracking

Those who are familiar with BizTalk Server will know about the advanced tracking capabilities it provides. When working with a messaging system, it is important to be able to see the messages that have been exchanged when troubleshooting technical or business issues. Azure BizTalk Services provides tracking of both EDI/AS2 messages and events for messages processed through a standard bridge.

After the parties and agreements are defined for an EDI or AS2 exchange, it is time to define an EDI Bridge to handle the message flow. I covered this aspect in a previous article. One step in configuring the EDI Bridge is to select the property and message tracking options. As seen in Figure 4, the system allows for tracking message properties on sent and received messages as well as archiving the entire message body.

2306-BizTalkMonitorEDITracking-25b6b247-

Figure : Configuring EDI/AS2 Tracking

The message-body archiving option for tracking is fairly self-explanatory. It saves the body of the message into Azure Blob Storage. The properties to be tracked are defined in another page of the bridge settings: the transform configuration. On both the Send Settings and Receive Settings (the two bridges or pipelines) the Transform page contains a section to define promoted properties as shown in Figure 5 using either XPath queries against the message body or data lookups. These properties can then, optionally, be tracked with the previous settings.

2306-BizTalkMonitorEDIProperties-30f18db

Figure : EDI Bridge Property Promotion

For AS2 exchanges, Azure BizTalk Services also supports NRR (Non repudiation of Receipt) which can be enabled on the AS2 agreement configuration. Do not confuse regular message tracking for NRR which has a specific meaning and potentially legal implications.

Tracking can also be used for standard bridges created in Visual Studio. Because these bridges are defined in Visual Studio and have much of their configuration specified there, the tracking properties and configuration are defined as part of the message itinerary.

Message properties are defined in the enrichment stages of the bridge, thereby allowing for XPath extractions, data lookup, SOAP, HTTP, and Brokered Messaging properties to be “promoted” as a named value. That property promotion is separate from tracking and allows for passing values along in the context of the message and possibly transferring them to outgoing messages.

However, once these properties have been defined, the tracking properties for the bridge can be configured to track, or store, the properties that are useful to have after a bridge has completed. As shown in Figure 6, an XML bridge can be configured to track only the properties that are deemed important to retain, or all properties including the message type(s).

2306-BizTalkMonitorTrackProperties-f19e0

Figure :XML Bridge Property Tracking

Regardless of the nature of the bridge, the tracked message events can be found in the Azure BizTalk Service management application. The tracking view allows for searching standard messages or EDI/AS2 messages and filtering by date rage and other criteria. As shown in Figure 7, the query interface is broken down between standard messaging , EDI/AS2 (protocol) messages, and batching events. Each interface provides search filter criteria to enable narrowing the results based on date, error level or endpoint URL. In addition, if an error surfaces on a client with a specific track or request ID, those values can be used to find the specific events related to the processing of that request.

2306-BizTalkMonitorTrackingPortal-dfd2e7

Figure : Tracking in the Azure Portal

One big difference in these two types of tracking is that the tracking of message bodies is only available for EDI/AS2 messaging and not XML bridges. Also, this portal does not let you search for or view the tracked properties that were configured. In order to view those properties, you need to use your favorite tool to query the TrackRecordMessageProperties table in the Azure SQL Database configured as part of the provisioning process. The records in this table are related to those in the PipelineTrackRecords table which contains the TrackID and RequestID found in the portal.

Tracking data tends to build up quickly which will increase the costs of storage in your subscription. Much like BizTalk Server allows for purging tracking data, Azure BizTalk Services provides a stored procedure and PowerShell command to purge tracking data older than a certain amount of days. The benefit of using the PowerShell command instead of the stored procedure is that the PowerShell command purges both the SQL tables and the Azure blob storage. Information on using the PowerShell commands can be found on MSDN .

Debug and Operational Logs

When trying to troubleshoot issues during development or even production, the operational and debug logs for the service can be quite useful. Operational logs keep track of the various operations that service managers take against a service such as creating the service, enabling it, scale operations, and any other changes made through the portal or REST API.

2306-BizTalkMonitorOperationalLogs-8a6e9

Figure : Azure Operational Logs

These logs are not unique to Azure BizTalk Services and are available across services in a subscription. As such there may be a need to filter the log based on subscription and service to narrow down the results to the Azure BizTalk Service being investigated. Time windows and event status or type can help narrow the results even further.

Using the operational logs is most useful to determine what changes may have been made that could be contributing to a new problem reported with the service. For example, if users report errors starting on Tuesday morning, a look at the logs might reveal that a change was made to the service late Monday night that is the likely culprit.

Debug logs, in contrast, are for deep dives into the underlying runtime happenings of a solution in order to isolate problems that might be hard to resolve based on the surface error messages. Azure BizTalk Services logs information into an Azure Storage Table. These logs contain information about bridge processing, assembly loading, artifact updates, and other low level processing. These logs are most often useful during development as they allow developers to discover the issues keeping their solution from completing successfully including issues with custom code, proper deployment of artifacts and bridge processing issues.

2306-BizTalkMonitorDebugLogs-9108c4c2-7b

Figure : Debug Logs

The logs are stored in a table named WADLogsTable. Depending on the tool you are using, it can be challenging to browse to this table because Azure BizTalk Services creates a great number of tables to store event logs and performance counters. If you have a tool that lists the tables alphabetically, this table will appear near the end of a very long list.

The information in the debug log is essentially trace output from bridges and deployment operations. When trying to find defects in logic, configuration or processing, the debug log will show each step executed in a bridge allowing a developer to follow a message as it passes through processing. If an error is logged at a certain point, control flow or routing follows an unexpected path, or other issues come up, they will be visible in the context of the bridge workflow. Although this is not a substitute for a full debugging experience, it does instead provide a log of each and every step in order to help isolate any issues.

Deploying BizTalk Services

As I mentioned at the start of this article, deployment and monitoring are those critical steps after the development of a solution. Unfortunately, unlike the many options for monitoring a solution, there are very few for deploying an Azure BizTalk Service solution.

While in development, the most common way to deploy a solution is to use the tools included in Visual Studio. These are in fact a façade over a set of REST APIs available to deploy schemas, maps, and other resources as well as to manage the listeners on a bridge configuration. However, the REST API only covers deployment of artifacts such as certificates, schemas and maps. It does not provide a means to manage trading partners or to deploy message itineraries and the associated bridges.

Another option is to use the management portal to individually upload schemas, transforms, certificates etc. to the BizTalk Service. Neither of these options provides much in the way of automation or repeatability. For example, a solution setup in a development environment ready to push to production would need to be deployed and configured step-by-step. Partners would need to be setup for EDI/AS2, artifacts deployed, and bridges deployed from Visual Studio 2012 or recreated in the portal in the case of EDI bridges.

Using the Azure BizTalk Services, PowerShell commands can provide some level of automation by enabling such tasks as the upload of artifacts, starting and stopping of sources to be scripted. They can do a lot, but they cannot be used to totally automate a BizTalk Service deployment because of the limitations on deploying bridges and the fact that EDI bridges are defined in the portal itself. This can be a real blocker for teams used to continuous integration or other automated deployment solutions. If your team depends on automation and continuous integration, then using the Azure IaaS option with BizTalk Server virtual machines may end up being a better solution than the PaaS solution of Azure BizTalk Services. The IaaS model provides more control over the environment and can leverage existing BizTalk Server tools for deployment and management.

Where are we?

In this article, we completed the series that explains the different options for working with BizTalk in Azure, the kind of solutions you can build, and how you can manage those solutions. The goal was to give readers an understanding of Azure BizTalk Services and IaaS BizTalk Server offerings in Azure to make decisions about its use and to answer questions about the capabilities. Hopefully, that goal was achieved for you.

Keep in mind that much of the new development and current efforts for EDI, B2B, and other integration work in Azure, is now focused on Azure Logic Apps and API Apps. The features available in Azure BizTalk Services are being exposed as API Apps and new features such as Business Rules are being introduced as well. As that offering reaches general availability, look to that platform to provide updated tooling and a more micro services approach to the various components of EAI and B2B messaging.