Azure BizTalk Services: Connecting to On-Premises Resources

By moving applications to Azure, you can't always avoid the requirement for accessing data from applications that are based within your intranet. You have a wide range of choices for doing this. Matt Milner explains that there are several technologies that can achieve this for BizTalk, but that will apply more generally.

In previous articles in this series I have focused on building BizTalk solutions in Azure. However, as many companies are finding, the cloud is not an all or nothing proposition. Many solutions built in Azure will rely on data and application logic found on-premises. In this article, I will focus on the different ways to connect BizTalk applications that need to communicate between on-premises and Azure.

Networking Solutions

BizTalk Server was built around the idea of being able to connect to a variety of transports and application interfaces using adapters. Unfortunately, almost all of the adapters and connectivity options rely on being able to communicate within or across networks. When you deploy BizTalk Server in Azure Virtual Machines (IaaS), by default your BizTalk Server machines will only be able to make outbound calls from Azure. These calls may or may not succeed depending on the resource you are trying to access. In the common case where you want to access an on-premises system, this will fail because the BizTalk Server will not have access to your internal network.

For these and other scenarios where you need to connect your Azure and on-premises resources you can leverage Azure Virtual Networks and VPN connections as I have discussed in previous articles. With this solution, you group your Azure resources including Virtual Machines, Cloud Services, and Web Apps into an Azure Virtual Network. You then connect that network to your on-premises network using one of the VPN solutions available (Express Route, site-to-site, point-to-site).

2277-1.x-emf-ab7dcfde-0958-43bf-8719-c4f

Figure : Connecting Resources with VPN

Using this solution your on-premises applications will be able to reach your Azure resources and vice versa. This solution provides an experience most like what you would have without a cloud component and is similar to a branch office solution. As such, the most important planning decision is to make sure the VPN solution you choose will meet your needs for throughput.

Probably the largest benefit of using a networking solution to connect your resources is that it involves the least amount of development effort. You do not have to change the way your BizTalk solution communicates with other systems, or use different messaging solutions to exchange data between BizTalk and your other applications.

Application Connectivity Solutions

Network solutions often provide the simplest solution from the application developer perspective, if not for the network administrator. However, there are several solutions for connecting application nodes across network boundaries even when firewalls and network address translators (NAT) are present. The following solutions can be used selectively by applications, and focus on connecting applications together as opposed to opening connectivity between all the applications on the network.

Service Bus Relay

The Azure Service Bus provides two modes of messaging, the first of which is relayed messaging. In this mode, messages can be sent to an Azure Service Bus endpoint in the cloud – even from behind a firewall in most cases – and relayed to another listening node that is also possibly behind a firewall.

2277-1.x-emf-c136051c-1694-47fe-a2b7-6b1

Figure : Connecting application nodes with Azure Service Bus Relay

Essentially this solution provides a rendezvous point on the internet where applications can connect and relay messages such that each application is reaching out of its respective data center. Most networks block incoming traffic but are not as strict about blocking this type of outbound traffic. For those that are, there are known port numbers that can be configured in the firewall for outbound connections to the relay.

Using the Azure Service Bus Relay, you can send messages from an Azure Virtual Machine to a web service in your on-premises data center. Likewise, you can send messages from your on-premises applications to your BizTalk Server instances running in Azure Virtual Machine instances. Because each endpoint is connected to the relay, the firewalls are generally not an issue.

Unlike the VPN solution, this solution does not limit you to a particular network. You can use this same solution to manage application communication between your BizTalk instances in Azure Virtual Machines and partner applications and services in remote data centers. You can create different relays for different messaging solutions, and you can restrict access to the relay to control who can send and receive messages.

Service Bus Brokered Messaging

While the Azure Service Bus Relay provides a convenient solution, it does have one potential drawback: it assumes both sender and receiver are connected when sending messages. In many solutions that is not realistic, or you may need more guarantees of message delivery. Brokered messaging over the Azure Service Bus provides a solution involving queued messages that can be sent regardless of the connectivity state of the listener. When the listener comes online, it will retrieve the messages from the queue.

2277-1.x-emf-e7352561-a28a-4232-8239-981

Figure : Brokered messaging with Azure Service Bus Queues

For example, you might set up an Azure Service Bus Queue for receiving messages into your BizTalk Servers running in Azure Virtual Machines or an Azure BizTalk Service itinerary. Using this configuration your partners or on-premises applications can send messages to the queue and the BizTalk solution in Azure will receive those messages. Using queues allows for messages to arrive faster than your BizTalk solution can process them and then allows for those messages to be stored if your BizTalk solution is currently down for maintenance. Alternately, you could gain the same benefits by using a queue to send messages to an on-premises application from your Azure BizTalk solution.

The design of queues is such that a message is received by a single receiver. That does not mean you cannot have multiple receivers, just that each message will only be received once. Multiple receivers allow you to scale your receiving application horizontally to increase throughput. Sometimes, however, you might need to send messages to multiple partners or receiving applications. You can handle that by using routing in either BizTalk Server or Azure BizTalk Services, but each requires quite a bit of configuration. Azure Service Bus Topics provide you with the ability to send a message and have multiple receivers get a copy of the message.

2277-1.x-emf-0cf8941a-1115-4535-9aee-95b

Figure : Brokered messaging with Azure Service Bus Topics and Subscriptions

When you create a topic, you can create related subscriptions to messages on that topic. When a message arrives on the topic and matches the filter criteria in the subscription, the subscription gets a copy of the message. Subscriptions are very similar to send ports in BizTalk Server with filters describing the subscription criteria.

Using topics, you can create solutions to send a message from your Azure BizTalk solution to many different receiving applications with each receiver having the ability to filter for only those messages that are relevant. For example, your application can publish a message to a topic regarding new inventory of a particular product in stock. Those partner applications interested in that product or category of product then receive a message on their subscription informing them of the new stock so their staff and systems can react. As another example, your Azure solution could publish a message when a new customer is brought on board. Several internal applications, each with their own subscription to the topic, receive the message and act on the information about the new customer.

Brokered messaging, whether using queues or topics, provides a model for communicating between application nodes using durable messages and gives you options for different message exchange patterns. Because the messages are exchanged through the Azure Service Bus, there are usually very few issues with firewalls and other network devices.

Hybrid Connections

Messaging is a core piece of integration solutions, but many times in your application design it is nice to be able to simply connect to a resource as you would normally. For example, rather than trying to create a service layer over a database, and then use the Azure Service Bus or some other messaging solution to cross network boundaries, it would be nice to be able to simply access the database as if it was in the same data center. That is the thinking behind Hybrid Connections.

Hybrid Connections are a feature of Azure BizTalk Services that allow you to define a connection to your on-premises resources, such as database servers (SQL Server, MySQL, or Oracle) and HTTP web services, and connect to them from Azure just as you would locally. For example, in a .NET Azure WebApp you could use Entity Framework or ADO.NET to connect to a SQL Server instance running on-premises by using the same connection string you would use if you were accessing it locally. This enables scenarios where you want to move part of an application from your data center to Azure while reducing code changes necessary to make it work.

Hybrid Connections actually leverage the Azure Service Bus Relay to relay TCP traffic between Azure and your data center. You must install a proxy in your datacenter which connects to the Service Bus and proxies data to your resources in the datacenter. When you create a new Hybrid Connection, you specify a particular resource such as a specific SQL Server instance or web service URL.

2277-2277-1.x-emf-6516fce0-3e22-4926-921

Figure : BizTalk Services Hybrid Connection

At this time, only Azure Web Apps and Mobile Apps are able to consume Hybrid Connections. This means you cannot use them from an Azure BizTalk Service or a BizTalk virtual machine. That does not mean they are not useful in a BizTalk solution. If your BizTalk environment is going to be on-premises but you have web applications or mobile applications running in Azure that need to send data to a BizTalk endpoint, you can leverage Hybrid Connections.

For example, if you have a web application that accepts orders from users and you need to send those orders to your BizTalk Server group running on-premises, you can use a Hybrid Connection. In this scenario you should set up an HTTP receive location in your BizTalk Environment and then create a Hybrid Connection pointing to that HTTP endpoint using the local address. After installing the Hybrid Connection proxy on a server in your datacenter, you can use the Hybrid Connection from your web application and post the data to the HTTP endpoint. This allows you to host the web application in Azure – taking advantage of cloud scale and features – while keeping your BizTalk Server installation on-premises.

BizTalk Adapter Service

Hybrid Connections enable you to connect your web and mobile applications to your on-premises resources, but that doesn’t help if you’re trying to connect from an Azure BizTalk Services bridge. Fortunately, there is a similar solution for this scenario known as the Azure BizTalk Adapter Service. This service also leverages the Azure Service Bus Relay to connect an on-premises proxy service with your cloud resources. While Hybrid Connections enable relaying TCP connections across network boundaries, the BizTalk Adapter Service enables relaying messages that are handled by adapters written to interact with specific line-of-business (LOB) systems.

To work with the BizTalk Adapter Service you must have the Azure BizTalk Services SDK installed along with Visual Studio 2012. You define the specific connection you want in the Visual Studio Server Explorer, providing the connection information for the line-of-business system (currently the supported systems are SAP, Oracle Database, Oracle E-Business Suite, Siebel eBusiness Applications, and Microsoft SQL Server). After providing the information, a new node will appear in the Server Explorer for that LOB application and you can drag it onto the design surface of your message itinerary.

In addition to the itinerary configuration, there are two other key runtime components that must be set up. The first is an endpoint on the Service Bus Relay which will relay messages from the cloud to your datacenter. The second is the BizTalk Adapter Service Runtime which is comprised of a management service and individual LOB application services hosted in IIS. These services act as the local proxy, they listen on the relay endpoint, host the LOB application adapters, and broker messages between the two.

2277-2277-1.x-emf-613b6f12-f603-40b7-bc2

Figure : BizTalk Adapter Service

Because you have to define the LOB interaction as part of the itinerary, these systems cannot be accessed directly from an EDI bridge. However, because you can chain EDI bridges to XML bridges, you still have the opportunity to build a solution that leverages the LOB applications regardless of the initial bridge where the message is received. Using Azure Service Bus Queues and Topics, you can also build a variety of solutions combining several bridges together. Using this model you could even make a solution where the LOB application itinerary is its own separate itinerary and receives messages from a queue or topic. This would make it accessible from numerous other inputs such as EDI bridges, XML bridges, or even other applications outside of Azure BizTalk Services.

2277-2277-1.x-emf-bbeb9f2c-8335-4036-b3a

Figure : Combining bridges to access LOB systems

Choices, Choices

With all of these choices you have a lot of options. The networking solution works very well if you want to have an environment that resembles an on-premises BizTalk solution with all resources available and that is connected.

The application solutions can be used based on the solution you are creating and the connectivity you require. Table 1 provides some examples of application connectivity options based on the resources in and out of Azure that you are trying to connect. This is not an exhaustive list and there may be more than one solution for each scenario. You might also have multiple aspects to your application that require more than one connectivity solution. For example, you can have an Azure WebApp that uses Hybrid Connections to pull data from an on-premises database during runtime then sends the data to an Azure BizTalk Services itinerary over a Service Bus Queue when someone places an order. This same queue could be used from other applications to submit orders as well. The itinerary then connects to an on-premises LOB system using the BizTalk Adapter Service to insert the orders into your system of record.

Azure On-Premises Solution
Web or Mobile App Database or HTTP web service Hybrid Connection
Azure BizTalk Service Line-of-Business Application Azure BizTalk Adapter Service
Azure BizTalk Service Web service, database or other application Service Bus Brokered Messaging
BizTalk VM Line-of-Business System Service Bus Queue + BizTalk Adapter Service
BizTalk VM Partner web service Service Bus Relay
Azure BizTalk Service Partner custom application Service Bus Brokered Messaging (Topic)

Table 1: Example application connectivity

Where are we?

Over the course of this series we have looked at IaaS and PaaS BizTalk solutions in Azure including the use of EDI and AS2. In this article we focused on connecting those solutions to your on-premises applications and resources using a variety of technologies available today. In the next article in the series we are going to look at the various mechanisms available for monitoring your solutions in Azure and tracking the data that is important to your business.