As a BizTalk developer you are familiar with schemas, maps, adapters, pipelines and all the other working parts of BizTalk Server. Now Microsoft has introduced several options for BizTalk in “The Cloud” or BizTalk in Azure. This support for BizTalk in Azure comes in two forms: running BizTalk in virtual machines hosted in Azure (Infrastructure as a Service – IaaS) or using BizTalk Services (Platform as a Services – PaaS). This article will help you understand these two models and how to work with them. Specifically: what works just as you would expect based on your experience with BizTalk Server on premises, what new features are available, and how to map your current skills onto the new PaaS offering.
Running BizTalk in Azure Virtual Machines – IaaS
One of the first challenges faced when getting ready to develop your first BizTalk solution is setting up the environment(s). You need to provision a SQL Server cluster, BizTalk Servers, often web servers, and usually SAN space for all the above. Now try building development, test, and production environments and you multiply the servers and resources that need to be procured or at least provisioned. This is where the IaaS offering in Azure can simplify your work by providing virtually unlimited resources and enabling you to very quickly provision the servers you need.
Windows Azure provides a gallery of Virtual Machine (VM) images that can be used to provision a server in an Azure data center. Figure 1 shows an example of creating a BizTalk Server 2013 image using the most recent Azure portal. Additional images are available for SQL Server, Visual Studio and other servers that you might need in your environment. Note that Visual Studio images may only be available as part of an MSDN subscription. The additional parts of the wizard allow selection of the CPU, memory, and networking among other settings.
In the IaaS model most aspects of development and management remain the same as an on premises solution. The same tools can be used for development, deployment and management. The major differences come in how you provision and configure the environment.
A typical BizTalk environment minimally consists of two BizTalk servers, high availability SQL Server, and an Active Directory (AD) domain controller (DC). This is true in Azure as well but the setup is slightly different as you need to consider differences in networking, connecting to on premises resources, and how to manage high availability and disaster recovery.
Because Azure provides high density hosting of computing resources in Microsoft datacenters, networking has to be handled differently for a BizTalk IaaS deployment. Azure enables you to define virtual networks into which you can place your virtual machines. These virtual networks provide a private IP address space that your virtual machines can share to be able to easily address each other, making connections between Active Directory, SQL, MS DTC, and services built with local network connectivity in mind much easier to manage. Without virtual networks, communication between virtual machines is challenging because of the networking limitations in place for security considerations.
While there are many benefits to hosting a BizTalk solution in Azure, many applications and resources that form part of the overall solution often remain on premises. For those scenarios you will need to consider how to reach your BizTalk environment from the on premises systems and how to reach those systems from Azure. There are several options varying in complexity of setup and applicability to application or protocol types.
Azure Virtual Network provides a VPN connection between your virtual network in Azure and an on premises network. VPN connections can either be point-to-site where a single server in your data center has a secure connection to your Azure network, or site-to-site where you setup a secure connection between the two networks. A point-to-site connection is as easy to setup as connecting your laptop to a VPN which is essentially what you are doing in this scenario. If you just have one or several servers that you need to connect to your Azure network, this is the simplest solution. For a site-to-site connection you need a supported external VPN device and IPv4 address already configured. You can then run a script to setup a connection between the two VPN networks allowing multiple machines in your environment to be connected to the Azure network without having to install or configure any software on the individual servers. Virtual networks enable access to Virtual Machines and Cloud Services in Azure (items that can be added to a virtual network configuration).
Azure Express Route goes beyond the virtual network solutions and provides a dedicated secure connection between your network in Azure and your on premises network. It is supported by many different network providers and provides more reliability, security (nothing goes over the public internet), faster speeds and lower latency than virtual networks.
When considering these options, virtual networks are generally recommended for development and testing scenarios while express route is recommended for production deployments, especially useful for BizTalk solutions when you consider a hybrid approach to disaster recovery (more in the next section). This comes down to the improved reliability and throughput of those networks. You should evaluate the network requirements for your solution and the pricing of each of these options to see what meets your needs.
Connecting to On Premises Resources
In addition to network level solutions for connecting your environments there are several application level options for connecting your BizTalk application in Azure to your on premises resources: Hybrid Connections and Azure Service Bus. Hybrid connections allow you to connect to on premises databases and HTTP resources from Azure websites and mobile services using regular connection strings and URLs. Service bus provides both message relay and brokered messaging options that can traverse firewalls and network address translators.
Hybrid connections are a feature of BizTalk Services and provide connectivity similar to point-to-site virtual networks in that one server on your environment is connected to Azure resources. Where they differ is that the entire server is not exposed to Azure, only a connection to a particular resource on that server. Additionally, the connection is initiated from Azure and therefore not useful for resources on premises that need to initiate a connection to an Azure resource. Today these connections are only available to websites and mobile services and therefore in a BizTalk solution most useful if you need to expose an on premises BizTalk resource to an Azure website or mobile service.
Hybrid connections rely on a service installed and configured on a server in your data center. This service acts as the gateway between connections from Azure and your local resources on that server or other servers in your environment. One useful feature of Hybrid Connections is that a single defined connection can be shared across multiple websites and mobile services.
Azure Service Bus provides two different modes of communication between resources: relayed and brokered. With relayed messaging both an on premises resource and cloud resource connect to the Service Bus at a pre-arranged address and messages are relayed through the bus. Since both services are initiating the connection, many firewall and NAT issues that often pose problems with cross data center communication are avoided. Note that if you have firewalls or proxies that block certain outbound ports you may need to configure exceptions to leverage the relay feature.
Brokered messages provide one-way queued messaging with publish and subscribe functionality. This type of communication is similar to how BizTalk internally handles messages on the Message Box and enables greater scale and reliability as well as enabling alternate messaging patterns such as fan out.
In an Azure BizTalk solution Service Bus can be used to connect your BizTalk IaaS application to on premises resources using either the relay or brokered option. Using the same configuration you can enable partner organizations to interact with your BizTalk solution without having to provide direct access to the BizTalk servers themselves.
High availability and disaster recovery
One of the benefits of cloud computing and using services such as Azure Virtual Machines is being able to leverage the infrastructure services of a huge datacenter. Virtual Disks are stored in Azure Storage which has redundant copies of data locally and optionally geo-redundant copies as well. Despite the many features aimed at limiting your downtime, some precautions are still necessary especially when it comes to SQL data.
Running SQL Server in Azure virtual machines provides some different options for high availability (HA) and disaster recovery (DR). Newer versions of SQL Server even support backing up directly to Azure Blob Storage. Review your options for HA and DR solutions using the supported Azure models for SQL.
In addition to backing up your data and having redundant copies of your virtual machine hard drive, you will also likely wish to have backups of your virtual machine images. You can use a couple of different approaches to back up your virtual machine including snapshotting of the disk image in blob storage or more traditional Windows backup techniques. Using virtual machines provides the options you would expect on premises as well as adding additional options that are cloud specific so that you can either use what you know or take advantage of options designed for the cloud.
Building solutions on BizTalk Services – PaaS
Having looked at the IaaS model you might have noticed something: you still have to install all the software and manage all the patches on all the machines running in your Azure BizTalk solution. The PaaS model aims to avoid all of that maintenance by providing you with a BizTalk service that you provision and into which you can deploy your solution. The only infrastructure decision you need to make is what features and scale you need to handle your business.
BizTalk Services is a newer offering and does not necessarily have all the capabilities of the decade old BizTalk Server product. Many of the core functions are present and many more are being added in future releases. The product as it currently stands includes rich messaging functionality, EDI processing (both X12 and EDIFACT) and connectivity.
One of the core needs in integration is connecting systems with disparate formats, transports, and security considerations. BizTalk Services provides this messaging connectivity in the form of Bridges. A bridge is defined with a message itinerary that describes the end to end processing of a message from the time it is received until it is sent out of the system. This type of end to end message processing definition has long been requested by BizTalk developers. As an example, Figure 2 shows a simple message processing itinerary that receives a message from a Service Bus Queue, processes the message and then routes the message to one of two destinations based on contextual properties in the message.
For BizTalk developers this type of processing is familiar. The Bridge encompasses the definition of a receive location (adapter and pipeline), routing configuration or subscriptions, and send ports. The “SourceQueue” item defines receiving a message from a Service Bus queue, which is essentially the adapter configuration. The “DestinationQueue” and “FanOutTopic” items define the equivalent of Send Port Adapters and a routing table defines the rules for routing messages as they finish the “XmlBridge” step.
The “XmlBridge” step can be expanded as shown in Figure 3 and includes all of the typical pipeline and transform logic commonly known as Validate, Enrich and Transform. The bridge starts with a stage to define the message types that will be processed. Despite being called an “XML” bridge this type of bridge can handle both XML and flat file documents. The “Decode” and “Encode” stages of the process enable receiving and/or sending flat files using flat file schemas. The “Validate” stage validates documents against a specific XSD schema just as in a BizTalk receive pipeline.
The transform stage can be configured with several maps and determines which map to use at runtime based on the incoming message type. Finally, the “Enrich” stages allow for adding or copying properties. An example of enrichment might be moving a SOAP header into the message context or adding values to a BrokeredMessage property before sending the message to the Azure Service Bus.
This example uses a one-way bridge but there is also a request-reply XML bridge that can be used when interacting with web services that provide responses. Additionally a pass through bridge can be used which only provides the “Enrich” stage for message processing which can be useful when processing non-XML messages such as images or documents.
Developers can also write code to execute before or after each stage in the bridge. The simple .NET interface you implement allows you to process the message and manipulate context properties much as you would with a custom pipeline component.
After you develop your bridges in Visual Studio along with your maps, schemas, and any custom code you then deploy the package to your BizTalk Service. Once you’ve deployed your bridge you can send messages via HTTP if your bridge is setup for that protocol or the bridge will start monitoring queues, FTP servers and other sources you specified.
In addition to messaging with XML and flat files BizTalk Services also support both X12 and EDIFACT EDI processing. Just as in BizTalk Server these interchanges depend on the configuration of partners, profiles and agreements that define the EDI processing and the deployment of EDI schemas for the transaction sets your environment supports. Configuration of the partners and agreements all happens through a web interface but provides most of the same settings you would configure in a BizTalk Server environment. For example, Figure 4 shows an X12 agreement being defined between two parties. Once the basic details are provided the system allows you to specify the specific schemas being used, authorization data, and other EDI processing settings.
In BizTalk Server EDI processing follows the same steps as normal processing using pipelines, adapters and maps so it is only natural that in BizTalk Services the EDI processing leverages bridges. After you have configured an agreement for two parties you then define bridges for that agreement, one for each direction of traffic flow between partners. Unlike the messaging bridges discussed earlier, EDI bridges are defined and configured in the web portal.
Each EDI bridge defines the receive protocol (adapter), transform, and routing logic. Messages can be received over HTTP, FTP, or SFTP and can be sent to the Azure Service Bus (relay or brokered messaging), Azure Blob Storage or to another BizTalk Service bridge. By being able to chain your EDI Bridge to another bridge you can receive EDI, XML, and flat file messages across the same processing logic. Additionally, suspended messages can be routed to any of the same destination types enabling rich error handling workflows.
On Premises Connectivity
Just as in an IaaS solution there will be a need to connect your on premises services with your BizTalk Service messaging solution. Because BizTalk Services is a PASS offering and a multi-tenant hosting solution the private network options are not available. To address this need BizTalk Services has extended the BizTalk LOB Adapter Pack with cloud capabilities so an on premises resource can be made accessible over an Azure Service Bus Relay.
In this model an agent or service is hosted on an on premises server. This agent is then configured with the connectivity and authentication information for the BizTalk Service and acts as the bridge between the cloud and your on premises applications hosting the LOB adapters for applications such as SQL Server, Oracle, SAP, and Siebel. When developing your bridges you can add the LOB applications as sources or destinations for messages using tooling in Visual Studio installed as part of the BizTalk Services SDK.
On the Azure side, Azure Service Bus relay endpoints are setup that provide the connectivity between the Adapter Service and the Bridges hosted in Azure. All of that infrastructure gets managed automatically as part of defining your bridge and deploying it to your service.
Management in the PaaS Model
Given that this is a platform as a service offering, the configuration, development and deployment operations are different from those used for either on premises or IaaS solutions. That said, many of the concepts from traditional BizTalk development will transfer to the PaaS model.
To get started with BizTalk Services you provision a service from the Azure portal. As part of this process you will be expected to select a particular edition and the number of units. The differences in editions are primarily about the number of artifacts you will be creating or resources you will be consuming as can be seen in Table 1. [Check the Azure website pricing
for up to date information about what is included in each edition and the current pricing.] Each unit enables you to multiply those limits. For example, buying two basic units means you would be able to use 50 Bridges and 100 Agreements in your solutions. More information on each of these resources can be found in the following sections.
Table 1: BizTalk Editions
|Level||Hybrid Connections||Bridges||Agreements||HA Capable||Adapter Connections|
|Free||5 / 5 GB||None||None||No||None|
|Developer||5 / 5 GB||25||10||No||1|
|Basic||10 / 10 GB||25||50||Yes||2|
|Standard||50 / 250 GB||125||250||Yes||5|
|Premium||100 / 500 GB||500||1,000||Yes||25|
In addition to the compute resources provisioned as part of your service you will also need to create or select an Azure SQL database and Azure Storage account for use by the service. The Azure SQL database will be used to store tracking information including message properties and content. The storage account is used to store various configuration settings and log files in blob storage. It is important to note that these resources are NOT included in your billing for the BizTalk Service and are charged separately so you will need to include that cost in any estimating you do.
Notice in this model you provision for the scale you require not based on number of virtual machines or size of SQL Server instances. This simplifies the planning, provisioning, and maintenance of your solution overall.
Once you have provisioned a BizTalk Service you can begin developing your solutions. Like a traditional BizTalk Server environment you will use Visual Studio to develop your solutions. However, there is a specific BizTalk Services SDK that must install in order to begin developing your solution Visual Studio 2012 (Currently the BizTalk Services SDK does not support Visual Studio 2013).
Several of the Visual Studio tools will be familiar to BizTalk developers such as the schema designer and the mapper. The schema designer itself is nearly identical while the mapper is similar but has some unique new features and functoids. Specifically the mapper provides many functoids related to list management; adding items and retrieving items from lists; making it easier to manage some previously challenging looping constructs. The mapper also includes a functoid that many BizTalk developers have longed for in BizTalk Server which extracts context properties from the source document and makes them available in the transformation process.
Additional designers enable the definition of message processing logic in the form of itineraries and bridges as noted previously. You can also create resource projects in Visual Studio which makes it possible to easily deploy items you will use in your solutions such as EDI schemas.
Much like a BizTalk Server solution is installed to a specific environment, your BizTalk Services solution must be deployed to your provisioned service. Because this is a service you cannot control all of the items available. Only the specific BizTalk Services artifacts and your custom message processing logic can be deployed to your BizTalk Service. This is the nature of a PaaS solution in that you trade off some control of the physical or virtual servers in order to have fewer parts of the solution to maintain overall. What you lose in control of the environment you gain many times over in the speed to provision and deploy your solution and the flexibility you get in the PaaS environment.
When redeploying your project you can also choose to “refresh the service after deploy” which means you are deploying new assemblies but don’t need to completely redeploy the service. This is especially useful if you are deploying a simple change to a schema or map and don’t need to create any new endpoints in your solution.
Where are we?
BizTalk in Azure provides many of the same messaging, routing, and EDI capabilities that are found in BizTalk Server, but with benefits offered by the cloud. Whether you choose the IaaS or PaaS model your experience with BizTalk Server development will serve you well. In future articles in this series you will have the opportunity to learn more about each of these models for building solutions and connecting to your on premises resources as you move application logic to the cloud.