If you are a developer then at some point in your career you are going to build an application that talks to other applications. In fact, these days most applications connect to other applications in some fashion be it through web services, FTP transfer of files, or some other form of messaging. Unfortunately not all applications expose the same type of interface or support the same type of security. Connecting applications together is often referred to as integration and it has many challenges besides those already mentioned.
Integration is often used as an umbrella term to refer to many things including Enterprise Application Integration (EAI), Business to Business (B2B), and Service Oriented Architecture (SOA). BizTalk is both a specific product and an umbrella term for Microsoft’s set of integration technologies. In this article you will learn about the challenges associated with building integration solutions and how the set of products in the BizTalk and Azure toolbox can help you meet those challenges.
This article assumes little or no familiarity with BizTalk Server. If you are already familiar with BizTalk Server and interested in learning about BizTalk in Azure, you should refer to the companion article Azure BizTalk Services: An Introduction for BizTalk Developers.
Integration simply put means making applications work together. Those applications might be in your company, in which case this is often referred to as EAI – connecting your enterprise applications together. The applications to which you are connecting might also be a partner’s application, often referred to as B2B. In either case, you might be using web service to connect those systems in which case the integration may be labeled as Service Oriented Architecture. At the end of the day however, it is all about getting applications connected.
There are several different kinds of integration but the two most popular are data integration and application integration. An example of data integration would be doing a nightly transfer of data from an Oracle database used by one application to a SQL Server database used by another application. Application integration generally involves the exchange of messages between the applications on a more regular basis. For example, every time a new customer is added to a CRM system a message would be sent to the accounting system so the customer could be created in that system as well.
There are tradeoffs between these two styles and no one option is always the right solution. One major distinction between the two is that application messaging often enables the reuse of not just data, but the application logic as well. For example, when creating a new customer in the accounting system, one aspect of that operation is inserting data into some SQL tables. It is highly likely however that the accounting system has business rules and validation logic that get applied in code not in the database. When using application integration a message is sent to the accounting system and generally the same code that is used to process a new customer directly in the application is used to process the message and create the new customer.
When integrating systems using messages there are many different considerations involved including message format (comma separated files, XML, JSON), message structure (schemas for example), transport (FTP, File system, MSMQ, HTTP), and security (encryption, authentication). Any application integration solution and the tools used to build it will need to at least address these considerations.
As mentioned previously “BizTalk” is both a product and a brand. For many years the term BizTalk was synonymous with the product BizTalk Server which has been around since 2000 with the latest release, BizTalk Server 2013 R2, being the ninth version. Over the years other products have been released under the BizTalk brand name including Microsoft’s RFID implementation and the BizTalk Adapter Pack for connecting to Line of Business (LOB) systems. Other tools are now packaged with BizTalk Server including Microsoft UDDI (Universal Description Discovery and Integration) Server and Host Integration Server (HIS). In fact, the Azure Service Bus and other technologies were originally introduced under the title of “BizTalk Labs”.
BizTalk Server is Microsoft’s integration server offering providing the tools and patterns needed to address the integration considerations mentioned. It does this through a series of capabilities around message exchange and business process management. For the purpose of this article the focus will be on the message exchange capabilities of BizTalk because those are the most commonly used features and the features that have currently been migrated to the cloud offerings in BizTalk Services.
Listed below are the various integration considerations along with the specific tools provided in BizTalk Server to meet those needs.
Message Format -> Message Schemas
When integrating systems there are various message formats used from flat files (delimited files such as comma separated values or positional files) to XML or industry standard formats like EDI (Electronic Data Interchange). BizTalk provides an editor in Visual Studio to define message schemas. These schemas primarily describe an XML representation of a message because BizTalk requires XML internally for some of the other messaging functions such as transformation. Developers also have the ability to define the translation of messages from non-XML formats into the XML representation. This means a single schema can be defined that describes a flat-file message and the corresponding XML message. These schemas are used at runtime by BizTalk to automatically translate flat-file messages into XML messages and vice versa. EDI messages are handled in the much the same way with the schema providing information about the EDI validation and structure as well as the XML representation of those messages.
Message Structure -> Message Transformations (Maps)
Two applications might both have a notion of a customer entity but each likely has a different representation (schema) of that entity. Message transformations, or maps, allow you to define the transformation of data from one message schema to another. Maps are defined using a mapping tool in Visual Studio but the underlying technology used at runtime is XSLT (Extensible Stylesheet Language Transformations). This is one reason BizTalk uses XML internally. However, the use of XSLT does not preclude mapping between different formats. Using the example from before of the CRM and accounting systems, the CRM system may have contact information for a customer in a CSV file that includes separate first and last name fields while the accounting system may have a single field for customer name in an XML document. The map would get applied after the CSV file had been translated into an XML document and be responsible for concatenating the first and last name from the CRM message into a single field for the accounting application.
Transport -> Adapters
BizTalk Server uses the notion of adapters to communicate with systems to send and receive messages. Adapters can be transport adapters such as HTTP, SMTP, FTP, and MSMQ/MQ; or application adapters such as SAP, Siebel, Oracle E-Business, and SQL Server. BizTalk ships with over 80 adapters included and there are several third party component vendors that provide additional adapters. Each adapter has the logic about how to send messages over a particular transport or use the appropriate API from a LOB application to send messages that can be processed by the target system or to receive messages from the system.
Security -> Pipelines, Adapters, and Single Sign On
Security in integration often falls into one of two categories: encryption/signing or authentication. Authentication refers to presenting credentials to an application in order to be allowed to send or receive messages. The adapters used to connect to different applications allow you to specify the credentials used during configuration and most can also accept dynamic credentials at runtime if needed. Enterprise Single Sign On is a component installed as part of BizTalk server that enables secure storage of credentials to be used by adapters. This system enables a user to securely submit a message to BizTalk and have it map that user’s identity to either a common / group identity or user-specific set of credentials for the target application. For example, a user could authenticate to BizTalk with their Windows credentials and then have an adapter automatically look up a different username and password for the HR system at later point in the message interaction to authenticate with when connecting.
Each message received by or sent from BizTalk passes through a pipeline. A pipeline is composed of a set of components that process the message in some way. A common use of pipeline components is to handle the encryption or signing of messages. For example when receiving messages a pipeline might be used to decrypt the message before translating and transforming it. When sending a message the last step in a pipeline often involves signing and or encrypting the message.
Using these components together developers can create messaging solutions based on a publish-subscribe architecture. Messages are received by BizTalk using an adapter and pipeline then published into what is known as the Message Box. Subscribers to the message then receive a copy to process and generally send the message out through a pipeline / adapter pair. This architecture lends itself well to scaling the processing as well as allowing messages to be sent to multiple subscribers. Subscriptions can be based on the content of a message or the context in which it was received.
BizTalk in Azure
BizTalk Server has traditionally been installed on premises in a company’s data center. With the release of BizTalk Server 2013 Microsoft supports installing BizTalk Server in Azure virtual machines to be run in a Microsoft data center. This model of using cloud hosted virtual machines for computing is referred to as Infrastructure as a Service (IaaS).
In addition, Microsoft now offers a new model for building BizTalk messaging applications known as BizTalk Services. BizTalk Services is a Platform as a Service (PaaS) offering which means there are no virtual machines to manage. Because this offering is also intended to address many of the same considerations as BizTalk Server, the capabilities and tools are similar.
BizTalk in Azure using IaaS
Running BizTalk in Azure looks very similar to running BizTalk in your own data center, especially if you are using virtualization. For a typical BizTalk installation in Azure a developer needs to setup SQL Server virtual machines in a high availability configuration, at least two BizTalk Server virtual machines, and at least one Active Directory Domain Controller. A typical BizTalk configuration for on premises or in Azure is shown in Figure 1.
Running your solution in Azure does require some planning related to the configuration of your environment and consideration of developer concerns such as application messaging and deployment.
While you can create BizTalk environment in Windows Azure it is likely that your solution will need to communicate with servers and resources in your data center. BizTalk is an integration product and therefore primarily facilitates communication between applications. Moving or creating your BizTalk environment in Azure does not mean your BizTalk solutions will be working in isolation. Your BizTalk environment will likely need to be able to communicate with line of business (LOB) applications in your data center as well as partners’ applications and services. This communication may originate from BizTalk or it may be initiated from the applications the BizTalk solution is intended to integrate. In order to manage this communication you will have to consider both network connectivity and security in addition to the considerations related to using Virtual Machines in Azure.
Using the management portal, or other Azure management tools, you can very easily provision virtual machines pre-installed with SQL Server, BizTalk, or plain Windows machines. As Figure 2 shows, when you provision these virtual machines you specify sizes and other relevant configuration details. Azure also enables you to define virtual networks into which you can place your virtual machines. These virtual networks provide a private IP address space that your virtual machines can share to be able to easily address each other making connections between Active Directory, SQL, MS DTC and services built with local network connectivity in mind much easier to manage. Without virtual networks, communication between virtual machines is challenging because of the networking limitations in place for security considerations.
Azure Virtual Network provides a VPN connection between your virtual network in Azure and an on premises network. VPN connections can either be point-to-site where a single server in your data center has a secure connection to your Azure network or site-to-site where you setup a secure connection between the two networks. A point-to-site connection is as easy to setup as connecting your laptop to a VPN which is essentially what you are doing in this scenario. If you just have one or several servers that you need to connect to your Azure network, this is the simplest solution. For a site-to-site connection you need a supported external VPN device and IPv4 address already configured. You can then run a script to setup a connection between the two VPN networks allowing multiple machines in your environment to be connected to the Azure network without having to install or configure any software on the individual servers. Virtual networks enable access to Azure resources such as Virtual Machines and Cloud Services.
Azure Express Route goes beyond the virtual network solutions and provides a dedicated secure connection between your network in Azure and your on premises network. It is supported by many different network providers and provides more reliability, security (nothing goes over the public internet), faster speeds and lower latency than virtual networks.
When considering these options, virtual networks are generally recommended for development and testing scenarios while express route is recommended for production deployments, especially useful for BizTalk solutions when you use both Azure and on premises servers for backup and recovery of your SQL databases. This comes down to the improved reliability and throughput of those networks. You should evaluate the network requirements for your solution and the pricing of each of these options to see what meets your needs.
There are different options for high availability (HA) and disaster recovery (DR) when running SQL Server in Azure virtual machines. Newer versions of SQL Server even support backing up directly to Azure Blob Storage. Review your options for HA and DR solutions using the supported Azure models for SQL.
You also need an Active Directory domain controller in your Azure BizTalk environment. When planning you will have to consider whether this will be a standalone domain in Azure, if you have a trust relationship between this domain and your on premises domain, or if you setup a read only domain controller in Azure that is part of your domain. Generally speaking, if your BizTalk solution will be accessing on premises resources, setting up the read-only domain controller provides the best performance in Azure, least work to connect to your resources, and the lowest cost for network traffic between the domain controllers.
Developing BizTalk solutions for deployment in Azure is in most ways similar to developing solutions for an on premises BizTalk environment. You will use the same tools to develop your solution artifacts and in many cases you can use the same tools to deploy your application. The latter might depend on the network setup choices discussed previously.
One of the primary considerations for development is connecting to the applications and services in your solution. As discussed previously, you can use network level configuration and Active Directory where appropriate to connect to your own applications. Another option for enabling connectivity between on premises applications, your BizTalk environment and partners’ services is to use Azure Service Bus.
Service bus allows you to send messages through an internet relay to which both applications connect enabling secure traversal of firewalls and network address translators. Additionally, Service Bus provides brokered messaging, much like BizTalk itself, using queues and topics. This brokered messaging model supports occasionally connected clients and a publish / subscribe model of communication. Using Service Bus along with your BizTalk deployment in Azure can greatly reduce the complexity of connecting BizTalk to other applications over the internet.
BizTalk in Azure using PaaS
One of the promises of Azure, or actually the cloud computing model in general, is a reduction in the management of resources. Using the IaaS model means you still have virtual machines to back up, patch, and otherwise manage. PaaS is about removing even more management concerns from your solutions. BizTalk Services is a PaaS offering that provides messaging capabilities similar to BizTalk Server.
Like BizTalk Server, BizTalk Services provides messaging based integration relying on message schemas, message transformations (maps), pipeline processing and adapter connections. Unlike BizTalk Server, however, there is no server management or setup. To get started with BizTalk Services you provision the service and choose a particular edition and number of units. This choice determines the scale characteristics of your service and the number of solution artifacts you can deploy. Check the Azure website for information about the different editions and their capabilities.
Messaging solutions in BizTalk Services are created using what is known as a “bridge”. A bridge is defined with a message itinerary that describes the end to end processing of a message from the time it is received until it is sent out of the system. As an example, Figure 3 shows a simple message processing itinerary that receives a message from a Service Bus Queue, processes the message and then routes the message to one of two destinations based on contextual properties in the message.
This bridge configures the receiving and sending configuration for messages much like the adapters described in BizTalk Server. The “XmlBridge” step can be expanded as shown in Figure 4 and includes all of the typical pipeline and transform logic commonly known as Validate, Enrich and Transform. The bridge starts with a stage to define the message types that will be processed. Despite being called an “XML” bridge this type of bridge can handle both XML and flat file documents. The “Decode” and “Encode” stages of the process enable receiving and/or sending flat files using flat file schemas. The “Validate” stage validates documents against a specific XSD schema.
The transform stage can be configured with several maps and determines which map to use at runtime based on the incoming message type. Finally, the “Enrich” stages allow for adding or copying properties. An example of enrichment might be moving a SOAP header into the message context or adding values to a BrokeredMessage property before sending the message to the Azure Service Bus.
This example uses a one-way bridge but there is also a request-reply XML bridge that can be used when interacting with web services that provide responses. Additionally a pass through bridge can be used which only provides the “Enrich” stage for message processing which can be useful when processing non-XML messages such as images or documents.
After you develop your bridges in Visual Studio along with your maps, schemas, and any custom code you then deploy the package to your BizTalk Service. Once you’ve deployed your bridge you can send messages via HTTP if your bridge is setup for that protocol or the bridge will start monitoring queues, FTP servers and other sources you specified.
In addition to messaging with XML and flat files BizTalk Services also support both X12 and EDIFACT EDI processing. Just as in BizTalk Server these interchanges depend on the configuration of partners, profiles and agreements that define the EDI processing and the deployment of EDI schemas for the transaction sets your environment supports. Configuration of the partners and agreements all happens through a web interface but provides most of the same settings you would configure in a BizTalk Server environment. For example, Figure 5 shows an X12 agreement being defined between two parties. Once the basic details are provided the system allows you to specify the specific schemas being used, authorization data, and other EDI processing settings.
In BizTalk Server EDI processing follows the same steps as normal processing using pipelines, adapters and maps so it is only natural that in BizTalk Services the EDI processing leverages bridges. After you have configured an agreement for two parties you then define bridges for that agreement, one for each direction of traffic flow between partners. Unlike the messaging bridges discussed earlier, EDI bridges are defined and configured in the web portal.
Each EDI bridge defines the receive protocol (adapter), transform, and routing logic. Messages can be received over HTTP, FTP, or SFTP and can be sent to the Azure Service Bus (relay or brokered messaging), Azure Blob Storage or to another BizTalk Service bridge. By being able to chain your EDI Bridge to another bridge you can receive EDI, XML, and flat file messages across the same processing logic. Additionally, suspended messages can be routed to any of the same destination types enabling rich error handling workflows.
Where are we?
BizTalk in Azure provides many of the same messaging, routing, and EDI capabilities that are found in BizTalk Server, but with benefits offered by the cloud. Whether you choose the IaaS or PaaS model you will be able to quickly provision the environment you need and start developing your solution. In future articles in this series you will have the opportunity to learn more about each of these models for building solutions and connecting to your on premises resources as you move application logic to the cloud.