Azure BizTalk Services: BizTalk in Azure Platform as a Service

Matt Milner continues his series on Azure BizTalk Services by covering the Platform as a Service (PaaS) possibilities in Azure.

2187-just_azure.svg

In a previous article I discussed running BizTalk in Azure virtual machines or Infrastructure as a Service (IaaS). While that model provides many great benefits of the cloud including cost model, ease of provisioning and scalability, it still requires the installation and management of SQL Server and BizTalk Server. For some solutions that model may work best. For other solutions the lower the barrier to entry the better. For example if your solution is to integrate 20 different partners sending you orders electronically, you would like to spend the least amount of time setting up your environment and focus on receiving, transforming or normalizing those orders and getting them into your order system. That is one example where using the Platform as a Service (PaaS) BizTalk Services might be a better fit for your solution.

BizTalk in Azure PaaS

PaaS solutions are about providing a platform on which your application can run allowing you to focus on the application logic rather than the infrastructure needed. For BizTalk Services this means being able to provision a service that can handle message-based application integration and quickly getting to work on the message processing logic required for your solution.

When you provision a new BizTalk Service in Azure you do not have to worry about number of virtual machines or sizing the SQL Server. Your focus again is on your application: What resources will my application require? As such, the provisioning is based on editions (Table 1) and units. Editions range from Free up to Enterprise while the units provide a scaling metric.

Table 1: BizTalk Services Editions

Level Hybrid Connections Bridges Agreements HA Capable Adapter Connections
Free 5 / 5 GB None None No None
Developer 5 / 5 GB 25 10 No 1
Basic 10 / 10 GB 25 50 Yes 2
Standard 50 / 250 GB 125 250 Yes 5
Premium 100 / 500 GB 500 1,000 Yes 25

The Free edition is provided as an option to use Hybrid Connections without having to setup an entire BizTalk Service and without charge. Hybrid Connections enable your Azure Website or Mobile Service to connect to on premises resources such as SQL Server or HTTP endpoints. We will discuss Hybrid Connections more in another article but this Free edition enables you to try them out in a limited capacity.

Bridges are the deployed definition of connections between messaging endpoints in your solution and will be discussed in this article. Agreements are defined in EDI solutions to define the relationship between two parties transacting business and the Adapter Connections are for use with the LOB adapter service. Both of these resources will be discussed in future articles that focus on those particular features of BizTalk Services.

Your solution may not require all of the resources provided in a given edition. For example, if you are not doing any EDI processing then the number of agreements will not matter to your solution. For a given edition you can scale your service to a number of units. So a Basic edition scaled to two units would provide 20/20 GB for hybrid connections, 50 bridges, 100 agreements and 4 adapter connections.

When you provision the service you will have to select an edition so make sure you have determined the right edition for you. Scale units can be changed as needed on an active BizTalk Service allowing you to manage changes in demand or growth over time.

When you provision anything other than a free edition you will also be asked to configure several additional parameters for supporting services in Azure. You will need to provide an Azure Storage account where configuration can be stored and an Azure SQL Database where tracking data can be stored. These can be existing items or you can create them specifically to handle your BizTalk Service depending on your preferences and requirements.

In this particular article the focus is on bridges as the basic building block of PaaS BizTalk solutions. Future articles will expand into 1) Hybrid Connections and Adapter Connections for connecting your on premises resources, 2) Agreements and EDI bridges for use in PaaS EDI solutions, and 3) Tracking and monitoring.

Bridges

Message-based application integration is about connecting systems to one another so the name “Bridge” becomes a very apt identifier for the definition of the connections in your BizTalk Services solution. In a BizTalk Services solution you define a message itinerary and connect sources and destinations using a Bridge. The itinerary defines the receiving of messages from a variety of sources, the validation, enrichment and transformation of those messages and the potential destinations for those messages based on subscriptions or filters.

Included in the Azure BizTalk Services SDK is the XML One-Way Bridge, XML Request-Reply Bridge and Pass-through Bridge. Despite the name, the XML one way bridge can process XML messages or flat file messages such as comma delimited files or positional files. Currently the request-reply bridge cannot process flat files.

Sources

Itineraries start with a message being received from one of the supported sources (Table 2). Some sources are polling based such as FTP while others such as HTTP use a push model. Once the messages are received they can be processed through the various stages of a bridge.

Table 2: BizTalk Services Sources

Source Description
FTP(S) / SFTP BizTalk Services polls a remote FTP server for files.
HTTP(S) Messages are posted to the HTTP address of the bridge.
Service Bus Queue BizTalk Services reads messages from a Service Bus Queue.
Service Bus Topic (Subscription) BizTalk Services receives messages matching defined filters on a Service Bus Topic.

Destinations

Once a message has been processed by a bridge it can delivered to a number of target destinations (Table 3). The bridge can send messages to multiple destinations and the choice of which destinations to use can be made at runtime through a set of filters.

Table 3: BizTalk Services Destinations

Destination Description
FTP(S) / SFTP Deliver a message to a remote folder on a server via FTP.
SOAP (one/two way) Invoke either a one-way or request-response web service.
Service Bus Relay (one/two way) Relay a one-way or request-response web service call through the Service Bus Relay Service.
Service Bus Queue or Topic Send a message to a Service Bus Queue or Topic.
Azure Blob Storage Write the contents of the message into Azure Blog Storage.

Message Exchange Patterns

When creating your messaging solutions you will likely find a need for both one-way message exchanges and two-way, or request-reply exchanges to connect applications. These two message exchange patterns are supported by the use of one-way or request-reply bridges in your message itinerary definition.

An example of a one-way message exchange would be receiving messages from an FTP source, processing the message through a bridge and then sending the message to one or more destinations such as FTP, Azure Blob Storage or a Service Bus Queue / Topic. In these exchanges the destination does not provide a response message. One-way messaging is useful for moving data from one application or system to another. If no validation or transformation is needed, a pass-through bridge can be used which only provides an enrich stage for enriching the message metadata.

In a request-reply scenario you might receive messages over HTTPS, process them through a bridge, then send the messages to a SOAP or Service Bus Relay destination. In this scenario the destination is expected to return a response message as the reply and that message will be go back to the original caller. One key feature of the request-reply scenario is that the bridge created will be a request-reply bridge. This essentially encapsulates two separate one-way bridges providing processing not only for the initial request message, but also for the response message. This means the response can be validated, enriched, and transformed before being returned to the caller.

Developing Solutions

Azure BizTalk Services solutions are built in Visual Studio 2012 using the Azure BizTalk Services SDK. After installing the SDK you can create a new project using the BizTalk Service template. There is also a BizTalk Service Artifacts project type but that is used for EDI solutions which I will cover in a later article.

When you create a new project you will get a default message itinerary. You can add schemas to the project that define your message types and transforms to map between those types to the project as needed. If you already have XML schema that define your messages you can add those to the project and use them directly. If you want to work with flat files I suggest using the Generate Flat File Schema item template and using the wizard to define the schema for your flat file. This greatly simplifies the process of creating schemas for delimited or positional files. The schema editor installed by the SDK is almost exactly like the one used in BizTalk Server without the notion of context properties and distinguished fields, so it will be very familiar if you have worked with BizTalk Server at all.

The mapping tool in Visual Studio (Figure 1) is based on the same tool in BizTalk Server but with several updates to make it easier to map repeating items, maintain scope, work with lists, and interact with context properties. Using this tool you can define the transformation of a message from one structure (schema) to another applying logic such as string manipulation, arithmetic, numeric and date formatting, and cumulative operations such as averaging or counting items. For the simple cases you can just connect items to copy values from one message structure to the other.

2187-BizTalkServices-04-Fig-1_Transform-

Figure 1: BizTalk Services Mapping Tool

With schemas and transformations defined you can start editing the message itinerary in your solution to define the sources, bridges (using your message types and transformations), and destinations using the itinerary designer provided by the SDK (Figure 2). Each component has properties that can be configured but unlike some other designers, you have to use the Connector item from the toolbox to connect each item to the subsequent items in the itinerary because the connections have specific properties.

2187-BizTalkServices-04-Fig-2_ItineraryD

Figure 2: BizTalk Services Itinerary Designer

While this model of being able to design the entire messaging interaction is an improvement over the model in BizTalk Server, it does have the drawback that connection information is contained in your development artifacts. To move from a development environment to production requires changing the connections in your development environment and deploying to the production environment.

Bridge Stages

Each bridge is composed of various stages of message processing logic connected together in sequence. These stages perform common tasks in message exchange scenarios such as enriching messages with metadata for routing, validating messages against a schema, and transforming messages from one structure (schema) to another.

In a one-way bridge the first stages are focused getting the incoming message processed and ready for the rest of the stages. This includes decoding the message if it is a flat file and validating the message against a schema (Figure 3).

2187-BizTalkServices-04-Fig-3_OneWayFirs

Figure 3: First Stages of a One Way Bridge

The Message Types stage provides the information needed by the Decode and Validate stages. In order to parse a flat file into XML or to validate a message structure the components need to know which schema or set of schemas to use. You provide that information by specifying the schemas for the messages expected to be received by this bridge. If multiple message types can be received then you can select all the relevant types. The type information available to you is determined by the schemas in your project. The Decode and Validate components do not have any configuration other than allowing you to specify if validation warnings should be treated as errors thus stopping message processing.

The remaining stages (shown in Figure 4) are about preparing the message to be sent including two Enrich stages, a Transform stage and an Encode stage. The Transform stage is where maps created in Visual Studio can be applied to the message transforming it into another message type. The Encode stage will serialize the message to a flat file format if the message type coming out of the map is defined by a flat file schema.

2187-BizTalkServices-04-Fig-4_OneWayLast

Figure 4: Remaining Stages in XML One Way Bridge

Message Enrichment

Enrichment is about populating messaging properties or metadata about the message. In the Enrich stage you configure a set of extractions that pull data from the incoming message and write them into a set of named and typed properties. The values for these properties can come from a variety of sources as shown in Table 4. The first Enrich stage in the bridge is for populating context properties that may be used in the Transform stage. Maps can use a function to extract property values and use them in the transformation to populate the destination message. This is a feature that has been requested by BizTalk developers for years!

Table 4: Enrichment Property Source Types

Source Description
SOAP Extract SOAP header values from the incoming message.
HTTP Extract HTTP header values from the incoming message.
Brokered Extract properties from Service Bus Brokered Messages
XPath Use XPath to extract a value from the message body.
Lookup Use a SQL query to lookup a value from a database.
FTP/SFTP Extract FTP location information from the incoming message.
System Extract identifiers or timestamps created by the system.

The second Enrich stage is for populating context properties used in routing the messages to destinations. The connector to each destination in the itinerary provides for the configuration of a filter to determine if messages should “follow” that connector and a particular message be delivered to the destination. These filters are logical expressions that can use the context properties. For example I might want orders with a total value over some threshold to go to a particular destination.

Each of these enrichment stages are about populating properties used strictly within the service for message manipulation or routing. There is another concept known as a Route Action that essentially provides the same functionality in reverse – you can use the available message properties or an expression to set destination-specific metadata properties. The Route Action is set on the connector between the bridge and a particular destination. For example with a Service Bus Topic destination the route action can be used to populate Brokered Message properties such as Label or Content-Type whereas for an Azure Blob destination the CACHE-CONTROL or BLOB-NAME properties might be set using these actions.

Route actions often work in tandem with the enrichment stages. Values from the incoming messages are extracted into message properties in the enrichment stages and then used to set destination properties in a route action. In this way an incoming filename or header value can be used to dynamically configure the destination. For example the filename from the FTP source might be used in the blob name on the destination Azure blob.

Deploying a solution

Once you have your itinerary and bridges ready, including configuration of the sources and destinations, you can deploy your solution to your provisioned BizTalk Service. You do this right from Visual Studio and then have options to manage various aspects of the solution using PowerShell cmdlets. Unfortunately, any changes to the configuration require you to deploy your solution again and refresh the system to pick up the changes. According to Microsoft this can take up to five minutes to refresh, which means somewhere host processes are probably restarting to pick up the new assemblies.

Other than deploying your artifacts, one of the key considerations is the state of your sources. In the itinerary designer you can specify the initial status of your sources in the property panel. This allows you to deploy the solution without having it start immediately. For example you might not want the FTP source to begin polling right away so you would set its initial status to “Stop” and then use PowerShell or the REST API to start the source when you are ready to begin polling. This functionality is useful at other time as well such as when you need to stop polling a source during maintenance windows.

What Can I Do With It?

With an understanding of what itineraries and bridges are and how they work, it is time to think about how to use them in a solution. Below are a couple of example solutions that show how the various sources, destinations, and message processing capabilities can be used to solve real business problems. This is only a sampling however as there are numerous ways to fit these pieces together in your solutions.

In addition to the examples below, EDI processing adds another layer of solutions options on the bridge framework. The addition of partners and agreements along with support for X12 and EDIFACT document schemas enables the bridges already described to be extended to more formalized message interchanges. I will cover the EDI capabilities of BizTalk Services in a separate article.

Solution: Bridge internet messages to on premises services

In this solution your company has an internal web service and needs to receive messages from partner organizations. The partner organizations cannot directly access the web service. The partner organizations may not have the capabilities to support all of the web service standards you have chosen to implement. The partner organization may only be able to post XML data over HTTP.

To complete this solution requires use of the Service Bus Relay to make your service accessible through the firewalls in your organization. However, the relay alone will not address all of your needs including message transformation and enrichment.

The solution involves allowing partners to post XML data to an XML bridge over HTTPS, transforming and enriching the data and then sending the message to a Service Bus Relay to be relayed to your on premises web service.

An HTTP source allows clients to post XML data to the endpoint for the Bridge (determined by the bridge relative address in the configuration in the itinerary). Then an XML Request-Reply Bridge handles two primary responsibilities: transformation and enrichment. A map is used to transform the incoming XML to the SOAP request body and the enrichment stages are used to create properties that can then be set on the Service Bus Relay destination, such as the SOAP action, which are required to make the web service call succeed.

2187-BizTalkServices-04-Fig-5_HTTPToRela

Figure 5: HTTP to Service Bus Relay

Because this is a request-reply exchange, maps and enrichment can also be performed on the response message from the relay. Note that there is no HTTP source specifically defined in the itinerary as HTTP as the bridge can always be activated with HTTP based on the address specified on the bridge itself.

Solution: Bridge files to internet messages

In this solution your company has positional flat files that are produced by a main frame application. These files are currently written to a file/ftp server by the application. You need to be able to pick up these files and deliver them to a Service Bus Topic where subscribing partners and applications can receive them. The partners are expecting the data to be in XML format following an agreed upon schema.

This solution involves receiving the flat file message from an FTP server, parsing the file into XML and then transforming the message to the agreed upon XML structure. Finally, the message is delivered to a Service Bus Topic where partners can subscribe for applicable messages.

The itinerary uses the FTP source to poll for the flat file messages and has a flat file schema defined and registered as a Message Type in the XML One Way Bridge. The bridge is used to transform the message from the structure of the flat file schema to the agreed upon XML schema. In the enrich stages of the bridge data is read from the message, including using XPath queries to create aggregate values such as sums or counts, and written to the message context. These properties are then written to Brokered Message properties in the Route Action on the connection to the Service Bus Topic destination.

2187-BizTalkServices-04-Fig-6_FlatToTopi

Figure 6: Flat File from FTP to Service Bus Topic

This solution takes advantage of the Publish / Subscribe model supported by Service Bus Topics but adds in the support for pulling files from the FTP server and transforming the flat file format to an XML format that is more easily consumed by partners. This type of bridging between the worlds of main frame and on premises computing with cloud solutions like Service Bus Topics is where BizTalk Services really shines.

Line of Business Systems

In addition to the destinations discussed so far, Line of Business (LOB) applications including Siebel, SAP, Oracle (DB and EBS), and SQL Server can be configured as destinations using the BizTalk Services Adapter Service. This service uses the Service Bus Relay to connect your cloud messaging solution to your on premises LOB system. Using this service you can expose operations on a system to a request-reply bridge and therefore sources. For example a client could POST XML to an HTTP endpoint on your BizTalk Service and you can bridge that request to a query or update in your SQL Server system returning the result to the client; all in a secure message-based solution. I will discuss the adapter service in detail in a future article.

Where are we?

In this article we covered the core messaging capabilities in the BizTalk Services PaaS offering. In future articles we will go into further detail on the EDI capabilities that build on these bridges as well as connectivity to on premises resources with Hybrid Connections and the LOB Adapter Service which are both part of BizTalk Services. In addition we will see how to use tracking to monitor your solution and collect relevant data.