In a previous article I discussed running BizTalk in Azure virtual machines or Infrastructure as a Service (IaaS). While that model provides many great benefits of the cloud including cost model, ease of provisioning and scalability, it still requires the installation and management of SQL Server and BizTalk Server. For some solutions that model may work best. For other solutions the lower the barrier to entry the better. For example if your solution is to integrate 20 different partners sending you orders electronically, you would like to spend the least amount of time setting up your environment and focus on receiving, transforming or normalizing those orders and getting them into your order system. That is one example where using the Platform as a Service (PaaS) BizTalk Services might be a better fit for your solution.
BizTalk in Azure PaaS
PaaS solutions are about providing a platform on which your application can run allowing you to focus on the application logic rather than the infrastructure needed. For BizTalk Services this means being able to provision a service that can handle message-based application integration and quickly getting to work on the message processing logic required for your solution.
When you provision a new BizTalk Service in Azure you do not have to worry about number of virtual machines or sizing the SQL Server. Your focus again is on your application: What resources will my application require? As such, the provisioning is based on editions (Table 1) and units. Editions range from Free up to Enterprise while the units provide a scaling metric.
Table 1: BizTalk Services Editions
|Level||Hybrid Connections||Bridges||Agreements||HA Capable||Adapter Connections|
|Free||5 / 5 GB||None||None||No||None|
|Developer||5 / 5 GB||25||10||No||1|
|Basic||10 / 10 GB||25||50||Yes||2|
|Standard||50 / 250 GB||125||250||Yes||5|
|Premium||100 / 500 GB||500||1,000||Yes||25|
The Free edition is provided as an option to use Hybrid Connections without having to setup an entire BizTalk Service and without charge. Hybrid Connections enable your Azure Website or Mobile Service to connect to on premises resources such as SQL Server or HTTP endpoints. We will discuss Hybrid Connections more in another article but this Free edition enables you to try them out in a limited capacity.
Bridges are the deployed definition of connections between messaging endpoints in your solution and will be discussed in this article. Agreements are defined in EDI solutions to define the relationship between two parties transacting business and the Adapter Connections are for use with the LOB adapter service. Both of these resources will be discussed in future articles that focus on those particular features of BizTalk Services.
Your solution may not require all of the resources provided in a given edition. For example, if you are not doing any EDI processing then the number of agreements will not matter to your solution. For a given edition you can scale your service to a number of units. So a Basic edition scaled to two units would provide 20/20 GB for hybrid connections, 50 bridges, 100 agreements and 4 adapter connections.
When you provision the service you will have to select an edition so make sure you have determined the right edition for you. Scale units can be changed as needed on an active BizTalk Service allowing you to manage changes in demand or growth over time.
When you provision anything other than a free edition you will also be asked to configure several additional parameters for supporting services in Azure. You will need to provide an Azure Storage account where configuration can be stored and an Azure SQL Database where tracking data can be stored. These can be existing items or you can create them specifically to handle your BizTalk Service depending on your preferences and requirements.
In this particular article the focus is on bridges as the basic building block of PaaS BizTalk solutions. Future articles will expand into 1) Hybrid Connections and Adapter Connections for connecting your on premises resources, 2) Agreements and EDI bridges for use in PaaS EDI solutions, and 3) Tracking and monitoring.
Message-based application integration is about connecting systems to one another so the name “Bridge” becomes a very apt identifier for the definition of the connections in your BizTalk Services solution. In a BizTalk Services solution you define a message itinerary and connect sources and destinations using a Bridge. The itinerary defines the receiving of messages from a variety of sources, the validation, enrichment and transformation of those messages and the potential destinations for those messages based on subscriptions or filters.
Included in the Azure BizTalk Services SDK is the XML One-Way Bridge, XML Request-Reply Bridge and Pass-through Bridge. Despite the name, the XML one way bridge can process XML messages or flat file messages such as comma delimited files or positional files. Currently the request-reply bridge cannot process flat files.
Itineraries start with a message being received from one of the supported sources (Table 2). Some sources are polling based such as FTP while others such as HTTP use a push model. Once the messages are received they can be processed through the various stages of a bridge.
Table 2: BizTalk Services Sources
|FTP(S) / SFTP||BizTalk Services polls a remote FTP server for files.|
|HTTP(S)||Messages are posted to the HTTP address of the bridge.|
|Service Bus Queue||BizTalk Services reads messages from a Service Bus Queue.|
|Service Bus Topic (Subscription)||BizTalk Services receives messages matching defined filters on a Service Bus Topic.|
Once a message has been processed by a bridge it can delivered to a number of target destinations (Table 3). The bridge can send messages to multiple destinations and the choice of which destinations to use can be made at runtime through a set of filters.
Table 3: BizTalk Services Destinations
|FTP(S) / SFTP||Deliver a message to a remote folder on a server via FTP.|
|SOAP (one/two way)||Invoke either a one-way or request-response web service.|
|Service Bus Relay (one/two way)||Relay a one-way or request-response web service call through the Service Bus Relay Service.|
|Service Bus Queue or Topic||Send a message to a Service Bus Queue or Topic.|
|Azure Blob Storage||Write the contents of the message into Azure Blog Storage.|
Message Exchange Patterns
When creating your messaging solutions you will likely find a need for both one-way message exchanges and two-way, or request-reply exchanges to connect applications. These two message exchange patterns are supported by the use of one-way or request-reply bridges in your message itinerary definition.
An example of a one-way message exchange would be receiving messages from an FTP source, processing the message through a bridge and then sending the message to one or more destinations such as FTP, Azure Blob Storage or a Service Bus Queue / Topic. In these exchanges the destination does not provide a response message. One-way messaging is useful for moving data from one application or system to another. If no validation or transformation is needed, a pass-through bridge can be used which only provides an enrich stage for enriching the message metadata.
In a request-reply scenario you might receive messages over HTTPS, process them through a bridge, then send the messages to a SOAP or Service Bus Relay destination. In this scenario the destination is expected to return a response message as the reply and that message will be go back to the original caller. One key feature of the request-reply scenario is that the bridge created will be a request-reply bridge. This essentially encapsulates two separate one-way bridges providing processing not only for the initial request message, but also for the response message. This means the response can be validated, enriched, and transformed before being returned to the caller.
Azure BizTalk Services solutions are built in Visual Studio 2012 using the Azure BizTalk Services SDK. After installing the SDK you can create a new project using the BizTalk Service template. There is also a BizTalk Service Artifacts project type but that is used for EDI solutions which I will cover in a later article.
When you create a new project you will get a default message itinerary. You can add schemas to the project that define your message types and transforms to map between those types to the project as needed. If you already have XML schema that define your messages you can add those to the project and use them directly. If you want to work with flat files I suggest using the Generate Flat File Schema item template and using the wizard to define the schema for your flat file. This greatly simplifies the process of creating schemas for delimited or positional files. The schema editor installed by the SDK is almost exactly like the one used in BizTalk Server without the notion of context properties and distinguished fields, so it will be very familiar if you have worked with BizTalk Server at all.
The mapping tool in Visual Studio (Figure 1) is based on the same tool in BizTalk Server but with several updates to make it easier to map repeating items, maintain scope, work with lists, and interact with context properties. Using this tool you can define the transformation of a message from one structure (schema) to another applying logic such as string manipulation, arithmetic, numeric and date formatting, and cumulative operations such as averaging or counting items. For the simple cases you can just connect items to copy values from one message structure to the other.
With schemas and transformations defined you can start editing the message itinerary in your solution to define the sources, bridges (using your message types and transformations), and destinations using the itinerary designer provided by the SDK (Figure 2). Each component has properties that can be configured but unlike some other designers, you have to use the Connector item from the toolbox to connect each item to the subsequent items in the itinerary because the connections have specific properties.
While this model of being able to design the entire messaging interaction is an improvement over the model in BizTalk Server, it does have the drawback that connection information is contained in your development artifacts. To move from a development environment to production requires changing the connections in your development environment and deploying to the production environment.
Each bridge is composed of various stages of message processing logic connected together in sequence. These stages perform common tasks in message exchange scenarios such as enriching messages with metadata for routing, validating messages against a schema, and transforming messages from one structure (schema) to another.
In a one-way bridge the first stages are focused getting the incoming message processed and ready for the rest of the stages. This includes decoding the message if it is a flat file and validating the message against a schema (Figure 3).
The Message Types stage provides the information needed by the Decode and Validate stages. In order to parse a flat file into XML or to validate a message structure the components need to know which schema or set of schemas to use. You provide that information by specifying the schemas for the messages expected to be received by this bridge. If multiple message types can be received then you can select all the relevant types. The type information available to you is determined by the schemas in your project. The Decode and Validate components do not have any configuration other than allowing you to specify if validation warnings should be treated as errors thus stopping message processing.
The remaining stages (shown in Figure 4) are about preparing the message to be sent including two Enrich stages, a Transform stage and an Encode stage. The Transform stage is where maps created in Visual Studio can be applied to the message transforming it into another message type. The Encode stage will serialize the message to a flat file format if the message type coming out of the map is defined by a flat file schema.
Enrichment is about populating messaging properties or metadata about the message. In the Enrich stage you configure a set of extractions that pull data from the incoming message and write them into a set of named and typed properties. The values for these properties can come from a variety of sources as shown in Table 4. The first Enrich stage in the bridge is for populating context properties that may be used in the Transform stage. Maps can use a function to extract property values and use them in the transformation to populate the destination message. This is a feature that has been requested by BizTalk developers for years!
Table 4: Enrichment Property Source Types
|SOAP||Extract SOAP header values from the incoming message.|
|HTTP||Extract HTTP header values from the incoming message.|
|Brokered||Extract properties from Service Bus Brokered Messages|
|XPath||Use XPath to extract a value from the message body.|
|Lookup||Use a SQL query to lookup a value from a database.|
|FTP/SFTP||Extract FTP location information from the incoming message.|
|System||Extract identifiers or timestamps created by the system.|
The second Enrich stage is for populating context properties used in routing the messages to destinations. The connector to each destination in the itinerary provides for the configuration of a filter to determine if messages should “follow” that connector and a particular message be delivered to the destination. These filters are logical expressions that can use the context properties. For example I might want orders with a total value over some threshold to go to a particular destination.
Each of these enrichment stages are about populating properties used strictly within the service for message manipulation or routing. There is another concept known as a Route Action that essentially provides the same functionality in reverse – you can use the available message properties or an expression to set destination-specific metadata properties. The Route Action is set on the connector between the bridge and a particular destination. For example with a Service Bus Topic destination the route action can be used to populate Brokered Message properties such as Label or Content-Type whereas for an Azure Blob destination the CACHE-CONTROL or BLOB-NAME properties might be set using these actions.
Route actions often work in tandem with the enrichment stages. Values from the incoming messages are extracted into message properties in the enrichment stages and then used to set destination properties in a route action. In this way an incoming filename or header value can be used to dynamically configure the destination. For example the filename from the FTP source might be used in the blob name on the destination Azure blob.
Deploying a solution
Once you have your itinerary and bridges ready, including configuration of the sources and destinations, you can deploy your solution to your provisioned BizTalk Service. You do this right from Visual Studio and then have options to manage various aspects of the solution using PowerShell cmdlets. Unfortunately, any changes to the configuration require you to deploy your solution again and refresh the system to pick up the changes. According to Microsoft this can take up to five minutes to refresh, which means somewhere host processes are probably restarting to pick up the new assemblies.
Other than deploying your artifacts, one of the key considerations is the state of your sources. In the itinerary designer you can specify the initial status of your sources in the property panel. This allows you to deploy the solution without having it start immediately. For example you might not want the FTP source to begin polling right away so you would set its initial status to “Stop” and then use PowerShell or the REST API to start the source when you are ready to begin polling. This functionality is useful at other time as well such as when you need to stop polling a source during maintenance windows.
What Can I Do With It?
With an understanding of what itineraries and bridges are and how they work, it is time to think about how to use them in a solution. Below are a couple of example solutions that show how the various sources, destinations, and message processing capabilities can be used to solve real business problems. This is only a sampling however as there are numerous ways to fit these pieces together in your solutions.
In addition to the examples below, EDI processing adds another layer of solutions options on the bridge framework. The addition of partners and agreements along with support for X12 and EDIFACT document schemas enables the bridges already described to be extended to more formalized message interchanges. I will cover the EDI capabilities of BizTalk Services in a separate article.
Solution: Bridge internet messages to on premises services
In this solution your company has an internal web service and needs to receive messages from partner organizations. The partner organizations cannot directly access the web service. The partner organizations may not have the capabilities to support all of the web service standards you have chosen to implement. The partner organization may only be able to post XML data over HTTP.
To complete this solution requires use of the Service Bus Relay to make your service accessible through the firewalls in your organization. However, the relay alone will not address all of your needs including message transformation and enrichment.
The solution involves allowing partners to post XML data to an XML bridge over HTTPS, transforming and enriching the data and then sending the message to a Service Bus Relay to be relayed to your on premises web service.
An HTTP source allows clients to post XML data to the endpoint for the Bridge (determined by the bridge relative address in the configuration in the itinerary). Then an XML Request-Reply Bridge handles two primary responsibilities: transformation and enrichment. A map is used to transform the incoming XML to the SOAP request body and the enrichment stages are used to create properties that can then be set on the Service Bus Relay destination, such as the SOAP action, which are required to make the web service call succeed.
Because this is a request-reply exchange, maps and enrichment can also be performed on the response message from the relay. Note that there is no HTTP source specifically defined in the itinerary as HTTP as the bridge can always be activated with HTTP based on the address specified on the bridge itself.
Solution: Bridge files to internet messages
In this solution your company has positional flat files that are produced by a main frame application. These files are currently written to a file/ftp server by the application. You need to be able to pick up these files and deliver them to a Service Bus Topic where subscribing partners and applications can receive them. The partners are expecting the data to be in XML format following an agreed upon schema.
This solution involves receiving the flat file message from an FTP server, parsing the file into XML and then transforming the message to the agreed upon XML structure. Finally, the message is delivered to a Service Bus Topic where partners can subscribe for applicable messages.
The itinerary uses the FTP source to poll for the flat file messages and has a flat file schema defined and registered as a Message Type in the XML One Way Bridge. The bridge is used to transform the message from the structure of the flat file schema to the agreed upon XML schema. In the enrich stages of the bridge data is read from the message, including using XPath queries to create aggregate values such as sums or counts, and written to the message context. These properties are then written to Brokered Message properties in the Route Action on the connection to the Service Bus Topic destination.
This solution takes advantage of the Publish / Subscribe model supported by Service Bus Topics but adds in the support for pulling files from the FTP server and transforming the flat file format to an XML format that is more easily consumed by partners. This type of bridging between the worlds of main frame and on premises computing with cloud solutions like Service Bus Topics is where BizTalk Services really shines.
Line of Business Systems
In addition to the destinations discussed so far, Line of Business (LOB) applications including Siebel, SAP, Oracle (DB and EBS), and SQL Server can be configured as destinations using the BizTalk Services Adapter Service. This service uses the Service Bus Relay to connect your cloud messaging solution to your on premises LOB system. Using this service you can expose operations on a system to a request-reply bridge and therefore sources. For example a client could POST XML to an HTTP endpoint on your BizTalk Service and you can bridge that request to a query or update in your SQL Server system returning the result to the client; all in a secure message-based solution. I will discuss the adapter service in detail in a future article.
Where are we?
In this article we covered the core messaging capabilities in the BizTalk Services PaaS offering. In future articles we will go into further detail on the EDI capabilities that build on these bridges as well as connectivity to on premises resources with Hybrid Connections and the LOB Adapter Service which are both part of BizTalk Services. In addition we will see how to use tracking to monitor your solution and collect relevant data.