Azure BizTalk Services: EDI

The Electronic Data Interchange has been around a long time. Azure BizTalk Services supports this important protocol for transmitting business transactions, and it is reasonably easy to set up, as Matt Milner explains

In a previous article I discussed the basics of the Azure BizTalk Services Platform as a Service offering. In this article I will focus on the EDI processing capabilities in that offering and how they build on both the base concepts of BizTalk Services and the EDI capabilities in BizTalk Server. Note that you can also create an EDI solution using virtual machines in an IaaS design, but as that solution is not much different from an on premises solution, I will focus exclusively on the PaaS solutions available.

EDI (electronic data interchange) has been around for many years and is used by companies to transact a great deal of business. Despite many claims of the death of EDI with the upsurge in the popularity of XML a decade or more ago, the protocol lives on and remains a large part of many businesses. EDI defines a set of messaging protocols and message schemas for various business transactions and comes in two flavors: X12 and EDIFACT. X12 tends to get used more by companies in the United States while EDIFACT is more popular in Europe and other parts of the world.

AS2 is a protocol commonly used in EDI exchanges to provide message integrity, confidentiality, and reliability. It is most commonly used over HTTP and is an additional wrapper over either EDIFACT or X12 message exchanges. Azure BizTalk Services provides support for both EDI and AS2 exchanges and the corresponding configuration and infrastructure needed.

EDI in Azure BizTalk Services

You might remember Table 1 from my previous article. For EDI solutions the Agreements column becomes important as it determines the number of agreements you can have in a particular service (an agreement is a pairing of two partners for message exchange and will be covered in detail shortly). Remember that you can also scale your service to multiple units so if you need 100 agreements you can either choose two units of the Basic edition or one unit of the Standard edition.

Level Hybrid Connections Bridges Agreements HA Capable Adapter Connections
Free 5 / 5 GB None None None
Developer 5 / 5 GB 25 10 1
Basic 10 / 10 GB 25 50 2
Standard 50 / 250 GB 125 250 5
Premium 100 / 500 GB 500 1000 25

Table 1: BizTalk Services Editions

Developing Solutions

Like the basic Azure BizTalk Service solution, some development is done in Visual Studio 2012 for an EDI solution. However, the only artifacts you develop for the EDI portion of your solution are schemas and transformations while your bridges and agreements are defined in the portal. You create the schemas and maps in the BizTalk Services Artifacts project template installed by the Azure BizTalk Services SDK and then deploy the artifacts to the service.

One of the options when downloading the SDK is to download the EDI schemas. The compressed archive is approximately 340 MB so you will likely want to expand the file onto a separate drive as you will only need to deploy the specific schemas you use in your solution. For example if you are only working with X12 purchase orders and their corresponding acknowledgements you will only need to deploy the 850 and 855 schemas instead of the hundreds of schemas included in the download. The EDI schemas can be added to your project directly without any need to modify them. That said if your requirements for a particular document type are different from the defaults (and when are they not) you can also modify the schema to alter settings such as which fields are required or to change a datatype or field validation parameters. As in BizTalk Server you can also deploy multiple versions of the same schema using different namespaces and then configure the appropriate namespace on the agreement settings. This allows you to use unique definitions of requirements for particular trading partners. I highly discourage this practice unless you absolutely cannot avoid it as it creates a lot more artifacts to manage and introduces complexity to your solution that is best avoided if possible. The reality of integration of course will dictate that this is not always possible.

Partners and Agreements

At the heart of any EDI solution in BizTalk is a lot of configuration. Because EDI message exchanges are focused on real business interactions the parties involved need to be modeled and agreements between those parties defined. If you have done any EDI work in BizTalk Server, the creation and management of partners in Azure BizTalk Services will be very familiar.

Partners are the definition of a company, department or other entity that is part of the business relationship you are modeling with Azure BizTalk Services. The partner (Figure 1) includes one or more business profiles that define the various identifiers the partner will use in a given exchange. These identifiers can be anything from a telephone number to a D-U-N-S number used to identify the organization, and are used in the documents as they are exchanged. In addition you can store basic contact information about the partner alongside the more technical details used at runtime.

2317-BizTalkEDI_Partner-dffbdeec-df07-41

Figure 1: Defining a partner in Azure BizTalk Services

Once you have partners defined you can begin defining agreements (Figure 2). An agreement combines two partners and the configuration of exchanges for one or more EDI interchanges. This includes information about which identifiers will be used, the EDI protocol (EDIFACT or X12) and all the various details of the interaction. There are far too many details of an EDI exchange to explain in an article about Azure BizTalk Services, but all the settings needed to manage envelopes, batching, counters, acknowledgements and the multitude of other settings required to make an EDI interchange succeed are available through the web interface.

2317-BizTalkEDI_Agreement-890d0963-55f7-

Figure 2: Defining an agreement in Azure BizTalk Services

Bridges

EDI message processing is handled by bridges much as the XML and flat file documents I discussed in the previous article. There are two major differences. First, EDI bridges are defined in the portal and not with a message itinerary in Visual Studio. Second, the EDI bridge will depend on the partner and agreement information at runtime to process the messages according to all the configuration defined for the given agreement.

You will also find that there are fewer options for bridge sources and destinations when working with EDI bridges. One interesting destination however includes sending messages to an Azure BizTalk Service bridge so the EDI processing can be an on-ramp to a standard message itinerary built to process XML or flat file documents. With this capability, if you are already processing XML orders for example, you could add the ability to receive EDI orders and process them in the EDI bridge and then have those orders processed through the very same bridge you had previously configured for the XML orders.

Something that can be somewhat confusing occurs when you create a new EDI bridge in the portal. When you define the bridge you configure “receive settings” and “send settings”. In reality, each of these settings becomes its own bridge and each shows up in the portal as a separate bridge. Clicking the link for either one will bring you back to the original definition dialogs for both the sending and receiving operations. In BizTalk Server terms you are setting up a receive location/port and a send port at the same time. The messages coming into the system on the defined receiving bridge do not necessarily have to exit the system on the corresponding send bridge, but they may. Also, messages can be sent directly to the sending bridge without having to process through a receiving bridge when you just want to send an EDI message.

Sources

EDI receiving bridges only support the two sources shown in Table 2 which makes some sense when you consider the systems doing EDI processing have been around for many years. Most of those systems are not going to be interacting with Service Bus Brokered Messaging or even SOAP web services. FTP and HTTP are very common protocols used to exchange EDI messages.

Table 2: BizTalk Services EDI Bridge Sources

Destinations

Once a message has been processed by an EDI bridge it can delivered to a number of target destinations (). The bridge can send messages to multiple destinations and the choice of which destinations to use can be made at runtime through a set of filters just like a standard bridge. Unlike the sources, the destinations represent both classic transports and current technologies such as the Azure Service Bus and Azure Blob Storage.

Table 3: BizTalk Services EDI Bridge Destinations

Suspended Messages and Acknowledgements

In any BizTalk solution you will experience messages that fail to be processed correctly due to some type of error. In an Azure BizTalk Service EDI solution you can specify what to do with those failed messages as there is not a management console like the one for BizTalk Server. You have the same options for suspended messages as you do for route destinations with the exclusion of FTP/SFTP. Again this makes sense when you think about how you will process suspended messages. Most likely you will want them to stay in your Azure solution so you can repair and resubmit the message for processing again.

An example of failed message processing would involve writing failed messages to a Service Bus Queue and then monitoring that queue in order to know about the failed message. Then you could have a custom application interface that allows the user to view the message and handle it. Handling it might mean contacting the sending partner to let them know they should fix and resubmit, or you can build your own resubmit pathway into your solution and allow the user to fix and resubmit the message right from your application. Of course there are many other options available, this is simply one example.

In an EDI interaction acknowledgements are a common requirement and both technical and functional acknowledgements are supported. When you create an agreement between partners you specify which types of acknowledgements should be sent. When a message is received by your Azure BizTalk Service, the acknowledgment is generated and the send settings on the agreement are used to send it back to the partner.

What can I do with Azure BizTalk Services EDI?

Azure BizTalk Services allows you to quickly setup an environment capable of handling EDI and AS2 messaging interchanges without complicated setup of a server environment. This capability enables several scenarios, of which the following are just a sampling. In addition to those shown below, you can of course create a brand new EDI / AS2 solution connecting to your existing applications on premises or in the cloud.

Accept EDI Messages into an Existing System

Your company currently has an order management system which can accept XML documents defining orders, invoices, etc. You have the chance to work with a big new customer, but they require that you support EDI and AS2 in order to do business with them.

In this scenario you can provision a new Azure BizTalk Service, setup your agreements and deploy the schemas for the message sets you will be using with the customer. Then you can create your EDI bridges to process messages between your company and the customer. The receiving EDI Bridge can receive the documents from the customer and then send them to an XML bridge where they can be mapped to your current XML format and submitted to your order management system using a Service Bus Relay for example.

Send EDI Messages from an Existing System

This scenario may be an extension of the previous scenario where you need to be able to respond to incoming EDI messages with the corresponding reply message; an order acknowledgement or invoice for an order for example. However you might also find yourself needing simply to send EDI messages to a partner organization because that is the format they can handle.

By provisioning the service as described above and creating the EDI Bridge, you will be able to send messages to the Send bridge over HTTP and then apply transforms/maps and route actions to modify properties and headers before the message is sent to the destination over the configured transport. Of course you can then receive response messages from the partner over the receive bridge.

The main difference here is that you can start an EDI send operation by using HTTP and XML messages sent from your existing application to invoke the EDI processing, as opposed to starting with EDI. Using these two bridges you can put a full EDI interchange process on top of your existing application interfaces retaining your existing logic.

Replace an Existing EDI Solution

Your company currently has a legacy system performing EDI operations but the software is no longer supported and you are constantly called on to troubleshoot issues with the processing of messages. The whole EDI system is running on a workstation computer under an employee’s desk (please don’t laugh, I’ve actually seen this scenario).

In this scenario you can provision a new Azure BizTalk Services deployment and deploy all the schemas you are going to need for your EDI transaction sets. In addition, you will likely need XML or flat file schemas that you use when interacting with the other systems in your solution and the maps between the EDI and non-EDI schemas. Define your trading partners, agreements and bridges and connect those bridges to the existing applications using FTP, HTTP, or across the Azure Service Bus using brokered messaging (queues and topics).

One of the biggest steps in migrating to a new system is managing your partners and all the related settings. Azure BizTalk Services provides a REST API based on OData that enables you to do most of the work of creating partners and agreements. You can pull data from your existing system regarding your partners and then use this API to create the corresponding entities in your Azure BizTalk Service deployment. You will need to use the PowerShell cmdlets for Azure BizTalk Services or the portal to upload any certificates, schemas, and maps. You’ll need to use the portal to finish up the process of creating the bridges and route actions.

You now have a full EDI / AS2 solution integrating your partners with your back end systems without having to provision hardware, install software or otherwise spend time on the infrastructure needed to build your solution. Better still, you can get rid of the legacy system and decommission or repurpose the workstation under that employee’s desk.

The Future of EDI in Azure – Azure App Services

In preview at the time of this writing, Azure App Services includes, among other things, API Apps which provide service functionality and Logic Apps which provide workflow functionality. These two services together will be the platform, moving forward, where EDI and AS2 functionality will be released. The current model, discussed in this article, will continue to be supported and is currently production ready and available, but most effort in new features is being applied toward App Services.

In the App Services model you will still have all the same concepts of partners, agreements, transports, and the functionality found in the bridges such as transformation, parsing of files, and routing. Each of these components can be found in an API Service in App Services. For example there are API Services available for Flat File Encoding, Transformation, AS2, X12, and EDIFACT processing.

In the App Services model you provision each of these services into your Azure Subscription and setup the required accompanying resources such as Azure SQL Databases or Azure Storage accounts. Once you have the services (API Apps) provisioned, you can build Logic Apps that aggregate those services into a process or workflow. In this way the parts are combined to make up your process much like bridges in the current model but with more flexibility as to the parts you want to combine and in what order.

Where are we?

In this series of articles I have so far covered the PaaS and IaaS options offered in Azure and Azure BizTalk Services for any type of messaging, now including EDI. In future articles I will cover cross-cutting concerns such as connecting your Azure applications to your on premises applications and the management and monitoring of Azure BizTalk Services.