BizTalk EDI
Build a Robust EDI Solution with BizTalk Server
Mark Beckner
This article discusses:
|
This article uses the following technologies: BizTalk Server 2006 R2 |
Contents
Developing an EDI Schema
EDI Mapping
Trading Partner Configuration
Transporting EDI Documents
Document Delivery through a Firewall
Handling Failed Documents
EDI and SOA
Electronic Document Interchange (EDI) is a technology standard that has been around for decades. So mixing it with a modern service-oriented architecture (SOA) and the latest release of BizTalk® Server may seem an unlikely combination. Yet EDI encompasses the largest share of real-world business-to-business commerce—nearly 90 percent of the current market—and is growing rapidly year over year. As companies relying on EDI evolve their IT architectures, the capabilities of BizTalk Server 2006 R2 are proving to be a reliable, robust, extensible, supportable, and intuitive way to solve both SOA and EDI infrastructure needs.
Prior to the BizTalk Server 2006 R2 release, EDI support in BizTalk was limited. There were adapters and accelerators that supplied the basic infrastructure for EDI solution implementations, but there were limitations in their capabilities, such as in how documents were validated. With BizTalk Server 2006 R2, the EDI functionality was formalized. It now allows for extensive document validation and numerous methods for transporting documents, including all of the reporting features common to enterprise-level EDI implementations.
BizTalk Server now provides the same level of service that many value-added networks (VANs) provide, with the additional benefit of the underlying BizTalk components that have been essential to enterprise integration solutions and SOAs. These include the development of business workflows through orchestrations, access to a business rule engine, extensive document-tracking capabilities, state management, and other similar functions.
EDI implementations in BizTalk Server 2006 R2 begin with developing the schemas that relate to the documents being traded. Once the documents have been defined, trading partners are created as BizTalk parties and their specifications are configured to ensure the proper processing and routing of EDI documents. Next, the specifics around how documents will be delivered are implemented through a combination of party configurations and BizTalk adapters. When solutions are in place, document flow can be monitored in real time through the use of EDI reports. All of these capabilities ride on top of the BizTalk infrastructure and benefit from all the standard components such as the MessageBox, orchestrations, ports, and pipelines.
The intent of this article is to introduce you to the EDI functionality within BizTalk Server 2006 R2 and to show that this functionality can enable you to integrate your EDI processes more easily with the rest of your enterprise. I will walk through several critical aspects of working with the new BizTalk Server EDI components, illustrating the different aspects of schema creation, document mapping, EDI delivery and transmission, and exception handling.
Developing an EDI Schema
To understand EDI schema development, first look at the specifics of the document structure itself. An EDI document can best be described as a simple text file consisting of three parts: the Header, the Detail, and the Footer. The Header defines from where the document came and for whom it is intended, the document type, and some date information. The Detail contains all of the business information that gives meaning to the document. For example, in the case of an invoice, the Detail consists of such information as line items, descriptions of sold products, pricing, quantities, and totals. The Footer consists of summary information about the Detail rows, such as the total number of lines included within the document.
EDI documents are formatted in segments, with each row of data containing a number of named segments. The format and composition of these segments are dictated by standards such as X12 and Electronic Data Interchange For Administration, Commerce, and Transport (EDIFACT). For X12 documents, the segments ISA and GS are considered to be the Header, the segments GE and IEA correspond to the Footer, and all lines in between the Header and Footer are the Detail (see Figure 1).
Figure 1 An X12 EDI Document (810—Invoice) (Click the image for a larger view)
For EDIFACT documents, all segments that begin with the characters UN correspond either to the Header (UNA, UNB,) or the Footer (UNT, UNZ), while all segments in between are the Details. Segments and rows are separated from one another with separator characters, and these characters can be different for different trading partners. In both document formats, data is generally separated by the asterisk (*) character, and rows are separated by line feeds, tildes (~), or any other mutually agreed upon character combination.
BizTalk Server 2006 R2 ships with thousands of predefined EDI schemas that function as starting points for all documents exchanged by trading partners. Generally these schemas are altered to reflect specific expected formats. Though EDI has document standards, the reality is that two trading partners who both exchange 810 Invoice documents may still have two different representations of the 810 and therefore require two different schemas. These schemas will be very closely related and may only differ in one or two segments. For example, one may truncate a street address at 50 characters while another allows for 100. But even this small difference requires that the default 810 XML Schema Definition (XSD) be modified and implemented separately for both parties.
The first step in schema development is to define the electronic documents that will be exchanged and to develop the corresponding schemas. Using the invoice example, you would begin by adding a copy of the default 810 Invoice schema to your BizTalk solution. Schema templates can be found in the \Program Files\Microsoft BizTalk Server 2006\XSD_Schema\EDI directory in the file MicrosoftEdiXSDTemplates.exe. Run the executable to extract the templates, then find the file named X12_00401_810.xsd and add it to a BizTalk solution in Visual Studio®. This XSD represents the superset of all data that can be part of an 810 Invoice.
Say you are going to define an invoice document exchange between Company A and Company X. The next step is to create an XSD representation of Company A's XML document; this will act as the source document when creating the EDI instance. In many cases it is unnecessary to modify the default schema instances. But in this case, assume that Company X requires N401 (city name) to have a maximum length of 10 characters instead of the default length of 30. To change the length, click the N401 node and look for the Maximum Length value in the property window. Enter the new value here. This will ensure that if a document tries to pass through the system with more than 10 characters, an EDI error will be thrown indicating an invalid document. During the mapping process, it will be necessary to truncate this field before mapping it into the EDI schema.
EDI Mapping
Suppose Company A has an XML representation of an invoice that needs to be mapped to the EDI standard before delivery. It also needs to map all invoice line items to the IT1 loop and place the total number mapped in the CTT02 node.
Prior to mapping, all of the schemas, whether specifically in EDI format or not, need to be defined and added to the solution. Company A has an XML version of the invoice data that needs to be mapped to 810 Invoice format prior to transmission. This XML data must have an associated BizTalk schema, and in this case, the schema looks like that shown in Figure 2.
Figure 2 Invoice Schema Excerpt
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns:b="https://schemas.microsoft.com/BizTalk/2003" xmlns="https://schemas.MSDN.com/Common/810/v1" attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="https://schemas.MSDN.com/Common/810/v1" xmlns:xs="https://www.w3.org/2001/XMLSchema">
<xs:element name="COMMON_810">
<xs:complexType>
<xs:sequence>
<xs:element name="TRANSACTION">
<xs:complexType>
<xs:sequence>
<xs:element name="HEADER">
<xs:complexType>
<xs:sequence minOccurs="1" maxOccurs="1">
<xs:element name="GUID" type="xs:string" />
<xs:element name="DOCID" type="xs:string" />
<xs:element name="DESC" type="xs:string" />
<xs:element name="PARTNER" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="ADDRESSES">
<xs:complexType>
<xs:sequence minOccurs="0" maxOccurs="unbounded">
<xs:element maxOccurs="unbounded" name="ADDRESS">
<xs:complexType>
<xs:sequence>
<xs:element name="TYPE" type="xs:string" />
<xs:element name="STREET" type="xs:string" />
<xs:element name="CITY" type="xs:string" />
<xs:element name="STATE" type="xs:string" />
<xs:element name="ZIP" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="ITEMS">
...
After the schemas have been defined, you will need to create the BizTalk map. In this case it will be a map that transforms the data from the XML version of Company A's invoice information into the standard EDI 810 Invoice instance. Mapping EDI documents is not unlike any other type of BizTalk mapping except that there are several complexities that are unique to EDI. In the case of the invoice, a total count of all line items in the IT1 nodes must be displayed in the CTT01 node. Let's look closer at this to demonstrate what is unique about this type of mapping.
Begin by looking at the IT1 repeating node, which represents line items on the invoice. As shown in Figure 3, there are two types of mapping occurring: simple source to destination (as in the case of PRICE to IT104) and complex (as in the case of TYPE, which determines whether a line item in the source is copied to the target).
Figure 3 Mapping Line Items (Click the image for a larger view)
In the case of the complex mapping there is a combination of two scripting functoids and a single equal functoid. The first scripting functoid contains logic that determines whether the value in the TYPE field is something that should be mapped across. If the value is equal to a certain string, then it will be added to the IT1 loop.
The logic of interest, though, is in the second functoid, which is connected to the IT101 node. The code in this functoid is actually keeping track of the number of line items that have been mapped across—not the total in the source document, but the total in the target document. This total is used both to populate the value of IT101 and to increment a global variable that can be used eventually to populate the CTT01 node.
The code for the IT101 scripting functoid, which declares and increments a global parameter accessible throughout the map, is shown in Figure 4. A global integer is created, and it's incremented by 1 each time a line item that needs to be mapped is sent across. With each line item that needs to be mapped, this value is incremented, and the resulting value can then be populated in the IT101 node. Once all of the IT1 line items have been mapped, the map can move on to populating the CTT01 node with the total number of line items that occur in the EDI instance. This value is contained in the global integer and can be accessed in a separate scripting functoid specific to the CTT01 node using the code shown here:
public int intAccessTotal()
{
return globalCtr;
}
Figure 4 Functoid to Ignore Certain Line Items
// declare global variable, accessible throughout all other components
// within current map
int globalCtr;
// declare function to increment and return the current value of the
// global count
public int keepCount (bool mapMe)
{
// the value of mapMe is the boolean returned by the equals
// functoid (see map)
if(mapMe)
globalCtr++;
// return the total value to be populated in IT101
return globalCtr;
}
Trading Partner Configuration
Two trading partners must be set up in BizTalk Server, one as the sender and one as the receiver. Trading partners are created as parties within BizTalk and configured through the BizTalk Server Administration Console. Trading partner configuration consists of a number of settings that enable BizTalk to determine what documents belong to which partner. When EDI documents arrive, BizTalk matches documents to trading partners by comparing what is defined in the document Header (or what is in the Applicability Statement 2, or AS2, envelope) with what has been configured for the trading partner.
For example, assume that the following ISA segment showed up in the Header of a document:
ISA*00* *00* *01*
BASECOMP12 *ZZ*TRADPART1 *
070407*1555*U*00401*000000025*0*T*>~
The sixth segment (ISA06) has the value of BASECOMP12, while the eighth segment (ISA08) has the value TRADPART1. Remember that segments are delimited by asterisk (*) characters. Here, the third and fifth segments are empty.
Trading partners can be configured as interchange senders or receivers. In this case, BizTalk compares the segments against how parties have been configured and finds that these values correspond to the configuration of trading partner 1 set up as a receiver (see Figure 5). Since the document has been matched to a party, the rest of the document now can be validated against the schemas associated with that trading partner. Documents found to be invalid are processed in one way, while valid documents are sent on to be processed by other EDI components.
Figure 5 Trading Partner Configured (Click the image for a larger view)
Transporting EDI Documents
Documents can be sent to trading partners via any protocol, including SMTP, File, FTP, HTTP, and many others. However, EDI standards support only VANs and AS2. VANs ensure that documents are valid, that they are routed to the appropriate recipients, and that there is a record of the transaction. AS2 is the technology that allows for the secure transmission of documents directly between trading partners using S/MIME over HTTP/HTTPS. There are robust features in BizTalk Server that enable either standard to be incorporated into a solution so that, whether trading partners want to use AS2 or the VAN, BizTalk can act as the single internal EDI solution.
While BizTalk Server 2006 R2 can eliminate the need for VANs, many trading partners continue to do their business over these networks. BizTalk facilitates the transmission of documents to and from VANs through the use of FTP and secure FTP.
Interacting with VANs via BizTalk may require the use of a third-party adapter, as many VANs require secure FTP. There are some limitations with the standard FTP adapter that may prevent its use in environments such as these. (Often the SYST FTP command is required by VANs, but the standard BizTalk adapter does not support this.) Regardless of the adapter used, however, the idea of connecting to the VAN is a simple process of interacting with an FTP server. EDI documents being shipped to trading partners via the VAN are uploaded via FTP, and EDI documents being retrieved from trading partners are downloaded via FTP. BizTalk is responsible only for the successful delivery and receipt of EDI documents, never for the actual delivery of the document to the trading partner—the VAN takes care of this. It is the duty of an administrator to check the status of documents on the VAN through the use of tools provided specifically for that purpose.
When looking at FTP and VAN communication with EDI in BizTalk, much of the value that the VAN provides, such as document validation, document tracking, and document delivery, is now available within BizTalk itself. In AS2 solutions, BizTalk is the "value-added network," providing all of the functionality without the costs associated with the VAN.
AS2 allows trading partners to interact directly with one another over secure HTTP. When an AS2 implementation is used in an EDI implementation, the port is exposed to the public via the standard HTTP adapter with AS2 properties set on the EDI party, or via a third-party BizTalk adapter. Either way, the core idea is the same: enable a secure transmission and validate the document. The secure transmission is handled through certificates, and the validation of the document is handled through AS2, party, and schema configuration in BizTalk.
An example of a party configured to allow for AS2 transmission over the standard BizTalk HTTP adapter is shown in Figure 6. In this case, AS2 properties are set on the party and act as an extension to the HTTP adapter. When documents arrive over HTTP, BizTalk first compares the information to the AS2 settings, then opens the EDI document. The EDI document is then routed just as if it had arrived through any other method. AS2 simply provides a secure way to transfer the data and any relevant envelope information.
Figure 6 Configuring Base AS2 Properties (Click the image for a larger view)
Document Delivery through a Firewall
An important concept to illustrate is how documents can be delivered to BizTalk Server on networks where BizTalk is behind a firewall and not available to the public. When a SOAP or HTTP receive port is added to BizTalk, the port will operate on the local IIS server, and all posted data will need to be posted to that local Web server. For example, when using the BizTalk Web Services Publishing Wizard, the Web service can be created only on the local IIS server where BizTalk Server is installed.
In an environment where this IIS server cannot be accessed from outside of the network, some sort of solution must be put into place to allow communications to be routed (see Figure 7). There are several solutions, including placing BizTalk Server in the public portion of the network or creating a reverse proxy that routes requests through a firewall and allows BizTalk to reside in a private portion of the network.
Figure 7 Document Delivery through a Firewall (Click the image for a larger view)
Communication through the firewall can be made programmatically as it is rarely acceptable to house BizTalk Server in a publicly accessible portion of a network—the security risks are simply too high.
There are several approaches that can be taken in the development of the Web service on the public IIS server in the public network. One is to create a gateway Web service. Another is to create a proxy Web service. The gateway Web service requires custom coding, just like any Microsoft®.NET Framework Web service, and complex transformations of interfaces to match those that BizTalk expects. The gateway accepts a request from the public and maps it to a request to the Web service behind the firewall. A proxy Web service, on the other hand, can be created by modifying what is produced using the BizTalk Web Services Publishing Wizard and placing the copy on the public Web server. Due to its simplicity, modifying a proxy Web service seems like the clear choice.
To create a proxy Web service, launch the BizTalk Web Services Publishing Wizard. Click on the appropriate options in the wizard and define where the Web service will be published (notice that the only option for where the virtual directory is created is on the local IIS server). When the wizard creates the Web service, it is actually creating a proxy that redirects incoming calls from IIS and places them in the BizTalk MessageBox where they can then be routed to the corresponding orchestration. This proxy Web service is coded automatically, and everything is preconfigured to work without additional user changes—as long as the incoming posts come directly to this Web server.
Enabling posts to come in on a public portion of the network and be routed to this Web service requires three basic steps. The first step is to create a proxy-to-the-proxy Web service. When a Web service is exported via the wizard, BizTalk refers to it as a proxy Web service. In order to call this Web service from external clients, a proxy Web service must be created on a public server that can route requests to it. This can be done by creating a virtual directory on the public IIS server with the exact same name and interface as the generated proxy. Once the virtual directory has been created, copy all of the generated files from the original BizTalk IIS directory to a virtual directory on the public server.
The second step is to modify the public-facing Web service so that, instead of posting the incoming SOAP information to the BizTalk engine, it reroutes the information to the internal proxy Web service on the IIS server hosting BizTalk Server. The code to modify sits in one file, which is located in the App_Code subdirectory of the virtual directory to which the Web service was published. The name of the file will depend on the name of the entity being published, but will always end with asmx.cs. Opening this file will show the class and Web method declarations, which provide a public interface to calling entities and enable these requests to be pushed directly to the BizTalk MessageBox.
To see what this generated code looks like, export an orchestration with a two-way SOAP port using the Web Services Publishing Wizard. Once the wizard has completed, open the asmx.cs file in the App_Code directory and look at the code. The name of the Web method will be based on the name of the operation as defined on the port in the orchestration. Within this Web method is code that takes the incoming post and transforms it to a format that can be posted to the MessageBox. Once the Web service pushes it to the MessageBox, it then waits for a response that corresponds to the outbound response on the orchestration's two-way port and posts this back to the calling entity. This code, once copied and placed on the new IIS server in the public network, must be modified so that it forwards the incoming post to the Web service on the BizTalk Server.
The modification to the original proxy Web service code can be accomplished using the code shown in Figure 8. The code overwrites the Web method in the original Web service and, instead of posting to the MessageBox, loads some configurable fields from the web.config file and posts the incoming request to the Web service on the BizTalk IIS server. The routing to the BizTalk engine is still handled by the original proxy Web service.
Figure 8 Proxy-to-the-Proxy Web Service
public BTSRedirect.SyncTransResponse Operation_SyncTrans(
[System.Xml.Serialization.XmlElementAttribute(
Namespace="https://GuardianProStar.BizTalk.Schemas.SyncTrans.SyncTransRequest",
ElementName="SyncTransRequest")] BTSRedirect.SyncTransRequest part)
{
BTSRedirect.SyncTransRequest objSyncTransReq = part;
BTSRedirect.SyncTransResponse objSyncTransRes = new BTSRedirect.SyncTransResponse();
BTSRedirect.GuardianProStar_BizTalk_Orchestrations_SyncTrans_SyncTransactions_Port_SyncTrans
objWebServiceMethodCall = new
BTSRedirect.GuardianProStar_BizTalk_Orchestrations_SyncTrans_SyncTransactions_Port_SyncTrans();
// authentication/credentials
string strWSUser = System.Configuration.ConfigurationManager.AppSettings["WSUser"];
string strWSPassword = System.Configuration.ConfigurationManager.AppSettings["WSPassword"];
string strWSDomain = System.Configuration.ConfigurationManager.AppSettings["WSDomain"];
string strWSAuthenticationType =
System.Configuration.ConfigurationManager.AppSettings["WSAuthenticationType"];
objWebServiceMethodCall.Url =
System.Configuration.ConfigurationManager.AppSettings["RedirectURL"];
System.Net.CredentialCache cache = new System.Net.CredentialCache();
cache.Add(new System.Uri(objWebServiceMethodCall.Url), strWSAuthenticationType,
new System.Net.NetworkCredential(strWSUser, strWSPassword, strWSDomain));
objWebServiceMethodCall.Credentials = cache;
// set the response equal to the return value of the call to the web service
objSyncTransRes = objWebServiceMethodCall.Operation_SyncTrans(objSyncTransReq);
return objSyncTransRes;
}
Because both Web services have the identical Web interface, either one can be called using the same client code. During development, therefore, you can work with the original Web service code as published by the wizard to validate that everything is working as expected. The proxy-to-the-proxy can be created during the final stages of testing and deployment without affecting any client code.
The final step involved in working with the proxy-to-the-proxy Web service is to modify the web.config file on the public IIS server so that the configurable fields are available. The fields needed by the new Web service are shown here:
<appSettings>
<add key="RedirectURL"
value="[redirection URL]"/>
<add key="WSUser" value = "UserName"/>
<add key="WSPassword" value = "Password"/>
<add key="WSDomain"
value = "DomainName [Not Always Applicable]"/>
<add key="WSAuthenticationType" value = "basic"/>
</appSettings>
These settings provide for the authentication values and the redirection URL. The actual list of configurable fields (and whether these are actually stored in the web.config file or elsewhere) is up to the developer, and is shown here for illustration purposes only.
The most complex post routing through the firewall is that involving Web services. HTTP posts (and AS2) can be handled in a similar fashion but do not require copying and modifying generated code from BizTalk. In the case of HTTP posts, these can be routed with fairly simple redirect patterns depending on the nature of the post. Many organizations can get mired in determining the right approach to routing traffic to and from their BizTalk servers, and it is important to understand that these types of conversations should never get overly complex, as there are plenty of simple solutions to this common problem.
Handling Failed Documents
EDI implementations are schema intensive. Every document that passes in or out of BizTalk is validated against a corresponding schema. If the document passes validation, it will be routed to the BizTalk engine and will be processed by the EDI components. If the document does not pass validation, however, it will fail and will, by default, remain unprocessed in BizTalk, flagged as invalid. By enabling error routing on the ports that transport EDI documents, these documents will be able to be routed to any BizTalk component.
For example, if a trading partner were to deliver an invalid document to BizTalk rather than allowing the document to fail and remain hidden, it may be desirable to notify an administrator. This could be accomplished by enabling error routing on the port being used for EDI transport. You could then create an orchestration that subscribes to all documents containing errors. This orchestration could then include all of the logic necessary to determine the reason behind the document's failing and notify an administrator.
To route failed messages to a subscribing entity, the port where the messages are arriving or from where they're being sent must have the Enable Routing for Failed Messages flag set. When this is set, any message that fails will be sent to the MessageBox contained in an envelope with information about why the document failed. One set of promoted properties, the ErrorReport group, will then be available to allow for routing to specific subscribers.
To set up a subscriber to listen for these failed documents, simply set a filter to listen for any document on the MessageBox that had a value in the ErrorReport.ErrorType property.
Say your company wants to subscribe to all failed messages that arrive from external trading partners. The initial receive shape on an orchestration could be set to listen for all documents with a value in ErrorReport.ErrorType. This would be done by dropping the receive shape at the top of the orchestration flow and setting it to subscribe to a port with a generic document type (such as System.Xml.XmlDocument). The filter on the receive shape could then be set by clicking on the Filter property of the shape and setting the Property value to ErrorReport.ErrorType and the Operator value to Exists.
If the filter needed to be further refined to restrict the document being picked up to those that specifically matched the Invoice 810 schema, for example, then the incoming port type could be swapped from the generic System.Xml.XmlDocument to the specific schema desired.
EDI and SOA
I have outlined the core EDI functionality in BizTalk Server, but how does this functionality fit into a service-oriented architecture that ensures the maximum reuse of components?
An EDI solution is really a series of processes that can be reused by numerous trading partners. BizTalk Server provides the infrastructure that enables these processes, and their accompanying data entities, to be easily plugged into an SOA and leveraged by a variety of other workflows and services across the enterprise. Additionally, in the context of EDI all documents can be transferred in a secure manner, are validated for content, and can be tracked. In the event of error, they can be fully recovered. If there is extensive processing that needs to take place, either through the core EDI engine or through orchestrations, state management is ensured. Because the solution is based on BizTalk Server, much of your work on orchestrations, error processing, reporting, and other elements can be leveraged across EDI trading partner relationships.
For more information on this subject, see BizTalk Server 2006 R2 EDI Functionality (available at msdn.microsoft.com/library/bb226440.aspx) and BizTalk Server 2006 R2 EDI and AS2 Tutorials (at msdn.microsoft.com/library/bb259975.aspx).
Mark Beckner runs Inotek Consulting Group, LLC, specializing in technical consulting and enterprise integration infrastructures. He is the author of Pro EDI in BizTalk Server 2006 R2.