One of the mixed blessings of BizTalk Server 2004 is the orchestration feature, a graphic process configuration tool with few equals. Incredibly flexible and exceedingly well-suited to business process design and integration, its correct usage is unfortunately also opaque in many applications and woefully under-documented (so far, anyway—we'll keep hoping!)
Another positive of BizTalk Server is the BizTalk Explorer side, where all the limbs of BizTalk flex and extend. Don't fail to appreciate this layout. The clean BTS separation between business logic (nested in orchestrations) and interface/database (in Explorer) isn't particularly conspicuous, since it's possible to do just about everything in orchestrations. But it's a separation you should keep in the forefront of your designer's mind, because it can buy you a lot.
Roll your own rollback
Here's a basic design principle that will often be applicable and perhaps appropriate to your particular distributed/integrated applications:
Send/receive, acknowledge and format the data shared with partners in BizTalk Explorer; unleash the data on your business logic and internal messaging in orchestrations.
Some of the BizTalk tutorials, samples, and labs you'll come across will employ this principle implicitly. Others will go another direction. Don't take either course blindly; instead, give careful thought to when (and in what form) you're going to need to handle your data as it makes its way through a process, and design accordingly.
Why? Because BizTalk Server 2004, sadly, does not offer rollback; its Health and Activity Tracking utility doesn't always permit you to resume the handling of a message; and BizTalk Explorer robustly facilitates the features of your process design that aren't business logic-oriented, making your overall implementation effort faster and easier to debug.
Isolating business logic in orchestrations and leaving the pitch-and-catch of data in various stages of dress in Explorer permits you to temporarily store data at any point in a process that you might want to do so. You can effectively create de facto queues for transitory data in a matter of minutes (even in a solution already deployed in production), where it can be captured for review or troubleshooting, copied and forwarded to need-to-know parties or ancillary processes, or sit patiently until rescued if something goes haywire downstream.
What goes where, and why
In a typical distributed business process, there are various transitions data usually has to pass through. It is received from some external source in some foreign format; it is acknowledged and interpreted; it is transformed into a friendly format; it is validated and parsed; it is analyzed; it is introduced into a business application, or stored for later application; it is culled for transitory reports; it is handed off (in modified form) to other down-line applications or to external partners. Along the way, people and other systems are often made aware of the progress of the data at any of these points. This scenario has many variations, but whatever the order of events, the steps above are fairly typical.
If you apply the principle presented above, then most of these steps will occur in BizTalk Explorer, rather than in orchestrations. Specifically, the only exceptions would be the introduction of data into a business process, and what is done with the data within that process. The creators of BizTalk might not be crazy about such a slim application of the orchestration feature (remember that just about all of these steps can be done in an orchestration, if you choose that route), but there are advantages to this segregated design approach:
- When a data-related error knocks an orchestration off kilter, it's often difficult to locate the error message, let alone correctly interpret it; data-related errors in BizTalk Explorer are more easily interpreted;
- A message that contains data, lost or suspended in an orchestration exception, may or may not be recoverable; in BizTalk Explorer, the data can be retained at various points, making the process restartable from those points even if the message is "officially" suspended;
- Anyone can see and understand what's going on in the BizTalk Explorer portion of a process, since it's basically just messages with date/time stamps making their way through a series of folders; BizTalk orchestrations, while easy to follow, nonetheless require considerably more effort to decrypt.
In a general scenario, a purchase order is received from an outside partner via TCP; a BizTalk pipeline catches it at the receive port, validating it and identifying the sending party, then returning a PO acknowledgment; the purchase order is re-mapped to an XML document; the XML document is parsed and transformed down to purchase order elements that are .NET typed, and the record is finally stored in a SQL PO table.
In our example, we're using BizTalk Explorer's send/receive adapter features to filter the inbound data and make decisions about storing it (in different stages) based on that filtering. In this particular scenario, we're archiving both the inbound order and the acknowledgment that went back; we're using a hot folder to pass the XML version of the purchase order into the orchestration and our archives function as a log that tells us not only what happened on our side, but what we told the sending partner.
Why? Because if something has gone wrong and the orchestration portion of the process is down, the purchase order will sit quietly in the folder until the process resumes. We don't have to back up any further in the process to restart; we have a time stamp on the interim version of the order that tells us when things went awry; and we can leave the BizTalk Explorer portion of the process running while we fix whatever's wrong, knowing that purchase orders will safely accumulate in the to-be-processed folder while we work.
Nuts and bolts
Finally, here are the nuts and bolts of the BizTalk Explorer side. There are several tricks you need to know:
Filter inbound data by sending party and document type
You'd perhaps conclude intuitively that you'd do this filtering in your Receive Port Location, but in fact where you want it is in the Send Port filtering. Open BizTalk Explorer from Visual Studio.NET View, and open up the Send Ports under the BizTalk configuration database (.BizTalkMgmtDb.dbo).
Adding one for the first time? Right-click on Send Ports to bring up the Add Send Port wizard. Right-click to Edit the port. Here's the thing to keep in mind: you can have multiple Send Ports for any particular message-receive from a partner (called Party in BizTalk Explorer), as you'll see below. Give each Send Port you create for a particular partner/process a name that makes clear what you're doing, like MyPartnerAcks, or MyPartnerArchive, etc.
When editing the Send Port, go to Configurations | Filters & Maps | Filters. Here, you can parse messages based on type (BTS.MessageType, for instance), to sort out acknowledgments, parse header data from detail data in multipart documents, and so on. You can combine this with filtering on the ID of the sending partner (based on a field in the header, perhaps) to route inbound data to the correct folder (see Figure A).
|Route inbound data|
Use "file" Transport Type liberally
When setting up multiple Send Ports for a particular partner or process, creating a "copy" of the data is almost an effortless exercise. Create a Send Port specifically for this task, and select FILE for Transport Type in Add Send Port. You'll need a URI, a location and filename (Figure B). For the filename, the default is %MessageID%.xml. Change this to .txt.
|Add Send Port|
Set up trading partners (Party) for multiple Sends; or send data to multiple destinations with Send Port Group
Parties in BizTalk are partners, identifiers for business entities with whom your process will exchange data. Some industry-specific accelerator packages, such as HL7, for BTS2K4 allow you to configure Parties for multiple Send Ports; that is, a batch of data arriving from that Party will, upon being identified from the message header, be sent to all the Send Ports for which that Party is configured. Alternately, you can set up a Send Port Group (Figure C), which will give you basically the same thing—a collection of Send Ports, bound to different destinations, filtering on a common condition or conditions.
|Send Port Group|
Use the appropriate pipeline when capturing data
The Send pipeline you choose when setting up a Send Port (Figure D) determines the format of the data when it lands in the file you've sent it to.
Your choice of pipeline depends on what you want done with the data when the file lands in the destination folder. That is, if it's to be passed another process, you'll want it in the form the process is expecting (often this will be XML). If you're setting up a "hold" folder where you want data stored for "rollback" at some sensitive point in a process, then you'd likewise want it in the format it would need to be in when the process restarts (again, often XML). If you want to archive the data, or pass it to a person for review or reporting purposes, you may want it in its original form (often simple text).
Whether you're using canned or custom pipelines, it's important to make the right choice for the right job, as you configure your Send Ports. Among the default BizTalk pipelines, for instance, the following pipelines yield the following results:
- Microsoft.BizTalk.DefaultPipelines.PassThruTransmit passes a message in the format it's already in (i.e., no post processing occurs in the "send" version of the pass-thru pipeline)
- Microsoft.BizTalk.DefaultPipelines.XMLTransmit delivers XML
The practice of separating interface/database processes from business logic is usually best, but there may be reasons why you need an entire process situated within an orchestration. When there's good reason to do so, by all means, take this course. We'll look at some scenarios where it is indeed the best practice, and some tips on how to organize such an orchestration, in an upcoming article.
Scott Robinson is a 20-year IT veteran with extensive experience in business intelligence and systems integration. An enterprise architect with a background in social psychology, he frequently consults and lectures on analytics, business intelligence and social informatics, primarily in the health care and HR industries.