XML integration: Buy or build?

The decision to build or buy XML messaging engines instead of continuing to hard code integration applications is a common dilemma faced by application architects. Both options offer unique advantages. Find out more about each solution.

Visual Studio .NET has rapidly become the platform of choice for building systems that support XML Web services standards. Because of its deep support for XML technologies, I’ve also seen it used extensively by corporations to build XML messaging solutions.

Even though XML has become the universal standard for data representation, companies still write and maintain programs that transfer data between applications and systems by converting one flat file format to another and then sending the file to a receiving system. This hard-coded program has a huge associated maintenance cost because every conversation between two systems has to be coded uniquely and updated whenever a file format changes on either side. However, companies are beginning to recognize the advantage of building or buying XML messaging engines instead of continuing to hard code these integration applications.

Basic functions of an XML messaging engine
Creating an XML messaging engine basically involves placing a layer between every system and the representation of its data. When an XML messaging engine processes a transaction, it first retrieves data from the source system and maps the data to an XML format. Next, it runs a transformation process, converting the source system’s XML format to an XML format for the target system. If the target system accepts the XML stream, it sends the data to the target system. If not, the messaging engine converts the data from an XML document into a format that the target system can understand before sending the data to the target system. This process is reversed when receiving additional data or response data from the target system.

If written properly, the XML mapping layer greatly reduces new development and maintenance. If the source system needs to send its data to a new endpoint, the developer can use the existing source map and only needs to develop the logic to perform the transformation to the target system’s XML format. If the source format changes, only the mapping engine has to be changed, and the target system mapping can remain unchanged. Now that the industry has standard protocols like TCP/IP, HTTP, and SMTP, and markup languages like HTML and XML, the plumbing is in place in most organizations to build integration applications. The real question is: Should you build one or buy one?

.NET Framework support
The .NET Framework does a great job of wrapping the protocols and markup languages and adding additional system support for creating custom XML messaging engines. For example, it is now trivial to create Windows services that can monitor pickup locations to start an XML messaging process.

Developing a workflow process that looks for uploaded or extracted files placed at a specific location is easy using the FileSystemWatcher class. The FileSystemWatcher class monitors the file system and raises events whenever a specified directory or file within a specified directory changes. This class can watch for changes within the directory to include subdirectories and can monitor directories on a local machine, a network drive, or a remote computer. Once your Windows service picks up the file, it can use the built-in XML support to process it.

The System.XML namespace includes support for XML readers or the XML Document Object Model (DOM), which can take source files and convert them to their XML document equivalents based on predefined XML Schema Definitions (XSDs). Once converted to XML documents, the built-in Extended Stylesheet Template support allows you to write XSL templates that automatically convert files between different document schemas. Also, you can load an XML document into a standard ADO.NET data set using its schema and use any coding language to write out a new data set, effectively performing a translation. The DataSet.WriteXML command will produce a well-formed, valid XML document from the resulting data set.

Once you have the converted XML document, you can either send that document to the target system or create a new file in the format the target system expects. You can produce a file from the transformed XML document, written to the target system’s expected format using an XML writer. Finally, you can use functions in System.Web to post data to a remote server using HTTP (System.Net.WebClient), FTP (System.Net.HTTPWebRequest), or sending it to the remote system using SMTP (System.Web.Mail).

Building your own XML messaging engine for a small number of installations or endpoints may be the most cost-effective choice. But it requires intimate knowledge of a family of XML protocols (i.e., XML, XSLT, XSD, and XQuery) and communications protocols (i.e., TCP/IP, SMTP, and HTTP). Unless you develop a custom workflow engine with a simple end-user interface, you’ll also end up developing custom code around the business process that consumes multiple XML messaging transactions. For example, if you have a system that sends orders to a remote system for shipping and then needs to receive shipping information to update the local database, you’ll have to wrap your XML messaging system with this business logic by developing custom code for each unique instance.

BizTalk solves several of these problems for you. First of all, BizTalk has a document editor that makes it simple to define transformation documents. The BizTalk Mapper allows you to develop transformations between two predefined XML documents. It even allows you to write custom functions (functoids) that can do specialized processing on data. Also, you can call your own processing routines from within a BizTalk transformation. BizTalk’s Message Manager allows you to build ports and channels that handle the underlying protocol plumbing without having to write any code. The BizTalk Administrator allows you to define receive events that replace the Windows services you’d have to create to handle the workflow. Finally, the BizTalk Orchestration engine uses Visio to allow architects or business analysts to visually design the workflow process around multiple transactions and systems and then turn the design over to a developer for implementation.

Choose your path
If your system designs call for complicated or repeated development of XML messaging systems, you should decide in advance to pursue one of two paths. The first option: Develop your own set of standard tools that extend the .NET Framework and allow you to create repeatable and supportable XML messaging solutions. The second is to begin by supporting BizTalk in your applications. Although BizTalk has an associated server licensing cost, you'll recover the cost of the license many times over in the cost of maintenance associated with a custom solution for a system with large numbers of connections or endpoints.

Editor's Picks

Free Newsletters, In your Inbox