A DataSnap application consists of two tiers: the DataSnap Server, with a remote data module that has one or more DataSetProvider components on it, and a DataSnap Client, with an xxxConnection component and one or more ClientDataSets connected to the DataSetProviders. Instead of a ClientDataSet, you can employ XMLBroker components, which are specifically used in InternetExpress Web server applications.

The DataSetProviders export the data to the outside world, and the ClientDataSets receive the data (and send requests as well as updates back to the DataSetProvider). Many DataSnap Clients can be connected to a single DataSnap Server, all requesting data from the server, which is responsible for the data throughput bottleneck that can occur.

Getting started
To start to work on this problem, it’s necessary to identify the amount of data that is actually sent over the wire. For this, we won’t look at the ClientDataSets (requesting and receiving the data). Instead, we have to look at the DataSetProvider component—sending the data in response to requests for data. Fortunately, the DataSetProvider component has useful virtual methods that can be overridden and used to include our “tracing” code, so you can get an idea of the actual data throughput.

Once you have an idea, you can start working on lowering this number. Of course, you should not start to optimize just because you can; there should be a reason. But once you have a few hundred clients connected to your DataSnap Server, all requesting megabytes of data, you’ll quickly have that reason.

DataSnap formerly known as MIDAS

The DataSnap technology, the new name for MIDAS, creates multitier applications in Delphi Enterprise (or C++Builder Enterprise). When I talk about DataSnap efficiency, I’m mainly concerned about the data throughput bottleneck, including ways to prevent it by limiting the amount of data being sent from the server tier to the client tier. Before you can limit throughput, you have to measure it.

TB42DataSetProvider
For the remainder of this article, I’ll assume that you have a DataSnap Server and Client available. See Delphi6\Demos\Midas\MstrDtl for a sample DataSnap master-detail project if you need one. To measure the amount of data, you have to replace the TDataSetProvider component with a special version called TB42DataSetProvider. This new component has a BytesTransferred property that will contain the amount of bytes that have been transferred since it was created.

It also overrides the InternalGetRecords method, to increase the value in FBytesTransferred and report the number of bytes transferred so far, and the CreateDataPacket method, to report the fact that a new data packet has been sent, including the number of records and the size of the data packet. These two methods, shown in Listing A, are called in response to a request for data from a DataSnap Client, which can be made by a ClientDataSet or an XMLBroker component.

Note that the code in unit eBob42PRO uses simple writeln statements in the InternalGetRecords and CreateDataPacket methods. This means you have to add the line {$APPTYPE CONSOLE} to your DataSnap Server main project to open a console window for this debugging output. Don’t worry that your DataSnap Server will give you an I/O error if you forget to do this, because the IsConsole check will make sure that the writeln statements are actually called only in a Console application.

Using this component is simple: Add the TB42DataSetProvider component to a Delphi package in the component palette and replace the normal TDataSetProvider on your remote data module with a TB42DataSetProvider.

ClientDataSet’s PacketRecords
One of the potential dangers of using a ClientDataSet connected to a DataSetProvider that is pointing to an entire table is that when you open the ClientDataSet, it will by default request all data from the DataSetProvider. This means that the entire contents of the table (at the server side) will be sent over the wire to the ClientDataSet. This is no problem with the database table examples that ship with Delphi itself, since, for example, the customer and orders tables contain fewer than 100 records. However, in real life, companies can have more than 100 customers (and hopefully, more than that number of orders), and it would take a while to obtain all these customers and have the contents of the table sent from the DataSnap Server to the DataSnap Client whenever you open the ClientDataSet.

One way to try to solve this is to use the PacketRecords property of the ClientDataSet component. This property will change the initial request for data to include the size (in records) of the requested data packet. So if thousands of records exist, you get only the first X, where X is the value of the PacketRecords property. By default, this property is set to -1, which means “give me all records.”

To get the next packet of records, the DataSnap Client has to call the GetNextPacket method, which not only gets the next packet, but also returns the number of records that were retrieved—so it’s not hard to detect when you’re at the end of the remote table and have retrieved all records.

A snag
At first sight, this approach seems to work just fine. But it has a potential downside: The end user can accidentally trigger an event that will result in obtaining all records from the remote table. For example, a user might click the Last button of a DBNavigator or press [Ctrl][End] in a DBGrid to jump to the last record. In both cases, to show the last record of a dataset, the DataSnap Client has no choice but to request them all. It’s impossible to contain only the first and last few records in a dataset without all the records in between them. Fortunately, the new TB42DataSetProvider component will immediately show that all records are sent, including the size of the total data packet.

The GetNextPacket method
If that’s not problematic enough, consider this: For the GetNextPacket method to work, the DataSnap Server somehow has to know which records the DataSnap Client received last time. In other words, the DataSnap Server is now stateful, retaining the state for all connected clients whose PacketRecords value are set to a value other than -1. And maintaining state for all these clients is a significant overhead for a DataSnap Server—something you don’t want, since it can prevent the server from upscaling to far more clients than a stateless DataSnap Server can handle.

Fortunately, there is a workaround for this problem. It involves setting the FetchOnDemand property of the ClientDataSet to False and using the OnBeforeGetRecord event handler of both the ClientDataSet and the DataSetProvider to pass a value in the OwnerData OleVariant, which will send the state information from the DataSnap Client to the DataSnap Server (so the latter no longer has to retain the state information by itself).

PacketRecords and stateless DataSnap servers
This workaround is particularly important because the PacketRecords doesn’t work without it when you are using truly stateless DataSnap Servers (which are the result of using the WebConnection or SoapConnection components). In those cases, the server has no idea which (new) records are expected by the clients. So in those situations you really must use the OnBeforeGetRecord events (from the ClientDataSet and DataSetProvider) to pass OwnerData that specifies the state of the client to the server. You’ll find specific details on this in online help.

Limit the number of records
To limit the amount of data that is sent over the wire, an even more useful technique is to make sure that the number of records is limited. You can do this by using a server-side filter or by using a query instead of a table. That way, the selection is done at the server side, and only a subset of the data is returned to the client. In practice this means that you don’t select the complete table with all of the more-than-a-million books available from Amazon.com, for example, but only the few hundred books about Star Trek or the handful on Delphi 6.

Limit the number of fields (views)
Apart from limiting the number of records, you can also get a significant space savings if you make sure not to pass all fields of the records. This may not always be practical. For example, you might need all fields in a detail overview. But in that case, you can define two views: one for searching (with only a few fields) and one for the detail overview (with all fields).

Don’t assume that if you remove the fields you don’t want at the client side, the server won’t send them over the wire. To demonstrate, employ the Biolife table, providing it (and all fields) by a DataSnap server and then connecting to it from an XMLBroker component (in an InternetExpress application). Even if you select only a few of the fields in the WebPage Editor, the full set of fields will be sent over the wire (including large fields like the Notes and Graphic) and placed in the XML data packet embedded in the resulting HTML file. So if you limit the fields you’re using, make sure you limit them at the server side first and then synchronize with the client side so both sides are using the same number of fields.

Summary
Writing multitier applications with DataSnap (or using any other technique, for that matter) requires careful thought about the amount of data being sent back and forth between the different tiers so that you can minimize network throughput bottleneck. The TB42DataSetProvider can be a big help in measuring the actual amount of data being requested. And the ClientDataSets can limit the network throughput by using the PacketRecords property to limit the number of records you request and to minimize the individual recordsize in each data packet. Just make sure that the server remains stateless.

This will not only increase performance today, it will also ensure that you can upscale your multitier applications to service even more clients than before.

More Delphi

Have you worked with the DataSnap application? Post a comment below and share your experiences or contact the editors.