Software Development optimize

Working with the OutSystems Agile Platform's Integration Studio

Part four of this six-part developer diary focuses on the OutSystems Agile Platform's Integration Studio, which is the tool used to tie the platform into external code.


Read the previous installments in the series: Getting started with the OutSystems Agile Platform, Learning the basics of the OutSystems Agile Platform, and Describing the OutSystems Agile Platform Service Studio experience.

One of the key things to understand about Rat Catcher is that its core functionality involves a fairly complex piece of code that is moderately resource intensive; specifically, it opens up a fair number of network connections at once to download content and then it processes that content.

When I made the move to the Agile Platform on this project, I knew that re-implementing this functionality would not be possible to do in Service Studio, primarily because Service Studio does not have provisions for multithreading. Another issue was that too much of the work involved parts of the .NET Framework that simply are not exposed through the Agile Platform. But I was not worried -- after all, I knew that Integration Studio would let me hook up my application to this code. Integration Studio is the tool used to tie the Agile Platform into external code, either by writing code as new Actions or by importing existing code as Actions and Entities.

The first thing I tried to do was to use Integration Studio to import the code and expose its members as entities and actions. While this was successful at a technical level, it did not work well for my needs. Because, in my code, I have one class in which you set values for a large number of properties, and then call an "Execute" style method on the instance. Is this necessarily the prettiest or the most "correct" way of doing things? Not really. But C# is what I call a "noun-oriented" language not a "verb-oriented" language, and algorithms really are "verbs" (maybe an F# rewrite is in order?). The end result is that what was brought into Integration Studio was fairly unusable.

After talking to Miguel Baltazar (the OutSystems Solutions Delivery Manager in the United States), he showed me in about three minutes a better way of handling this situation. Instead of importing the code directly into Integration Studio, I could create a new action in Integration Studio, and then edit the code for that action in Visual Studio. From there, it would be a snap to add references to my algorithm code and call it. I did exactly that (Figure A), and with only a little bit of effort (mostly to copy data from the results of my code into the entity formats), I was up and running. Figure A

The original implementation of the extension to access my algorithm (aka the Mongoose Engine). As you can see on the left, I had to create two structure types and use them in the action definition.

This worked great throughout my development, but there was a little snag that forced a change, and the change was a very beneficial one in the end. You see, I had used the Parallel Extensions Library (PFx) in my algorithm code for the multithreading. When I wrote the code, PFx was only available in the .NET 4 beta and as a CTP for .NET 3.5, so my code was written in .NET 3.5 and used the PFx CTP. The problem was I discovered that the CTP was not licensed for actual production use. Even though it was rock solid for my needs, I knew that I would have to move to .NET 4. I asked OutSystems if the Agile Platform supported .NET 4 yet, and it didn't. This meant that I would need to wrap the algorithm code in a Web service and then consume the Web service.

I went ahead and did this, and while it was not as bad as I thought it would be, there was some pain involved. The biggest problem with moving to WCF was deploying the service on my server. It was a royal pain in the neck, and there were some aspects to not having a true application installer that caused me some issues (for instance, the installer I made for my original test application would create the Event Log sources used). After a few wasted hours trying to figure out what was wrong, I got the event sources made, and the WCF service was working. From there, it was a cinch to point Service Studio to the deployed Web service. I did find one little bug, which I am sure the technical team will fix very soon, and you can see the details of the problem on the message boards. Incidentally, this was my first time reaching out to the community via the message forum, and it was a great experience.

Figure B

Service Studio with the MongooseService Web Reference, and the referenced action front and center in my action's workflow.

So, what was the silver lining here? Scalability. You see, in the back of my head, I've been very afraid of what will happen if Rat Catcher really takes off. I have a front end that requires virtually no processing to be done, just a little bit of data validation, some user management, and that's about it. There is a little bit of management for the monthly reports as well, really just setting which jobs are included and the settings on those jobs. But the backend processing is where the real work gets done. That's where the algorithm gets called from. My fear has been that, with heavy loads, I'll need to start adding servers not for the day-to-day operations, but for the nightly report batches. Now that I have all the heavy lifting put into a WCF Web service, this is no longer a problem. If I find that the nightly reports are a problem, it is no issue at all to scale the WCF Web service out, or perhaps to move it to Microsoft's Azure service. At that time, I will need to modify the service to accept a batch of requests and send a batch of results back, but that will only be an hour or two of work.

While I now have the main processing algorithm being imported as a Web service directly in Service Studio, I still have some other minor utilities functions being used through Integration Studio. Right now, I had to fork my backend code, one branch for the .NET 4 processing algorithm, and one branch for the older 3.5 code that is being used by Rat Catcher through the Integration Studio extension. But as soon as the Agile Platform supports .NET 4, I will be taking that code in Integration Studio and re-referencing the newer code branch. I am perfectly happy with this, since the code is locked down and not going to change any time soon.

Figure C

This is a screenshot of what a user sees after entering a new document into the system.

Rat Catcher is almost ready for its public beta! I just need to test the scheduled report system. The UI needs to be improved a bit, and I am waiting on the graphics designer to get the logo to me before I change the color scheme.


Disclosure of Justin's industry affiliations: Justin James has a contract with Spiceworks to write product buying guides; he has a contract with OpenAmplify, which is owned by Hapax, to write a series of blogs, tutorials, and articles; and he has a contract with OutSystems to write articles, sample code, etc.


Get weekly development tips in your inbox Keep your developer skills sharp by signing up for TechRepublic's free Web Developer newsletter, delivered each Tuesday. Automatically subscribe today!


Justin James is the Lead Architect for Conigent.