Enterprise Software

Converting batch load processes into real-time processes

More on one college's efforts to drive inefficiency out of their systems and processes and to allow individual departments to focus more on providing services to our students and less on data management.

Earlier this summer, I wrote about a business process improvement project that we are beginning to undertake at Westminster College. Since that time, I've held meetings with departments on campus and am beginning to see a number of methods by which we, as a campus, can be more efficient. I'm not done yet meeting with departments and have yet to compile a formal report for the rest of the executive team. However, over the rest of the summer, I have learned a great deal that will help us along the way. Namely, we shouldn't limit ourselves to simply enhancing existing processes by making them electronic. Instead, we must think much more holistically and include a lot more buzzwords!

Kidding aside, besides undertaking this project, some other things took place this summer, making me realize that, in order to be successful, our business process improvement project must be expanded to include a service-oriented architecture element. Specifically, my database person left the organization. Bear in mind that my IT organization totals eight people, myself included. Over the years, we’ve accumulated a lot of responsibility for managing processes across campus. From initially loading incoming students into our dining and printing systems to running batch programs that handle student billing, our application support person/DB person has a hand in many critical functions. However, over the years, all we’ve managed to do is shift data management burden from individual offices into the centralized IT function. Sure, in the past, all of these processes were performed manually in each individual office and, with the batch programs now in place, that burden is much less, but has now shifted to IT. Like other departments on campus, we’re struggling to keep up with our regular workload and these manual processes don’t make it any easier for us to truly innovate to help people solve their problems.

Worse, even with the pseudo-automation in place, we eventually must revert to manual processing. For example, in our current scenario, once we’ve batch loaded our dining system, changes made after the initial load are handled manually and there are two systems involved. Our housing office makes changes to our primary student system and then, via email, notifies dining who makes the manual change in their own system. Obviously, in a number of ways, this is less than ideal. Besides being extra work, each time a person has to get involved in a process, the possibility for error increases.

To help solve the IT-time-crunch problem as well as the problem above, we’re going to begin to look at ways to convert these batch load processes into real-time processes. For example, instead of doing a batch load each summer, as soon as housing activates a student’s meal plan, that information should be automatically created in the dining system, which is a food service point of sale system. When a student makes a meal plan change, that information should also flow automatically to the dining system. Middleware needs to be developed to facilitate this inter-database communication. This development must precisely follow the human logic that would be taken for each type of change.

We simply need to develop processes and procedures that keep repetitive manual labor to a minimum. For example, when we get a donation, someone prints a letter, delivers it to the President, has it signed and then delivers it to the mail room. Why can’t an overnight process simply print letters on the printer in the President’s office to take some of the labor out of the process?

I suppose that this is a sort-of service oriented architecture project that will happen in conjunction with our business process review. Going hand in hand, these efforts are designed to drive inefficiency out of our systems and processes to allow individual departments to focus more on providing services to our students and less on data management.

Next week, I’ll share with you some preliminary thoughts on how our business intelligence/dashboard efforts fit into this mix as well.

About

Since 1994, Scott Lowe has been providing technology solutions to a variety of organizations. After spending 10 years in multiple CIO roles, Scott is now an independent consultant, blogger, author, owner of The 1610 Group, and a Senior IT Executive w...

4 comments
Fairbs
Fairbs

Can this concept be utilized in an 'older' Cobol environment where a change in one flat file type database can be propagated to other flat file type databases or to a more modern SQL db? Are middleware or SOA concepts part of the solution?

PonderousMan
PonderousMan

This topic is critical for Higher Ed, and it's great to see someone discussing what they are actually doing to address these sorts of issues. I work at a Big Ten university, and we see the same challenges ahead of us - just that the scale is a whole lot bigger, since a number of our "Colleges" (Engineering, Business, etc.) is likely the same size as (or bigger than) all of your school. As a result, even talking about how to move forward - getting the support and buy in from leaders across campus - is a huge task in itself. I look forward to hearing more detail about how you are going to move your batch processes to SOA, since that is a huge issue. I'm not convinced that all batch processes can be moved to a simple SOA model, and I'm worried that building "compound" services may not scale well - but we're so far from dealing with the practice side of this, I don't think we'll hit those roadblocks for some time.

PonderousMan
PonderousMan

The answer is yes, from what I've heard. Though our organization didn't follow this path (we still run an old school mainframe and send data files via SCP), we had some serious discussion of doing this sort of thing over ten years ago, and the tools for it have gotten better (and cheaper) since then. From the SOA/Middleware perspective, I think the big transition is to change your thinking from "Data feeds" to "services". So instead of sending a file of student info to Housing, Housing asks a "service" that provides student info (with proper security, authentication, etc.), and then gets the data. The Web Services model sort of expects this would be done student by student, but whether they payload is a single record or a batch of data is not really an issue for WS. As I understand it, there are a lot of tools out there that can offer a web service "front end" for everything from COBOLs to IMS queries. So you might start to look One last caveat - this all assumes that the customers - the "consumers of the services", in SOA-speak - are ready to ask for the data using Web Services. Since a lot of satellite systems tend to be written in much more recent tools, this is almost always doable - but if a small satellite IT shop (small dept, contractor, etc.) isn't ready to make WS calls, then you may not have any choice but to send them data the "old fashioned way". That said, many good SOA stacks allow you to actually call a service from an ESB or similar tool, and then send the result via SCP or some such to a target. In this case, the "requestor" is technically on the source end, but it still lets you move in the right direction. We've done this for a few "feeds", and it seems to work fairly well. Once you have the SOA/Middleware stuff up and running, then you can start to move either end - the source data or the customer requests - into a more "pure" SOA model, where the consumer makes a WS call, the ESB routes it, and a WS processes the request, gets the data using current tools (SQL, perl, etc.) and passes it back through the Web Service chain.

Scott Lowe
Scott Lowe

David, You're probably right in that not every batch load process can, or really should, be converted into a real-time SOA model. For some processes, the time and effort required to do so could be prohibitive and too complex. We basically need to look at each process, map the entire decision tree and replicate it in code. Complex on a good day! With less than 1,000 students at Westminster and you being a big 10, you're probably right that our scale is much smaller! However, I'd be willing to bet that our overall projects would have a whole lot of similarity. On the buy-in side, most of the executive team, including the president, is behind the effort and sees the benefit. Quite frankly, I believe that this type of effort is a gold mine for institutions like ours. Scott

Editor's Picks