Software Development optimize

Developers need to lead the way in a business revolution

Developers, we are rapidly approaching a major inflection point (we may already even be there). The amount of code we write to pretend that legacy systems are not legacy is beyond too much.

The recent discussion about the programming paradigm needing an update spanned a number of topics (the discussion is so dense that it could probably make up a couple book chapters). One theme that kept coming up is that the current state of the art in business computing is horrid.

Developers, we are rapidly approaching a major inflection point (we may already even be there). The amount of code we write to pretend that legacy systems are not legacy is beyond too much. The rapid adoption of cheap multicore x64 CPUs in the server room and on the desktop means that parallel processing for CPU intensive tasks is no longer the realm of the Big Iron developers. All of the mainstream development environments have too many layers of abstraction between the underlying data and the application itself; this indicates that either the current data storage methods are inadequate or current development environments are inadequate. Too much code lacks proper validation because developers rely purely on the data type, which is completely unrelated to the business logic. We're constantly choosing between a code generator that writes garbage code or missing deadlines because their languages require 5 LOC to write an accessor -- and that is without any validation or data constraints.

This is going to collapse under its own weight.

The more I work with on interesting projects, the more the cracks are showing. The languages that are well suited to really advanced computing techniques are not well suited to business computing for a variety of reasons. Some of the reasons are technical, some of them are not, and most of them are valid. After all, a language that supports lazy evaluation or set theory or is based purely on lambda calculus is not likely to be very well suited for the typical serial data transformations that most business computing really is.

The issue comes back to something that Alan Kay first made me aware of in one of his papers: the history of computing. In a nutshell, current business computing is the direct descendant of the data processing or information processing folks. These are the people for whom the computer was to primarily replace the accountants and record keepers and other desk jobs that amounted to paper shuffling. In the discussion about the programming paradigm, Chad Perrin (Apotheon) mentioned that most "applications" are really a database browser with a customized interface. This is more or less correct.

Most folks who spec out software development projects cannot see past this. Rather than asking, "How can I do a better job?" they are wondering, "How can I get out of performing these tedious, repetitive tasks?" There are two major reasons why this happens. First, it's a lot less challenging (i.e., you examine what you do now, find the parts that the computer can easily do, and have the programmers code those parts). It's tough to actually analyze your business process. The other reason is people would feel threatened if they did anything else.

Think about it: When writing the specs for a tool they use, no one is going to spec themselves out of a job; they are going to make that job easier. The results are software designs that handle the tedious parts of a bad process instead of enabling a better process. The users leverage this to do more of the bad process instead of freeing themselves completely to learn a new job.

Nothing will change until developers show the business folks that we are capable of revolutionizing their business and not merely automating it -- that is the IT world's contribution to this dilemma. Instead of playing the part of an active partner, we act like a doormat. When we transform that business process into code and say to ourselves, "Gee, this looks like a really bad process," why are we only saying that to ourselves and not the BAs, SMEs, and users? Why are we not offering more than what is requested? For example, "OK, I think I understand the spec" can become "I understand the spec, but I think I can add some value here too, and maybe make this 'smarter' as well as 'automatic.'"

This is a difficult but needed transition. As much as I am down on agile development methodologies, they are well intentioned, and my hat is off to them for trying to do something about it. After all, the goal of agile development is to be a much more responsive and fully fledged member of the IT-user partnership, and to encourage the user to better leverage the development team. Most of my reasons for not being pro agile development are cultural; it takes a very special development team and some very special customers to make it work.

This speaks volumes about the current business environment. Remember that many of our current business leaders were junior clerks at companies run by people like Robert McNamara. His management techniques ("systems analysis") worked very well in many instances. The role for computing in such an environment was to replace the filing cabinet with a computer program, to provide humans with the aggregated information needed to make an informed decision about a complex subject. Does this sound familiar?

Developers have two choices

Option #1: We can proceed down the current course, which is to allow businesses to continue running on 80-year-old principles despite having the technology to create new management principles. This course will lead us towards ever complex systems to handle the exponential increase in data volume (ironically, without any correlated increase in data significance). The abstractions needed to make this work are also coming at an exponentially fast pace. Look at how long COBOL ruled and then was supplanted by SQL. But SQL was not enough by itself, so we added ODBC to cover it up, then systems like DBM, ADO/ADO.NET, etc. on top of that, then ORM systems, "typed datasets", and now layers upon layers to hide that. At the current pace, we should be seeing a new data abstraction layer every six months in a year or two. Option #2: We can reject the current course and lead the way out of this mess. Organizations that are jettisoning this mindset are reaping rewards. Companies that programmed the business model into the system have freed their leadership to spend their time keeping an eye on the ever-shifting business landscape and not with their eyes glued to some "data dashboard" with their finger "on the pulse of the organization" waiting for its heart to stop. These smart companies are doing well.

What's your take?

Would you prefer to: allow leadership to figure out when the patient is about to die, or help leadership keep their eyes on the road so the wreck doesn't occur in the first place? I opt for the latter.

J.Ja

About

Justin James is the Lead Architect for Conigent.

41 comments
logistic
logistic

Hi, please revolutionne only things who does not works in a very good manner ... STOP to develop all kind of gadgets. I have a lot of very interesting revolutions in my "boxes" .. but I see that there is not any person who is open for the really revolutions and improvements. But it seems to be simple to revolutionne procedures with fake gadgets and fake improvisations. Excemple: Credit card processing does works very, very fine and in a very secure manner ouver Internet if we do not change the basic procedures, instead of an adaptation of these basic procedures. But if we change the basic procedures and replace these procedures by very crazy procedures (and this is the case in 90% of the Internet credit card processing) then we produce false systems and we shut down the interest of the Internet.

Absolutely
Absolutely

Another separate thread that's pertinent to the questions Justin raises here is "Ease of use - but for whom?" http://blogs.techrepublic.com.com/opensource/?p=111 Expanding the Linux v. Windows stereotypes discussed there to the inclusive Tech v. End-(L)User stereotypes hinted at here, what business really needs is to stop accommodating morons with no productive skills beyond typing, and mangling a spreadsheet if heavily assisted. Apps like MS Excel are written from a condescending assumption, that the end user is too stupid to understand how its tools work and needs all the logic hidden behind a colorful facade. There's your "bottleneck", and the source of all the business processes that had to be designed "broken". Fix that.

Justin James
Justin James

Beleive it or not, Excel (as do the rest of the office apps) expose most underlying functionality to the end user than just about any other app I can think of. I am sure that OpenOffice does the same. Even the most basic of users can put together a dinky macro, and from within the macro system, every single piece of functionality that the application has is directly exposed to the end user to manipulate as they see fit. Furthermore, Microsoft has put out the VSTO package, so that programmers may work with the Office stuff in the language of their choice. So I think that Excel was actually a fairly bad choice of examples. I do agree 100% with your point, by the way, in that I feel that most apps should be exposing a lot more functionality, particularly for "power users", but Excel was just a bad example. :) J.Ja

Absolutely
Absolutely

If ( example < optimal && point == scored ) {printf ("absolutely (in Cartman voice): Sweeeeeet!") } Plain English: I agree that Excel is not the quintessential example of what I described, but I won't go so far as to agree that it's "bad", either. I think a lot of "average folk", ie entry-level office workers, could understand, and effectively use, SQL with less training than their taskmasters, on average, estimate. A dedicated language for "spreadsheets", ie databases [i]sans[/i] many-to-one capabilities, would be silly. Instead, smart businesses could just teach Power Users a subset or an analog of SQL, to be used within purely "rectangular" datasets where spreadsheets are useful. This idea just occurred to me, and I was dreaming when I typed this, so sue Prince if I go astray. ;)

Justin James
Justin James

A huge fear of mine is data security. How many millions of indentities at this point have been lost, possible discovered by the wrong people, because someone with no clue loaded the data onto their laptop to work from home or on the go (just like in the ads, when we all work from the beach or the park bench!) and they lose the laptop? Stuff like that scares the pants off of me, it can sink million dollar businesses in an instant. Some doof with a copy of Access demands a raw dump from IT so they can run a report, takes it home, then someone does a "snatch 'n grab" when they see the laptop in the backseat of the car. And of course, the file is right on the Windows desktop, labeled "SSNs for HR". Argh. It would take an extremely aggressive campaign of data sanitization to give this to users. Probably more effort than it is worth in most cases. J.Ja

Absolutely
Absolutely

Maybe user privileges should only include limited views of mission-critical servers, expanding based not only on their job title, but also on demonstrated ability to not accidentally jam the server with a giant Cartesian product. A recent thread about phishing of salesforce.com got me thinking about the levels of access to valuable data that "Joe User" has in a lot of corporations. Creepy.

Justin James
Justin James

In fact, what you end up seeing is a lot of business users using Excel as an ad-hoc database, because no one showed them Access or gave them enough rights to the SQL server. It requires a very safe "Sandbox" for those users to work in, though, so they can't accidentally jam the server with a giant Cartesian product, or lock a table while running a 3 hour query, preventing anyone else from using it. J.Ja

jslarochelle
jslarochelle

While I certainly agree about the general idea I think that doing it in practice will be very difficult. It`s certainly worth trying but I don`t see this happening as a revolution. Inertia, insecurity and other psychological factors will make the whole enterprise slow and difficult. Those who want to tackle this will have to be patient and keep at it. JS

Justin James
Justin James

I agree that trying to change people and organizations is difficult. That's why it is so rare to see a company truly turnaround, and it is why marketplaces are so cyclical. A bunch of mature players suddenly get outfixed by a young, nimble startup, a few of the Old Guard go under or dissappear in some M/A action, then the startup becomes the juggernaut on "cruise control". Success almost always turns the small, nimble companies into juggernauts, or kills them. It is indeed possible to drown in success. I think most of the organizations that are able to wed IT to the business are most likely going to be startups, and I suspect that (at least for a while), it is only the first few people hired who get it right... because as you expand, and have to hire more people, it is unlikely that everyone you hire "gets" your culture. In fact, if they worked anywhere else, chances are, they *don't* "get" your culture. That is the paradox of success. :) J.Ja

aureolin
aureolin

You're missing a piece of the pie. There's a triumvirate needed to do what you're asking: 1) The Customer or business person 2) The Developer who's going to implement the changes 3) The Business Analyst who's going to help person 1 understand the changes that need to be made to the process, and person 2 how to make it happen. Without person 3, the business analyst, you're just going to wind up with programs that "replace the filing cabinet".

Tony Hopkinson
Tony Hopkinson

The problem is information flow. Leave the developer at implement and you end up with a program that looks like a filing cabinet as well as replaces it. Developers should not just implement, most business analysts base what they ask for on what they know can be implemented. Given who does what, the disconnect is obvious. How many times have you seen an attempt to do a web application that requires rich and stateful for instance? Why is it a web application. There couild be a good customer based reason, did they understand the business consequences behind that constraint though. If they did, why did they end up with a piece of crap delivered over budget, and over time? It can all be sorted out all it takes is for those involved in the process to communicate with each other instead of at. All it takes. ROFL

Justin James
Justin James

I didn't necessarily forget the BAs... I just think many of them are completely ineffective, and understand neither "business" nor "analysis". I have seen too many specs produced by a BA that are filled with useless "use cases" like "user clicks the button marked 'OK'" (hello? no kidding!). Now, if BAs were doing things like constructing useful prototypes, providing input into the process itself and acting as true *business* *analysts* (with emphasis on both words!) they are indeed a great addition. J.Ja

Tig2
Tig2

If you want a BA that understands business, you have to hire for that skill set. As a project manager, I have discovered that I fit a hundred job descriptions, including BA. No one seems to be able to decide what it is that a PM or BA actually does so finding qualified people is tough. When I write a requirement, I want to also define what business need will be met when that requirement is met. So "ability to present demographic data in a weekly report" has to be coupled with "satisfying marketing need to see shifts in the demographic population being served, thus allowing for trend analysis and greater flexibility". If you want that level of granularity (I do), you have to require it and hire it.

Justin James
Justin James

You could not be more right... the PM and BA skill sets are really poorly defined, so hiring for them is pretty tricky. And from what I can tell, most companies want a PM to really be "I/O system for a Gantt chart" and a BA to "write use cases and document what the user thinks they want." Neither of those are particularly helpful on a software development project. Gantt charts are only as good as the information put into them, and we all know that for development, it is pure guesswork. And just about anyone can listen to the users talk about what they want; it takes a special person to use that as a starting point instead of the expected ending point. J.Ja

Tony Hopkinson
Tony Hopkinson

and works with development to achieve a solution is indeed a godsend. One that knows the domain and effectively provides a solution, a ginormous pain in the arse. A BA that only knows how to document a use case but can't get to what it means, is a screw up waiting to happen. You've seen the ones who spend half your budget coming up with a design that couldn't or shouldn't be implemented. If the project is being run correctly they should be pulled up short after about a day.

rkoontz
rkoontz

Management Needs To Listen To Us First!

Justin James
Justin James

... just listen with an open mind. :) J.Ja

SnoopDougEDoug
SnoopDougEDoug

Unfortunately in many (most?) companies there are the IT folks and their are the SMEs (subject matter experts). In most cases management sees little value in either group learning about the other, much to the detriment of every IT project. SMEs should teach devs about the business process and devs should teach SMEs about the development process. Without cross-fertilization we will always have apps that do exactly what the SMEs *think* is best, but users hate.

Justin James
Justin James

Doug - That is spot on. I too have seen that the SMEs seem to be totally divorced from the "on the ground" reality of the people doing the work, then give the programmer something not in line with reality. The programmer realizes it is a mistake the moment they deal with an end user, but all too often, it is way too late. An even worse SME is one who can quote the current process to you with no understanding of *why* that process exists or why it is the way it is. A great example for me is some work I was doing a while ago, I was working to put into code a paper government form with strict legal requirements. The SME knew how to fill out the form, and had all sorts of (quite good) suggestions to make the Web version much better. Unfortunately, she didn't see the legal requirement that said that no changes to the form whatsoever were allowed when making a computer version of it. Luckily, I caught that requirement before coding started, but only because I was reviewing the legal documents to see if there was anything in there regarding technical requirements. J.Ja

AMaldonado
AMaldonado

I definitely opt for the latter, and have seen mild improvements over the last few years in some of the organizations I've worked with. It does seem that the culture of the company has to embrace technology for what it can be instead of what it has been in the past in order for opt 2 to succeed. Companies that have implemented CRM tools and other workflow software use their data to implement more process centric business models there by freeing up human capital for innovation. I think where I've seen the successes are where the company leadership has a strong understanding and focus on IT innovation, and where the company's business model has embedded IT into every aspect of it. Companies like this have people with roles such as Business Architects, Marketing Analysts that come from IT backgrounds, and programmer/analysts that have strong business skills. Aligning Business and IT into one entity in every aspect of the company will allow the cultural shift to become a reality and IT to provide more innovative next generation business concepts.

Justin James
Justin James

I think in the future, many business people will have some sort of IT background, and know much better how to leverage it. Another thing I look forwards to is that as more non-IT managers get IT exposure, they'll lose that "Gee, wow! I sent an email!" and get to "OK, so the system sends email, now tell me, how will this actually improve the business?" I have seen too many managers buy into something rediculous because they were awed by some dinky feature. I think a little "familiarity breeds contempt" could be a good thing, especially if it breeds skepticism and critical thinking, not contempt. J.Ja

Tony Hopkinson
Tony Hopkinson

Some technical types when they come up with a product. I feel much of the time they are sticking to what they know and missing out on all sorts of opportunities. It's often missed how much the way Product Manager's and BA's word the requirement constrains development. That new concept, or lateral shift just won't happen and you end up with same sh** different technology and try to sell it as progress. The other thing they do is limit what they ask based on what they think is feasible, sometimes they don't even mention it. Not even the cleverest design team can deal with that. All this talk about everything on the table, a new way, blah blah might be happening at these strategy meetings. All I see though is two buttons, a caption and yet another f'ing grid. What we need to do is get involved early, clue them in on possible and not possible, stop them throwing good ideas away, because they don't know how to implement them. The latter is our job, but it's not going to happen if we don't see them.

Mike Page
Mike Page

The job of a developer who is an employee is to build systems to management's specifications. The job of a consultant developer is to take a critical look at the specifications and offer management better options should they exist. If all developers had this mindset then they would add more value to the business. They would be more on the level of valued collaborators with management rather than grunts that work in the coal mines.

Justin James
Justin James

I think what you are saying is really quite similar to what I meant when I said that developers need to start acting as active partners, not doormats. A partner makes suggestions for improvement, and injects themselves into the process. A doormat sits around and waits to be invited to the party, then acts like Eeyore when they aren't. :) J.Ja

Tig2
Tig2

Go out for those "smoke" breaks. Take a cup of coffee with you. No one says you have to smoke. Those 5 minutes ARE important. So pace. Drink your coffee. Stand on your head. But go out there and get that face time. My dad used to play with his pens- areo-space design engineer back in the day when they actually drew the stuff- but when he quit smoking, he realized the same thing you are finding. So he took his time outside and brought a troublesome pen with him. He took his "smoke" breaks to fix his pens.

Justin James
Justin James

Sheesh Tony, I thought you were better than this... do it at the company Holiday Party, duh! They will all be in the same room, and *drinking* too, so it is a slam dunk! All joking aside, I quite smoking a few months back, and I discovered that as a result, my visibility to management dropped significantly. The 5 minute quick chat that others now have with the bosses on smoke breaks is working against a lot of the things I pushed for on those 5 minute smoke breaks. I actually said that to one of the managers, "sorry I quit smoking, now I have no idea what is going on." Not that I would ever recommend taking up smoking as a way of getting in front of management, but I am pretty surprised at how important it was to work just to get those 5 minutes of undivided attention every few days. J.Ja

Tony Hopkinson
Tony Hopkinson

all the medium bosses, who hide behind the smaller bosses isn't he? At the moment I'm having to explain up the chain of command, and I'm getting zero feedback on why, if and whether the idea good or bad went anywhere. There's a lot of tradition bound types above me, I have to get at each one individually as they infected with new = different = bad. That's what I want the forum for, so I can get them all together. All I want is to get them to accept the possibility of change, it's turning out to be, well difficult.

Justin James
Justin James

If you essentially gate crash, yeah, you get ignored. But there are ways to inject yourself into a process and not be ignored. The trick is to show the value to the Big Boss and have them formally invite you in. Besides, if the Big Boss doesn't see the value once it has been explained, nothing will go anywhere anyways. :) J.Ja

Tony Hopkinson
Tony Hopkinson

You can gate crash the party, you'll look like a twat, and no one will listen to you... The difference is the party is being held for the consultant and everybody better have a real good time because these chicken drumsticks are the gourmet variety.

Tony Hopkinson
Tony Hopkinson

What if they are specifying drivel ? Do you know why consultants get listened to ? Because you have justify the cost of bringing them in. Whether they come out with complete bollocks, or simply dress up an idea we came up with is immaterial, as a manager you do not want to explain why you spent all that money for nothing. I should know I was a consultant developer for six years, it's one of teh reasons I got the job. Collaboration is two way street, if all I'm here for is to batter out code for dumb ass requests, how is it going to happen. I'm begging for a forum, not drive, ambition or knowse.

jtech100
jtech100

"Consultant" is a very vague term. Also, I don't think that most developers are qualified or experienced enough to make suggestions to "revolutionize business processes". The folks with expertise in Business Analysis or with actual experience in the business processes may be more qualified.

Absolutely
Absolutely

[i]OK, point taken! But I was thinking of the inexperienced ones - straight out of university.[/i] Can you, or could you when you were straight out of university, understand dollars & cents? Profit margins & expenses? So can I. And when I write a program to handle the related data, yes, I can contribute to improving, not just automating, the business processes. I don't need a business degree for that because like most decent programmers, engineers & scientists I learned the relevant mathematics [u]before[/u] I entered university. $ is just another unit of measure, whose meaning is much less subtle than a Watt or a unit of Torque.

jtech100
jtech100

OK, point taken! But I was thinking of the inexperienced ones - straight out of university.

Tony Hopkinson
Tony Hopkinson

doing us down, they don't need help. I've been developing for twenty years, so I probably am 'qualified'. Certainly I'm experienced enough to have seen something somewhere somewhat like what's needed. But that's by the by, what I was alluding to was a forum to throw some ideas in the pot, that the business could use, not tell them how to do it. We are making money so they must have that bit down, I can see how some things might make them more if they knew about them. Big yourself up, no one will do it for you.

Mike Page
Mike Page

A developer should take the time to understand the business processes being modeled and try to determine if they are could be improved. Blindly implementing requirements without understanding the underlying processes increases risk that the system will cumbersome, inefficient, or a failure. Work as a partner with your stakeholders as opposed to a servant, and you will be highly valued.

duc.rider
duc.rider

I certainly agree with the idea of IT's role in business. I have seen this work first hand. Like when you show an exec some cool (and useful) product you've built and they start talking about to everyone else. It's the kind of stuff that makes me do what I do. But in the end of the article, if I reject option #1, what does option #2 leave me to work with? I'm not sure what it is you are proposing.

rclark
rclark

Back 10 years or so ago, we were drowning in a sea of requirements that did not add value but did add automation and complexity. At that time, DP was king and we basically told the user what they needed and how it would be implemented. That led to computer slaves and did not make us popular with the user base, even though it sometimes saved the company a lot of money. We pleased CLevel and alienated the rank and file. Then we went too far the other way and allowed the rank and file to buy what ever they said they needed to get the job done. Productivity was king. Bottom line was ROI, regardless of whether it was sustainable. Both of these over time have been shown to be unworkable. Like many other business rules, we now work under a new paradigm that says you tell me what, I'll tell you how, and the Clevel will say when and how much. Everyone maintains control in their specific area of expertise, and we don't build solutions that solve nothing, we don't waste money on projects that can't be completed, and the rank and file are invested from project start. All other models are us vs them, and that is a handicap that is very difficult to overcome, regardless of CLevel directions. The model I like best is for the user to ask for a solution, the IT people to design a solution, the CLevel to approve the solution and the whole team to implement. Much more flexible. The funny thing is that most projects are a lot less complex once you get all the egos out of the way and just roll up your sleeves and study the problem. And in this model, IT doesn't have to be salesmen.

techrepublic
techrepublic

Please explain what you mean by "CLevel". This jargon is not industry standard. Can't we just all speak and write in plain English?

rclark
rclark

Most companies have approval and capital acquisition models that have purchase authority for large projects at Board of Director level. The Board of Directors often give limited approval authority, or at least gateway authority to the executive management. Executive management is often made of Senior President/Vice Presidents. In tech terms the C-Level executives. While they often don't have the authroity to approve major capital projects, they do have the requirement for planning and bringing to the Board of Directors a plan that the Board will entertain and perhaps approve. When everything lines up, you have a project plan. My point mainly was that 10 years ago, we forced projects through that process. We were the initiators and the users lived with what we pushed, regardless of their wants or needs. Bottom line return on investment. If we could make the company money by putting a software package in we went for it. The problem with that is that there are seldom and cut and dried solutions that work regardless of user activity. So projects failed when users would not support the solution. So then we went to another model that basically catered to the wants and needs of users. Their productivity went up, but the effect on the bottom line was horrendous. Sure you can make a worker more productive. But if you spend twice the capital to do it, you negate the advantage of the productivity. So now we are in a model of the user asking for help, IT designing a solution and presenting it to the executive decision makers, then the user selling the executives on the need. We are out of the loop except for designing the solution and implementing it. We are once more techs, and not king makers and power brokers. At this point, it pretty much is a lot less stressful, we have happier users, management is getting it's monies worth in software purchased, and the Board doesn't get nickeled and dimed to death with small projects that have limited usefulness.

Justin James
Justin James

CLevel... CEO, CIO, CFO, etc. It's quite common to call that layer of management "CLevel" (or "C level" or "C-level"), to be honest, but it usually is followed by the word "executives" (like "C Level executives") for clarity. Hope that helps! J.Ja