Efficiency, Code Reuse and Artificial Intelligence — we sat down with one of the inventors of UML and the RUP to talk about how the software industry has to change in the next five years.
He's one of the fathers of modern software practices. In the late sixties while working at Ericsson he invented both sequence diagrams and use cases, and in later years worked on the SDL, UML and the RUP. We caught up with Dr Ivar Jacobson to hear his thoughts on where the industry is today, and where it will head in the future.
Builder AU: What do you think of the state of software engineering today?
Ivar Jacobson: What I see when I travel and talk to customers, participate in conferences and have discussions with experts around the world is that software development is very much still an immature discipline. We still rely on too much old work. Personally I am convinced we will change that dramatically, but it is a very slow process.
I have been working on process improvement and new technologies for many years now, starting with component based development and then adding to that object orientation and now aspect orientation. I've been involved with new technology since the sixties, and I've been more optimistic than most — it's my nature, but we are still struggling with basic stuff for many reasons. We will continue to struggle for a couple of decades more, I think.
We spend too much time on no brain work in this industry. There are measurements that we spend around 80 percent of our time not doing anything new, just following patterns of different kinds and that are well known, rather than actually creating something. Which is paradoxical really because the whole industry wants to be seen as creating something. Everyone wants to be viewed as the little genius but the fact is that 80 percent or more of the time they spend doing no brain work. That is something I've been working with for as long as I can remember to improve, but there are many reasons it is hard to fast forward here. But it will happen eventually.
What do you think is the most important thing to change?
To really automate and remove no brain work you need to find some identifiable patterns and apply these patterns over and over again with slightly different inputs — that makes it hard because it's not reusable code only. I think reusable code has increased, we definitely use more reusable code than we did 20 years ago — not as much as we could — but we use more. When it comes to patterns you need to have different parameters based upon the context. They're very context centred. So what programmers do is use patterns, but they have to add the context themselves. That is what makes it so slow.
I have since 1981 described a vision where we are assisted by intelligent agents. An intelligent agent that understands what you're doing in software. The only real difference between this kind of software and normal software, so to speak, is that its rule driven. You use rule technology to describe them, you can't really describe it in a reasonable way using pure Java. You can do it, but it's very, very costly. So you have to use some kind of rule technology to do it.
These rules trigger based on context, and then a pattern is applied. Theoretically that would remove the lost 80 percent. Data Consultancy Services (DCS), which is one of the big Indian companies, have applied technology that we've developed and they have cut their costs to 80 percent in two months for each project that applied it, but the potential is much higher than 20 percent. They also increased their quality, so they work with more consistency.
There is no reason to be creative in everything you do when you develop software. You should be creative in what problem you solve and in areas that are new, but as I said 80 percent at least in software is no brain work.
That's one of the things I think we need to change, I mentioned also software reuse, there is so much more to do here. The best way to improve productivity is to reuse things that already work. The problem here is that the industry doesn't really have any good platforms. We have too many platforms which prevent development of programs that can be used widely. I think we will someday see these things converge as well. Aspect oriented technology, which hasn't really taken off, helps up even today, but will help us even more when we get better support on different platforms.
Do you think then that using artificial intelligence is a way we can increase productivity?
Artificial intelligence is something that has been around for more than 50 years. In the 80s it was hugely hyped, not as big as the Internet, but it still had a lot of hype. With the Internet though, it's still there, it's still very useful, it's not dead in any way. It's the same thing with artificial intelligence. It was hyped, it was blown up like a balloon and it burst, but there are still valuable things in it and intelligent agents are one of these valuable things that are really practical. I used to say there is nothing as practical as a good theory. And a good theory is to use rules in some cases as they reduce the amount of code you need to write. That's all there is really.
The other thing when you talk about artificial intelligence is that a lot these ideas have been around for 25 years. A lot of things have happened since then, back then you had special machines to run expert systems — you don't have that now. Expert systems were still big software systems, there were no component types, intelligent agents were more like components. You had one agent for each thing you wanted to do, or for each responsibility you had. So an analyst could have one agent, a developer may have many agents, a tester others and so on — it was very nice componentisation. There are many things that have changed since then but the ideas are very similar to back in the '80s.
It's practical and it's used, we have used it ourselves since 2000. We have tried it in several large organisations, and it's a very slow process because it breaks so much, it's in conflict with so many things. First of all the real hard programmers don't want to think that the work they are doing can be done by an agent, so they resist right from the beginning. For companies like DCS though there really is no choice, they recruit 10,000 people a year and they need to get them to do some consistent work. How do you do that? There's no other solution — they know that and they've expressed it. So all these Indian companies that compete with us over low cost are now going to be much more productive as well.
Aspect oriented programming hasn't become popular among developers, why do you think this is?
There are basically two reasons, there is a conflict between programmers and people who develop systems. In the computer science world there is a conflict. The most outspoken computer scientists fight about whether aspects are good or not. Some of them say, and this is what I've heard since I started talking about these sort of things in 1978 at Ericsson, is that it's a cutting technology — you can create so much damage if you use aspect orientation. Technically you can create all kinds of patches that destroy software you have developed. That's a very programmer only perspective. If you have a methodology which starts from identifying a good aspect instead of just having a programming language you have a very different situation.
We have incredibly good results from using aspect orientation, but we don't talk about the programming language, we view the language as a facilitator, but you can still develop aspect orientated software without using AspectJ. It would be better if Microsoft or IBM had a better platform for development, but they don't do it, I guess because of the conflicts between the two camps in computer science: people who think it's devastating to use aspects, and people who think it's the best way to do. So long as they are fighting nobody really does anything.
Is there a camp inside computer science that is scared of new methodologies becoming used that will make the programming knowledge they have obsolete?
Definitely! Absolutely. Not in computer science, but in the programmer field. These people who don't really give much attention to using methodologies, people who believe more in people themselves and don't really believe in any particular methodology that can help them. They are scared of this sort of thing.
People who follow a methodology don't really produce anything. You need to have both, you need to have a very practical way of working. I don't like to use the words methodology or process any more, these words have become dirty, but you need a way of working and you need a way of working consistently in a team. If you have a large organisation you don't want different teams to work in different ways for no real reason, because it prevents you from reusing components. Even more importantly you want to reuse people in different places in the organisation — and if they use completely different vocabulary and terminology they will come in as a freshman. That is very expensive for companies, so they want to have as much commonality as possible. They can move people around, they can reuse not only software, but ideas, business solutions and so on.
The landscape has changed so much already, how do you think software development will have to change to adjust to future changes?
The way we have worked with software has changed quite dramatically, but not consistently. In some camps today we talk agile, today everyone is agile, if you go back 10 or 15 years nobody talked agile, but that was still what they were doing, it went without saying. So agile is really not a lot of new things but it's focus is put on people, on social engineering, rather than just technology.
Twenty years ago I remember that object management group had tried to categorise all the different object methodologies, in '92 or so, and they had 26 object orientated methodologies. I would like to call them "software engineering" methodologies. Today there is only one left, the Unified Process that we developed, which became the Rational Unified Process. It started in Sweden by the way, in my company Objectory in 1987, which was acquired by Rational — when it became the Unified Process. Then it was named, for some business reasons I never really liked, the Rational Unified Process. So that's another camp.
The third camp is CMMI, which is process improvement. This is something that Indian companies have been extremely successful in using, and it has given them a high status, because people believe that if you are on level five then you must build really good software — which is not necessarily true. Anyway, it is popular and there is something really good in it.
These three camps are really not compatible with one another, which means that you either belong to one or the other. But they solve completely different problems!
It's really amazing that they are three basically non overlapping types of methodologies, but you have to belong to one or the other and it's very hard to mix and match. That is one of the things I have been working a lot with the last two or three years. We believe that a branded process like RUP, or XP, is out — because such a thing is just a collection of practices. Instead practices will become first class citizens, and then you can mix and match practices from different camps, put them together into what you can call your own process. Process will be downplayed, we will all describe practices instead.
So we have developed a platform in which you can describe any kind of process you like, and then you can publish them, and they can come from any platform, not just RUP, which I always felt uncomfortable about. So that is the practice we have developed. We are giving this platform away by the way, on our website www.ivarjacobson.com.
We consider one of the damages we have done to the software industry is by branding processes. Every branded process has to grow, to be successful it has to steal ideas from every other successful process, then it adds and becomes bigger and bigger and bigger and eventually nobody can understand it. Instead, by talking about single practices, you can learn one at a time, you don't have to completely change how you are working, like RUP, you can just add one practice at a time. We have also completely redefined the unified process, instead IBM defines five practices and you can take one or two or all of them. We have also integrated these with practices from the agile world. Agile practices are about social engineering, you can take one or two of these as well and compose them with the software engineering practices and you have your process. So we definitely are downplaying the importance of defined process, we instead talk about practices and composing practices.
How has this been received by businesses that are already using a branded process?
We don't tell them to forget their process. They use RUP, for instance, or they use XP or SCRUM, or whatever. We ask them: "Where are your pain points?", and then we suggest practices that could help them. Then we take the practices that they have, and add in a few new practices. In these systems we then have to remove practices that otherwise would overlap.
In this framework we have developed for working with practices we have mechanisms that allow us to do that significantly easier. So we don't ask people to throw away what they have, not any more, I should say, we suggest they add things and maybe they have to remove something, small manageable things. It's a very different approach, and they love it.
Do you think that modelling is still so important?
Modelling has always been important and will always be important. I think the reason that it has not been 100 percent successful is that the tools have not been that good, they have been too heavy. This is a typical example of an industry that is immature, once you have developed tools and the tools look like they are good, people like them and so on, the vendor has very competent people — they know what's missing — so they start to develop a second generation tool. These next generation tools become too clumsy, and too hard to use, so you get a backlash. The tools that you can find from the major vendors today are on the one hand significantly better than the tools we had 10 years ago, but they are not as popular, because people want very simple tools. They don't want tools that require a lot of training to use.
The industry is making serious mistakes in strategy when it comes to tooling. We are now in the next generation of tooling again, I think we'll see tools that are very simple but can scale up, instead of being big and hard to learn and needing to scale down. We will see a new generation of tools, the tools today are very good if you take the effort required to learn them, but thats a lot of effort.
It's the same thing with process, we don't talk about big process anymore. One of the laws of nature, that we have known for 30 years or more but ignored, is that people don't read books! Not process books or language books. So why did we write big processes like the unified process? Well, that's because we really wanted to help people write better software. But that doesn't make sense because people don't read anyway. So in our new work we only focus on the essentials, we talk about the essential unified process.
The Essential Unified Process is described in less than 100 pages, whereas the unified process is described in several thousand pages. There is a reason of course, when I wrote what became the unified process, I could sell it and the price tag was dependent on the number of pages, I could sell it for $22,000 per head in the late '80s, but when I sold it they could do whatever they wanted, they could change the pages if they really wanted to. It's a little bit funny looking back on it, we think its very strange that people had that perspective at that time, but there definitely has been a change. We think that you should only document the essentials, the really really important things, and the rest you should either learn by training, or learn it by working with people who already know it, or, and this is the only way that I think is really systematic, you use intelligent agents.
How well does university education in software engineering prepare developers for the real world?
I had a meeting yesterday with Beijing university deans, at the school of software and microelectronics. I started off by saying that people in most universities are only teaching computer science and not really preparing people for developing software. The reason for that is that most of them really had never developed anything useful, how can they teach people do develop software? Then they described to me that in the US, and I guess its similar in Australia and the rest of the western world, the universities have a computer science department, and most of the professors have a computer science background and have never really developed software. They're very proud of their own mathematical and formal approach, so they don't really teach software engineering.
I've seen that very closely at the university I worked at myself, I worked at MIT in Boston for a whole year and got to know many of the most famous professors in the world at that time, and I did the same thing at the Royal Institute of Technology in Stockholm, the Chalmers institute of Technology in Gothenburg and Lund institute of Technology in Sweden. These are very good schools for software, but they have very limited understanding of commercial software out there, engineering you might say. One very famous professor told me he thought UML stands for "Undefined Modelling Language", and in some aspects I think he's right, but it really has been incredibly successful for many years in large companies.
With UML we just took the modelling people were already doing and cleaned up the mess, if it's not quite good, then maybe still it's the best. We had done modelling for 40 years in telecoms, and we're still doing it, so I have no doubt we'll increase the use of graphical languages. Graphical languages need clean semantics, however, and we don't have it yet. UML is pretty good, it's got much better semantics than any other modelling language, apart from SDL, which was developed in telecoms as well, but it's not good enough.
Once we clean up we'll do another round of that I'm sure, and again and again and again. I think the ideas will be very similar, some of the sub languages we used have been proven very, very successful, such as sequence diagrams for example, but there needs to be a clean up. Now Microsoft is promoting domain specific languages. That's another approach with a similar goal, namely to model. I think that that is very interesting. I don't want to say that it is the end result, because it's not, but it's a good step forward. I think we will see UML will also go through a similar process where we make it simpler, because it's too complicated, it's the same thing as RUP. So what we are doing is creating what we call the Essential Unified Modelling Language. The EUML is less than 20 percent of UML, but it will be enough for 80 percent of applications. This industry is moving very slowly to some better place.
Do you think the model of developing how we work on software, the processes or practices, is iterative in itself?
That's a big question. It's a very good question though. I think it's all down to the particular organisation, they should always iteratively adopt practices, they should never go for big bang changes, rather they should take one practice at a time. When we talk about us as a community, it happens all the time. We develop iteratively our understanding about how to develop, how to do use cases, how to do architecture. To say it's iterative is just to use the word "iterative" just a little too much, I think it's more like refactoring.
If you really look at ideas, the big ideas that we really use when we develop software, there are not so many, and they don't change dramatically. It's the way we present them, the way we adopt it and the way we combine things that change all the time. This is where we put a lot of effort in, instead of creating completely new ideas. There are new ideas coming though.
Take service oriented architecture, the world is full of these buzzwords. Service oriented architecture is a great thing! It's just not new at all, we just package it in a new way and call it something new. It's the same thing with Extreme Programming, but that's what we like, so we'll continue to do that. It's good stuff, very good stuff, it's just not new. It's new packaging.
Is there a problem with too much new packaging?
Yes, there is. When process becomes branded, and it's the next new thing that companies should buy then people run away from what they already have and move everything over to it. Then they find out that it doesn't solve all the problems that they had and that they need a new silver bullet. The software world is very immature, it still runs after these silver bullets, one after the other. We just need to slow down a little on that. It will happen by nature, but it could take another 10, 20, 30 years, who knows. I see progress, but it's slow.
Just take component based development, we used component based techniques at Ericsson in 1967, it took almost 30 years before component based development started to become mainstream. It really is the only way to develop software, but it took 25 years.