It's common for many programmers to be suspicious of modelling, but it doesn't have to be that way. We talk to IBM's Scott Ambler about how you can use agile modelling for any project, and why you probably already are, without realising it.
Builder AU: What does agile modelling mean to developers in the Web 2.0 world?
Scott Ambler: The goal of agile modelling is to describe a collection of principles and practices for modelling and documentation, preferably on agile projects, but if they're not so agile then that's OK too. What we've seen is that the major use was in XP [extreme programming] to make modern documentation more explicit, or with the RUP (Rational Unified Process) to sort of ramp down some of the bureaucracies and make it as streamlined as possible.
It just describes ways for you to be effective with thinking through some of the things you're doing without taking a hit from needless documentation. From an agile point of view it gives some explicit strategies for how to protect yourself from the bureaucrats who want you to do way too much documentation, and provides some advice to govern some of the things you're doing.
Some of the more extreme rhetoric in the agile community really motivates some people to do the wrong thing, not picking on agile fans, but that's probably the wrong way.
"Hey, I did all this modelling and it didn't have any affect on the product whatsoever, what a waste!"
On the perception of modelling.
What do you think the general opinion from programmers is on modelling in business?
I think that a lot of programmers struggle with modelling, for a lot of reasons.
First, they haven't got good training in it. I don't think that schools do modelling justice at all, they might never for all I know, but they certainly don't do it well now.
A lot of the time developers will show up to their first job and the first time they do modelling they'll almost always be put into one of two dysfunctional situations. Either they'll be put into a project team where you're given all this modelling up front, and then you ignore it towards the backend. So they see all this wasted effort around modelling documentation and then they say "Hey, I did all this modelling and it didn't have any effect on the product whatsoever, what a waste!" And so they get bitter about modelling.
Or worse yet, they'll do the work, they'll successfully deploy the project into production and then someone will come along and say: "Well, now we need to spend the next two months writing all the documentation we should have to make it look like we followed the process." That is a complete waste of effort, that's just somebody justifying their job, it has nothing to do with delivering value. A lot of developers get bitter about this sort of stuff.
Another common issue is that they struggle to distinguish between modelling and documentation. If I'm doing a sketch on a whiteboard then that's a model, but it's not a very good document. What will happen — and in a way the fault lies with the vendors because we want to sell CASE tools — is that we try to convince developers that modelling has to be done with these heavy tools.
Well no, not all of it — and let's just observe the fact that a lot of modelling will happen on whiteboard and it will happen on paper, and that's perfectly fine. When you want to get more sophisticated though, you need more sophisticated tools. For example, I've got pretty good modelling skills, so I'm far more effective using a tool like RSA (Rational Software Architect) or RSM (Rational Software Modeller) to do my modelling, and then generate my code and generate my database stuff than I am "hand-jamming" it.
It's just crazy writing code when I can generate it — I'm assuming that the tools generate decent code here. The problem is though, that it requires a heck of a lot of skill — if you don't have that skill, and haven't taken the time to get it or been in a situation where you can work with someone who does it, then that's pretty rough. A lot of developers are finding that given a choice they'd do more modelling, but they haven't had the opportunity to learn it well.
We definitely need to distinguish between models and documents, they're totally different concepts, and while some models will evolve into documentation, many will not and that's perfectly fine.
How strong is the connection between modelling and documentation?
It's very strong in the traditional world, which is why we're all so hung up on it, but really it's not that strong at all. Ninety percent of modelling is done on the whiteboard and in a recent survey we did, modelling turned out to be the fourth most effective practice being done on teams. Paper modelling was not that popular, for all the stories about User Stories and CRC cards and whatever, which is what I like to see.
The vast majority of it is done in these throwaway kind of models. But you can make a very strong argument that writing a test before you start writing your code is actually modelling, you're specifying your program before you start writing it. Just because I'm not doing a UML diagram doesn't mean I'm not modelling.
Most models will not evolve into documentation, some will, but a common thing you'll see in an agile team is whiteboards everywhere and people drawing models and so on, and over time you'll see them evolve. What will happen when you get down to writing documentation is that the models that survived on the boards are the useful ones, the ones you put into your tools and you write-up and make look good and so on.
On the educational side, what do you think needs to be done to fix this situation?
Well I think universities struggle with a couple of things: Firstly they just don't have near the funding they need, and they never will, that's just the way it is. They also, for whatever reason, tend to shy away from group work. Modelling is a group activity and you need multiple people to do it, and you need to work together. If you are doing assignments and you are asking individuals to sketch stuff, then that's great, but they'd probably just as well be banging out code. They also chop things up into different courses, so you get the Java course, the database course, the mathematical theory course, so what happens is that the focus is just on numerical programming or whatever and they never bring the full lifecycle in.
The other thing is that they don't do projects, they'll do problem sets or they'll give you an assignment to go off and build something, but they don't say "Let's work on this system for the next two years of your degree" and so for two years they're working on different aspects. You're not getting any real world experience.
I'm not saying that it's easy to do, but these are the things that they need to start doing. I was working at the University of Toronto years ago, and one of the things we did, and it was really hard to do, was in this group work course we told students that they'd be developing a system all the way, then in the middle of the way through we yanked all their stuff and replaced it with work that had been done in previous years and said "Okay, now you're maintaining a legacy system, what are you going to do now?"
They were shocked, there was all this whining and complaining about how evil we were, but that's the real world. In the real world you have to maintain someone else's code. I've run into people years later and they've said that it was the only course that was actually useful to them, and it's because that's what's happens, that's real. You need to simulate things like that, and it's hard.
I want to know why can't a third year computer science course just take on an open source project, why couldn't the assignment be that these three people work on some project they want and they have to show evidence that they added a feature that somebody was actually interested in — they designed, implemented, tested and delivered it. That would be the entire work. That's an easy thing to do and it would be much more useful.
Just because I'm not doing a UML diagram doesn't mean I'm not modelling
How important to do you think agile development processes become when you've got a more globalised development model?
It's critical for a couple reasons. First, people are doing it, so for people doing offshoring, why can't the offshore team work in an agile manner? Nothing's stopping them. Why can't the onshore team work in an agile manner? Nothing's stopping them either.
So the issue becomes how you communicate between those teams, and you have to be smart about that. How do you organise your teams in such a way that you can still be effective when your team's on the other side of the planet? It's never going to be totally easy, but you can be smart about it.
Give as much responsibility as possible to each location. So if I've got a sub-system that I'm offshoring, then offshore almost all of it. If the onshore team does the requirements and the architecture and then they hand that off to offshore team, you've just sucked all the life out of the project. If you've done all the big thinking onshore, you're trying to minimise risk, but you've actually increased it.
If I hand an architecture document or a detailed requirements document to the offshore team and say "build this", well, what are you going to do if you're the offshore team? You'll put your junior staff on it because it's all laid out for you — this is what happens, anyone senior isn't going to want to do that work, you're just a coding monkey at that point.
You need to give as much responsibility as possible — I've actually seen some very successful teams where they flew their business people to India and that was it, you had a project manager coordinating on the other end, but that was all there was. It was a lot less risky and a lot less expensive in the long run. This is actually a very interesting way of working, but you've got to have a heck of a lot of trust in the people working on it.
What would you say to a developer that is dismissive of agile modelling as being unnecessary?
Well first of all, if you're doing agile [software development] you're really doing agile modelling without realising it.
So I start asking questions like, "Are you doing any whiteboarding?" and the answer's yes. It's hilarious, if you go and look at an XP team in operation there's always a whiteboard in the background with diagrams on it, or a corkboard with index cards, and those are both agile models.
So they're doing this modelling stuff but they're not recognising it as modelling. The planning game in XP — that's a modelling activity, right? So if you're doing a one month iteration and you're spending the first half day or day planning out what you're doing for the next month then a lot of that is going to be modelling, you've got a bunch of people together in a room, you're talking it out, filling out your cards and going to the whiteboard, and drawing, sketching and arguing and so on. That's modelling. So they're doing it, they're just not calling it modelling.
If a developer is dismissive about agile modelling I would have to question whether they're doing a good job
What happens as a result is that they talk to management, and they talk to other senior developers, and since they don't understand and can't communicate what they're actually doing then they're dismissed by senior management.
Management thinks "Oh they aren't doing any modelling, you low-end scumbag hacks". This is what they're thinking because in the '70s and '80s they saw this sort of thing happening by the code fixers and now what's happening is many of these extremists seem bound to the traditionals like code fixers and so they're written off — they're not the same, but it sounds like it.
So for all the talk about communication in the agile community we really struggle sometimes to communicate what we're doing, and this is a critical thing. If you're scaling, if you've got any sort of complexity to deal with, if you're offshoring, then you're going to be modelling, you are going to be documenting. Even if you're not modelling, you'll still be documenting because nobody can deliver a system without documentation. That's just crazy right, you need user docs, support docs, system docs, it's just the way it is. I would invite you to be really smart about that, and that's what agile modelling is all about, being smart about it.
If a developer is dismissive about agile modelling I would have to question whether they're doing a good job, and I can't imagine not modelling, and still doing a good job.
A lot of developers are not worried by the idea of whiteboarding, but rather restricting it to a set of rules ...
Yeah, but the thing is what team is not doing some kind of modelling? I mean, just because it's on a whiteboard, or it's in RSA or it's in Visio or whatever tool you're using — that's not a commitment to overbuilding.
It doesn't mean it's got to go off to fantasy land and build all these useless frameworks for months and months and not deliver business value. What's wrong with these people who have no control over themselves to not overbuild? So you can choose to gain the benefits of modelling without taking the hit of overbuilding or over documenting. And it's a reasonable choice to make, most teams do it, they just don't know how to describe it.
Do you subscribe to the idea that code generated products are generally of a lower quality than hand written projects?
It all depends on the tools. So for example I would challenge anyone who does any kind of database development to write everything by hand. Use any of the leading data modelling tools and they generate great DDL, they generate great triggers. If you're hand jamming DDL, what planet are you on? If you have to handwrite a trigger, what the heck is going on? What a waste of time, that's just crazy. It was interesting, when I was writing my database book I had to relearn DDL, because it's been years since I've written it. I just load up a CASE tool, model what I want press the button and boom, out pops the code. Why would I write that?
On the Java side of things, why would I write class stubs, or constructors, or all this code for doing OR mapping and stuff like that? I can't imagine writing that level of code anymore. Sure, there's some tools that generate not so good code, you've just got to find the right tools.
Beyond mechanical triggers and the like, there's always been a move to abstract out the programmer from the whole process.
I've heard this vision for over 20 years now. It was crap then and it's crap now.
At the end of the day you can't do that, you need such a high level of skill to make that work, you need such good tools, and at the end of the day if I've got those skills I can still code it faster.
What you really want to do is visually model the things that make sense to visually model, code the stuff that makes sense to code and have tools that allow both and do the right thing at the right time.
That's still not easy, but it's realistic. You're never really going to have a 100% modelling site, there's only a handful of teams that could do that, it's such a niche thing. You always discover that it's very small, very narrow and they're the only team in a company of 10,000 doing this.
It's so rare to see it in practice; sure it's possible, but why bother?
So you've mentioned the need for modelling skills and code generation skills. Where do you pick up those skills beyond just finding yourself in that sort of environment, learning from someone who happens to have them?
It's pretty tough, I mean you can take courses. You can learn to program in a course I guess, but you don't [learn to program] right, you learn as you do it. You've got to do the work, so when you're learning this stuff you can get some training and that'll give you some ideas, but you need some mentoring, you need hands on experience doing it — it's going to take time to learn.
If you think though that everyone is going to have a 30 or 40 year career then it makes a lot of sense to put a little bit of effort into this. Yes, it's going to take several years to pick up these sorts of skills, but percentage wise, it's not such a bad thing.
You've got to get hands on experience and you have to work with other people who know how to model. There's nothing special about modelling, it's hard to learn unless you have someone to talk you through it.
So if it's mostly down to experience, what do businesses need to look at if they're looking to hire new people for a modelling job?
A couple things. The things I'd look for in people: existing experience is good, but are they willing to learn? Are they willing to work side by side with others? Are they willing to work iteratively and focus on software? We've been talking about modelling a lot, but the reality is your goal is to build software, it's not to model.
I think a lot of the professional modellers out there, that's what they struggle with, because what they've been taught for decades: "do this modelling up front, you're the business analyst, hand your stuff off to someone and they'll take it from there".
No I don't need that, I need people who can do a bit of analysis, a bit of design, a bit of coding, a bit of testing and then iterate. I don't need someone who's just going to model, or just going to code or just going to test.
I think that's the major difference we're seeing now in the community, people with all-rounded skills. So we've been calling that "Generalised Specialist", other people call it "Craftsperson" or "Renaissance Developer", the Gartner group has this new word: "Versitalist" — it's a specialist that's versatile, or a specialised something. It's a brutal word, they must have not tried to pronounce it. I don't know what they're thinking, but the idea's right.
Any final thoughts?
I think main thing is just to developers — be fair to modelling.
I think for developers who are a little down on modelling and documentation — it's not a black and white world, modelling is not evil, documentation is not evil, the goal is to be just enough.
For programmers, you can definitely become significantly more productive if you get some modelling skills. For professional modellers, people who have been told to do all this detailed modelling up front, they can become more productive by doing less modelling and by doing it just in time.
There is significant evidence out there, if you choose to look into it, that a detailed requirements specification early in the lifecycle is actually a bad practice not a good practice. There's evidence out there that doing a detailed design early in your project has no actual effect on the outcome of your project. You're equally as likely to succeed as you are with no design if you do a whole bunch of detailed design up front, and those are both extremes.
It's not a black and white world, I'm not saying don't do any design, I'm not saying do a whole lot of design, agile modelling is all about finding that sweet spot somewhere in the middle of just enough design, just enough requirements for the situation at hand.
This is what gets missed — the people at either end have to come to the middle. There's no coherent reason to model up front, it's the worse time to model, you can't think things through up front, we know this, we've seen projects fail time and time again from doing this stuff. If you have the skills to model everything up front today, then surely you'll have the skills six months from now when you actually need all that information, why can't you wait? There's no good excuse to do everything up front.
You're equally as likely to succeed as you are with no design if you do a whole bunch of detailed design up front
The longer you wait to gather information the more chance there is that it's information you actually need — you can ask more intelligent questions. If I model something today then I know the least amount of information about my models, if I model six months down the track then I have six months more domain knowledge — I can ask better questions. My stakeholders have seen six months worth of software delivered, they can give me better feedback.
So it's far more efficient to take an agile modelling approach than traditional modelling. It's the bridge building theory, software development is a lot like building a bridge. Well, no it's not.
The only people telling me that are people who don't build bridges. Although it's interesting, the people who have experience building bridges and building software will tell you that building bridges is a lot more evolutionary than we think.