Most organizations don't have a formal apprenticeship program for software developers. There is no journeyman for software development. However, most software developers today learn as much through their interaction with other developers as they learned during their formal education.
So if your software developers are learning how to be software developers through their experience at your organization, what are you teaching them? It's time to evaluate what the software developers in your organization are learning.
The need for education
As an industry we still suffer from project failures, cost overruns, poor client satisfaction, and a general malaise. Failure rates for software development projects are wildly different depending upon who you listen to. Publicly reported rates vary from 5% at the lowest to over 70%. While neither of these numbers is particularly believable they illuminate the fact that there is still a problem. If software development was steadfastly producing quality software all of the numbers would be in the same range.
However, the mere fact that there are failures doesn't indicate what the causes for failure are. Surely there are a variety of causes for project failures. Just as surely some of those causes are causes that are well understood by those at the forefront of software development.
Understanding the impact of poor requirements gathering, poor project management, poor change control, improper tool selection, etc., are all examples of classic problems with software development which we, as an industry, know leads to project failure. However, despite this we continue to make the same mistakes over and over again.
The reason for this is expressed as the gap between "state of the art" and "state of the industry." "State of the art" is simply the ideal world where someone would know everything about software development that can be known. It would be an ideal understanding, without bias, for various methodologies, tools, and techniques. "State of the industry" on the other hand is the average knowledge of the practitioner working in software development. While there is no one average person, you can develop a persona to identify the traditional software developer.
In today's world the typical (average) software developer has a good chance of using an integrated development environment (IDE) and source version control. Both of these things are important because they form the basis of effective code management. However, the typical software developer doesn't understand more than two software development methodologies. The software developer is unable to identify where one software development methodology might be more useful than another.
Similarly, the typical software developer doesn't have exposure to or experience with implementing patterns. Patterns are essentially template solutions that describe a generic approach to a common problem. Although most software developers have heard of patterns (most frequently the factory pattern) they don't have any idea what they mean or why they're important.
Testing techniques are generally haphazard at best. Some developers perform unit tests, other developers test via the "throw-it-over-the-wall" method which relies upon other developers, quality assurance professionals, and end users to identify their bugs.
The typical software developer continues to allow change to happen to the code without change control either because of user requests or because of their own desire to do something more interesting. The lack of change control creates ripple effects which combine with other changes to become the tsunami which overturns the proverbial project boat. They do this not from a desire to intentionally sink the project but instead because they do not fully comprehend the impact of these small changes.
How we learn
Before you set off to buy a bookshelf full of the latest books in software development and a collection of the classics, you may want to pause. Reaching a level where every software developer reaches a relative level of "state of the art" through reading is unrealistic. Most people can't fully understand what they're reading without practical examples that they can apply quickly after reading it.
Think back to every article you've read. Have you been able to implement the suggestions or guidelines from every article? Of course not! You can't absorb everything you read. Our brains are not wired that way. While we each have different levels of being able to understand and integrate the material we read, no one is perfect in this respect.
Humans learn best by experiential learning. That is, we learn best by experiencing the situation where we need the learning. Reading something is fine but the more involved we get with the idea the better our retention of it.
When I defined "state of the art" above I indicated it was a theoretical state. It's something that no one individual could ever meet. That is because of the experience based learning that we do as humans. It's impractical to expect that a person has had experiences that would allow them to understand the best practices on every aspect of software development. For instance, the best practices for Web development, where multiple iterations and visual appearance are important, are completely different than best practices for embedded software where quality and reliability are important because updating embedded software is difficult.
However, all is not lost. We do have the ability to move everyone closer to "state of the art" by encouraging focused reading and development in specialized areas and then allowing this information to be spread to the rest of the team through experience.
That's the positive side of teaching through experience, allowing individual team members to focus and leverage that learning throughout the entire team. It's something that moves your team closer to "state of the art". However, most organizations are unintentionally moving developers away from the "state of the art" by developing and reinforcing bad habits.
In most organizations there are the basic processes. There's some sort of a requirements gathering process. There's typically some sort of a design phase. Some folks, particularly those who've used universal process (UP), Rational Unified Process (RUP), or some agile methodologies may know these by other names such as elicitation and investigation. Although they are typically much shorter in duration, they perform the same fundamental purpose – to understand the problem enough and to create a structure for a solution.
Most organizations also have some form of documentation, no matter how minimal. It might be a "project charter"-like email from the executive sponsor, a request database, or some other form of information repository but it typically exists.
The challenge is, however, that many organizations skip these steps in the process, bypass documentation, and proceed headlong into development. Everyone instinctively knows that this is a bad idea. There's a mountain of evidence that the "code and fix" methodology (if it can be called that) doesn't work. However, when pressed for time we eliminate the time reserved for elicitation and investigation.
What does this teach software developers on the project? It teaches them that these practices are not essential (In Fred Brook's words, accidental) to the development of software. Nothing is further from the truth; however, it is the message that is conveyed. It's clear that these steps and the associated documentation can be skipped when speed dictates.
Ignoring for the moment the probable consequences of this course of action, this has a second order effect. Developers begin to think that some of the work that they're doing (elicitation and investigation) are really the organization just making them do busy work. Instead of being seen as a technique for making software development projects successful, they are now a burden levied upon the developer.
With a four year old in my home, I'm painfully aware of how few times I have to do (or say) something for it to be repeated. It reinforces my resolve that it is never acceptable to eliminate or scrimp on elicitation, investigation, or documentation. I know what skipping or undercutting these activities even a few times will mean in terms of the long term retraining that will have to be done. (If you have a child, think about what happens when they've been a grandma and grandpa's house for a few days.)
That is not to say that elicitation must be a document heavy three month process – it does, however, mean that the goal of understanding all of the requirements for the current iteration or phase as good as is practical must be met. It means that you shouldn't relent if there's a critical portion of the solution, which is both essential to the operation and needed for the current iteration, which remains nebulous after the time for elicitation has been exceeded.
Most software developers are lagging behind the "state of the art" due in part to the lack of time for training, the need for more experience, but most painfully because we are teaching software developers that parts of the process (no matter what methodology you subscribe to) are accidental – and are therefore not essential to the success of the project. Protecting the minimalist implementation of the process is essential to continuing to support the professional development of software developers.