Jeff De Luca says his first law of information technology is, "IT is 80% psychology and 20% technology". Although I agree wholeheartedly, I find this somewhat annoying because it means the greater part of software development is actually about learning to communicate and work with other people rather than just designing and writing source code.
It wouldn't be too bad if all software projects were small enough to be designed and built by one or two developers working closely with a customer or other domain expert. Then it is relatively easy to agree informally how to work best together.
Unfortunately, many modern software development efforts are simply too large to be attempted by two or three people. The only option is to use a larger team of people (or multiple, coordinated, small teams). These teams need some formal rules and guidelines about who does what and when if the effort is to stand any chance of success. These rules and guidelines form the project's process.
I have worked on projects of different
sizes in different industries in different parts of the world. No two of these projects have had
exactly the same ways of doing things even when they are supposedly
following the same process. This is simply because no
two projects have had exactly the same people working on them or
faced exactly the same challenges.
More recently, the projects I have worked on have used agile approaches like eXtreme Programming, Scrum, and Feature-Driven Development (FDD). Several of the others were significant projects that successfully delivered using more traditional approaches. The projects that have delivered successfully did not have a particular process in common. What they did have in common was a good team of people.
Good teams have people with the right mix of technical skills, experience, motivation, and personalities. Motivated people that can work well together are as important, if not more important, than technical skills and experience.
Training, coaching and consultancy can help fill gaps in skills and experience but it is largely a waste of time for people that are not motivated or that have no real desire to learn. Improving the morale and motivation of a team requires skilled leadership and management.
Take a poor process, one that is either too abstract, incomplete, heavily over-specified, or simply does not fit with the company culture or maturity level. Combine it with a poorly-motivated team missing key required technical skills, short on experience, and whose members have not learned to work well together. Now there is a great recipe for failure!
In cases like this, fixing the
process is only a part of the
answer and not the most important part. In fact, a poor team
by definition will find it very hard to fix the way they work.
Speaking from experience, they may not even be able to agree on
what areas the process should cover or at what level of detail to
stop defining rules and guidelines. Process never makes up for
In contrast, a good team of people
will usually find ways to work around problems and fill in the gaps
in a poor or incomplete process. It is part of what makes them a good
On the whole, I have been fortunate to work on good teams, some very good. This has largely been due to the project managers and technical leads on those teams working very hard with their managers and HR departments to ensure they have good people assigned to or hired for the project.
The bottom line is that process
improvement is much more effective when the people side of things
is also addressed, and this is more than simply sending individuals
on a range of technical training courses.
Even if the best were only twice as good as the rest, it still makes financial sense to pay twice as much to employ them due to savings made in office space, equipment, and other facilities. When they might be ten or even twenty times better, the financial argument for paying more to employ them makes even more sense.
The trick of course, is attracting the best software people to your organisation, recognising them, and then keeping them. This, of course, takes considerable effort. It is far easier for a software development organisation to peg the salaries they offer to the industry average and base hiring decisions on little more than the general appearance and contents of a candidate's resume.
Unfortunately, as the old saying goes, 'You get what you pay for!'. If you pay industry average salaries you will generally get industry average analysts, designers, and developers, and need to hire far more of them to reach the same productivity levels that hiring well above the industry average would.
It is not even as if attracting and keeping good people means paying disproportionately more for them. A good office environment, professional development opportunities, feeling part of something special, working with other good people, and so on, are often as important. Using good software processes is part of that but the amount of time and money some organisations spend on their software development processes compared to the amount of time and money spent on their human resources processes is disproportionate.
People vs. Resources
The reality for many software team leads and project managers is that they have a team of mixed abilities, varying amounts experience, with different personalities. The trick is to find effective ways of working together that get the most out of the team, taking advantage of particular strengths and mitigating any particular weaknesses.
Unfortunately many software development processes and supporting project management tools seem to be written as if people are interchangeable, consistent, logical, software-producing machines. This is ridiculous.
People are emotional and fallible. They can be inspired and motivated one day, and frustrated and demoralised the next. They can be highly creative and resourceful but they also get tired and fall sick. Some work better in the morning, others in the afternoon, some simply do not function well at all on Tuesdays for some strange reason, and some hate Monday mornings, for example.
Software development processes that consistently deliver results have had a large dose of psychology applied in their creation. They are not simply lists of roles, tasks and artifact templates. They are flexible enough to adapt to a teams' particular strengths and weaknesses. This is why I like Feature-Driven Development so much and dislike process improvement programs that do little more than throw products like the Rational Unified Process at software teams and expect productivity to improve.
Putting People First in Process Transformation
The agile manifesto states that it values “Individuals and interactions over processes and tools”. It puts people and the ways they work together at the center of things. Processes and tools support these collaborations. For organizations undergoing an agile transformation, this ‘people first’ idea is proving useful in introducing and establishing agile practices.
One thing I have learned over the years as both a developer on the receiving end and a consultant on the other side of the fence, is that simply deploying a software product or mandating a particular process rarely produces the desired transformation in the way people work. It is all too easy for people to continue applying the same thought processes and working practices while using the terminology of the new tool and process. ‘Play it again, Sam! … but this time call it something different.’
This is not something particular to agile approaches. I have seen
the same thing happen when introducing UML
modeling tools as a replacement for generic
diagramming tools, requirements management tools for monolithic
requirements documents, and configuration
for basic version control of source code. A shift in paradigm needs
more than a switch in technology.
Those old enough to remember the switch from structured programming languages like Pascal and C to object-oriented programming languages like C++, Java and C# will remember that for a long time, developers simply used these new languages in much the same way has they had the old ones. What we needed to realize the benefits of these new languages was a change in the way developers thought. The programming languages supported this new way of thinking but they could not cause the change. The same is true of the shift from traditional software development management to agile approaches.
The ‘people first’ principle has worked well for several teams I have worked with that wanted to adopt agile approaches. We concentrated initially on establishing the values, principles, and practices of the approach within the team, postponing any discussion about the use of software tools to manage the project in which they would be piloting the agile approach. Instead, we started with index-cards and a large corkboard.
The idea was for individuals to get a feel for the new ways of working without having to worry about operating a particular software product. With this approach,the team was able to concentrate on establishing the new ways of communicating and interacting, instead of learning the idiosyncrasies of a particular user interface.
Once the team had two or three iterations (sprints) under their belt we started to discuss how software tools might help improve management of the project. For example, one member of the team was having difficulty because he worked from home - in another country - part of the time. He had to attend standup meetings by phone, call or e-mail his daily updates for the corkboard to the scrum master, and so on. Another problem this team had was the growing size of the backlog. The product owner was struggling to manage the stories coming in from multiple different sources. Because we had established the new way of thinking, the selection of tooling focuses on addressing real problems experienced by the team.
From agile management tool vendor’s perspective, it might seem counter-intuitive not to push your product hard from the start but it actually makes considerable sense. It is all too easy and common to blame software tools for frustration caused by lack of understanding, bad communication, and poor process definition. Agile transformation programs are no different. As a vendor looking to establish a lasting relationship with a client, you do not want to see a process-improvement program struggle or fail. You certainly do not want your products cast wrongly as one of the causes of that struggling or failure. And yet, ‘Play it again, Sam!... but this time call it something different,’ is likely to happen if a tool is viewed as the key to transforming traditionally managed teams into agile teams.
The Key to Process Transformation
One of my favorite quotes on process improvement initiatives comes from Peter Coad’s book, Java Modeling in Color with UML, it starts with ‘We think most process initiatives are silly’. I particularly like the paragraph that says, “No amount of process specification will make up for bad people. Far better: Staff your project with good people, do whatever it takes to keep them happy, and use simple, well-bounded processes to guide them. ” In fact, I would go so far as to say process over-specification gives non-performing people something to hide behind. Agile processes strip away that cover exposing weaknesses in the ways individuals, teams, and communication/interfaces between teams work.
However, exposing dysfunctional teams and working practices is not the same as fixing them. Management-imposed change that does not win the hearts and minds of those on which it is being imposed is almost always fails to deliver its full potential benefits. I certainly recognize these problems and not just with agile process adoption, but with any major process improvement initiative including standardizing on a new tool set.
In fact, there are some facets of such programs that I have come to recognize as patterns:
The ‘Sheep-dip training’ pattern:
Most process improvement teams realize that staff will need training in the new process or tool set. However, with one eye on the budget, they opt for a training program that introduces everyone to basic the mechanics of the process or toolset.
Unfortunately, this is a little like giving soldiers basic training in a new type of warfare or equipment and sending them into battle without trained NCO’s and field officers.
- The ‘If we build it, they will come’ pattern:
Some people seem to believe that if the process improvement team create the perfect description of the new way of working on a intranet site (or worse, in a single large MS Word or PDF document), everyone will read it and follow it. I have even known one client that, on deciding to adopt agile, disappeared for a couple of months to write a large document to describe how to do agile in his department!!!
This is like HQ addressing the lack of trained NCO’s and field officers by sending more detailed written orders to soldiers already under fire.
- The ‘Centralized coaching team’ pattern:
A team of experts, possibly external consultants, is set up to visit, assess and advise teams on issues they are encountering with the new way of working.
To continue the military metaphor, at regular points in the battle, officers from HQ appear to advise briefly on the new tactics and then hurry back to the safety of the HQ.
- The ‘The disappearing external consultant/coach’pattern
End users of a new tool or process need expert guidance to get over the initial learning curve and to apply it within their particular project context. The easy solution is to parachute external expert consultants and coaches into these teams. Unfortunately, again the budget often limits this to part-time or a few weeks only.
Having your process documented on a website is not wrong. Giving users basic training is not wrong. Providing access to experts and coaching is not wrong … but it is not enough. The fundamental problem with the patterns above that turns them into anti-patterns, is one of pigs and chickens.
Note: In the UK, we talk about ‘having skin in the game’. In other words, having grazes and bruises from being an actual player rather than supporting or coaching from the sidelines. It seems a more appropriate metaphor for a process called Scrum but the pigs and chickens thing has stuck. Pigs have 'skin in the game'. Chickens are well-intentioned,interested spectators willing the team on but not able to contribute in any concrete way to the final result.
In the patterns above, chickens, not pigs, are providing the expertise whether written or face-to-face.
Personally, I believe the key to effectively achieving a change in culture and a sustainable transformation is to:
- Identify the technical leaders within projects;
those that are “self-driven to produce quality results on time …
combine technical ability with enough people skills …are trusted and
respected by both their managers and fellow developers, are determined
to make the team succeed, and usually live the work.” (Chief
programmers, Chapter 2: A
Practical Guide to FDD).
- Sell them the vision: if you cannot sell
these people on the benefits to them, their colleagues and the
organization of the new way of working then something is wrong. Either
these people are not the real technical leads, you are not
communicating the advantages of the change well enough, or the new way
of working is actually a pile of ‘emperors new clothes’.
- Provide in-depth training and on-going coaching.
It is better to have a single lead person trained in-depth who can
coach his teammates through the basics than to have the whole team
trained on the basics with no-one on the team to turn to when the
basics are not enough. Providing initial training is not enough though.
When the pressure is on, the temptation to return to previous ways of
doing things is often too strong to resist. If technical leads are to
work for you, they need on-going support and coaching, and a means by
which they can support each other. Good external coaches (expert
chickens?) can help here.
- Let the technical leads continue working on their projects. Some fail at the final hurdle by doing 1-3 above and then assigning or scheduling the technical leads to coach other projects, effectively turning them from pigs into chickens.
To summarize, if you can produce a change in the behavior of the lead pigs, the other pigs will, by definition, follow. However, pigs will not follow chickens for long because chickens are simply not pigs.