On planning – why?
tl;dr
- Planning is important even for agile teams.
- Planning must be done by the people who are going to do the work, because the most valuable part of planning is the process.
- If done right, planning helps you to avoid doing stupid things, prevents blockers, makes you faster, and enables better decisions.
Intro
I am a planner. Have been one since my childhood. I plan everything, from my meals to software development projects. I even plan the unplannable. I have children – the most unpredictable things on this planet – and still make plans. Sometimes they even work. :-)
In my first job as a professional software developer I worked in product development. The company sold market data distribution software to financial institutions. Development was financed in part through licensing fees, but mostly by customers paying us for extensions (new functionality and new interfaces). As these were all fixed price contracts, planning (and its ugly twin, estimation) was vital because any mistakes were going directly to our bottom line. We became really good at it as a result.
Nowadays, I work in agile teams, and there is a strong headwind against planning anything. I get where this is coming from, and I myself plan much less now than I did when calculating fixed price quotes 15 years ago. However, I still think planning is a key skill to be successful in the long term.
In recent years, I have noticed a certain bafflement among my fellow software developers – and not just them – when the need arose to make plans that extended more than two weeks or so into the future. I think this is a shame. So I decided to put some of my thoughts on planning in writing in the hope that someone might find them useful.
In this post, I will examine the question of why we should plan. In a follow-up post, I will write a bit about how I approach planning.
Why plan?
The Manifesto for Agile Software Development values
Responding to change over following a plan
and this is often taken as an argument against planning. It is not, for two reasons. The first is mentioned right in the next sentence (emphasis mine):
That is, while there is value in the items on the right, we value the items on the left more.
This sentence is just as much part of the agile manifesto as the proceeding ‘X over Y’ statements, but often ignored. Yes, there is value in following a plan – and thus, in having a plan –, even if responding to change is more important.
For the second reason, I turn to general Dwight Eisenhower, who gave us this insight:
Plans are useless but planning is indispensable.
While one should certainly be cautious about applying military doctrine to peaceful endeavors like software development (wait, why are you laughing?), it is certainly true that warfare is highly dynamic and much older as a discipline, and thus might have something to teach us about dealing with volatile situations.
So, why are plans useless?
Presumably because, as Helmuth Graf von Moltke has it,
No plan survives contact with the enemy.
Enemies will not adhere to your plan. To the contrary, enemies will try their very best to destroy your plan, and battle field commanders who try to stick to their plans in the face of intransigent enemies will probably lose.
As to the applicability of this adage in software development, have you ever had the opportunity to
- … watch the first user interact with the UI you carefully designed?
- … support a team who is trying to use an API you specified?
- … undertook to finish a task today, only to be interrupted by a (real or imagined) emergency and making no progress whatsoever?
Case closed.
In effect, Eisenhower, von Moltke, and the agile manifesto are all saying roughly the same thing, which is that the value of having or following plans is limited, the difference being a matter of degree, not of principle. I assume this is due to the fact that enemies actively trying to foil your plans make them really useless, whereas the chances of success are significantly higher in software development, where it is usually ‘just’ life getting in the way. The value of following a plan as long as it remains practicable lies in increased efficiency. More on this below.
Why is planning indispensable, then?
After all, if planning is the activity that generates plans and plans are of limited value, how come planning is deemed to have a higher value than the plans it generates? The reason is simply this: the primary purpose of planning is not to generate plans.
Wait, what?
I repeat, the primary purpose of planning is not to generate plans. The primary purpose of planning is to prepare you to reach a goal. While planning will typically produce one or more plans, those are almost incidental. The real value of planning lies in the hard thinking that goes into it and in the shared understanding developed among the planners. To come up with a plan, the planners need to examine the current situation and clarify the goal. They need to explore options, assess risks, and devise countermeasures. They will come up with ideas only to find – on closer examination – that they are unworkable. In many cases, only one path from the current situation to the goal is written down as a plan. If everything goes well, it may work. If anything disrupts the path that was written down, it becomes worthless. But if planning has been done well – and the people who have done it are still available – the hard thinking that has gone into it will enable them to plot a new path from the new current situation to the goal and follow this instead.
This is why the real value lies in the planning activities and not in the final artifact called ‘plan’. This is why Eisenhower values planning so much, even though he deems plans useless.
Fine, you say, but this all sounds very generic. What does planning do for you that you don't get if you skip it?
Benefits of planning
First off, planning prevents you from doing really stupid things. This is a typical illustration of development progress for an agile team.
Beginning at the bottom left, the team takes small steps in the rough direction of the goal at the top right. After each step, the team reflects on what they have accomplished and decides on the next step. The assumption is that the world is volatile and the goal may shift, so you need to readjust your direction regularly, which is why you probably will not move in a straight path but a kind of zigzag line. And if you subscribe to this line of thinking fully, there is no point in looking further than the next step because every effort you make to plan ahead will be wasted if the underlying assumptions change.
However, there is a problem with this picture. It looks as if the team were able to move in any direction it wanted. Even though it is operating in a volatile world which ostensibly influences the direction the team is going, the picture does not show this. It does not show any constraints at all. My experience is that some things are indeed volatile and may change at a moment's notice, but other things do not, even if you want them to. For example, the design and feature set of the web app you are building may evolve very quickly, but the legacy system you are relying on for half of the functionality will not – because it simply cannot. So the reality may look more like this.
The team is not completely free to choose its own way because the environment it is operating in imposes constraints, such as the (in-)flexibility of the legacy system. If you approach this situation without looking ahead and continue iterating towards your goal the same way as in the picture above, you will be in for a nasty surprise.
A little planning will allow you to spot such obstacles and plot a way around them. This does not mean you should stop taking small steps and re-examining your approach regularly. It merely means that you take into account not only the goal, which may be a long way off, but also the major obstacles.
As you can see in the picture, the team still adjusts its approach continuously, and the plan also changes accordingly. Even though the team has taken the same number of steps as above, it is much nearer to its goal when taking the obstacle into account.
Obstacles may also include critical resources your team depends on – and which may be unavailable or only available at certain times. Planning allows you to anticipate needs and arrange for them to be met when they materialize. What use is it to only consider what you are going to do for the next two weeks if your progress depends on another team's work, and this team has a lead time of a month? If you know beforehand you are going to need an API change from another team, you can coordinate with them to get it done in time. If you only know when the need has become immediate, you will be stuck. Mind you, planning does not guarantee that you will have what you need when you need it, but not planning guarantees that you will not have what you need when you need it. And planning pushes you to think about this dependency and may even surface some way to decouple your work from the other team's schedule so you will not be blocked even if they fail to deliver on time.
Planning enables better decisions. Why? Because good decisions take time. There is a concept in agile software development called ‘the last responsible moment’, which says you should delay commitment to a decision until the moment when not committing becomes more costly than committing. The reason is that deciding later allows you to make more informed – and thus better – decisions. For this to become true, however, learning needs to take place between now and later. Otherwise, the decision becomes just later, not better informed. And learning does not happen by itself. The team actively needs to create opportunities for learning along the way. And why should the team do this if it does not know that it will need to make the decision – if it has not planned for it? Planning in this case does not mean anticipating the result of the decision. It means anticipating the need for the decision, identifying important decision criteria, and ensuring that the team gathers enough relevant experience to make the decision well.
Planning allows you to move faster. This may sound counterintuitive if you think of planning as a large effort that happens before you get anything done. It doesn't need to be so. You can plan continuously as you go along. Your plan may turn out to be wrong, so advancing in small steps and continuously reassessing your position and the best way forward is still advisable, but the thinking you have done during planning will help you get this done very quickly.
In one extreme case, I have seen a team flounder after taking a successful step. How could they not know what to do next if everything went just as they wished? If they had thought a bit about the whole journey it would have been immediately clear what the next step should be. As it stood, they had to hold an hour long meeting to determine what they should do next.
Not planning is a bit like walking around with your eyes closed in broad daylight – or maybe with your eyes glued to your smartphone. You can do that, but you have to go slowly and feel your way forward lest you run into a wall and hurt yourself. You can go forth with much more confidence, less risk of hurting yourself – and faster – if you look ahead. You still have to slow down when approaching a corner you cannot see around, and if it is foggy you cannot see as far ahead as in bright sunshine, but it still helps.
Also, plans are rarely completely wrong or useless. Remember, the adage was coined for war – an extremely adversarial situation where the other party is actively trying to sabotage your plans. Software development is not that (at least in my experience). In general, the world is unpredictable, yes, but small adjustments to software projects are still more likely than drastic 180° turns. Some plans even work. Often, some details are off. Less often, major adjustments are necessary, and it is likely you will be able to anticipate some of them. And events which change everything are rare. Case in point: when the COVID-19 pandemic hit, it drastically changed my personal life; it affected how I worked to a large degree (I went from 100% on site to 100% remote in a week); it changed the goals of the project I was involved in at the time – not one bit. It also did not change the EU tender process we were going through, nor did it change our technology evaluation criteria. The details of our plans were adjusted, but the plans were not completely invalidated and helped – rather than hindered – progress.
When to plan
So should you plan everything all the time? Probably not. As I mentioned, I am a planner at heart, but even I admit you can overdo it. Very detailed plans, in particular, are pretty much useless, because details are very likely to change anyway. So when should you plan? I think planning is most useful in the following situations.
- Feasibility check: You want to check whether a project is possible before committing time and resources to it. Example: your product owner has an idea for a major new feature, and it is not clear whether it is even possible to build. Planning prevents you from embarking on a fool's errand and helps with the definition of the first step if the project is deemed feasible.
- Implementation start: You are about to embark on a new non-trivial project. Example: the above-mentioned feature has been added to your backlog as an epic, and you are ready to begin work on it. Planning allows you to spot potential problems and identify key decisions early on, and to provide for the necessary learning opportunities to make the decisions well-founded ones.
- Direct deadline: A project absolutely must be finished at a certain point in time. Example: your contract with a service provider is expiring and you must have integrated their successor before this. Planning helps you to move faster and allows you to spot and circumvent obstacles which could jeopardize the deadline.
- Indirect deadline: You are providing a service to a different team, and they have a direct deadline for which they need something from you. Example: a web shop needs to do a release for Black Friday, and they need a new feature in your API for it. Here, too, planning enables faster progress and helps you deal with obstacles proactively.
- Dependency on scarce resources: Your team needs expertise or resources external to the team, which are hard to get on short notice. Example: you want to build an integration with a legacy system maintained by a third party and need integration support. Planning ahead enables you to anticipate this need and to coordinate with the support provider in good time.
These are scenarios where a little planning can make the difference between quick success and abysmal failure. In other situations, the effect may not be as noticeable, and an argument can be made that planning is not necessary.
Conclusion (and the single most important success factor)
To make this clear, I am not in favor of overly detailed planning and strict adherence to plans. This – shall we say – legacy practice goes hand in hand with rejecting – or even denying – change and results in software that is delivered late, if at all, and often fails to meet customer needs. However, the current predisposition against planning seems to me to be an overreaction to this kind of overplanning. The solution to bad planning is not no planning at all. It is to learn good planning, and to apply it in the right way.
To me, this means recognizing – like Eisenhower – that the value of planning lies not in the plans produced (which can become useless very quickly), but in the process itself and particularly in the learning that takes place during planning. This learning enables you to execute faster while avoiding stupid mistakes. It also enables you to manage dependencies and make better decisions by planning for learning along the way. And it enables you to be – yes! – more agile.
Given that the value of planning lies in the process, the most important factor is the people involved. If the plan is made by group A and the execution of the plan is assigned to group B, you lose all the learning and usually hand over only the plan – the least useful result of planning! To reap the benefits of planning, you therefore must involve the people who are going to be responsible for execution.
If you only remember two things from this post, please let it be these two:
- Planning as an activity is valuable for agile teams, even if the plan is not.
- But only if planning is done by the people who are going to do the work.
Thank you for reading, and happy planning!
Update: See my follow-up post for my personal approach to planning.