On planning – how? (1)
In my previous post, I made a case for planning even in agile teams. Next, I will describe my personal approach to planning in agile teams, which consists of these aspects:
- end-to-end planning,
- iterative refinement,
- planning forwards and backwards,
- the critical path,
- risks and alternative paths.
This post covers the first two aspects. I will address the other three in a follow-up post.
Ground work
The primary purpose of planning is to prepare me to reach a goal. For this, I need to know – or figure out – three things: where I am now, what the goal is, and what means to getting closer to the goal I have at my disposal.
Knowing my current position is essential because only then can I put my position in relation to the goal and determine the direction in which I will need to go. How do I get to Reykjavik? If I am in Keflavik, I will probably get a car and drive along the coast. If I am in Nuremberg, I will probably fly. How do I replace my payment service provider (PSP) with another one? The answer depends heavily on the structures and processes that are already in place.
Knowing my goal is important, because, well, if I don't know where I want to end up, there is no point in preparing to get there, is there? In practice, the goal is not always well-defined from the start. I may, for example, know that I want to visit Iceland, but not yet have a specific town or place in mind. Or I may know that I want to replace my PSP, but not yet have a detailed picture of the end state. This is fine. I can narrow down the goal during planning. But it still needs to be specific enough so I can tell when I have reached it.
Knowing the means at my disposal greatly aids in planning because it allows me to focus on possible paths forward as opposed to getting hung up on impossible ones. Means include actions I can take, time, money, work capacity, skills, material, and tools, all of which are usually limited in some way. I have observed that we (myself, but also colleagues) tend to gravitate towards our respective default methods when solving problems and forget that others also exist. As the saying goes, if you only have a hammer, everything looks like a nail. But even if the toolbox is full, if the hammer is lying on top because it is the most used tool, people will still squint very hard to make their problem look like a nail – rather than search through the toolbox. A software engineer, for example, may default to adding code for any problem, forgetting that sometimes organizational changes can work better. Explicitly considering what I have available before entering focused planning mode can open up alternative (and sometimes better) paths I would not have thought about otherwise. It can also prevent me from pursuing impossible plans once I discover what means are not available to me. It may, for example, be impossible to get additional resources for a project because of budget constraints.
Once I have a more or less complete understanding of my starting position, my goal, and available means of getting there, I can begin planning in earnest.
In the following I am going to use the examples from above. For the traveling example, let's say I want to travel from an address in Nuremberg, Germany, to a hotel in Reykjavik, Iceland, on a certain date. For the IT example, let's say I want to replace my current payment service provider A with the new PSP B, and this must be finished 6 months from now.
(Side note: I have never been to Iceland, so the details I am going to add to this example may not be factually correct, but they should serve to highlight the kind of thinking that I put into planning. I have, however, done a payment service provider migration. You can watch my talk about it at the WeAreDevelopers DevOps Day 2023 [free registration required].)
In the case of traveling to Iceland, for example, the means available do not include direct flights from Nuremberg to Iceland. They do include various indirect means of travel.
In the case of the PSP migration, the means available include a development team (partially occupied with other activities going on at the same time), our infrastructure, and the new PSP.
End-to-end planning
I start planning with a rough outline of the whole journey from my starting position until the project is done, i.e., I have reached my goal completely. Then I iterate on it by refining parts of the journey, validating assumptions, identifying risks, and examining alternative paths.
This is a core part of my planning strategy, and it bears repeating because I have so often seen people miss this.
I always plan the complete journey from start to end. There must be no gaps in the middle of the path. Steps may be roughly specified, i.e., details may be left out; steps may be unvalidated; their feasibility may be unclear; they may be fraught with risk; but they must not be missing.
As long as my plan contains gaps, it is not really executable because I cannot move from one step to the next directly. And worse, there is no way to know whether it is at all possible to make it executable. Small gaps may contain huge amounts of hidden complexity. Making the effort to close those gaps has the potential to expose issues that, if unaddressed, may sink my project. By exposing them, I make them addressable.
Traveling example:
I'm going to fly from Nuremberg to Keflavik airport via Frankfurt and drive from Keflavik to Reykjavik with a rented car.
Notice the gap in the middle? How am I going to get from Keflavik airport to the car rental? Forgetting this step may not be a problem if the car rental I end up using is directly at the airport, but it may turn out badly if there is some distance to cover. A more complete plan would be:
I'm going to fly from Nuremberg to Keflavik airport via Frankfurt, get to a car rental somehow, rent a car, and use it to drive from Keflavik to Reykjavik.
It is not yet clear which car rental or how I am going to get there, but it is clear that this step is necessary and will have to be dealt with at some point.
A first attempt at planning our PSP migration could look like this:
We are going to implement the API of the new PSP, migrate users from the old to the new PSP in batches, and disable the connection to the old PSP.
However, an important step is missing: before we can actually migrate users, we need to change the general terms and conditions to refer to the new payment service provider. Thinking of this early is particularly important because there are legal deadlines to consider which may have knock-on effects on the following steps. A more complete plan could look like this:
We are going to implement the API of the new PSP, update the general terms and conditions, migrate users from the old to the new PSP in batches, and disable the connection to the old PSP.
I always plan the complete journey from start to end. How to get startecd is a point that is often glossed over, but it is a critical part. If the journey does not start where I am now, if I cannot make the first step, there is not going to be a second one, not to mention the last one where I reach my goal. Getting started can take a lot of time and effort which, when not taken into account, can nix my whole plan.
Look again at my traveling plan and try to see what is missing at the beginning!
I'm going to fly from Nuremberg to Keflavik airport via Frankfurt, get to a car rental somehow, rent a car, and use it to drive from Keflavik to Reykjavik.
My journey begins at an address in Nuremberg, not at the airport. Getting to the airport is the first step. Going by subway, this step can take up to an hour. I will probably have luggage to check in, which will take time, too. So, an even more complete plan would look like this:
I'm going to walk to the subway, take the subway to Nuremberg airport, check in, fly from Nuremberg to Keflavik airport via Frankfurt, get to a car rental somehow, rent a car, and use it to drive from Keflavik to Reykjavik.
Now, for the PSP example:
We are going to implement the API of the new PSP, update the general terms and conditions, migrate users from the old to the new PSP in batches, and disable the connection to the old PSP.
What is missing at the beginning? Can we start implementing the API right away? No, we cannot. First, we need to clarify API usage for our use cases with our technical contacts at the new PSP, and we need access to the PSP's test environment. Only then can we begin to implement the API. Starting from where we are, the plan thus looks like this:
We are going clarify API usage with our technical support contact, and the new PSP is going to give us access the their test environment. Then we will test our scenarios manually, before implementing the API of the new PSP. We will update the general terms and conditions, migrate users from the old to the new PSP in batches, and disable the connection to the old PSP.
I always plan the complete journey from start to end. When is a project done? Over the years, I have seen various attempts to define done-ness, and claims that some project or other was done, and most of them excluded certain aspects – sometimes for expediency, sometimes because the person claiming done-ness had only limited responsibility, e.g., a developer who considered a task done as soon as the code was written. I think such narrow definitions of done do not make sense in the context of teams with true end-to-end responsibility.
For me, the answer is simple: a project (task, user story, whatever) is done when there is nothing left to do. Done is ‘done’ in the perfect tense, which means the doing has taken place in the past. If part of the doing is still in the future, it is not done. Is it tested? If not, it is not done. Is it deployed? If not, it is not done. Is it released to users? If not, it is not done. Is it documented? If not, it is not done. And so on.
In my opinion, as soon as you start to distinguish between ‘done’ and ‘done done’ and maybe other kinds of ‘done’, you enter a zone of semantic diffusion where it is no longer possible to talk about when something is really finished. After all, when ‘done’ no longer means the doing lies in the past, what does ‘finished’ mean?
Consequently, when planning, I take care to not leave gaps at the end, either. It is easy to miss important bits that happen after the ‘main part’, leading to problems later on. Be careful not to fall into the Pareto trap, where the final (unplanned for) 1% take 99% of the time!
The key question I always ask is, ‘If we have done all this, have we reached our goal completely, or is there still something to do?’ If there is, I add it to the plan and repeat the question until the gap between the end of the plan and actual goal completion has been closed.
I'm going to walk to the subway, take the subway to Nuremberg airport, check in, fly from Nuremberg to Keflavik airport via Frankfurt, get to a car rental somehow, rent a car, and use it to drive from Keflavik to Reykjavik.
OK, but what about the hotel I actually want to get to? Can I drive there or is it in a pedestrian area? Does it have parking space? What about returning the rented car? Let's complete the plan:
I'm going to walk to the subway, take the subway to Nuremberg airport, check in, fly from Nuremberg to Keflavik airport via Frankfurt, get to a car rental somehow, rent a car, and use it to drive from Keflavik to the hotel in Reykjavik. I will park the car in the hotel garage and return it to the rental company's Reykjavik branch the next day.
Now for the PSP migration:
We are going clarify API usage with our technical support contact, and the new PSP is going to give us access the their test environment. Then we will test our scenarios manually, before implementing the API of the new PSP. We will update the general terms and conditions, migrate users from the old to the new PSP in batches, and disable the connection to the old PSP.
Once all users' payments are being processed by the new PSP, are we not done? From the point of view of the users, yes. But not from the point of view of the development team. Payment integration is not a one-shot fire-and-forget project; it is something we will continue to maintain and extend for a long time. The integration component must be maintainable to remain economical, which, among other things, means not keeping dead code around. So there is at least one additional step to do:
We are going clarify API usage with our technical support contact, and the new PSP is going to give us access the their test environment. Then we will test our scenarios manually, before implementing the API of the new PSP. We will update the general terms and conditions, migrate users from the old to the new PSP in batches, and disable the connection to the old PSP. Then, we will remove all code dealing with the old PSP.
Only then are we really done.
Now I have a rough, but complete plan. It covers the whole journey between our starting point and the ultimate goal. While it is probably not ready for execution, it will serve as the basis for refinement and further discussion.
Iterative refinement
The first rough draft of my plan will probably be wrong, but it has an important property: it is complete. I have found it much easier to do planning with colleagues once I have this scaffolding in place because it serves as an anchor in discussions about feasibility, details of a particular step, risks, and possible alternatives.
There is a meme on the internet, that goes something like this. ‘If I don't get an answer to my question on a forum, I simply post a wrong answer with a different account, and someone will immediately jump in to correct me.’
The same goes for plans. If I need input from stakeholders who are reluctant to provide information or just not strongly motivated to take time to support the project, showing them a complete, but unvalidated plan more often than not prompts them to correct me.
Similarly, willing contributors who have not yet been deeply involved in the project are often missing some context. For them, too, it is easier to join in – and provide valuable input – if they can see how the issue under discussion fits into the big picture. When discussing a particular part of our journey, I often get questions like, ‘Yes, but what happens next?’ or, ‘What do we do with this result?’ Because the plan is already complete, I always have an answer. Sometimes, the answer is satisfactory. Sometimes, preceding or following steps will need to be adjusted as a result of the discussion. Sometimes, additional steps need to be introduced, or steps need to be reordered or merged. The impact of the step under discussion on the plan becomes clearer if we discuss it in the context of the complete plan rather than looking at each step in isolation.
In a way, this is like continuously integration-testing code changes. If I make an incompatible change to the code of a component in a project with good integration tests, a test will break and thus prompt me to restore compatibility (by either making my change compatible, or propagating my change to the dependent code). Likewise, if I change a part of my plan so it no longer fits with the rest, knowing how the part is linked to the whole enables me to make the necessary changes so the plan stays coherent.
It is OK for the plan to evolve unevenly, i.e., for some parts to be pretty clear while others are still hazy. How detailed each part needs to be depends on what I want to get out of my plan. If I do not have a hard deadline, it's probably enough to focus on the next couple of steps and keep the rest of the journey in the back of my mind to spot any potential problems. Using my example of dealing with a legacy system from the previous post, the plan could look like this.
beyond that, it gets hazy (but still complete).
If I do have a hard deadline, I need to get to a granularity that allows me to roughly estimate each step's duration to ensure my plan fits plausibly into the remaining time period. One way to do this is to plan forwards and backwards (see my next post for more details).
If there is a dependency on some third party that needs to be integrated into my plan, I may need to go deeper on the affected parts of my plan than I otherwise would to ensure a smooth integration.
In any case, the important point is, as long as I have an end-to-end plan, I can iteratively refine each part of the plan as much as is useful, and having the end-to-end view enables me to spot changes that affect other parts of the plan and deal with them accordingly.
Conclusion
I find it important to do planning end to end, i.e., from my current position until I have reached my goal completely, and to do it iteratively because
- the resulting plan is actionable immediately (begins where I am right now);
- this approach enables effective collaboration;
- it allows me to ‘integration test’ changes to the plan, ensuring consistency;
- when I'm done, I'm done.
I use this approach for major endeavors, especially risky ones, projects including deadlines, and if there are dependencies to consider.
In my next post, I will write about how I use forwards and backwards planning to meet deadlines, explore the concept of the critical path, and explain how I deal with risks and outline alternative paths.