By Merlin Carter
Last week, we published a discussion on post-mortems where we talked about how to learn from a critical incident. Armed with your hard-won knowledge, how do you prevent the same thing from happening again?
Old habits die hard, which is why you need to flag any issues early in a project. But not too early. There’s a subtle art to the timing and delivery of warnings. This is what CTO Stephan and I discussed in another edition of Tech Insights.
If you prefer to read, there’s a summary of the discussion below.
When is the right time to escalate?
Merlin: Stephan, have you ever been in the following scenario?
You’re working on a project, and you have a bad feeling about it. Maybe the technology is unstable, maybe the team isn’t working well together, or maybe it’s just taking too long.
Whatever it is, you need to act.
- On the one hand, you might be able to address these issues before they get worse.
- On the other hand, your stakeholders need to know about any serious factors that might endanger the project.
You don’t want to escalate unnecessarily, but you also don’t want to wait too long before you raise these red flags.
How would you approach this problem?
The higher the stakes, the earlier you should act
Stephan: Yes, that’s a very tricky scenario. I’ve definitely been there. My general rule of thumb is to look at the potential risk to the business and the probability of failure. If the issue is likely to have a huge impact, I would raise it earlier rather than later.
You can also look at the urgency of the issue. Is the problem likely to escalate tomorrow or in a few months?
Generally speaking, the consequences of raising problems too late far outweigh the consequences of raising them too early. If you raise a problem too early, you might be accused of crying wolf or making a fuss about nothing. Perhaps people will take your warnings less seriously.
But at least you’re triggering a discussion about the problem — you wouldn’t be raising a red flag unless you were genuinely concerned right?
The “BER airport” of the software industry — does it exist?
Merlin: Yes, I think in previous discussions, we’ve referenced the infamous Berlin airport and how people thought it was going to be finished in 2012 until the very last minute. And then it was postponed by almost a decade.
Have you seen anything like that happen in the software industry? Where people waited too long before they announced that the project was running late?
No, but there are technical and organizational red flags
Stephan: Not exactly, but there are situations that have given me a bad gut feeling.
Here’s an example of something that I occasionally see in the field:
Suppose that you have a microservice architecture that has a lot of interdependent services. These services all communicate with one another. But you don’t have a good solution in place for when a service becomes unresponsive.
Normally, I recommend having a circuit-breaker pattern that kicks in when a request doesn’t get a response. Instead of flooding the service with repeat requests, the requesting service would wait quite a while before attempting the request again.
This pattern is essential for applications that run in high-traffic environments or that depend on external services. If you forget this pattern, you’re creating a ticking time bomb. An unresponsive service could trigger an enormous clusterfuck that brings your whole system down.
The other thing is these kinds of architectural issues are expensive to resolve. You need to take the time to do a proper investigation and select the right tooling or framework to improve your architecture.
So that’s one technical red flag that I would raise very early because the risk to the business is very high.
You should also look out for organizational red flags. One example is any influential employee with a strong personality who tends to dominate the rest of the team. This person might be impatient, dismiss the opinions of others, and is generally reluctant to consult the team for important decisions. They make all the key decisions on their own.
That’s not how a team is supposed to work. You don’t want a team full of rockstars, you want people who contribute equally. Otherwise, if all of the load and responsibility is placed on specific individuals, your team could burn out. So that’s another red flag.
How to make anyone feel comfortable enough to raise red flags?
Merlin: Yes, I think the importance of team culture is often underestimated. I was just reminded of the book “The Unicorn Project” and the concept of psychological safety that was mentioned in the story.
The idea is that you foster a culture where people feel comfortable raising issues regardless of their seniority. So a junior developer can say, “Hey, I think we’re going in the completely wrong direction”, without being dismissed or ignored by their more experienced colleagues.
How do you foster a culture where everyone feels comfortable raising red flags?
Lead by example and make sure that everyone contributes
Stephan: I think you have to lead by example. You have to be transparent about your own mistakes and share the things that you yourself have learned. Nothing is perfect, and mistakes happen. So you need to show that you’re willing to learn from your mistakes too.
That hopefully leads to a culture where you treat failures as learning opportunities. Like I said in our post-mortem discussion, you find out exactly what happened and why rather than focusing on whose fault it is.
Merlin: Yes, we’ve already covered mistakes, but I’m talking about things that are more subjective. For example, architectural decisions.
For example, suppose that a senior engineer has built some microservices, but each one is trying to do too much. And then, a junior engineer reviews the code and suggests that the microservices should be broken down into smaller components so that each endpoint has a single responsibility.
In this case, there’s the potential for a clash of opinions. And you have the dynamic of a junior developer criticizing the strategy of someone more senior.
How do you ensure sure people feel it’s safe to challenge ideas regardless of seniority?
Stephan: To be honest, I think it should be perfectly natural for a senior developer to have their ideas challenged.
If someone questions your work and asks why you built something in a certain way, you can see it as a mentoring opportunity rather than a debate. Often, someone with less experience might not understand the bigger picture, and this is your chance to fill them in.
Also, junior developers aren’t stuck in their ways and often ask interesting questions. There’s usually a good reason for building things a certain way, but sometimes there are no easy answers. So questions like these can help you rethink certain decisions and see things from a different perspective.
And how do you foster this kind of culture?
When starting a project, ask your team for their opinions about how to implement it. Take their input seriously and try to make sure that everyone gets the chance to speak. Make sure that no one stays silent, and encourage people to be honest.
If you see someone senior dismissing the opinions of a more junior colleague, you need to act quickly. That’s a clear red flag. Let them know that’s not the culture you want in your team. If they want a more hierarchical organization, your startup probably isn’t the right place for them. Make sure they know where you stand on team culture.
What if you’re just behind schedule?
Merlin: Got it. So we have technical red flags and organizational red flags. But what about the simplest kind of red flag? Which are estimation and timing.
In this scenario, the technology is fine, and the team works well together, but you just underestimated the complexity. Perhaps there were certain things that you didn’t know when starting the project.
So you have to raise a red flag about the planned delivery date — it’s no longer realistic.
You don’t want to cause panic too soon, but a lot of other decisions might depend on this delivery date. So you need to give people the time to adjust their planning.
How do you get the right balance?
If you’re running the project properly, you can adapt to delays well in advance
Stephan: Well, the thing about underestimated projects is that there are different levers you can pull to fix them. You don’t necessarily have to sacrifice your deadlines. Maybe you can reduce the scope instead?
For example, you could pull in a business stakeholder and make sure that you have your priorities straight. Work with them more closely to reduce the communication overhead and the time it takes to get feedback. Maybe you have a kind of “situation room” similar to the one we described in our critical incident discussion. Once you’re communicating more closely, you might discover specific features that you can delay.
Also, if you’re managing the project properly, a scheduling red flag shouldn’t come as a surprise. Ideally, you’re tracking your progress very closely, and perhaps you’ve already raised “yellow flags” about the timing.
If you need to remove significant features from the scope of a release, then you escalate that to management and present them with a clear choice.
- Either we reduce the scope and keep our original milestone, then deliver the postponed features in a follow-up release “X” weeks after the main release.
- Or, we keep our original scope and delay the release by “X” amount of time.
Merlin: Wow, yes, we could open up a whole discussion on project management methodologies and Agile development. I think we need to give that subject its own dedicated session.
For now, I think we’ve already got enough food for thought on when to raise red flags, so thanks once again for your input, Stephan.
Stephan: My pleasure, any time.
By the way, if you want to know about how we approach projects vs products check out the very first edition of Tech Insights below: