Monday, May 15, 2023

Why agile software engineering transformations fail...and sometimes succeed

All of my fellow Agile and Software Craft practitioners have experienced failure.  It happens far more often than success, and usually for obvious reasons.  Obvious or not, there are some clear precursors to a successful transformation.

To support high-performing teams, you MUST have the following pillars:
  • Solid DevOps infrastructure and practices
  • Skill in key software engineering practices
  • Teams who are engaged, motivated, and empowered
  • Manageable (as in, not-too-much) technical debt in your codebase
If you don't have these things, you need to create them before pushing too hard to be "agile", and certainly before measuring success or failure of your efforts.

Fortunately, there are clear paths to creating all of these.  Unfortunately, there are very few organizational cultures that have the ability to follow these paths.

DevOps Infrastructure and Practices

This is the easiest one to see and remedy (though as with all of these, it does require effort).  If you don't have a cloud-based DevOps infrastructure, there are plenty of experts in the marketplace who can create one for you.  If your teams don't follow good practices (code checkin discipline, a continuous integration (CI) pipeline based on infrastructure-as-code, clear PR rules and automated deployment), these things are all trainable.  They are also the easiest way to improve the developer experience, which has major upsides around retention and productivity.

In other words, if you're lacking in this area, consider tackling your DevOps first. Or at least in parallel with any other pillars.

Software Craft Practices

The "Software Craftsmanship" movement of the previous decade spelled all of this out, and it is still true today.  It is usually what engineers think of when they say "Agile".  Most software engineers who learn these practices wouldn't want to work without them.  Unless they are missing the other pillars.

Cross-Functional, Self-Organizing Teams

Another way of saying this is, "empowered teams."

This is the easiest pillar to create, but many organizations struggle the most with it.  It requires that you drop command-and-control-based management practices, and instead empower teams to do their best work.  Leaders need to actively seek out areas to trust their people.  Leaders also need to raise their tolerance for mistakes and setbacks, and instead appreciate how their teams learn from these things, rather than punishing them.

Trust is difficult in many organizations.  It is a daunting challenge to find a balance between giving people room to learn, and risking damage to the organization's reputation and assets.  Nonetheless, you simply cannot have adaptable, high-performance teams without it.  

However, it can be done, even in less risk-tolerant industries like finance.

A Flexible-Enough Codebase

Another way of saying this is, "Don't carry too much technical debt."  If you aren't familiar with the term, "technical debt" is a way of describing how difficult code is to change.  From a code perspective, it means long, tightly-coupled modules, opaque method and variable names, and a lot of duplication.

If you have too much tech debt, every change requires deep knowledge of the code, and this forces specialization.  If your team members must be hyper-specialized, then only Person X can work on Product Y - it just takes too much time to cross-train everyone unless they're constantly in the code.

Fortunately, there is a simple (but time-consuming) way around this.  "Legacy Rescue" techniques offer a path to clean up the parts of the code that are touched the most.  The payoff for doing this starts slowly, but ramps up quickly, and the end-result are faster production deployments with far fewer surprises.

What If We Have All Of These?

Odds are that you don't have all of them, but if so, congratulations!  You need only start practicing a flexible agile methodology, with the key cadence meetings, and your engineering teams will no longer be a bottleneck.

Of course, that's when you'll find out how agile your product development process is...