Why agile software engineering transformations fail...and sometimes succeed

All of my fellow Agile and Software Craft practitioners have experienced failure.  It happens far more often than success, and usually for obvious reasons.  Obvious or not, there are some clear precursors to a successful transformation.

To support high-performing teams, you MUST have the following pillars:

• Solid DevOps infrastructure and practices
• Skill in key software engineering practices
• Teams who are engaged, motivated, and empowered
• Manageable (as in, not-too-much) technical debt in your codebase

If you don't have these things, you need to create them before pushing too hard to be "agile", and certainly before measuring success or failure of your efforts.

Fortunately, there are clear paths to creating all of these.  Unfortunately, there are very few organizational cultures that have the ability to follow these paths.

DevOps Infrastructure and Practices

This is the easiest one to see and remedy (though as with all of these, it does require effort).  If you don't have a cloud-based DevOps infrastructure, there are plenty of experts in the marketplace who can create one for you.  If your teams don't follow good practices (code checkin discipline, a continuous integration (CI) pipeline based on infrastructure-as-code, clear PR rules and automated deployment), these things are all trainable.  They are also the easiest way to improve the developer experience, which has major upsides around retention and productivity.

In other words, if you're lacking in this area, consider tackling your DevOps first. Or at least in parallel with any other pillars.

Software Craft Practices

The "Software Craftsmanship" movement of the previous decade spelled all of this out, and it is still true today.  It is usually what engineers think of when they say "Agile".  Most software engineers who learn these practices wouldn't want to work without them.  Unless they are missing the other pillars.

Cross-Functional, Self-Organizing Teams

Another way of saying this is, "empowered teams."

This is the easiest pillar to create, but many organizations struggle the most with it.  It requires that you drop command-and-control-based management practices, and instead empower teams to do their best work.  Leaders need to actively seek out areas to trust their people.  Leaders also need to raise their tolerance for mistakes and setbacks, and instead appreciate how their teams learn from these things, rather than punishing them.

Trust is difficult in many organizations.  It is a daunting challenge to find a balance between giving people room to learn, and risking damage to the organization's reputation and assets.  Nonetheless, you simply cannot have adaptable, high-performance teams without it.  

However, it can be done, even in less risk-tolerant industries like finance.

A Flexible-Enough Codebase

Another way of saying this is, "Don't carry too much technical debt."  If you aren't familiar with the term, "technical debt" is a way of describing how difficult code is to change.  From a code perspective, it means long, tightly-coupled modules, opaque method and variable names, and a lot of duplication.

If you have too much tech debt, every change requires deep knowledge of the code, and this forces specialization.  If your team members must be hyper-specialized, then only Person X can work on Product Y - it just takes too much time to cross-train everyone unless they're constantly in the code.

Fortunately, there is a simple (but time-consuming) way around this.  "Legacy Rescue" techniques offer a path to clean up the parts of the code that are touched the most.  The payoff for doing this starts slowly, but ramps up quickly, and the end-result are faster production deployments with far fewer surprises.

What If We Have All Of These?

Odds are that you don't have all of them, but if so, congratulations!  You need only start practicing a flexible agile methodology, with the key cadence meetings, and your engineering teams will no longer be a bottleneck.

Of course, that's when you'll find out how agile your product development process is...

Software quality vs rapid growth: are they incompatible goals?

Note: I'm specifically talking about software development organizations, since that's where my experience is.

 

With that out of the way, the short answer is "yes".  Once a company grows beyond a certain size (somewhere between 300 and 1000 people), it simply can't sustain shared values across all of the teams.  Here's why:

 

quality

"Software quality" describes how easy software is to understand, change, and run.  High quality software, in my experience, is only created when teams have:

• High safety - team members feel it is safe to raise issues, acknowledge failures, and admit when they don't know how to do something.
• Comprehensive automated tests - the software is safe to change because an automated test suite exists and is run every time a change is made.
  
• Sound infrastructure - a CI/CD pipeline and modern code versioning tools ensure that it is easy to see what changed when, and build + test + deployment is simple to execute whenever a change is promoted to production.
  
• Good practices - to me, this is usually XP practices.  Things like teamwork (pairing / mobbing or just good code reviews) and test practices (TDD and a balanced test pyramid) are what create the top 50% of code quality.  Everything else really just supports the practices.

There is more to this such as good design and data engineering practices, as well as good agile backlog management and product development.  The core of my experience is in software development, though, so I see things through that lens.

growth

When I refer to "growth", I'm talking about growing a business.  [Edit: more specifically, a consulting business, where "growth" = "more people", since people are the source of revenue.] 

I have spent a lot of time in consulting over the past 15 years, and most consulting businesses aim to grow by 50% or more whenever they can.  I have worked in one location where the software engineering team (and the revenue) doubled in size of the course of a year.  (And it wasn't a small team to start with.)

Growth is good, right?

Well, not always.

the conflict

Very high growth comes at a high cost.  When you increase the size of your team by more than around 25% in a year, it becomes very hard to sustain good practices.  There simply isn't enough time to show the new hires "how we do things here", so they fall back on the way they did things before you hired them.  If you are very stringent about hiring (i.e., only people who already profess to follow the same practices), this can still work, but such people are pretty rare, so it's an unlikely scenario.

Instead, what usually happens is that there is confusion and some level of conflict on how things should be done.  Particularly on teams with more new hires than old hands.

A focus on growth also means that there is considerable pressure to take on work that isn't well-suited to a high-performance team.  If a client doesn't want to update infrastructure and insists that your team works with their non-agile process, the response is often "let's start that way, and we'll reshape the project over time to be better suited to our team."  

But that last part doesn't happen.  There just isn't the time or energy to fight the battles you'd need to fight to make that a reality.  Not when the team is busy trying to create a working product in a sub-optimal environment.

a middle ground

Here's a possible answer: Grow at a sustainable pace.  

Target 10%, with 15% as a stretch.  Say "no" to work that is a poor fit.  

Build a reputation and market the hell out of it.  

If demand gets really high, don't just raise your rates across the board, or you'll filter out most of the work that your people are best suited to do.  Tell your clients that they need to change in order to get value out of what you do, and offer to help them change.

why we don't see this in practice

In large companies, the people setting strategic goals are too disconnected from the day-to-day operations to realize that they are creating problems (and attrition) from the people delivering the product.  They are also humans, and as a human, I know that I'm really good at fooling myself.  

It's easy to tell yourself that you need to grow whenever you can in order to pay bonuses and keep people employed.  The truth is that when you grow too fast, you'll have to lay people off later because of economic boom & bust cycles.  

 My % growth numbers may be a bit naive.  Maybe they should be twice that.  But one thing I am sure of: Growth targets in big-box consulting are far too high to be sustainable for everyone except the executives.

Meditation for Nerds

 

This is a very short guide, based on my own experience and what I have read.  I may update it over time, since the topic is very important to me.   Best to get something down to start with, though, so...

Why Meditate

You can find a lot of reasons why in popular literature.  My own experience is:

• I'm better able to handle stress (I actually feel less of it, rather than being able to handle more).
• I'm less reactive, and less likely to get caught up in someone else's anger.
• I sleep better.
• I understand my own emotions and wants better, and I have much more control over them.
  
• I'm better able to understand the emotions and wants of others.
  

How to meditate 

There are many different ways to meditate.  Most "concentration "practices boil down to:

• Choose an object of meditation, and pay attention to it as well as you can. A common object is the feeling of breath at the tip of your nose.
• Your mind will wander, pretty much constantly, whenever it does, gently bring your attention back to the meditation object.

Most "insight" practices boil down to almost the same thing, but with an extra step:

• Choose an object of meditation, and pay attention to it as well as you can. A common object is the feeling of breath at the tip of your nose.
• Your mind will wander, pretty much constantly, whenever it does, note what thoughts were present when you realized your attention had wandered.
  
• Gently bring your attention back to the meditation object.

That's it. 

That makes no sense.  Why would that help with anything?

The main reason is that it gives you a clear view into your own mind.  It teaches you that your default state is for your attention to wander from object to object, constantly.  It also brings to light the thoughts and your reactions to them, mostly either attraction or replusion.  Knowing this in yourself, it becomes easier to see in others.  Understanding that basic truth at an experiential level makes your self-understanding and understanding of others intuitive, which means you are able to apply this understanding at a level below conscious thought.  This means that the thinking part of your mind can be used to infer deeper meanings because you don't have to work as hard to understand the basic context of the situation.

Another reason is that the simple exercise of deliberately and repeatedly redirecting your attention to an object you choose increases your ability to keep your attention where you choose, which seems obvious.  It is powerful because you can keep your attention on a person who is expressing intense emotion, rather than reacting to the emotions this triggers in you. 

That sounds too easy.  How long does this take?

 Seems like forever, sometimes.  My own experience is that after a few days of daily sits, I felt like I was a bit less reactive.  After several weeks I was sure of it.  After several years, I feel like I have a fundamentally different relationship to myself and other people.  It isn't blazingly obvious in any given situation or context, but I find that I'm content much more often, and reactive much less often.

Answered in another way, I found that I got good results by sitting quietly for 10-15 minutes per day when I started doing this, and that was my schedule for several years.  The important thing seems to be consistency, rather than duration.  That said, I aim for 30-45 minutes per day now that I'm 10+ years into my practice, and I find that I get more out of it. 

That still sounds too easy.  

I have found that I get much more out of it from reading on the subject, and trying a lot of variations on the basic techniques.  I also find that having a "core" practice is helpful. 

When I say "variations", I'm referring to traditions with millenia of history behind them.  Much of that history is concerned with things that I don't find useful (the divinity of the buddha, the definition of an adept meditator, obedience to a teacher or practice, etc.).  As a non-theist, I simply skip those parts and work with the rest.

One of the most useful variations which I have tried is "Metta", with the closely-related "Tranquil Wisdom Insight Meditation".  In this practice, the "object" is a feeling of happiness and goodwill toward yourself and others, rather than the feeling of the breath.  There are stock words that help to bring up this feeling ("May I be free from suffering, may I be free from ill willl, may I be filled with loving kindness, may I be truly happy").  There is a lot of useful writing on the subject.

Another is "noting", where you pause to notice what your current thoughts are.  This can be done as part of a breath meditation (whenever your attention wanders, note before returning to the breath as stated above), or while doing any undemanding or repetitive activity.

References 

There are a LOT  of works on this subject.  The one I have found most useful and accessible is "The Mind Illuminated", by John Yates a.k.a. Culadasa.  It is the closest to a detailed instruction manual that I have found.

For a much more in-depth (and much less accessible, IMHO) but still mainly secular take, Daniel Ingram's "Mastering the Core Teachings of the Buddha" is hard to beat.  Full disclosure: I bought it a year ago & haven't read all of it yet.  It has given me a number of useful insights and ideas for my practice, but I think it was written for meditators who are much more advanced than me.  Also, the author makes no bones about being an opinionated S.O.B., so be prepared.

More to follow, probably.

Greater Than The Sum Of Its Parts: Leading High-Performance Teams

A good leader doesn't tell the team what to do.  They model the behavior that they want from the team, and learn how to encourage people who get on board with it.

So what types of behavior should a leader demonstrate?

Look in the mirror

Every leader wants a high-performing team, but very few do what it takes to actually create one.  Building high-performance teams is hard.  It involves a lot of work, and a lot of risk on the part of the leader.  It is much easier to go along with the status quo, and "manage up" while you aim for the next promotion.
If your team isn't performing the way you want them to, it's your fault.  Accept that and you have a powerful incentive to learn and change.  Even if you have organizational obstacles, even if your boss seems like a short-sighted jerk, you're the leader and you have the responsibility and the power to effect change.

A good team deserves a good leader.  To become one requires a significant investment on your part.  On the upside, becoming one is one of the most rewarding work-related experiences you will ever have.


Do, don't tell

Telling your team what you want or what you think isn't a convincing way to lead.  Your actions speak much louder than your words.  Leading means doing the things that you want your team members to do. 

Failure is safe

"I screwed that up" is a very powerful phrase.  Simply admitting error is good, but admitting you made a mistake and you don't feel bad about it is better.  Failure is a much better teacher than success, and being able to quickly learn and move on is the central characteristic of a high-performing team.

Individuals don't succeed, the team does

For the leader, this means not taking credit for the team's work.  Have different team members run the team's demo.  (Regular demos are central to agile software delivery, but they should be part of any team's regular cadence.)  You should also look for opportunities to praise people's good work in front of your boss or any other stakeholders, but try to frame it in terms of collaborative work.  Don't worry - you'll still look good in front of the boss.  A great team-builder is much more valuable than someone who does great individual work.

Don't solve the team's problems for them.   Whenever an individual comes to you for answers, use that as an opportunity to facilitate collaboration.  Get at least one other team member involved in the conversation.  Encourage them to share ideas.  Act as the secretary to capture their ideas and feed them back.  It doesn't matter if they don't come up with the same approach that you would - they'll be much better at implementing their own idea than they would be implementing yours.

Encourage Experimentation

This goes along with "it is safe to fail."  You want the team to test out new ideas, but not get bogged down if they don't work out.  "What's the simplest thing we can do to see if this will work?" is a powerful question.  "What did we learn?" is another.



These things are powerful whether you're delivering software, building a marketing campaign, or managing corporate finances.  Having a team-oriented, fail-forward mindset is a great benefit whether you call your team "agile" or not.

photo credits

Alex Iby

José Alejandro Cuffia