Why agile software engineering transformations fail...and sometimes succeed

All of my fellow Agile and Software Craft practitioners have experienced failure.  It happens far more often than success, and usually for obvious reasons.  Obvious or not, there are some clear precursors to a successful transformation.

To support high-performing teams, you MUST have the following pillars:

• Solid DevOps infrastructure and practices
• Skill in key software engineering practices
• Teams who are engaged, motivated, and empowered
• Manageable (as in, not-too-much) technical debt in your codebase

If you don't have these things, you need to create them before pushing too hard to be "agile", and certainly before measuring success or failure of your efforts.

Fortunately, there are clear paths to creating all of these.  Unfortunately, there are very few organizational cultures that have the ability to follow these paths.

DevOps Infrastructure and Practices

This is the easiest one to see and remedy (though as with all of these, it does require effort).  If you don't have a cloud-based DevOps infrastructure, there are plenty of experts in the marketplace who can create one for you.  If your teams don't follow good practices (code checkin discipline, a continuous integration (CI) pipeline based on infrastructure-as-code, clear PR rules and automated deployment), these things are all trainable.  They are also the easiest way to improve the developer experience, which has major upsides around retention and productivity.

In other words, if you're lacking in this area, consider tackling your DevOps first. Or at least in parallel with any other pillars.

Software Craft Practices

The "Software Craftsmanship" movement of the previous decade spelled all of this out, and it is still true today.  It is usually what engineers think of when they say "Agile".  Most software engineers who learn these practices wouldn't want to work without them.  Unless they are missing the other pillars.

Cross-Functional, Self-Organizing Teams

Another way of saying this is, "empowered teams."

This is the easiest pillar to create, but many organizations struggle the most with it.  It requires that you drop command-and-control-based management practices, and instead empower teams to do their best work.  Leaders need to actively seek out areas to trust their people.  Leaders also need to raise their tolerance for mistakes and setbacks, and instead appreciate how their teams learn from these things, rather than punishing them.

Trust is difficult in many organizations.  It is a daunting challenge to find a balance between giving people room to learn, and risking damage to the organization's reputation and assets.  Nonetheless, you simply cannot have adaptable, high-performance teams without it.  

However, it can be done, even in less risk-tolerant industries like finance.

A Flexible-Enough Codebase

Another way of saying this is, "Don't carry too much technical debt."  If you aren't familiar with the term, "technical debt" is a way of describing how difficult code is to change.  From a code perspective, it means long, tightly-coupled modules, opaque method and variable names, and a lot of duplication.

If you have too much tech debt, every change requires deep knowledge of the code, and this forces specialization.  If your team members must be hyper-specialized, then only Person X can work on Product Y - it just takes too much time to cross-train everyone unless they're constantly in the code.

Fortunately, there is a simple (but time-consuming) way around this.  "Legacy Rescue" techniques offer a path to clean up the parts of the code that are touched the most.  The payoff for doing this starts slowly, but ramps up quickly, and the end-result are faster production deployments with far fewer surprises.

What If We Have All Of These?

Odds are that you don't have all of them, but if so, congratulations!  You need only start practicing a flexible agile methodology, with the key cadence meetings, and your engineering teams will no longer be a bottleneck.

Of course, that's when you'll find out how agile your product development process is...

Developer Skills: Drawing

You aren't a good developer if you can't draw.

Fortunately, if you're a human, you can.  Drawing isn't an inborn talent, it's a technical skill that can be learned.  Write the previous sentence on a piece of paper to prove it to yourself.

There.  You just drew a whole bunch of letters quickly and (hopefully) legibly.  Words are a complex set of shapes that need to be drawn in a particular sequence to have meaning.  It's the same basic skill you use when drawing non-character shapes.

Drawing isn't a binary skill that you either have or do not have.  It's a continuum, and even at the shallow end (people with barely-legible handwriting), you have enough of it to communicate ideas visually.

So that changes the top line of this post to:

You aren't a good developer if you don't draw.

Visual communication is much more powerful than text.  (This study found 65% retention over 3 days for images vs text.)  We also absorb visual information much faster than text.  (This link goes nowhere.  Do your own research you lazy bastard.)

Communication is a key software development skill.  Consider how often you need to talk to people, and how badly things have gone wrong when you don't.  Design sessions, customer collaboration sessions, backlog grooming, story estimation, etc, etc, etc.

And yet, how often do we get up an draw a picture in an estimation session?  How often do we do it when talking about design?

Some years (decades, actually) ago, I started drawing simple box & line diagrams whenever talking about design.  Now I don't even bother to have a design conversation without that.  Even a simple nested box with a few labels in it will speed things up considerably.  It gives us something to point to and summarizes basic assumptions.

So, do you want to supercharge your development skills?  Don't bother with Angular 2.0, typescript, SpringBoot or DropWizard.  Practice drawing!

Why don't you?  What excuses have I heard?

• I don't have any talent.
• I'd look like a five-year-old.
• I have gone paperless.

All of these really mean, "I'm embarrassed that my simple boxes and lines aren't as good as what a professional sketch artist can do."

They certainly don't need to be.  The easiest way to get better (in fact, the only way) is to just do it. For yourself if you must, but preferably to show others.  Their drawing skills are probably just as bad, so what have you got to lose?

Still too embarrassed?  Fine, here are a few simple ways to improve:

• Switch back from digital notes to a paper notebook (I like the Levenger Circa system, but Staples' Arc is just as good) and doodle in it for a while.  (You can still capture photos of your notes to stick in Evernote, OneNote, Keep or whatever.)
• Google the word "doodle" (or whatever else interests you) and copy whatever appeals to you onto paper.
• Google "learn to draw" and choose from websites and YouTube videos.
• Buy a book and work through it.  I liked "Learn to Draw in 30 days", but there are loads of others to choose from.

Bonus: Get others to draw!

I find this is much more powerful if you can get other people to draw on a whiteboard with you.  It clarifies what they are saying, and forces them to give you something concrete to point to.  

This is particularly useful in determining if A) someone is trying to convey and important concept that you don't understand, or if B) they're a clueless idiot blowing smoke.

Give it a try.  At the very least it will give you something to do in boring meetings, while making you look engaged.

It's More Fun With Two: Pair programming done right

A couple of months ago, I blogged about some of the mechanics of pair programming.  This post is more about the attitudes around it and supporting practices.

Pair programming is often held up as the one true path to good code, but there are some dissenters.  Many of these people aren't naysayers, but instead believe that while pair programming has it's place, so does writing code in blessed solitude.  Some (at least anecdotally) think that pairing is the root of all evil.

The truth isn't somewhere in between.  Pair programming really is coding nirvana, but only if you do it right.

Keys to Success

I have spent the last three years in a software development organization that has strong values around pair programming (after spending 20 years in 23 other organizations that didn't).  It is pretty consistent - most of the seven teams practice nearly 100% pair programming.  Most are pretty good at it.  Some are awesome at it.  In my experience, the following keys are what allowed them to reach levels of never-before-seen awesomeness:

• People - you have to eliminate assholes from the organization.  If you have developers who belittle or bully other developers, they will destroy your ability to make this work.
• Frequency - you have to do it a lot.  By "do it", I mean pairing pretty much all of the time, and swapping pairs daily or almost daily. (Some people advocate swapping even more often, but I don't have direct experience with that practice - yet.) Do this for at least 2 months before allowing any solo work.  You simply can't get good at it unless you have a lot of practice.
• Urgency - build and maintain a sense of urgency across the team.  All of the developers should be motivated to get stories all-the-way to done as soon as they can, with zero defects.
• Communication - pairs should be constantly discussing, whiteboarding, and calling over a product owner (or proxy) to clarify details.

Benefits

I'll assume for the moment that your organization doesn't practice pair programming, or does so only on a part-time basis.  Why should you go to the trouble?  What will you get out of it?

First of all, there are some sound psychological reasons.  From a programmer's perspective, it is just plain more fun to work with a pair all day.  You keep each other honest, which means that you never go home at the end of the day feeling like you haven't accomplished much. Productivity really is its own reward.  It's hard to overstate the profound impact of going home at the end of the day - almost every day, with a sense of accomplishment.

It also feeds our social needs.  You spend the day in a relaxed conversation (and occasional heated debate), talking about something you both enjoy and understand.  How often do you spend 6 or more hours talking to your significant other about something you're both into?  Once the team has been together a while, you'll also get a sense of camaraderie, as well as building up a lot of inside jokes.  I haven't worked at many jobs where I felt as close to my coworkers as I have on projects were we constantly pair.  (This is also a good reason to look for a sense of humor when you interview people.)

Finally, there are a number of excellent business reasons.  Here is a logic-chain for you to consider:

• When developers are committed to pairing and feel the need to work fast, they work harder to understand each other.
• To help understand each other, developers usually draw pictures on a whiteboard or paper.
• When people communicate visually as well as verbally, they understand what the other person intends to do much better.
• When people understand each other's intentions and plans really well, they can point out flaws, solutions to the flaws, and additional opportunities for improvement much more quickly and effectively.
• When developers are communicating quickly and effectively, you get the benefit of two programmers' experience in a fraction of the time, with almost no software defects.

This means high-quality code done quickly.  There aren't too many customers who would say no to that.

Pitfalls and objections

Of course, no plan ever survives contact with the real world.  There are a lot of things that you have to adapt to.  Here are a few of the problems I have seen and their solutions.

Problem:

How do you get a team to feel a sense of urgency, communicate a lot and pair all of the time?

Solution:

Tell them what you want them to do and why.  Coach them to do it.  It's simple, though not easy.  Hire an agile coach or two if you can't do it yourself.  And remove any jerks from the team.  (Like I said, simple, but not easy.)

Problem:

Noise!  Isn't it really distracting to work in an environment with constant chatter and movement?

Solution:

This one mostly solves itself.  You'll be amazed at what people can adapt to.  If the acoustics in your space really suck, look into some sound absorbing ceiling tiles and baffles.  They can make a pretty big difference.

Problem:

The developers on my team just aren't committed to this.  They're not jerks about it, they just don't believe in it.

Solution:

This is one of the most difficult problems, and probably the most common.  The only answer that I have seen work is to have the committed people significantly outnumber the non-comitted people.  Culture tends to follow the majority.  If you have to, start with a small seed team and outnumber them with some contractors with the right outlook and skills, then slowly (every month or so) add another person to the team.  When it gets large enough, fission the team into two teams and repeat.  This is expensive, but worth it.

Problem:

There really are some things that are best done solo.  How can I tell my team that they have to pair on them when common sense dictates that it's more efficient to split them up?

Solution:

There are probably far fewer thinks that are better done un-paired than you think.  Enforced pairing at the outset helps the team identify more clearly what they are.  Just tell them that - they'll probably get it!  Give them a time-frame if you must, but it should be no less than 2 months.  After that, he more obvious things like fixing simple UI bugs can be done solo… though if you're doing it right, you may find that you don't have any UI bugs, simple or otherwise.  I'm just sayin'...

Problem:

I can't sell this to my business.  They'll never let us do it!

Solution:

How hard have you tried?  Can you enlist some help in selling the idea to them?  Just as important - do you really need their permission, or will your boss (or his/her boss) provide enough air cover to let you get it done?  If not, maybe your business is filled with jerks and you should just find another job.  I have worked in automotive companies before and I feel your pain.

Summing it up

Pair programming really isn't just the best way to get a project done quickly, it's also the most fun.  Most programming shops don't practice pairing because they haven't tried doing it without understanding the keys to success or the challenges they were likely to encounter, then came to the conclusion that "pair programming doesn't work".

Doing it right really is simple: stress communication and urgency, and insist that your developers pair 100% of the time until they get good at it.  It's a winning formula.

Discussion Question

What other problems could derail implementing pair programming in your shop?

Why are we having a "Post-Agile" debate?

I just read a post by Kurt Häusler to the Software Craftsmanship Google group this evening.

The background for this is a long-term debate about what exactly "Software Craftsmanship" means, and what it's major focus should be.  I think that Kurt did a great job spelling out two of the major points of view:

It seems a lot of people suffering from "agile-fatige" and looking for something post-agile are in two camps. One camp seems to be competent leaders working with poor developers. For them things like Scrum have bought some improvements in the organization and they despair that after all this "agility" the code is still crap. These people crave something like craftsmanship to light a fire amongst their developers. The other camp seems to be competent developers suffering under poor leadership. They see the XP practices and clean code as obvious to the point of being trivial, and wonder why people like Bob Martin are calling for more focus on code, when it is clear, to them, that future improvements lie in better management practices, better understanding of value streams and improved relations between customers, management and developers. These people are currently looking at things like lean and kanban, and finding a lot of valuable stuff there. 


My own perspective is that we don't really need a "post-agile" movement or focus.  Agile (with it's multiple meanings) is still not the norm in most organisations, large or small.  Getting organizations - particularly large organizations - to do more than pay lip service to agile methodologies should still be our main focus.


I wonder if the drive to come up with a "post-agile" methodology or movement is really being pushed so a new crop of developers can have their names associated with it.  Is it more of an exercise in branding than engineering or craftsmanship?
  
Photo by Lovefusion Photography

Too much testing, too little clue

A couple of interesting things shook out of today's retrospective.  It turns out that we're well behind on our targets for the current project, and two of the culprits seem to be:

1. Too much Selenium testing
2. No card wall

The evils of Selenium

The right way to do Selenium testing is pretty well covered in Patrick Welsh's blog and in other places.  Talk about great timing - Patrick was at the office today to give a lunchtime talk on this very subject.

The team decided to handle the Selenium problem by writing tests for only the things that we absolutely had to test through the UI.  Most of us (we'll have to revisit this) also agreed that we would keep the Se tests as small as we could get away, with while still retaining a reasonable level of test protection.


VersionOne or card wall?

The second item is something I should have brought up before, but I wasn't sure of it until I had been on the project for a month.  We have been using VersionOne as our story management system, and doing all of our updates through it.

It's a nice tool and fun to play with.  Just one problem - I wasn't  aware that we were behind this week (as in others) until after it was too late to do anything about it.  Two reasons came to mind for this:

• I'm lazy.  I didn't realize that VersionOne had a Story Board view because I didn't take the time to learn the ins and outs of the tool.  The task-oriented view has far too much information on it to give me an overview.
• We didn't have a card wall that I can just glance up at to see how many stories were still not done and who was working on them.

The team focused on the second bullet - lack of card wall, and I think that is a wise decision. 

It doesn't matter  how good your electronic tools are.   There is no substitute for being able to just look up and see status without any mouse clicks.  It is also very powerful to be able to stand up with another team member or two, and point at cards as you talk.  The simplicity of the setup is what makes it so powerful.


What goes on a card...and on the wall?

I did a quick search for examples of story cards, and this one (scroll down a bit after you click the link) will work.  The best setup I have seen adds some refinements to the cards and their organization on the wall:

• Write the names of the people working on it on the card, or have mini-cards with team member's names on them that you can post next to the card their working on.
• Have some sticky notes handy with a bright color.  If the story is blocked for any reason (even if it's something you expect to get resolved the same day), write a few words on the note describing the block and stick it on the card.
• Organize the cards in columns by status, such as "Dev Ready", "In Progress", "Test Ready", "Test Verified" and "Business Verified".
• Write a "definition of done" where appropriate for each status and post it at the top of each column.  For instance, "In Progress" would have bullet points like:
• Unit tested
• Integration tested if it has external dependencies
• Selenium tested if it has UI components
• Reviewed in both IE and Firefox if it has UI components

I doubt that we'll go to such lengths this late in the project, but it's something to consider for the next one.  It isn't a panacea, but it does solve several problems.

Above all, it's simple.

(Image courtesy of J'Roo's Flickr page)

Agile Purity: How Agile Is Agile Enough?

A Tale of Two Projects
I have been fortunate enough to work on two "green field" projects in a row. Both are/were "agile", but the way they're being managed is quite different.

The first project is/was characterized by:

• The business folks knew at the outset what market they wanted to engage, but not exactly what they wanted the application to do. The direction was "figure it out as you go".
• Virtually no up-front documentation.
• Frequent changes to existing features.
• Considerable 1x1 time between the product owners and developers.
• We had no BA and one QA specialist was assigned shortly before production release.
  
• The product, content and software development people all sat together in one big room.
  
• The system is currently in production maintenance mode, with a possible version 2.0 in the future.
  

The second project is characterized by:

• We are in "Sprint Zero" - doing technology and design spikes now.
• We have detailed specifications of many required features.
• We have several wireframes, scenarios and related documentation available up-front.
• We have a BA assigned to the team and two QA specialists.
  
• The business team is not currently sitting with the development team (though we expect / hope to change that once we start Sprint 1).
  

Better / Best?
The big question is, "which is better?" I think that depends entirely on which developer you ask. There are certainly advantages to having documentation and someone with a formal BA role. You don't need to have difficult conversations with the product people - that's the BA's job. One of the main disadvantages is that "you don't need to have difficult conversations with the product people" - these are the conversations that drive a better understanding of what the feature/system should do. (Determining if it does it right is another conversation entirely.)

Project 1 did get a bit wearing when we would add a feature, then remove it, then add it back in. It was the most fun I have had on a software project, though, and our velocity was enormous.

We'll have to see how things play out on Project 2. My expectation is that the documentation and specialization of roles will make some things go more smoothly. My fear is that we will find out quite late in the game that some core components of the system aren't quite what the business & customers need, and will be expensive to change.

Expect a detailed side-by-side case study in a few months, and more blog posts on related topics in the interim.