Metrics

I once worked with CTO that used lines of code as a metric to highlight his department’s achievements. What behavior are we driving here? If you were a developer and found a more efficient way to execute something, would you refactor 200 lines of code down to 30? If you are being evaluated based on lines of code, you would rather have as many as possible, wouldn’t you?

A few months later, the same individual, who by that time had attended the CSM and CSPO classes, wanted to motivate the teams by comparing the story points of one team against the others! Isn’t that a great idea? Of course, it is NOT!

How many Scrum Masters and Agile Coaches have to “fight” against that? Or the tendency of management to create targets for velocity? Just because we did 28 points instead of 24, it does not mean the team got better.

I know of a large financial services organization whose governance department employed a substantial team to come up with new metrics and associated visualizations. What behaviors emerged?

One of their dozens of metrics was called “commitment reliability” and it was used punitively. You can easily guess the behavior that it created. Teams either became nearly paralyzed trying to ensure exactness in their estimates or they started to “pad” their estimates just in case something went wrong. How does this align with the Agile Manifesto? Especially, the part that says, “Individuals and interactions…”

Then I did some digging. The first thing I learned was that if the teams didn’t pad the numbers, the Scrum Master would. All this to appear “green” in the dashboard because, according to them, “leadership only looks at the reds.”

Further digging and I found that the Scrum Masters eventually realized that the report was generated three days after the end of the Sprint. Instant solution! Go fudge the numbers and make everything fit!

I am sure that you have your own examples of bad behavior driven by metrics. However, I am not trying to suggest that all metrics are bad. Metrics must be used properly. They can never replace thinking.

I am a firm believer that metrics can be used properly and effectively to provide an incentive to teams and leadership while following the three pillars of Scrum: Transparency, Inspection and Adaptation.

There are, however, several pitfalls to avoid. There is that “little thing” called velocity. Velocity can be based either on story points or simply on how many user stories were completed during a Sprint. Most teams are mistakenly led to believe they must always increase velocity. Focusing entirely on velocity will lead to shortcuts, such as reduced testing, which in turn leads to decreased velocity due to rework and fixes.

I always suggest reading about the Hawthorne Effect: what is measured will improve, at a cost, as well as Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure.

Metrics can be used as part of Kaizen, continuous improvement. Continuous improvement it is what you commit to. Things do not become “perfect” instantly and when they get “perfect” they very likely will not stay that way for very long.

Before you decide on which metrics to adopt, it is important to consider:

Also think about the differences between the metrics a team uses internally and those metrics that have a broader audience. For example, a team may want to track the elapsed time to build and run unit tests to determine whether that process needs improvement. That metric is internal to the team and has a completely different objective than a business value burn-up chart, which should be communicated outward. I am quite aware that there is always a certain reluctance in radiating metrics outside the team or the development department. When this is the case, remember the imperative for transparency!

Metrics should reinforce Agile principles. Metrics should also foster conversations and provide frequent and regular feedback. Be aware of how metrics are used or perceived to be used. The misuse of a metric—often by leadership, but not always—is likely to promote intentional manipulation of the metric, a dysfunctional behavior referred as “gaming.” (See the example above.)

Then there is the issue of productivity. How do you measure that? Most people will say that you don’t or can’t and I agree. Scrum and Lean tell us that the primary measure is value delivered, also referred as return on investment (ROI). ROI is a high-level measure that is not subject to “gaming.” You know how much you spent to develop that feature that nobody uses, don’t you?

Can you spell poor product management? I am going to talk about this in another blog entry.

Speaking of value, outcomes and the value created are not measured in many companies. I read many studies that attribute that fact to our tendency to measure what is easy. An outcome can be, for example, to increase revenue, improve the usability of a product, or increase user satisfaction with the Android version of your product. These examples may appear obvious, but many organizations do not measure what is important!

When I ran development departments, I tracked the resulting business value of releases and shared them with the whole department. It is far more rewarding for teams to see the impact their work had for the company. It becomes clear why they worked on those features and it does wonders for team morale. When you share release results, you will notice that teams become engaged, focused, and better at making decisions.

Engaged, focused teams also produce a lot of discussion about the items in the product backlog. The example above is just one of many times that I saw items of questionable business value being quickly excised from the backlog through research and discussion. Incidentally, we also excised the bad product owner who came up with that feature that had so little value…

I bet that leadership will come onboard much faster and easier when they see the results of this approach.