Metrics

by Luiz "Q" Quintela

Metrics is a subject that has a very polarizing effect. In an average Agile Development shop, there is always a lot of time, effort, and money spent gathering metrics. As a coach, you hear a lot of questions such as, “How can we use metrics to improve our Agile roll out?” or, “How can we use metrics to improve our Agile practice?

Basically, there are those who demand a series of metrics and targets and those that question the validity of metrics and what you are trying to achieve. Keep in mind that very often metrics do not drive the behavior we would expect.

Lines of Code

I once worked with CTO that used lines of code as a metric to highlight his department’s achievements. What behavior are we driving here? If you were a developer and found a more efficient way to execute something, would you refactor 200 lines of code down to 30? If you are being evaluated based on lines of code, you would rather have as many as possible, wouldn’t you?

A few months later, the same individual, who by that time had attended the CSM and CSPO classes, wanted to motivate the teams by comparing the story points of one team against the others! Isn’t that a great idea? Of course, it is NOT!

How many Scrum Masters and Agile Coaches have to “fight” against that? Or the tendency of management to create targets for velocity? Just because we did 28 points instead of 24, it does not mean the team got better.

Commitments

I know of a large financial services organization whose governance department employed a substantial team to come up with new metrics and associated visualizations. What behaviors emerged?

One of their dozens of metrics was called “commitment reliability” and it was used punitively. You can easily guess the behavior that it created. Teams either became nearly paralyzed trying to ensure exactness in their estimates or they started to “pad” their estimates just in case something went wrong. How does this align with the Agile Manifesto? Especially, the part that says, “Individuals and interactions…”

Then I did some digging. The first thing I learned was that if the teams didn’t pad the numbers, the Scrum Master would. All this to appear “green” in the dashboard because, according to them, “leadership only looks at the reds.”

Further digging and I found that the Scrum Masters eventually realized that the report was generated three days after the end of the Sprint. Instant solution! Go fudge the numbers and make everything fit!

Metrics Drive Behavior

I am sure that you have your own examples of bad behavior driven by metrics. However, I am not trying to suggest that all metrics are bad. Metrics must be used properly. They can never replace thinking.

I am a firm believer that metrics can be used properly and effectively to provide an incentive to teams and leadership while following the three pillars of Scrum: Transparency, Inspection and Adaptation.

There are, however, several pitfalls to avoid. There is that “little thing” called velocity. Velocity can be based either on story points or simply on how many user stories were completed during a Sprint. Most teams are mistakenly led to believe they must always increase velocity. Focusing entirely on velocity will lead to shortcuts, such as reduced testing, which in turn leads to decreased velocity due to rework and fixes.

I always suggest reading about the Hawthorne Effect: what is measured will improve, at a cost, as well as Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure.

Metrics can be used as part of Kaizen, continuous improvement. Continuous improvement it is what you commit to. Things do not become “perfect” instantly and when they get “perfect” they very likely will not stay that way for very long.

Adopting Metrics

When properly applied, metrics are a terrific way to influence behavior, highlight areas of improvement and eliminate waste.

Before you decide on which metrics to adopt, it is important to consider:

  • How many metrics?
  • How long should we use them?
  • Who chooses which metrics?
  • What behavior are we trying to influence?
  • What are we trying to improve?

Also think about the differences between the metrics a team uses internally and those metrics that have a broader audience. For example, a team may want to track the elapsed time to build and run unit tests to determine whether that process needs improvement. That metric is internal to the team and has a completely different objective than a business value burn-up chart, which should be communicated outward. I am quite aware that there is always a certain reluctance in radiating metrics outside the team or the development department. When this is the case, remember the imperative for transparency!

Scrum’s transparency will surface issues within your organization and its processes. It will not fix them, but it will show you where to focus your efforts.

Metrics should reinforce Agile principles. Metrics should also foster conversations and provide frequent and regular feedback. Be aware of how metrics are used or perceived to be used. The misuse of a metric—often by leadership, but not always—is likely to promote intentional manipulation of the metric, a dysfunctional behavior referred as “gaming.” (See the example above.)

Productivity and Value

Then there is the issue of productivity. How do you measure that? Most people will say that you don’t or can’t and I agree. Scrum and Lean tell us that the primary measure is value delivered, also referred as return on investment (ROI). ROI is a high-level measure that is not subject to “gaming.” You know how much you spent to develop that feature that nobody uses, don’t you?

Can you spell poor product management? I am going to talk about this in another blog entry.

Speaking of value, outcomes and the value created are not measured in many companies. I read many studies that attribute that fact to our tendency to measure what is easy. An outcome can be, for example, to increase revenue, improve the usability of a product, or increase user satisfaction with the Android version of your product. These examples may appear obvious, but many organizations do not measure what is important!

When I ran development departments, I tracked the resulting business value of releases and shared them with the whole department. It is far more rewarding for teams to see the impact their work had for the company. It becomes clear why they worked on those features and it does wonders for team morale. When you share release results, you will notice that teams become engaged, focused, and better at making decisions.

I can tell you one story where a developer got curious about a huge feature and the cost necessary to create it. It turned out that very few people would have used it. After some digging we determined that very expensive feature would only bring in $20K additional revenue for the year! Speaking of poor product management!

Engaged, focused teams also produce a lot of discussion about the items in the product backlog. The example above is just one of many times that I saw items of questionable business value being quickly excised from the backlog through research and discussion. Incidentally, we also excised the bad product owner who came up with that feature that had so little value…

I bet that leadership will come onboard much faster and easier when they see the results of this approach.

Suggested Metrics

I often get questions about what metrics I would suggest. Obviously, I am going to say that is up to the teams and their leadership to collaboratively decide, but if you are looking for ideas, some of the following are a good starting point for discussion, in no order of importance:

  • Customer Satisfaction
  • Business Value per Point
  • Defect Count
  • Technical Debt
  • Team Morale
  • Time to Market
  • Return on Investment

Whatever you decide to track and report, don’t use it to replace thinking!

Concentrate on radiating information and how it influences outcomes. You want to be able to make informed decisions quickly and accurately. As teams evolve and behaviors change, consider the “shelf life” of your metrics and replace them when they are no longer useful. Focus on the behavior you want to encourage and keep the outcomes in mind, because as the Agile Manifesto says, “Working software is the primary measure of progress.”