Your transformation toolkit

Advance your enterprise testing strategy with our transformation toolkit.

Learn more


Metrics: The good, the bad and the misunderstood

Organizations focused on continuous improvement no doubt rely on metrics throughout all areas of the business. When used effectively, metrics can provide valuable insights on the effectiveness of DevOps processes and strategies. On the contrary, when the wrong metrics are tied to business initiatives, it can lead to costly inefficiencies across areas such as software delivery, software quality, and security.

In this episode of DevOps Unbound, host Alan Shimel, (now Techstrong Group) CEO and Editor in Chief, is joined by co-host Mitch Ashley, CEO of Accelerated Strategies Group (now Techstrong Group), Helen Beal of DevOps Institute, Viktoria Praschl of Tricentis, Frank Ohlhorst, principal analyst at Accelerated Strategies Group and Alex Hidalgo of Nobl9 in a candid conversation on how to use DevOps metrics effectively. Key points touched on include:

  • The downside of having too many or too little metrics
  • The number one factor when it comes to establishing DevOps metrics for your organization
  • Why selecting DevOps metrics should involve the entire team

View the full session video below and you can read the complete transcript.

Just because you can, doesn’t mean you should

Alan opens the discussion by recalling a quote that an old mentor shared with him some years ago to the effect of, ‘everything can be measured, and we measure everything…and based upon these measurements we determine who wins and who loses, and by how much.’ After a moment of reflection, he presents the panel with this question: “Is this still true or have numbers lost their meaning and/or importance?”

Helen: “I’m not sure it’s ever been true…I think what I’ve always observed is just people fall into one of two camps. So, either they’re not measuring enough – meaning they don’t have enough metrics, or they have far too many. We need to have the facility to be able to get to the metrics that we want, but we have to be able to choose them and be able to think of them as an adaptable framework.”

Frank: “I think it also comes down to an issue of context. People can measure whatever they want out there and they often do, and they spend so much time measuring, that they don’t take into account the context of the measurement. You can measure certain processes in the DevOps process and they really don’t matter when trying to achieve a particular goal.”

Alex continues the sentiment by warning us to not base our metrics on other large, successful companies simply because they are successful. He adds, “One of the big problems that I often see is that people end up trying to measure themselves against others. They hear ‘Oh, at my last company’ or, ‘I heard at Google they’re able to deploy X number of times a day and let’s aim for that.’ But why? Do you need to deploy that many times a day? Is it applicable to you or your application or your organization or your customers? Are you gaining anything from trying to do things the way you know others are? Every situation is unique. Every company is unique. Every organization is unique. Every application is unique. And that means you really need to take a step back and think about what is right for us and what do we need to improve upon?”

Taking Alex’s point one step further, Viktoria offers her perspective on the importance of establishing a baseline while understanding that “metrics will change over time as you mature your processes”. 

If you want a cake, you have to measure the flour

Alex makes a tasty analogy to this segment when he says “you measure flour to bake a cake; you don’t just collect flour. Otherwise, you’re not going to end up with a cake.” The idea of course alludes to being intentional about your desired outcomes. Sure, you can leverage metrics models of others having achieved similar outcomes to what you’re aiming for, while keeping in mind that the goal is to continuously improve. But how do you know if you’re getting better at the things that matter most?

Frank: “I think people run not only into an issue of where do I establish metrics? Or why am I doing metrics? But also, how you interpret those metrics. ‘Hey, we accomplished something’, but when it really comes down to it, what did we accomplish and how did that improve the process? And I think people are losing sight of that.”

Mitch: “I really like the word ‘outcome’. Have we achieved an outcome when we’re developing software or creating an experience or something out of the process? Because ultimately, that’s what matters. We can improve the things within it. We can automate the processes within how we create software. But it’s all for a purpose… if we’re not doing a better job of delivering on an outcome, maybe those things we’re working on internally are sort of navel-gazing, if you will, looking internally but it isn’t really that valuable. Let’s work on the things that we can connect to an outcome. That’s what I would propose.”

Helen goes on to share a recent experience at the DevOps Enterprise Summit, where she was nudged by colleague and friend, Jon Smart, to embrace the term ‘time to learning’ vs ‘time to value’. Adding “We’ve moved from time to market, to time to value, and now we’re in time to learning, where we’re recognizing that something only becomes useful when we’ve seen it have value and we’ve learned from it and decided to do something else.” She continues, “It’s the same as this kind of impact-driven development as well, this idea that we really focus on having an idea about what we’re trying to do, and then measuring that it did what we thought it was going to do.”

The dark (and bright) side of metrics

Not all metrics are good metrics. According to Alan, “There’s a dark side of these metrics, which is that in the absence of facts or in the absence of knowledge, we try to fashion metrics as a way to compensate. And so, we compensate by coming up with what some may consider are frankly nonsense metrics.”

Referring to earlier in the session where the panel touched on the importance of collecting enough data to train a model, Helen responds “it’s about data volumes again, in this kind of assessment environment it is quite important to have enough data to weed out variations from people that maybe don’t care or are skewing for other reasons.”

On the upside however, everyone agrees that of course there is tremendous value in the right metrics. So, the question then becomes, how do we discern between ‘good’ metrics and the not so valuable or ‘bad’ metrics?

Metrics shouldn’t be inflicted on people. They should belong to the team.

– Helen Beal

Helen: “The most important thing about measurement is being conscious about what you’re doing.” This goes back to the reason why we are measuring in the first place and equally important is making sure that everyone on the team is aware of that ‘why’.

Frank: “Making sure people on the team know why we are measuring things is key, not only from a cultural perspective, but also from a value perspective.” This allows for clearer outcomes and a much better understanding of what metrics are needed. 

Closing out this engaging conversation, Mitch remarks, “what I’m struck by is the richness of just the word ‘metrics’ and what all it can mean and the depths of it and aspects, whether it’s the human side of it, whether it’s the ‘why’, whether it’s the outcome…I think there are several good takeaways  from this, and so I hope it’s helped everyone think about maybe why you’re doing what you’re doing and how you want to lead that going forward.”

Simply put, when used appropriately, metrics provide tremendous value to an organization when it comes to delivery performance, assessing your teams’ strengths and weaknesses, shaping product roadmaps, and so much more.

[Watch the full “Lies, Damn Lies and Metrics” episode on demand]