Goodhart's law

When a measure becomes a target, it ceases to be a good measure.

People game measures. When a metric is identified as important and subsequently becomes a target for optimization, people naturally adapt their behavior to maximize that metric, often at the expense of the underlying goal the metric was designed to measure. This adaptation frequently leads to unintended consequences and can even subvert the original purpose of the measurement.

There are a few clarifications that need to be made here: usually, there’s a single big measure that people are using to “keep score.” The belief is often that if we push 1 big number up (or down) that something meaningful is going to happen.

For engineering organizations, this impulse usually shows up to solve 3 different problems: velocity, quality, and people management.

Counterpoints & nuance: It’s useful to remember the actual quote at this point, “when a measure becomes a target, it ceases to be a good measure.”

The quote does not mean to not measure. It means to be very careful about messaging to people that you’re looking at that measure, because people will over index on it and give you only that measure, not whatever the underlying thing is.

Software engineering teams should be primarily targeting to regularly deliver things that customers want.

That’s it. That’s the target. That’s what you should message over and over.

In practice:

If you choose to implement metrics, they should be several, and they should be measures in tension with each other. They should not be a single “big number go up”.

Some principles about engineering metrics:

1. Measure collections of things in tension
2. Measure the process to improve the process
3. Emphasize whatever is your current constraint
4. Focus on outcomes rather than outputs

The DORA framework (DevOps Research and Assessment) is a research-backed model for measuring and improving software engineering performance. It has 4 key metrics that capture both the speed and stability of software delivery: Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Recover (MTTR).

The primary value of this framework is that it’s a balanced set of metrics. If you are shipping lots of features of low quality, it’ll show up in the change failure rate and potentially the mean time to recover. If you optimize to try to avoid ever shipping a bug, then you’ll find deployment frequency or lead times to change spiking.

For more details, read: Charles Goodhart, “Problems of Monetary Management: The U.K. Experience,” Papers in Monetary Economics Vol. 1, Reserve Bank of Australia, 1975.