Measuring Goals
6 minute read
In my last post, we talked about common metric anti-patterns that are focused on the process instead of value. In this installment, we will cover alternatives that can help us remove waste and improve the flow of value.
Defining Terms
Goal: Something we want to achieve that will improve the performance of our organization. In most businesses, performance can be measured by profitability, sales, etc. If we have goals focused on the business, we’ll probably perform better.
Signal: Something that tells us how we are tracking against our goal. A signal is something that may not be able to be measured directly, but only by inference. For example, quality and value are notoriously difficult to measure, so we must find proxies for those that give us insights.
Metric: An objective measure we can use as a proxy for a signal. More on this later.
Goals
So what are our goals? Some good goals for any enterprise are:
- Value: We want to improve the value of what we do by creating, saving, or protecting money.
- Sooner: We want to realize that value sooner with higher quality, less waste, and shorter lead times.
- More innovation: We want to find even better things to deliver and better ways to deliver them to delight our customers.
- Stability: We want solutions that our customers can depend on and we want engaged teams who own those solutions.
Signals
Looking at our goals, there are some keywords we can pick out for signals:
- Money: It’s difficult to track metrics that directly improve income, but we can track metrics that reduce costs. Income will depend on how good the product ideas are, but if we control our costs by reducing waste then it costs less to run experiments to find better ideas.
- Quality: We want higher delivered quality.
- Lead Time: We need to iterate faster and shorten the time between idea creation and idea validation with the end-user.
- Stability: Our customers should be able to depend on our services when they need them and only teams that have high morale and ownership can provide us the level of stability we need.
None of these signals can be measured directly. Even lead time has too many dependencies to be an objective metric that will show meaningful trends. However, we can measure things that give us insights into these.
Metrics
Money
Money and Waste are tightly coupled. More waste costs more money. So if we reduce waste we reduce costs. Waste is a function of process overhead, handoffs, delays, rework, etc. So if we want to reduce waste, we need to measure things that discourage the creation of waste.
To reduce waste, we need to reduce the batch size of changes, deliver them more quickly and validate that they meet customer expectations. We can track this using delivery frequency and defect rates. Depending on the context, customer NPS may also be a valuable insight.
Quality
To improve quality, we need to focus on effective quality gates that provide a stable signal as early as possible. Ultimate quality is determined by the end-user, not adherence to the development specifications (recall from my last article that they are probably wrong), so our most important quality metrics are delivery frequency and **defect rates.**These are trailing indicators, but there are upstream metrics we can measure that will help predict those outcomes. How frequently is code integrated into the trunk and tested? How long does it take to complete a development task? How long does it take between when a change is made and when it delivers to production? Code integration frequency, development cycle time, and build cycle time can be used to reduce the batch size further. If we are integrating code more frequently, the change-set size is smaller, easier to review the tests, and faster to get feedback. Smaller development tasks have more clarity as to how they should behave and contain fewer useless features. Driving down the time from commit to deploy improves the efficiency and effectiveness of the quality gates which also drives out additional waste and cost. It’s critical though that we keep the delivery frequency and defect rates in mind as the critical metrics. They safeguard quality.
Stability
Measuring the stability of an application is relatively straightforward, but that stability will be short-lived without a stable, high morale team. The same goes for efficiency. We can drive to faster delivery with fewer defects and achieve those goals in the short term while burning out the team. That results in turnover or disengagement that will then cost us all of the gains we’ve made and possibly put us worse off than we started. We should ask the team how they feel. Do they own their solutions? Are they provided with the opportunities to keep the code healthy and provide feedback about the expected value of a requested feature? Are they merely a feature factory that is accountable only to dates? Are people fleeing the team, or worse, the company? They need to be business experts who are invested in the business problem, own the solutions, and feel like valued contributors instead of replaceable cogs.
Lead Time
To reduce the lead time, we reduce the internal cycle times for development, build, and delivery. Delivery cycle time depends on development cycle time which depends on build cycle time, so the priorities should be obvious. Again, driving down the cycle times acts as a forcing function for waste reduction and quality improvement.
Metrics Case Study: Formula 1
F1 racing reams measure and continuously improve. Races are not won by the car that can go the fastest. Races are won by consistent usage of the accelerator, brakes, steering, and transmission to reduce wasted motion and get the most from the available speed. They are also won by the teamwork required to drive down cycle times that impact the goals.
KISS
We don’t’ need dozens of metrics for high-level insights. Yes, there are other things we can instrument to find out “why”, but for getting insights into the “what” that will move us towards our goals, we need to measure in combinations that inform our signals.
- Code integration frequency: Tested changes to trunk per day
- Development cycle time: Duration from “started” to “delivered”
- **Build cycle time:**Duration from “integrated” to “delivered”
- Delivery frequency: Production delivery rate
- Defect rates: Defects created relative to delivery frequency
- Team engagement: Team morale and ownership survey
Quality: code integration frequency, development cycle time, build cycle time, delivery frequency, defect rates, and team engagement.
Waste: code integration frequency, development cycle time, build cycle time, delivery frequency, defect rates, and team engagement.
Lead time: Delivery cycle time, development cycle time, build cycle time, and team engagement.
Stability: Operational stability metrics and team engagement
Metrics are not goals; they give us insights into the goal signals. If you measure something in isolation you will risk degrading something else. Always use offsetting groups. Ask yourself, “If we measure this what other things that we care about could go wrong?” and then measure those concurrently. Resist the urge to come up with a “score” that represents everything. If you over-simplify metrics, you will fall into chaos. Our goal isn’t to control the process. Our goal is observability into the value stream so we can improve it. Remember that product development is done by people and for people. People are not machines. Don’t measure people. Measure the flow and give people the observability to improve it.
Written on February 10, 2021 by Bryan Finster.
Originally published on Medium