5 Minute DevOps: Solving the Uncertainty Problem

I recently gave a talk called “Agile Rehab” at QConNYC 2023 about the techniques we used to transition from delivering once per quarter to delivering multiple times per day. We did this by ditching SAFe and taking Continuous Delivery and software engineering seriously. We asked ourselves, “Why can’t we deliver today’s work to the end-user today?” and relentlessly improved our engineering practices until we could. Along the way, we found several techniques for solving that problem. One that was very useful for our improved workflow was Behavior Driven Development (BDD).

What’s BDD? BDD isn’t a technology or a coding practice, though many confuse it for both. BDD is a process where we define how something should work from the consumer's perspective in a declarative, testable way.

First, we needed to own the quality gates in the pipeline and have testing lead development rather than handing it off to a “testing team” after the fact because CD requires “Test During Development.” However, having a “testing mindset” was not a habit we had. We needed to build that muscle.

Second, we needed better requirements. You cannot write tests first for things you don’t understand. Having a product owner hand over half-baked requirements and say things like, “Oh, you can decide that during development,” followed by, “That’s not what I meant!” and endless debates about if it’s a defect or an “enhancement” isn’t useful doesn’t add business value. We had to fix these to meet our delivery goals.

I stumbled onto Liz Keogh’s BDD material while trying to come up with solutions for these. I highly recommend digging into her material. It was very helpful for developing the correct point of view; “describe the interaction with the consumer as if done by magical pixies.” In other words, if implementation details leak into the scenarios, remove them and stick to behaviors. You don’t “select a country from the dropdown.” Instead, “you choose a country.” The test still passes no matter what UX we decide on. BDD seemed like an easy win. It would help spread the right mindset for testing, make everyone a stakeholder in the outcomes, and provide a contract for “story complete” with the business. “It will do exactly this, and we all agree to that.” No more arguments about “defect” or “enhancement.” There are no enhancements in product development anyway, only “what’s next.” As my wife Dana says, “Every release is missing the next feature!”

Defining Scenarios

It took several iterations to develop a workflow that was efficient and effective at achieving clarity. When we developed this workflow, we were a team of 15, including a BA and a Product Owner. On a smaller team, fewer steps may be required. Try and see.

Drive Out Uncertainty

Define “Ready”

We had a set of specific acceptance criteria we tested our work decomposition process with:

If a task met that criteria, it was ready to start. The whole team was involved with this decision, and everyone was encouraged to throw up a flag if they disagreed that a task met this. We kept improving the task until it did.

Drive Out Friction

When we first tried this approach, we had the entire team meet to refine stories. That was painful and slow. Large meetings are good for pushing out information. They are less useful for creating things. To make that better and to allow the entire value stream to be more productive, we created two meetings.

In the first meeting, a group of 3 or 4 met to create rough drafts. Following the “Three Amigos” pattern, we had people with product, testing, and development knowledge get the scenarios to about 70% complete. Then, we’d schedule a meeting with the rest of the team. In that meeting, we’d review each story, review the scenarios we created, and have the team do final edits. The team would add scenarios, clarify the language, or even remove some scenarios. As the team learned more about the business domain, they became more comfortable challenging the need for some scenarios. A scenario not implemented is the best kind of development because it’s code that will never need support. Since we were using a pull system of work, rather than the misguided practice of assigning work, anyone could be developing any scenario. Therefore, everyone was personally invested in all of the scenarios being good.

After defining all of the scenarios, we’d organize them by priority and, if required, use them to split stories. Since each scenario was a unit of behavior that was independently deliverable, as long as we kept related positive and negative behavior tests in the same story, we could split them into tiny, independently deliverable units.

From Story to Task

Results

We tried this for the first time after I got frustrated with the time wasted arguing about how many points a story was. We still had “feature complete” habits at the time, and our stories were massive. So, we applied this technique. We met, defined all of the scenarios, story pointed the scenarios (they were also too big), added up the story points, and decided the “story” was 54 points or about 90% of our velocity. Then we went through technical decomposition to identify which services needed what changes, defined any service contract changes, split up the work, and at the end of the 2-week sprint, everything integrated cleanly.

When I left the team, everyone had learned how to “think” in tests, business knowledge grew across the team, and the coding style and application knowledge were leveled because everyone touched everything. The delivered quality was higher because we’d driven out uncertainty. There were no more sprints. Since the average time to deliver a story to production was 0.89 days, we had moved to a continuous flow of work and “just in time” story refining. That also meant no need for story points and endless debates with a deck of cards. The work sucked less.

Developers want to code

Cool. I want to get paid to do my hobbies, too. Alas, getting paid requires work. It can’t all be fun. The work is not coding. The work is deciding what to code. Coding is the activity of converting that work into usable value. I enjoy coding. It’s fun watching things grow and seeing the machine dance to my commands (or not). I don’t enjoy decomposing features. I don’t know a developer who does. If I find one, I’ll make sure they are on my team. However, driving out uncertainty before development is the real work, and the outcomes are universally poor if the expectation is that developers will be handed work that’s “ready to code.” Using a disciplined engineering approach to drive out uncertainty makes this less painful. I promise that it’s less painful than the outcomes of “I just want to code.”

Free Resources

For more tips on this topic and many others for improving the flow of delivery, check out the information at DojoConsortium.org.