5 Minute DevOps: Organizing for Failure

It can be entertaining, and sometimes depressing, to dig into the history of manufacturing and compare the lessons learned to the lessons we continue to struggle with in software development. One example is the documentary I saw on the challenges faced by the Ford Edsel. There were so many lessons on the UX research, design, manufacturing, and marketing of Ford’s new brand that apply to developing any product. I suggest everyone look into it. Ford’s quality process was interesting, to say the least.

In 1958, Ford started to manufacture the Edsel. It was a polarizing design, but it had every innovation Ford could think of. Since this was an entirely new design, they didn’t yet have an assembly line for it. So, they modified an existing assembly line to interleave Edsel manufacturing with another existing model. So, seven cars were the existing model, with every eighth car being an Edsel. The Edsel wasn’t just new bodywork. It had a new engine and many new innovative and complicated options that resulted in a more complicated assembly process.

So, as an assembly line worker, your processes changed and became more complicated every eighth car while the assembly line continued moving at a constant speed. On top of this, there was long enmity between the United Auto Workers union and Henry Ford’s son Edsel Ford. The combination of these issues led to “assembly challenges” at a higher rate than normal. However, Ford had a well-defined QA process to ensure the line kept moving, outputs were maintained, and dealers got only the best Ford could deliver.

• As each car rolled off the line, it went through road tests and quality inspections. It was then given a quality score.
• All of the quality scores for the day’s production of both models were averaged.
• If the average daily quality score met the minimum quality score, the cars were shipped to the dealerships.

Since Edsel’s represented 1/8th of production and their quality scores were averaged in with the other 7/8ths, cars often shipped to dealerships with major issues to be resolved. Sometimes, missing parts would be thrown in the trunk of the car with a note to the dealerships so they could be installed there.

This process worked fine for Ford, considering their competition, until changes in US tariffs allowed Toyota to disrupt the market.

The Toyota production system was created to reduce costs and increase quality so that they could compete. Core to the TPS is reducing waste by minimizing batch size, inspecting quality at every step, and preventing poor quality from moving to the next step. They made quality concurrent with manufacturing. This proved highly effective for increasing market share and allowing them to pivot to market needs rapidly. The Toyota system is something other manufacturers have worked to emulate for more than 50 years.

After watching the documentary, Ford’s quality process sounded very familiar to my experiences in the past delivering software.

• Develop features, often with shifting priorities and pressure to meet deadlines for the scheduled release.
• Throw it over the wall to the testing team, who log defects
• A go/no-go decision is made on the large batch of changes based on the criticality of the defects.
• If the minimum quality is reached to ship, then the release goes to the customers, and the operations team fixes the defects based on customer complaints.

This process hasn’t been a viable business model for manufacturing for more than 50 years, yet it’s all too common in software development.

A friend of mine told me that his organization’s QA team had built a framework to make it easy for developers to test every change on their desktops fully. They were teaching teams TDD and BDD and were moving quality checks as close to the source of defect creation as possible. They were also focusing on the speed of feedback. A developer could get quality feedback from a fully integrated test in less than 3 minutes. They were taking Toyota’s lessons to heart and rapidly improving the teams’ delivery abilities.

My friend also told me that one of the development managers in his organization is pushing back on the whole idea. “Why are we wasting time on this? Why are we asking developers to test? We have a testing team.”

I wonder how many Toyota factory managers demonstrate that level of ignorance about auto manufacturing?

Is your organization executing a continuous delivery development process with small, tested changes and rapid quality feedback to the developers? If not, when will you finally catch up to where manufacturing disruptors were over 60 years ago? Software is eating the world, but only for those who can move from artisanal development to modern software engineering.