I recently read Dr. Deming’s “The New Economics”. The book introduces what Dr. Deming calls “The system of Profound Knowledge”. It’s a four part theory for understanding and improving systems. Understanding variation is one part of the system of profound knowledge.
You see, variation is a natural trait of life and in systems. I we implicitly understand this. Deming explains how this implicit understanding becomes an explicit aspect of improving systems. Let me explain with a simple example.
There are two types of variation: common cause and special cause. Common causes are expected and natural. In other words, they are part of the system.
Consider your drive the grocery. You know that you’ll hit some red lights and green lights. Some days you may get lucky with lights and never have make a stop.
Don’t mistake that luck for anything special. That’s common. The system is operating normally. That’s just variation at work.
What if one day on your way to got a flat tire that delayed you four hours. That’s a special cause. That’s something outside the ordinary.
Consider the “lucky with the lights” scenario. Now imagine the opposite where you hit a frustrating amount of red lights.
You may mistake this for a special cause then try driving to avoid all red lights. That’s all of naught because there’s nothing you can do about the lights.
Deming calls this the first mistake: attributing special cause variation to common cause variation.
The second mistake is the opposite: attributing a common cause variation to special cause variation.
Now returning back to the drive to the grocery store. You could assume that flat tires are expected events, uncommon but just a cost of doing business and leave it there. That’s a mistake because special causes can and should be eliminated. You could eliminate, or at best, drastically reduce the possibility of a flat tire by conducting routine vehicle maintenance.
These mistakes and examples demonstrate how to approach special and common cause variation. Special causes should be investigated, mitigated, and at best eliminated. Common causes are inherent to the system. They cannot be eliminated. They may only optimized thus minimizing variation.
So why is this important to software delivery?
Well the work of software delivery—whichever way you measure it through lead time, deployment frequency, change failure rate, or mean time to resolve–is effected by variation.
So, if we want to improve lead time, then we start by identifying the common and special causes that produce different lead times. If we detect special causes, then the system is chaotic. Optimization is not possible yet. We must stabilize the system by eliminate special causes.
At this point the average and limits of variation are predictable with high confidence in the near future. Quality and quantity are predictable. Costs are predictable. “Just in time” begins to actually take on meaning.
Now we can set ourselves to task improving the processes.
Dr. Deming’s work describes improving systems by first identifying the stable or unstable nature of the system. Here’s my challenge to you: identify the common and special cause variation in your own work. If you’d like to learn more, then I recommend Dr. Deming’s work on statical process control, Dr. Shewhart’s control charts, and—of course—Dr. Deming’s book a “The New Economics”.