Considerations Before Using Data to Draw Conclusions: The Importance of the “M” in DMAIC (Part 3)

Written by Jennifer Christie, MBB, Senior Services Manager of Productivity Improvement  |  Part 3 of a Series on Lean Six Sigma

Data enables us to measure performance, conduct root-cause analysis, make predictions, and set targets, but our ability to see and understand a process is only as good as the data we get. The metrics we choose, how we measure, and the variation in the measurement system can enable clear vision or cloud our ability to see. If the variation in our measurement system is too large, we may not be able to detect changes or shifts in our process.

When using Six Sigma’s DMAIC (Define, Measure, Analyze, Improve, and Control) methodology for process improvement, we dedicate an entire phase to our measurement system. The measurement system is validated before analyzing any root cause so we can have confidence in our results.  We assess the variation in our measurements for adequate:

• Repeatability,
• Reproducibility,
• Resolution,
• Accuracy, and
• Stability

Repeatability

Repeatability measures how well a person or machine measures the same part consistently. Does the gauge give the same result each time? Imagine two archers. The first archer hits the target in a very different place each time, while the second archer consistently hits the same spot. The second archer has greater repeatability. This is what we are looking for in the measurement system: consistent results within the same gauge or appraiser.

Reproducibility

What happens if we change gauges or appraisers? Do both gauges or appraisers get the same result? Reproducibility refers to the variation we see between gauges or appraisers. This is especially important if we have a measurement system that relies on people.   People can have different definitions of what to measure. For example, if an operator is recording time for a package delivery, is the delivery time when the package arrives at the dock or when it is opened and scanned into the system? Assessing how different individuals measure time for the same packages evaluates reproducibility.

Resolution

Can you see enough detail? Resolution refers to the smallest change your measurement system can detect. Resolution can be affected by the instrument or gauge’s capabilities or the way we record the data. A resolution of 0.01 is greater than 0.10. An operator recording the day of delivery will not give enough resolution to track hours if the time is not recorded. In a measurement system with poor resolution, we cannot discriminate at the desired level.

Accuracy

Accuracy refers to how well we can measure a true value. This can be assessed by comparing measurement results to a known standard. An accurate measurement system has little deviation from the true measurement.

Stability

How a measurement system performs over time or under different expected conditions is its stability. For example, does the gauge drift as the room gets hotter in the summer? Monitoring stability helps determine calibration intervals.

We can evaluate the measurement system criteria through tools like the Gauge R&R and Attribute Agreement Analysis. By measuring a series of samples across the operating range, we can quantify the variation between and within appraisers and against a standard. For continuous metrics, we typically look for overall variation in the measurement system to be less than 10% and that of an attribute system to be less than 20%. Of course, this can be more stringent if there are high-risk consequences to our measurements. We are looking for a measurement system that allows us to see the process across the operational range with accuracy and precision. Too much variation within the measurements makes it difficult to detect changes in the process or to even know how our process is performing.

Sometimes it is not possible to measure a sample or part multiple times to assess repeatability or reproducibility. Under those circumstances, we try to measure against a standard (true value) when we can. When evaluating your measurement system, ask yourself: “What could go wrong?” Assess those failure modes to determine how much they could be impacting the data you see.

Tips for Measurement Systems

Know the operational definition. Many times, we are looking at historical data that has been pulled for a project or even reported regularly on a scorecard. Look at the source of your data. It isn’t uncommon to find that the data we are making decisions from has more variation than expected. Always ask what the operational definition of the metric is. For example, when counting defects, define exactly what is categorized as a defect. Or if measuring time, find out when the clock starts and stops and whether it is recorded consistently. This is true for people as well as automated systems. Check to see that automation is capturing what you think it is.

Make sure the data you use for decision-making is representative. Does that data set represent the actual process in the timeframe expected? This means we have enough data and, if we are sampling, we collect in a manner that represents all relevant subgroups. More data isn’t always better. You should have enough data to give adequate power to make decisions while making sure that you capture samples that represent the population you are interested in.

Continuous data is often preferred, but don’t be afraid of discrete data. Continuous data has more inherent information, can require fewer samples, and has more statistical analysis available. However, sometimes discrete data (e.g., binary pass/fail, counts) is the best or only measure available. Use the best metric for your purposes. If you can, try to get a continuous variable; this may require some creativity. For example, if you are concerned whether something is late or not (discrete), consider collecting a timestamp. We can always draw a line to determine when something is late.

Do quality checks on data entry or downloads. We can’t assume data is entered or downloaded correctly. Especially with very large data sets, randomly check with an independent source if the data is entered correctly. This doesn’t have to take long but can save a lot of frustration later.

Six Sigma and continuous improvement leaders pride themselves on data-driven decisions, but we must have good data to begin with. While validating a measurement system can take time, it ultimately results in better analysis and results. You can learn more about the techniques used to collect and validate measurement systems through courses offered by MEP.

Writer: Jennifer Christie, 317-275-6810, jennchristie@purdue.edu