Three reasons for misleading gas analyzer accuracy specifications

A funny thing happened on my way to helping a colleague make sense of the specifications of a metabolic phenotyping system. There were many strange things to mull over. Not the least of these were oxygen and carbon dioxide analyzer accuracy specifications that appeared to be the product of smoke, mirrors, and absinthe-infused coca tea sipped through a crazy straw.

Then it struck me: A sense of déjà vu all over again. I've seen similarly bloated and unwordly specifications made by the manufacturers of human metabolic measurement equipment. Why is it that companies that serve the biomedical market feel compelled to exaggerate analyzer specifications beyond the bounds of credibility? Curious minds want to know.

For that reason, I'm transcribing my spiral-bound anthropological notebook and presenting here my own, brief, analysis of the phenomenon.

Reason one is specialization. Scientists and researchers in general know a great deal about their subject (well, most do) but outside their area of expertise, not so much. Thus, they are vulnerable to exaggeration and outright untruth in areas outside their field of expertise. This is especially with regard to matters such as accuracy versus precision, and measurement theory in general.

Reason two is inconsistent terminology. Accuracy is an elastic concept, and unless you've had some exposure to measurement theory you will consider accuracy, quite reasonably, to be a measure of relative error. We can all agree that 0.000% accuracy implies that a measurement is made without error. But what does "0.04% accuracy" mean? This is the accuracy that one metabolic phenotyping system manufacturer claimed for their third-party O2 and CO2 analyzers.

UPDATE added in 2017:  Some legacy metabolic phenotyping system manufacturers have now jumped the shark!  Not content with claiming 0.04% accuracy, they are now claiming "0.001% of reading accuracy". Their previous fictitious specifications were already unreachable, and now, they have managed to make them 40 times more unreasonable and dishonest. But back to the main text: 

Here is where things get interesting. In order to evaluate the accuracy of an analyzer, you must use it to measure an accurately known standard, and then determine the error in its measurement of that standard. Say, for example, you pass an exactly known 1.0000% CO2 standard gas through the analyzer, and the analyzer reads 1.0004%. In that case, the error (in percent of reading) is 100 * (1.0004 - 1.0000), or 0.04%. Notice the number of significant figures? in order to measure this degree of accuracy, our CO2 standard gas must be exactly 1.0000%, not 1.0001% or 0.9999%.

Here is where the real world pokes its muzzle into this ideal world and barks, loudly.

Because the fact is, it is literally impossible to obtain such accurate gas mixtures. Not difficult. Impossible. This is for a variety of technical reasons I'll explain if there's some demand for it. But bottom line, the most perfect span gas anyone can buy is 1% accuracy. This means that the actual concentration of the CO2 span gas mentioned above will be anywhere from 0.99% to 1.01%. So as you can see, in most cases - and always, with CO2 analyzers - any accuracy figure of better than 1% is garbage, pure and simple. The sole exception to this is if you manage to persuade a national standards laboratory to create some 0.25% or so accuracy span gas, for which you will pay many thousands of dollars. In that case, you are equipped to evaluate accuracy down to 0.25%, if you make the assumption that the only cause of error is from the standard gas concentration. Which is highly questionable, as any student of measurement theory will tell you. Plus, you can only measure the error and thus the accuracy of the analyzer at that one concentration, because there is no way of mixing that gas to create a lower concentration without introducing a new source of error in the 1-2% range.

But 0.04% accuracy? Srsly?

There is one accuracy specification that allows this amazing specification. As Dr. McCoy would say, it's accuracy, Jim, but not as we know it. And that is to express accuracy in absolute terms at some point in the analyzer's measurement range. In the case of the system specifications I was looking at, the maximum CO2 concentration it could measure was 2%. Let's say the analyzer was fed a nominal 2.00% span gas, and actually measured 2.04%. In that case, its absolute error could be stated as 0.04%, and I suspect that this is exactly what the drafter of that metabolic phenotyping system's specifications had in mind.

Was the metabolic phenotyping system manufacturer trying to mislead its potential customers? I leave that for the reader to decide.

Bearing on that last question, here is an interesting fact. The metabolic phenotyping system in question uses a combination O2 and CO2 gas analyzer made by Siemens, a very solid and reputable industrial process-control firm. Siemens is not given to flights of fancy. So, what does Siemens give as the accuracy specification of that analyzer, as shown in the analyzer's downloadable instruction manual?

In fact, Siemens says exactly what they should say: "Calibration error: Dependent on accuracy of calibration gases". This is the only accuracy specification they give. They do, however give two other specifications that are important when considering accuracy: Repeatability and linearity deviation, both of which are 1%, again showing that the claim of "0.04% accuracy" for that same analyzer is, shall we say, imaginative.

Reason three is peer pressure. For some reason, there is a red queen race going on among biomedical equipment suppliers. Some of them are tripping over themselves to invent fantastical accuracy specifications that will impress researchers who are not used to thinking critically about accuracy specifications - not because the researchers are stupid, or credulous, but because they make the assumption that equipment manufacturers and resellers are being as honest as they (the researchers) are.

And that isn't always the case. Worse than that, paying uncritical attention to accuracy figures may lure researchers into a lair which, in hindsight, they will wish they had not entered. And that's a pity for their research, for their funding agencies, and for human curiosity.