Testing for System Integrity.

I was part of an examination panel for a PhD candidate this week.  Her discipline was on the testing of water to ensure that it was microbiologically safe for drinking.  

One of the research areas focused on the relationships between seemingly random events and came to the conclusion that the events may not be so random, and indeed one event (contamination of water with primarily vegetation related bacteria) may in time herald the future of a second, perhaps more dangerous outcome (contamination of water with human or animal fecal content).  

During the examination, I was pretty dismissive of the hypothesis, and suggested that random events are random events, and correlations between random events can sometimes occur more frequently than normally expected, but it doesn’t mean that they are anything other than random events.  

Now, upon reflection, I am not so sure I got it right.  Indeed I think I not only overstated my objection, I might have indeed got it wrong.  Darn.

Cities and towns test their drinking water on a regular basis, based on regulatory requirements, and also after some people get sick.  They can not test for every microorganism, so they test mainly for bacteria which may not in themselves cause illness, but which may be associated with other microorganisms that can cause illness.  It is cause “surrogate testing” and is the norm for drinking water cleanliness testing all around the world. 

When they test the water, it is usually done after the water has been drawn from its source, gone through a form of filtration, then after chemical treatment, and then gone through water pipes out to a tap.  By the time the water reaches that point, it may not be sterile, but it should be sufficiently clean that the surrogate bacteria associate with vegetation or waste should not be there.  

Intermittently (and rarely) a problem occurs and sufficient bacteria survive the process and end up being detected.

The place where I got it wrong, was that I assumed that when the bacteria appear in a test sample it was only because a massive dump, either of vegetation or waste occurred in the water at its source.  What I had not considered was that even if a dump occurs, the system of filtration and treatment should take care of it.  
But if the system is malfunctioning for any reason, then the bacteria can sneak through and show up as a positive test.  

Testing drinking water is not only about testing the quality of the water, but is about testing the integrity of the cleaning system.  It is a simple, but largely effective Quality challenge of a complex system.    
It fundamentally is like keeping a thermometer in a refrigerator to make sure the temperature is staying down, or testing an autoclave to make sure that it is getter sufficiently hot.  

When a positive test comes through that should be a signal to the folks along the water system that there may be a problem.  They may have been lucky if only vegetation bacteria got through, the next time it might be waste related bacteria.  Don’t think “reprieve”, think “check the system for weakness”.  

Water contamination events may occur at random, but if the testing suggests a problem, and the system ignores the warning, then when the bad thing occurs, the randomness is gone. 

I was thinking product and not system.  I was thinking too narrow and too concrete.  

I feel badly about it because I preach “think system” all the time.  Proficiency Testing is not about getting a test wrong it is about challenging the system to make sure it is working properly.  System testing is about error prevention and warning of potential risk.  That is why proficiency testing, and systems checks can be so valuable.

It is not about product integrity; it is system integrity that matters.

Many years ago there was a character on television that would go off on a crazy tangent argument based on her own confusion about what someone had said.  When the point was made clear, she would face the camera and say.  “Oh, well that’s different.  Never mind!”.

That’s how I feel now.