Benchmarking Clinical Microbiology

There is a universal appeal to finding a quantitative measure for assessing laboratory quality.  In some respects it is an appeal that I would like to resist because it becomes an ugly  part of inter-laboratory comparison (My laboratory is better than your laboratory.  My laboratory director can beat up your laboratory director).   But at the same time as a proficiency testing provider, one of my tasks is to provide some sort of quantitation of performance so that laboratories can have some benchmarks useful for monitoring their own level of performance and to use those measures as part of the continual improvement process. 

As a PT provider we assess responses to challenges and grade them.  In Clinical Microbiology correct responses for any result have to be more than accurate; the report needs to provide appropriate and relevant information.  So, in our program a grade of 4 is given for an acceptable response that is considered accurate and appropriate and relevant.  A grade of 3 indicates also an acceptable response, although not completely complete.  On the other side a grade of 1 is given for a response that is unacceptable because it is inaccurate or incomplete.  A grade of 0 indicates a response that if given for a clinical sample could lead to harm or inconvenience to patients.
Not all PT programs use that type of scale, but we have used this for near 30 years and have a sense of its strengths and weaknesses.

So with those values in mind we can review how well or unwell do laboratories perform on proficiency testing.  As can be seen in Figure 1, the vast majority of laboratories score 4’s and 3’s on the vast majority of challenges.  The number of 1’s is small, as are the 0’s.

So the first message should be that performance on individual proficiency testing challenges that are less than perfect should be viewed as a signal that perhaps there is a difference between what your laboratory considers as accurate and appropriate and relevant and what a panel of peers considers.

A second is what percentage of laboratories is able to earn 100 percent of their achievable score.  (If a laboratory performs 4 challenges and each as a perfect value of 100, then their achievable score is 400).  From the above you would expect that near all laboratories would get 100% or near 100%, but not all challenges have equal values  Some are more complex than others and therefore contain more gradable elements.  And not all laboratories are equal performers.  Some have ready access to well trained and experienced staff and sophisticated testing equipment while others do not.  
Each year we look at the proportion of laboratories that are able to achieve at least 90 percent of their achievable score.    While most are at or near 100 percent, there are always some that fall below.  On average, over the last 9 years, 75 percent of clinical laboratories earn 90 percent or greater of their achievable score.  Large complex laboratories tend to score higher than smaller laboratories.   

So some benchmarks to consider for a clinical microbiology laboratory’s performance.  On an individual challenge basis, any result that is not accurate and appropriate and relevant should be considered as an indicator for improvement.  On an annual basis achieving less than 80 percent of your total achievable score should be considered as another indicator. 

One might fairly argue that how a laboratory scores on Proficiency Testing does not necessarily reflect on clinical performance.  That would be a difficult relationship to prove or disprove.  But if PT samples look like and act live clinical samples, then the relationship may be more plausible.  

For a fuller examination of the information used to benchmark clinical microbiology performance values, check out the Annual Report at www.CMPT.ca