The “Guiding Principles” of External Quality Assessment – PT/EQA

 

                               

This presentation is part 2 of a series highlighting 
Proficiency Testing as a Laboratory Quality Partner.

This series is put together by Dr. Michael A Noble BA MD FRCP the material founder and now retired chair of CMPT – Canadian (formerly Clinical) Microbiology Proficiency Testing Program.  The content of the series iis based on national and international experience and expertise in both providing and teaching about Proficiency Testing (EQA). 
Dr Noble continues to serve on the advisory committee of CMPT and collaborates with Oneworld Accuracy.

The International Organization for Standardization (ISO) includes in one of its essential standards ISO9001:2015 (Quality Management) a list of what is calls the Principles of Quality Management that includes: customer focus; leadership;  engagement of people; process approach; improvement; evidence-based decision making; relationship management.

In my opinion citing of Principles should be an essential part of every standard.  With disappointment, ISO does not include similar preambles to either ISO15189: 2012 (Medical Laboratories – Requirements for quality and competence) or ISO17043:2010 (Conformity Assessment – General Requirements for Proficiency Testing). 

The later document, with particular interest in Proficiency Testing (PT/EQA) does however contain a preamble that speaks about a series of what it calls ten “typical purposes for interlaboratory comparisons”.  I have not included the full list here, but I strongly encourage readers to check the Introduction section of the standard.

I agree that in the widely diverse universe of laboratory testing, PT/EQA can have a wide number of “purposes”.  But within that list of purposes there are some that go beyond purpose and are core to what we do.  I argue they could and indeed should be separated out as more than “purposes” and would be better defined as “guiding principles”, which I think are qualities based on key values that help businesses and customers make beneficial decisions and create a distinct culture.

I don’t know if this list is going to be seen in the next version of ISO17043 (2023), but in my opinion I think they are worth consideration, even if they are not included going forward.  The list includes:.

A.   Evaluation of the performance

·       The evaluation of processes including stability and reliability of reagents, supplies, equipment and the environment is readily covered in the laboratory by Quality Control and internal audits. 
The external evaluation of performance can be done by direct observation by accreditation bodies but can only be done at scale through PT/EQA. 

B.   Identification of problems and initiation of actions for improvement

·       Highlighting problems  (non-conformances or opportunities-for-improvement) is done, not in search of penalties, but to light errors that potentially can harm customers.  
PT/EQA clients and providers alike should understand that laboratories that have a perfect track record with EQA/PT may still have errors, but at a frequency below the detection limits of PT/EQA.  More importantly, laboratories having even a few deficiencies in their PT/EQA challenges is making many more errors that are not being identified.  Laboratories that investigate their EQA/PT including in clinical samples, often find the same errors have occurred before.

C.   Provision of additional confidence to laboratory customers

·       Let me tell you a story.  Increasingly laboratories are putting their accreditation certificates out in the front lobby of their institution where the public can see it, but their PT/EQA certificates hang on the walls in the laboratory.  We in the laboratory know that we are doing well; what we need to do is share that information with the clients that use the laboratories that we assess.  It is the customers who benefit from knowing that the laboratories are performing well.

D.   Education of participating laboratories based on the outcomes of such comparisons

·       Education is more than a purpose; without an educational foundation PT/EQA is only serves as an error detection system, which can prompt some facilities to finding opportunities to cheat and collude. 

·       Another story… in our program we include an educational narrative in all our reports.  It is usually not more that 400 words but provides perspective and learning principles.  In a number of surveys of clients over the years, the smaller laboratories without access to journals and conferences  identified our critiques as their most available and most important educational information for continuing education.

E.   Establishment of the effectiveness of test or measurement methods

·       I include this within my list of PT/EQA principles, but I can concede that not all would agree.  In fields where there are a variety of choices on measurement methods, PT/EQA can offer some unique opportunities for study.  When samples are both homogeneous and stable, and there is a strong informatics foundation for analyzing performance, and there is a certain homogeneity in the testing laboratory pool, there can be value in looking at test methodology on side-by-side comparisons.  Only PT/EQA studies can create information on studies done by multiple laboratories looking for a common answer.  This can be an important part of the educational value-added through PT/EQA.

 

In summary I argue there are some guiding principles that should be central to every PT/EQA program.  I applaud the original crafters of ISO17043 in identifying these. I only wish they had identified them as more than just “purposes”.

But while I am on the topic let me offer a sixth Principle which probably should not only be included, but should be first on the list…   Quality First.  I think this speaks for itself.

Understanding the "Why" of Laboratory Proficiency Testing

 

Understanding the Why of Laboratory Proficiency Testing
(External Quality Assessment – EQA)

This presentation is part 1 of a series highlighting  
Proficiency Testing as a Laboratory Quality Partner.

When a doctor or patient or family members is told that a laboratory has processed a sample and come to an answer or conclusion, the people involved usually assume that the answer the have been given is correct, and usually they are correct.

Unfortunately that is not always the case.   While most errors are of little consequence, sometimes the error can lead to harm to people and public health and can lead to complications and sometimes worse.   What studies indicate is that most errors are either not detected or are detected but not reported. 

What the world needs is a program that can monitor laboratories, detect errors help them learn and avoid bad outcomes. Fortunately that program exists and is known as Proficiency Testing or External Quality Assessment (EQA)

Proficiency Testing (EQA) is single most important method to ensuring that laboratories are capable to performing tests in a manner that will provide accurate, relevant, understandable information, in a timely manner. 

Other techniques such as Quality Control (QC) can ensure that the materials and reagents and equipment that are being used are working as expected, and QC can also ensure that the environment in which the tests are performed is safe and meets requirements, and the staff have the knowledge and understanding necessary to perform the testing. 

Measuring test performance is not easily addressed by QC.  

Test performance measurement can be done in two ways.  Accreditation bodies often perform direct observation which means an auditor watches a person perform a certain test.  If they do it consistent with the written procedure, the auditor records the test was done in conformance. 

EQA measures differently by looking at the result.  If the result is consistent with the expected result, it is deemed as acceptable; if it is not consistent then it can be rated as “essentially consistent” or “not consistent” or “not consistent and dangerous”.

EQA can determine if test performance results in deriving correct results at high frequency every year.  Accreditation body observers have few opportunities to watch people performing tests, maybe 2 or 3 once every year or two.   EQA monitors laboratories 3 or 4 times a year with a spectrum of tests resulting often in 80 to 100 observations every year. 

Both approaches can detect non-conformances or errors (or opportunities for improvement), but the more observations there are the higher the probability of detecting problems. 

Studies on laboratory errors indicate erors fall into two main clusters; one group that are "one-offs", the other that are more repetitive.  When EQA picks up an error, it may be difficult to say what type has been detected.  When one considers that laboratories can do thousands if not millions of tests every year, the EQA-to-Performed Tests ratio is always going to be very small. But when a laboratory has even just a few errors,  the statistics of probability strongly suggest that this is a laboratory having repetitive problems.     

So if a laboratory can participate in EQA and not have errors detected,  that does not mean the laboratory is  error-free; it means that the error frequency is sufficiently low and they were not picked up.  On the other hand, when EQA does pick up even a few errors what it assuredly means is that the laboratory is making LOTS of mistakes… the vast majority of which are remaining undetected.

So, when the quality focused laboratory is notified of an EQA error, the staff and management are not only obliged to  investigate that error to figure out why it happened and put in corrective actions, they ALSO must do a look back and see if they have made a similar errors on patient samples  in the past.  

Bottom line…

Laboratories are busy active places generating lots of important information, but all laboratories can make errors, some of which can cause harm.  Programs like External Quality Assessment (or EQA or Proficiency Testing or PT) exist to help laboratories become aware of errors so that they can focus on cause and correction.   

Consider your EQA program as one of your QUALITY PARTNERS.