Welcome to the world of DTP reporting

As a rule I have stayed away from the getting engaged in the formal politics and laboratory Quality, especially the politics outside of Canada.  But a I find it difficult to leave this alone, because it is an area about which I have some well thought out opinions. 

This month the US Department of Health and Human Services (HHS) proposed to improve access to laboratory test results by bypassing laws in several states that require patients to get the data from their physicians.  Proposed rules changes would over-ride the patient privacy provisions of CLIA 1988.  and the HIPAA (Health Insurance Portability and Accountability Act)  1996.  If approved laboratories would be able to release results to the treating provider, the referring laboratories, and “authorized persons,” including the patient. 

The proposed lab regulations were part of a patient “empowerment” package with the intent to lead to better health and healthcare.

At the moment count me as “unconvinced”.

First I should mention that I have discussed this change with a number of people.  Many think the rule REQUIRES laboratories to provide direct-to-patient (DTP) reporting.   I have interpreted the rule more as being permissive rather than regulatory.  Laboratories have HHS permission to over-ride the state restrictions and report directly to patients.  The text does not require them to do so.  If I am right, that should take a lot of pressure off the laboratories.

Regardless, in North America the relationship between clinicians and laboratories has been a continuous work-in-progress.  Clinicians have not always ordered the right test the right way.  Smart laboratorians have figured out the way to notify and make suggestions for improvement.  The other ones just do the test “nilly-willy” with little interest in results validity. 

At the same time laboratories have not always get the test right the first time or developed processes for correcting reports after the fact.  Sometimes reports get some names wrong or send results to the wrong people. 

Importantly, because the conversation has always been between laboratories and clinicians the two groups have developed a special language based on medical jargon.  One needs to be “in” to understand what the reports say. 

With this new rule, I suspect all that will have to change.

In my province, each year we have about 5 thousand reported errors for about 5 million tests.  If we take into consideration the error results not reported, the error rate may be double.  While most are inconsequential, almost all have an amended report.   By extrapolation, in the US there are now around 1.5 billon tests done annually, so you can estimate somewhere between 1.5 to 3 million amended reports annually. Most of these will be inconsequential, but with DTP reporting many of the amended reports will be going directly to patients.  

This is likely to have impact on a lot of patients.  Since patients have a lot more invested in their own health and confidentiality, I wonder how many will get upset getting results that amend false positive and false negative results.  Will patients understand that results of tests with a high degree of uncertainty can dance all over the place and still be acceptable?

Will report jargon will be confusing.  Will they understand terms like “Beta haemolytic streptococcus” or “Group A streptococcus” may be the same or may be different?  Will the understand the significance of “many RBCs in urine” or the term “monoclonal peak”? 

I wonder if patients who receive reports amended reports or unclear reports will be accepting or will they respond with anger.  I worry that angry patients sometimes react via lawyers and the media. 

To avoid this laboratories are going to have to make a lot to change quickly.  Error detection is going to have to be a lot sharper and the language of reports is going to have to change.  I worry that smaller laboratories not in a financial position to make the necessary changes will close shop, leading to another wave of laboratory closures and buy-outs and consolidations. 

I am not arguing that changes should not need to be made.  Indeed, they should be made.  We should be more aggressive in reducing repetitive error.  We should be clearer in our language.  We should be regularly incorporating informative notes and cautionary notes.  But all this should be done based on rock solid Quality principles like PDSA and process monitoring. 

We will see what happens.

Constructive outrage about Education

In my previous entry I mentioned my mini-survey on Linked in about issues the group of laboratory professionals considered as “burning”.  The following is one of the responses:

“Continuing education on the topics of Lean/Process Improvement and Effective Management/Leadership. Med techs receive no training in these two important areas during their college courses, and this causes many laboratories to be less efficient and more dysfunctional than they need to be. Better management + improved processes = laboratories that are superb in every way!”

Since the survey was designed to be anonymous, I know nothing about the writer of the comment, other than the person is a laboratory professional that connects to Linked in.  While they might know of me, there is no purpose or advantage for them to be stroking my ego.  I therefore take the comment at face value.  That being said, I am very supportive of the sentiment being expressed.

Education providers are an interesting group.  On the one side, their personal interest side, educators tend to be a group prone to being progressive and promotional.  As a general rule, if you want to know what is current thought, talk to a teacher.  On the other side, the teaching professional side, they tend to be dreadful stick-in-the-muds.  When it comes to the content of their teaching, the rule of thumb is “change nothing”.  The process to change curriculum in many (most) universities and colleges is made difficult beyond difficult.  There are departmental committees, faculty committees, senate committees.  Everyone needs to have the opportunity to discuss and debate forever.

When it comes to Quality, with the exception of business schools and Engineering, you might think that the concepts were never developed.  (Most Business school curricula will have a course which includes Frederick Winslow Taylor, Water Shewhart, and William Edwards Deming, and most Engineering schools will have curricula on ISO 9000.) 

But in Medical Laboratory Technologist programs, there has been essentially little progress beyond Levey and Jenning, with perhaps the inclusion of the rules of James Westgard, which essentially are all about how to use and interpret Levey-Jenning charts.  Quality in these programs means Quality Control.  Period.  End Stop. 

That also is true when it comes to training programs for Pathology and Laboratory Medicine Residency programs.  And in my experience the same is true in medical science oriented graduate studies programs. 

So we continue to have laboratorians with absence of understanding of critical issues like how to reduce error, how to evaluate risk, how to reduce poor quality costs, and how to implement effective change.  As a colleague said today, their working strategy for change and improvement is IDEA - SHOOT - AIM!

There are programs that fill the void, mainly on a Continuing Education level.  For example there are courses available through the American Society for Quality, and Clinical Laboratory Manager Association (CLMA) and the Michener Institute (Toronto) and of course, my course The Certificate Course in Laboratory Quality Management through the University of British Columbia (see www.POLQM.ca).  And I am sure there are lots others, but they all have the same common characteristics; they are voluntary and they are provided as post-graduate continuing education. 

One can wonder why this programmatic void exists.  In my opinion it is a mix of disinterest, absence of imagination and visioning, and above all, inertia, all the hallmarks of stagnation.  And that is just sad.

And so to the anonymous responder to the survey.  Congratulations for speaking up here.  Now, take that same idea forward to your laboratory and organization and college or university.  Take it forward with constructive outrage. 

Demand change.

And in the meantime, contact me at www.POLQM.ca  

PS:  I have had a few more responses to my survey (http://www.surveymonkey.com/s/BurningIssuesB).  The ranking of burning issues remains the same, with stronger values.  Feel free to participate over the next few days.

Burning Issues in the Medical Laboratory

I have been joining a number of groups of laboratory professionals on Linked in.  It is something like Facebook but a lot less about pictures and pals and a lot more about professional interest.   I think it is worth a look.

I posed a question to some of the medical laboratory oriented groups about topics relevant to medical laboratories.  I presented them with 6 issues (so called Burning Issues) and asked the group to rank them 1 (not very important) to 6 (very important).

Response to the question was designed to be anonymous, without name or demographic, and with a short turnaround.  The results represent a single moment in time opinion.  I had no influence over any of the responders other than they are users of professional social media and prepared to put forward opinion. 

In 24 hours I got back about 1.5 percent response; certainly not overwhelming, but with a reasonable range of repeatability.  The graphic of the results of the ranked averages is seen below.

Without being too over-reaching, the results fell into 2 general groups.  The first group was the three issues of reducing the incidence of laboratory error, human resources hiring and retention of staff, and the meeting requirements for regulation and accreditation.  The second group is has a more quality/business focus and included semi-quality semi-business reduction of costs of poor quality along with the distinctly business oriented issue of changing technologies and changing demographics impacting on the business side of medical laboratories. 

The order may reflect that group participants are more operational (and maybe quality) oriented than business oriented.  That would account for the the business oriented issues – cost and new technologies and changing demographics  being lower ranked.   

Actually the two least ranked are critically important to the business of medical laboratories.  Today’s laboratories are in perhaps one of the most vulnerable transition periods because technology and patients are changing more significantly than at perhaps any other time.  A few years ago microbiologists used to say that if Robert Koch showed up in a modern microbiology laboratory today he might be intrigued by the equipment, but he would certainly understand all the sugar reactions upon which we still predicate many of our test results. Today as we move to more and more molecular Robert would be blown away.   That also means that what few medical technologist training programs there are would be similarly left in the dark.

North American demographics are also impacting on laboratories in a big way.  We old folks may see ourselves as the drivers of laboratory activity, but we would be wrong.  We may have more visits to the laboratory but we get fewer tests per visit and tend to have less intensive investigations.  Laboratory activity is more intense about the very young and middle age.  We require more time and generate fewer tests (and dollars).  Let’s face it we are very inefficient.  Reduced testing means lower revenues and lower interest from institutional administrative forces. That is a not a good thing.

The second question in the survey was a write-in section.  In the first part I had offered the 6 choices.  In the second participants offered one of two issues that they thought should have been included as burning issues.

Here is a partial  list of issues deemed as burning by a group of laboratory professionals participating in Linked in interest groups:

1. Increasing, often confusing, and frequently contradictory regulations
2. Increasingly complex and ever-changing billing standards
3. Continually improve quality in order to reduce cost
4. Effective communication skills and  open dialogue with upper and middle management.
5. Focus on safety in the workplace.
6. Retaining qualified staff; a meaningful career (ladder) for technologists.
7. Technologists dumbed down by outsourcing.
8. Quality Management
9. Decentralized specimen collection.
10. Quality management.
11. Make sure that the results of laboratory tests reflect current research and that the end users understand what the results mean.
12. Doing more with less!  implementing Lean tools for efficiencies
13. Functionality of information systems and smooth implementation of same.
14. Decreasing number of Medical Technology and Cytology Programs.
15. Continuing education on the topics of Lean/Process Improvement and Effective Management/Leadership.
16. Decrease in MT programs and graduates as current workforce retires.

I am going to let this survey run a little longer to allow the numbers to gel.  If MMLQR readers would like to participate, they can at:  http://www.surveymonkey.com/s/BurningIssuesB

Much more on this to follow.

I'll have a Grande Latte with that Proficiency Testing, please.

I have mentioned before my views on the close associations between Quality and Leadership and Innovation.  The other day I was watching television in a mindless sort of way, when I got to see a telecast of Howard Schultz, chairman, president and CEO of Starbucks giving a presentation at the Vancouver Board of Trade.

Now I know that Starbucks coffee is not for everyone, but one would be hard pressed to not recognize the company as pretty much a world wide success.  Recently I was in Ankara Turkey and still had my daily double espresso. 

Schultz has an interesting story with Starbucks because he built it up the first time, and left the company when it decided to chase growth over quality, and then was asked to return when the company was going sour in a hurry.  Three years after his return, the company is back on top.

During this presentation he talked about several key issues.  The first was that successful companies are driven first from their core principles.  Second, that human resources have to be at the center of decision making.  Successful companies ensure that their staff are front-of-mind.  Third, that in the “eco-system of enterprise”  companies that give back to customers succeed.  Not giving back in the sense of kick-backs, but in the sense of being a contributor to education, outreach, leadership and support. 

And fourth that innovation that is visible to and beneficial for the customer is critical.  Companies ignore the opportunities of innovation at their peril. 

Customers are no longer interested ONLY in price.  Customers are more comfortable when they perceive that they share your values and see you as part of a “thread of trust”.

Now I understand that most of this is way outside the scope and authorities (but hopefully not the interests) of the public sector institution based medical laboratory.  The drivers of decision making in the public sector institution may not see themselves in the same way as Starbucks does.  But it seems to me that the core principles are pretty central to every successful organization.

And for Quality Partners including Standards Development Organizations, and Accreditation Bodies, and Proficiency Testing Providers, and Reagent/Equipment Suppliers, and to some degree, Quality Consultants, the iteration of principles has to be absolutely “spot-on” advice.

Organizations need to know what they are good at and what their core values are.  If they don’t know, then how can customers know what to expect. 

And trust is critical.  To the extent that your customers have control, they will stick with the folks that they trust.  But watch out when the thread of trust is broken.  When you are perceived as only about growth and money and serving your own interests only, and not about providing support, then problems will blossom.

And then comes innovation that is visible and relevant to the customer.  Are we writing standards that matter?  Are our proficiency testing samples a boon to quality improvement and to today’s needs for continuing education?  Along with reagents and supplies, do we provide the element of real innovation that is meaningful to the laboratory?

Are you a value-add, or just the same old-same old?

CMPT spends a lot of time, effort, energy, and money one these very issues for year.  We like to think we are successful, although not quite to the same level as Starbucks. 

If you visit www.CMPT.ca you can see our latest Annual Report for 2010-2011

Meet Dr. Watson

Michael Laposata is professor of pathology and medicine at Vanderbilt University School of Medicine in Nashville.  A number of years ago he coined the terms “pre-pre analytic” and “post-post analytic” to define the interface between the clinician and the laboratory.  One the one side, if the clinician orders the wrong test (pre-pre), then everything that follows is for not.  On the other side, if the clinical does not understand the result or misinterprets it (post-post), the same holds true. 

There are procedures that can be put into place to reduce error at these two critical points.  Electronic ordering systems can insert some specific questions that will direct the test orderer in one way or another.  And incorporation of advisory notes and cautionary notes and interpretive notes into the report can provide some assistance.  But to date neither of these systems is widely used in laboratory practice.  We tend to be comfortable staying within our self-imposed boundaries.

But it appears that the next generation of solution makers is just on the horizon, and its name is Watson.

For those who do not watch American game shows, IBM’s most recent super hero is a mega networked computer named Watson.  Watson is the son (or maybe the grandson of Deep Blue, the computer that beat the chess genius Garry Kasparov.  Watson is today’s best example of how far computers have moved towards faster-than-human artificial intelligence.  Basically one loads up Watson with about 200 million pages of information.  When asked a question it can process the information of about a million books per second and determine the best connected answer.  Watson is likely to provide recommendations for the right test at the right fashion at the right time, and give recommendations for the most appropriate interpretation.

In a head-to-head challenge with the 2 best humans ever to compete in the knowledge game show (Jeopardy), Watson was faster and over 98 percent correct.  In a two day competition, Watson “cleaned their human clocks”.

I raise this because the IBM team has decided that the next best target for Watson is to partner up with a medical insurance company to tackle the science and art of medical diagnosis and treatment.  The outcome is likely to be predictable.  Like the Olympians say “Higher – Faster – Stronger”.

It is the modern linear pattern.  In the early chaos, we find leaders, and in leaders we find body of knowledge authorities.  Doctors (as an example) became the repository of knowledge.  The internet changed all that with the democratization of knowledge so that everyone could believe that they could be an expert.  Watson goes the next step. 

I suspect that in a world with so much information, nobody can be a better expert than a supercomputer.  Watson will take the art out of diagnosis and fill in with lightening speed relational and probability analysis.  It will know and integrate what people forget. It will exclude personal bias and myth and “n-of-one” information. 

There are factors that make forming diagnoses more challenging than winning on Jeopardy.  Good clinicians are influenced by visual and tactile and olfactory clues that will not be available to Watson.  And depending on how one asks a question about symptoms, one can get a different response.  So I suspect that on a good day and in a one-off situation, the human diagnosticians will come out not too bad.

But on the long haul, we all tend to be burdened and distracted by so many side issues (hunger and heartburn, noises, smells, fatique, taxes, spousal tiffs, parking tickets) that Watson will do better.

Now what we will need is a machine that will improve the pre-examination phase, the next step along the way.

If Watson is here now, can HAL be far behind?

For a course in medical laboratory Quality with a Twist, visit www.POLQM.ca

Registration is now open.

Ten years later.

As I write this entry I am listening to radio descriptions and reminders about the crisis that started in New York city on Tuesday September 11, 2001, and over the subsequent decade covered much of day-by-day activity around the world. 

What many have likely forgotten, exactly one week later the laboratory community  including medical, industrial, and research laboratories became caught up in the security tsunami when it was announced that anthrax spores were being found in a number of letters.  The US postal service was being used as a vehicle for a bioterrorism attack.

The events of the day were followed closely and concern.  Letters were posted in the United States to a number of politicians and media outlets.  Nineteen people became infected and 5 people died. Many speculated that the anthrax was part of the same terror process of the week previous.  Others suspected it was a home-grown event, perhaps inspired by, but independent from what else was going on  But as the days went by and the cases stopped, and a source was not identified, the focus of the media moved on  Eventually a person was named as a possible (and later probable) suspect, but reviews of the evidence pretty much refuted the official speculations.  In the minds of many that followed the events with interest, the perpetrator or perpetrators remain unknown. 

So did the anthrax attack have any lasting impact on the laboratory community?  Well try to walk into many laboratory facilities these days.  Security guards at all the entrance, and lots of electronic security locks.  Freezers that store microorganisms are more likely to be behind locked doors.  As to whether any of these procedures has made us safer is pretty unlikely.  All sorts of folks including students and staff and faculty still wander through the corridors and go into all sorts of places with all sorts of opportunity.  Someone bent on mischief and a modicum of imagination would be unstoppable.  But just as with airport security, we have gotten used to the added expenses and inconvenience, and I suspect that they will never go away.

Some think that the anthrax episode gave impetus to the laboratory quality movement, but that would be a huge overstatement.  The creation of ISO 15189 had begun 6 years previous, and the international Transport of Dangerous Goods stated some 10 years before that.   And the CLIA process in the United States started in the mid-1960s.  
There has been some activity about forming bioterrorism laboratory networks, and bioterrorism related quality assessment, but it would be fair to say that this has not remained front of mind. 

I will share a personal story.  About 8 years ago I started a process to get an exemption from TDG required packaging and labelling for transportation of Quality Assessment samples.  My argument was that our PT samples were so small in volume that they would represent nil risk to anyone.  I knew the events of 2001 were too fresh, but if we don’t start then no one ever has a chance to say “YES”.

 I was presenting my argument to a senior committee with little interest in supporting the proposal.   “If an airplane was to go down and someone was hit with a PT package there could be terrible consequences”.  My response was that if a plane went down between the jet fuel, the falling metal, and the falling bodies and all the released blood, tissues, body fluids, and body parts, my little package would be a pretty insignificant risk.  Indeed it would probably get incinerator from all the free jet fuel anyways.  

They were not impressed with the logic, and to date the answer to the request is still “NO”.

Microbiology, Error and Risk

About 12-14 years ago I was involved in a university affiliated hospital laboratory that got dragged into the consolidation fever of the time.  The plan was to move most of the laboratory down the road with a few exceptions to provide a rapid service for patients in the Emergency Department.  It was proposed (NOT by me) that one of the tests that should remain would be gram stain analysis of cerebral spinal fluid (CSF).  Fortunately we were able to convince some still rational minds that this would be a very bad idea; a highly critical test being performed by under-trained, under-experienced cross-discipline technologists would have far too great a risk for poor staining, poor reading and inappropriate interpretation.

Jump forward to today; the subject came up again.  With our proficiency testing program we have a fair number of very small laboratories, many in rural parts of northern Canada in a number of provinces.  Most of the work done in these laboratories is chemistry or haematology using modern analyzers.  Rarely or intermittently they are asked to do a gram stain, sometimes on swabs collected during surgery, sometimes on inflamed knees, and sometimes on CSF to help confirm a possible diagnosis of bacterial meningitis.  These laboratories would do one or two stains every couple months.  Some do as few as one or two stains a year. 

As part of our annual review process we have been contacting these laboratories to ask some specific questions about what we as PT providers can do to enhance our educational, technical, and quality oversight value.  It was interesting how many asked if we could act as an advocate on their behalf to stop doing the stains. 

As a microbiologist and a qualitologist and an advocate for patient safety, this seems to be as poor an idea today as it was 15 years ago.  The opportunity for harm is high.  Gram stain reagents that are under-used or improperly used get contaminate with bacteria or things taking on fungal forms.    Inexperienced eyes can confuse residue with gram positive cocci.  Are those dots probably grunge, or are they maybe Staphylococci ? Inexperienced eyes miss fragile organisms such as Haemophilus influenzae or Neisseria species.

Can I put a risk value number to this?  Not absolutely, but there is lots of evidence in a variety of laboratory skills that if a skill is not kept active errors occur frequently.  (The same is true when workers are too busy or too tired).  But if a critical error occurs only once a year in a laboratory performing 10 gram stains a year that is a 10 per cent error rate.

This seems immensely unfair to the technologist who is put in a position of stress and blame, to the physician who has to decide on the quality of information, and especially to the patient.  Pretty much lose-lose-lose.

As a PT provider there are a number of things that we can do.  One is a variant on traditional PT.  In this situation we send common slides to the laboratory for staining and have them return them to us for evaluation of the quality of the stain.  Another is to provide knowlingly negative slides.  Any positive report is a critical error.  The other is to ensure that we provide the maximum number of quality samples that we can so that their experience level can be grown and  maintained. 

While we may not have a legal obligation to take the extra step to monitor and improve performance, we at least have an ethical obligation.  After all, none of the other quality partners are in quite the same situation to provide the additional monitoring. 

More on this subject later.


PS: Registration for the Certificate Course in Laboratory Quality Management has begun with a lot of interest.  A number of changes this year addressing new ISO standards and a look at CLIA.  Those interested in more information can go to www.POLQM.ca or contact ubcpolqm@gmail.com

Quality and the Tower of Babel

For those of us in the Northern Hemisphere, September tends to be the end of summer and the return to more active work.  Welcome back.

According to the Bible (Book of Genesis) after the Great Flood all the people spoke a single language.  They migrated to the land of Shinar (modern Iraq?) and decided to build a city with a great tower.  God had distrust of this large group and their motives.  He confused their communication by creating the many different languages.  The work on the tower ceased and people scattered across the face of the earth.

Over the last while I have been communicating with a wide variety of groups that seem to have a common purpose to make health and patient care better.  But somewhere along the way they have become very fragmented and increasingly disconnected. 

Some of them fall into that collective group of quality partners.  This is the group of standards development organizations, accreditation bodies, proficiency testing providers, educator groups, professional organizations, and equipment and reagent suppliers.  Add to that the public representatives through the legislators, regulators, litigators, and the media.  More recently I have increasingly had activities with the groups, mainly in the clinical Patient Safety arena, and a new group (new to me) focused on Medical Diagnostic Errors.  

Some of these groups have splintered off even more distinctly, focused on Quality in resource limited regions while others are more interested in the opportunities in the wealthier countries, and others still fall somewhere in between because their interests are in the resource limited regions that are found within the wealthy countries. 

Like the aforementioned Babel, each group seems to be developing its own sets of definitions and jargon and discussion groups.  Each has its own set of meetings, which not surprisingly all occur at or around the same time of the year, but sufficiently distant that it would be difficult to combine.  
In Canada I recently discovered that three groups all with common interest have meetings planned for June 2012, two of them in the same hotel but a week apart and the third separated off by about a two hour drive and an additional week.  What was interesting to me was that none of the groups was aware that the others were meeting or even planning common discussions.  And Canada, while big in geography, doesn’t have all that many people.  How “Shinar-like” is that.  (Am I adding to the confusion by creating even more new words?). 

The problem (at least in my mind) is that as we all spin out further and more disparately, the opportunities for common goals get muddied and muddled.  Rather that building a large tower to the heavens we end up with a bunch of low-rises or maybe townhouses. Not that low-rises and townhouses are bad things, but I suspect that was not our collective primary purpose.  

As it turns out, all these groups appear to have a single vehicle that potentially can draw them together.  LinkedIn, the original social media has forums for all these groups.  If you have the time and inclination, a person can connect to all these interested parties and find the common links. 

To explore this further go to www.linkedin.com and hit the Group Directory.

While you are at it, for those interested in our on-line course in Laboratory Quality Management, go to www.polqm.ca.  Our registration for January 11, 2012 as begun.

Benchmarking Clinical Microbiology

There is a universal appeal to finding a quantitative measure for assessing laboratory quality.  In some respects it is an appeal that I would like to resist because it becomes an ugly  part of inter-laboratory comparison (My laboratory is better than your laboratory.  My laboratory director can beat up your laboratory director).   But at the same time as a proficiency testing provider, one of my tasks is to provide some sort of quantitation of performance so that laboratories can have some benchmarks useful for monitoring their own level of performance and to use those measures as part of the continual improvement process. 

As a PT provider we assess responses to challenges and grade them.  In Clinical Microbiology correct responses for any result have to be more than accurate; the report needs to provide appropriate and relevant information.  So, in our program a grade of 4 is given for an acceptable response that is considered accurate and appropriate and relevant.  A grade of 3 indicates also an acceptable response, although not completely complete.  On the other side a grade of 1 is given for a response that is unacceptable because it is inaccurate or incomplete.  A grade of 0 indicates a response that if given for a clinical sample could lead to harm or inconvenience to patients.
Not all PT programs use that type of scale, but we have used this for near 30 years and have a sense of its strengths and weaknesses.

So with those values in mind we can review how well or unwell do laboratories perform on proficiency testing.  As can be seen in Figure 1, the vast majority of laboratories score 4’s and 3’s on the vast majority of challenges.  The number of 1’s is small, as are the 0’s.

So the first message should be that performance on individual proficiency testing challenges that are less than perfect should be viewed as a signal that perhaps there is a difference between what your laboratory considers as accurate and appropriate and relevant and what a panel of peers considers.

A second is what percentage of laboratories is able to earn 100 percent of their achievable score.  (If a laboratory performs 4 challenges and each as a perfect value of 100, then their achievable score is 400).  From the above you would expect that near all laboratories would get 100% or near 100%, but not all challenges have equal values  Some are more complex than others and therefore contain more gradable elements.  And not all laboratories are equal performers.  Some have ready access to well trained and experienced staff and sophisticated testing equipment while others do not.  
Each year we look at the proportion of laboratories that are able to achieve at least 90 percent of their achievable score.    While most are at or near 100 percent, there are always some that fall below.  On average, over the last 9 years, 75 percent of clinical laboratories earn 90 percent or greater of their achievable score.  Large complex laboratories tend to score higher than smaller laboratories.   

So some benchmarks to consider for a clinical microbiology laboratory’s performance.  On an individual challenge basis, any result that is not accurate and appropriate and relevant should be considered as an indicator for improvement.  On an annual basis achieving less than 80 percent of your total achievable score should be considered as another indicator. 

One might fairly argue that how a laboratory scores on Proficiency Testing does not necessarily reflect on clinical performance.  That would be a difficult relationship to prove or disprove.  But if PT samples look like and act live clinical samples, then the relationship may be more plausible.  

For a fuller examination of the information used to benchmark clinical microbiology performance values, check out the Annual Report at www.CMPT.ca