One of the names that is used for proficiency testing is "Inter-Laboratory Comparison". There are many reasons to get rid of this name and indeed the concept because it implies that the point of the exercise is to somehow do "better" than other laboratories. All that does is foster being duplicitous (i.e. cheat), and to view PT as something other than a quality measuring tool.
But at the same time, there are some advantages of creating a consistent method of grading performance, so that laboratories can monitor their own progress and performance, and provincial or state authorities can measure group performance. In our program we developed a measurement which we call “Percent Achievability”
In our program, we have 4 classes of laboratories: (A) complex , (B) Intermediate, (C) small, (C1) pack-and-ship. Each group gets a set of samples that matches its level of complexity. (A) laboratories get more samples and more complex samples. (C1) laboratories get fewer and limited.
Each sample or subcomponent of a sample, is graded in the same manner. (4) Full value (3) Acceptable – with minor error (1) Unacceptable – major error and (0) Unacceptable – potential negative impact on patient care.
To measure Percent Achievability” we would take the score that a laboratory achieved over a year, and divide that by the score they would have received if all their challenges had received Full value. For example: if a laboratory did 10 tests and received 8 Full Value (4) and 2 Acceptable – minor (3), their score would be 38, and their %Achievability would be 38/40 (95%).
We have tracked this now for a decade and have seen that over the years category (A) laboratories have maintained a consistent % Achievability near 95%, while smaller laboratories have not maintained the same level of success. (see graph).
First of all, I believe this graph to be a reflection of reality, and I think we can account for how and why this has happened. As laboratory consolidation has occurred larger laboratories have been able to retain their expertise and competency, while smaller laboratories have not.
So my question is …
Is this a phenomenon that you are seeing in your jurisdiction?
Dear Dr Noble,
ReplyDeleteYour latest blog story is of great interest to me as I have often asked myself a number of questions regarding the grading of CMPT's.
For example, I have often wondered about the accuracy of such gradings.
I also work in a laboratory that process CMPT's at regular intervals. I have even attended a discussion workshop explaining several CMPT results.
At this workshop it was implied that smaller laboratories may have (probably have) lower results compared to larger laboratories due to lesser expertise as well as due to less sophisticated (or shall we say 'of lesser technology') type of testing (ie analyzers, etc).
Yet, having participated in a number of CMPT's myself I wonder if there is a way to test my theory as well?
It is my understanding that a CMPT specimen should be handled no different than any other patient sample coming through the laboratory. And, provided this procedure is being followed as such, it would then be evident of a certain accuracy in patient testing. However, is this what smaller laboratories in fact are following?
It is my theory that some laboratories (probably the smaller ones) are more likely to follow the rules by handling the CMPT specimen very similar to a patient specimen. Whereas there is no doubt in my mind, other laboratory see the CMPT as a 'special' category and thereby working (and re-working) the specimen to death. Naturally, the laboratories with the sufficient resources to process and re-process a CMPT specimen that way, should inadvertently also receive higher marks.
On the other hand, if my theory is correct, then it would stand to reason that several laboratories are actually 'cheating' - and they are not being penalized for it. So really, what good does a CMPT grade do?
Sincerely,
Just another follower that enjoys reading your blog
PS. I wonder what would happen if CMPT would actually send an anonymous patient sample - under a false name...rather than CMPT numbering and lettering?
Thanks for your comment. I agree that looking at and interpreting graphs can often be in the eye of the beholder. I can convince myself of anything if I spend enough time trying. That being said, the graph is pretty clear; one category has remained very stable over the last decade with its mean percent achievability, while the others have not. Additionally there appears to be a progressive year-over-year pattern for the decline, Additionally, as the categories change from A-B-C-C1 the pattern of decline increases.
ReplyDeleteFirst, as far as I am aware, we are the only program using a Percent Achievability assessment, and this may all be statistical gibberish. But it does carry a certain logic, and deserves some validation. But let’s for the moment pretend the information is valid.
One interpretation is that the category A laboratories are all “gaming” their proficiency testing, while the other categories are not. That might account for their consistency, but does not explain the progressive patterns in the decline.
Another factor might be that over the decade we have made our challenges progressively harder, but a review of them does not support that.
Another factor might be that there has been a progressive disinterest in PT in laboratories which is reflected by decreasing performance. If anything, when we look at our correspondence, and communications, and satisfaction surveys, we see the opposite.
Over the last decade provinces in Canada have consolidated laboratories, and through the process have moved technologists to larger laboratories. Finding technologists to work in rural areas is a continuing challenge, and smaller laboratories have borne the brunt of that. Also funding for continuing education has decreased.
So, I come back to my first interpretation, with caution, not intended to demean smaller laboratories, but rather for governments to recognize that current policies are potentially doing harm, and it is time to start putting resources back into the local laboratory.
Remember when one of our political parties used to promote good health care as “closer to home”?