Saving Science: The Argument for Quality Assurance

About a year ago, I wrote on the subject of challenges in the medical research laboratory community, with particular reference to an article in a newspaper article and a published article by John Ioannidis.  [ See: Quanlity and the Research Laboratory  - http://www.medicallaboratoryquality.com/2012/12/quality-and-medical-research.html ]  The discussion continues with the October 19, 2013  edition of The Economist: Unreliable research - Trouble at the lab.

The article presents to a number of studies that point out that the process of peer review is badly flawed (not really a surprise!!) but it points to an actual viable solution. 

The story references John Bohannon, a biologist at Harvard, recently submitted a paper on cancer research 304 journals describing themselves as using peer review. What made this article different was that (a) the paper was sent under a pseudo name and (b) it was intentionally flawed with errors in study design, analysis and interpretation of results. What was problematic was that 157 of the journals accepted it for publication.

Consistent with that observation, Fiona Godlee, editor (1998) of British Medical Journal, did a similar study by sending a similarly flawed article to more than 200 of the BMJ’s regular reviewers. On average, only 25 percent of flaws were reported.  No one picked up 100% and some reported none.

BMJ then did a follow-up study in which they told reviewers it was a test; despite this performance did not improve.  And a subsequent study from UCSF indicated that over time reviewer performance did not get better, it got worse.

This is pretty gruesome.

When credible journals like the Economist get on a subject, you may want to dispute some or all of their facts and ideas, but the reality is that their international impact on public and government and funding organization opinions is substantial.  Articles like this can make 1% or 5% or more impact on research funding, without causing a single eye to be bat.  In a time of economic fragility, saving a billion or two from research budget can seem like an easy way to deal with budget crunch.  We can ignore it, or slur it, but we do this at our collective peril.

Wearing my “Gee, what a surprise” hat, let me say that all this supports my argument that institutions need to clean up their act considerably.  More importantly it reinforces my argument that traditional published knowledge is no more reliable than information well presented by a trusted person in a standardized structure, but submitted electronically through instruments including blogs.  

But wearing my other “let’s get on and fix this” had, it seems to me there is a viable solution.  Medical and non-medical testing and calibration laboratories can be improved through quality assessment.  In our own recent examination, 75 percent of laboratories that were correcting proficiency testing error found system errors that impacted on their routine testing.  [See: http://www.medicallaboratoryquality.com/2013/08/proficiency-testing-does-improve-quality.html ]

I propose the following:  all credible journals need to develop a quality assessment strategy for all manuscript reviewers. 

1.    Reviewers are selected by virtual of their knowledge and skills in particular disciplines.

2.    Selected reviewers are retained for a finite period of time.

3.    Selected reviewers are expected to participate in proficiency testing of their reviewing skills on a regular basis, arbitrarily twice a year. 

4.    Reviewers who are found proficient, can continue on with reviewing.

5.    Reviewers who are found to have opportunities for improvement, either demonstrate their improvement, or move on.

It seems to me this is a viable solution, and is consistent with a variety of proficiency challenges that we do on a regular basis.  Acquiring manuscripts that are deemed flawed or not flawed would not take a long time.  Editing for freshness would not be particularly challenging.  

The review process would improve.  The quality of published literature would improve.  The confidence in published literature would improvement.

A healthy solution for the concerned patient

For those living in the lower South West corner of British Columbia surrounding Vancouver (aka the lower mainland), there was a very interesting article in the Vancouver Sun newspaper about CMPT, and POLQM, and our conference written by the health reporter  Pamela Fayerman.  The URL to her blog is:  http://blogs.vancouversun.com/2013/10/19/dr-michael-noble-a-microbiologist-with-a-passion-for-patient-safety-in-lab-testing/

Following the newspaper publishing the story which included a reference to MMLQR, I received the following comment: “I have been dealing with a particular lab for years now, I had paternity testing done there in 1994. It was HLA testing, there is no chain of custody of any sample. The lab has for years refused to release the work supporting their report. I wish to have it independently examined by my OWN genetics person. If you are currently reviewing lab error and have found all the problems mentioned in your news article on Saturday, imagine what is being hidden by labs like this particular lab who are refusing to release the file for independent review. This is no longer error, it is fraud that is being hidden to protect themselves. What lab error rate does in fact actually exist on DNA labs and or hospital labs conducting DNA testing. Does this review include this test? Please advise me of this. This sort of review would be most helpful. on”

I thought long and hard about whether I should allow the comment to be published, and ultimately decided that I would, first removing the name of the laboratory, in part because that would match the anonymity of the writer.  My reason for publishing was because it points out many of the principles of Quality that I support.

First off, this comment represents the voice of an interested party, whose feels that their concern and complaint has gone unresolved for now near 20 years.  Without taking sides, I can only state the obvious; 20 years is a long time, and whatever steps have or have not been taken are not producing a desirable outcome.  Conflict is not being resolved.  

Second, this invested person, as part of the general public, is asking some pretty basic questions, concerning laboratory error rates DNA labs and or hospital labs conducting DNA testing.  Whatever the circumstance that has led to the question, it is a question worth asking.  

In British Columbia we have a healthcare Quality Council that has taken a major and bold step forward by creating the software to allow people to report healthcare errors.  The program is voluntary and to the best of my knowledge the results are not publically available; but they are available and distributed to certain people and groups in authority.  

In time, sooner or later, I suspect that process is going to change.  At some point I can believe that reporting error rates will become mandated.  Patients will demand it, the public will demand it, and I suspect politicians will be put in a position of having to address the demand.  

From my perspective, the argument for regular reporting is too compelling to be ignored.  Ours is a public system, funded with public resources to address public interest.  We have a system that tolerates the reporting of physician incomes from the public purse; the argument for reporting error makes far more sense than reporting salaries.  

We do have intermittent situations where public notice of error occurs, such as when we locally had problems with X-ray reports, or when we had errors in reading breast biopsies in Newfoundland.  That is far too late, and gives off a poor message.  It points out the BIG problems without context.

Dealing with unhappy consumers likely occurs regularly, and I suspect more are resolved without going through court.  But we don’t know that.  Reporting error rates on a regular and open process would likely show that errors occur somewhere in the vicinity of 1-10 per 1000 people, probably closer to 1-2 as opposed to 8-9, generating a Six-sigma metric greater than 4.  Where it is lower we would have the active opportunity for improvement.

That would be a lot better than what we have now.

The importance of meetings

The importance of meetings

The importance of meetings is that people meet. Today we have to change that a old adage a bit, because people can also “meet” over video conference and webcast and text conferencing;  all these have value, but from my perspective are not the same as a good old fashioned face-to-face  meeting and having the opportunity to share common live experiences.  

Over the last few days we have been hosting the Quality Management Conference for Medical Laboratories here in Vancouver.  We have about 130 people here, talking, eating, and sharing experience in one room.   The audience included the full spectrum of laboratory professionals interested in laboratory quality.  There were laboratory technologists, administrators, leaders and directors, investigators and consultants, and of course, very important, partners and sponsors.  There were presentations and discussion, agreement and (respectful) disagreement, and good food and entertainment.

It was, from my very biased perspective, a very successful meeting of tangible minds in a live reality experience.

F-t-F meetings are actually becoming a lot harder to orchestrate, at least in Canada because resources for hosting meetings and resources for attending meetings are shrinking.  Both the public sector and private sector are feeling the squeeze.  But laboratory administrations do need to recognize that Quality improvement does not happen in a vacuum; improvement needs to be linked to continuing education.  Education without meetings can and does occur, but books and webcasts and staring at a computer can go only so far.  And while I appreciate that industry is under its own set of pressures from competition and Merger and Acquisition (M&A), cutting off its audience doesn’t make things better; it makes things worse.  A single meeting brings the sponsor together with a hundred or more contacts.  Everybody wins.

There were a number of themes that arose during the two and a half days:

·       There are links between medical laboratory quality and standardization in all its guises.

·       Quality and Safety and Risk and Error are intimately linked and improved through appropriate application of Quality tools

·       The critical role for Quality education at all levels.  (over all consensus is that starting on-the-job is way too late!)

·       Laboratory Quality has huge impact on patient care, and its corollary, patients can and do make their mark on laboratory management through voicing concerns and compliments and complaints. 

More information to follow.

This was not our “first rodeo”; over the years we have put on about 25 conferences.  This was our most ambitious, and from many perspectives, our most successful. 

The presentations will soon all be posted and we will share the link.

Our most immediate task is to make sure people fill in the Evaluation form.  I don’t know why, but getting that part done is often like pulling teeth.  I am hoping that we can get to 15%, but even that may turn out to be a lot of work.

I am starting to think we have to start building in incentives for submitting an evaluation.  I

The Value of the AGM

This is a busy time for the Medical Laboratory Quality arm of University of British Columbia; we had the Annual General Meeting for Clinical Microbiology Proficiency Testing (CMPT) last week and this week we have the Quality Management Conference for Medical Laboratories by the Program Office for Laboratory Quality Management (POLQM).  That is a ton of work, and if it weren’t for all the people that work with me, and for W Edwards Deming, it would have been a disaster.  

CMPT and POLQM staff are all strongly committed to Quality and that and a lot of hard work make the two programs work; but it was our collective commitment to ISO9001:2008 and Deming’s PDSA that created the structure that allowed us to be as organized as we are.  

The AGM went very well.  While we don’t unfortunately have records readily available, to my recognition we have been hosting an AGM since at least 2000; perhaps longer.   It is a convenient time for all the committee members and our stakeholders, especially the provincial accreditation bodies with whom we work, to get together in one room and sort out what is working with CMPT and what needs improvement.  The following day, the clinical bacteriology committee gets together in camera, and selected the next set of challenge samples that will be sent out between May 2014 and April 2015.  

Perhaps the most important part of the AGM from my perspective is when I share with the group how we have fared with last year’s Goals and Objectives, and more importantly the AGM is when I announce our new set of  G&Os for next year.

Last year we had a total of 6 G&Os, 4 were program oriented and 2 were Quality oriented.  All 6 were met.  

 This year we have a new set of 9 that address fiscal issues including personnel and hardware requirements, and importantly a plan to move forward on a new recognition pathway.  (As we progress forward, I will write more.)  It should be an exciting year.

One of the G&Os not listed at last year’s meeting, but added in as an extra in mid-year was the intent to allow medical laboratories that use CMPT products and services to send representatives to subsequent meetings with the view that being  present and more importantly being actively involved  during Question and Answer sessions could help create stronger relationships with those laboratories.  

It was a good idea, and well planned and promoted, and had some success.  We had people who planned to come, although one had to withdraw.  What I was hoping to have, questions arising during Q&A, didn’t occur.  Perhaps these new AGM attenders felt intimidated by the questions from the regular attenders, or perhaps they were content to just sit and listen; I can’t say.  But, I think the old adage of “you can’t win the lottery if you don’t buy a ticket” applies.  If you are not present during the Question and Answer period, then you are less likely to have your questions asked or answered.  

So I count this as a partial victory; a bunch read the announcements, some responded and a few attended.  I consider that as a good start.  Maybe next year a few more will attend and we will get a few questions from the floor.

As a final comment, within ISO 17043, the standard for competency assessment for proficiency testing programs, one finds requirement for management review and review meetings.  That is standard good quality, whether it is derived from 17025 or 9001.  What is not mentioned is any requirement or recommendation or a note suggesting that programs consider hosting a stakeholder meeting to bring together interested parties, including the customers to talk about product and service.  

I think that is a flaw in the standard that could and indeed should be addressed in future editions.