Even committed Qualitologists can make mistakes.

Even committed Qualitologists can make mistakes.  Darn. 

This is the 10^th season of our virtual classroom on-line course, and one might think that we have the process down pat.  Each year we review the information from the previous year, retain what is still relevant and appropriate, and make revisions where needed. 

Every year we make a few “big changes”.  This year it is a new Module on modern tools for Quality and an additional Quiz.  These “big” things take a little more time and attention and care.  And that is where I messed up.  (Again, Darn!).

Quiz 1 is completed at the end of Module 1.  It is an on-line auto-graded multiple choice exercise of 10 questions that should take no more than 30 minutes.  We allot an hour.

When the Quiz 1 was completed, the grades were surprisingly and disappointingly low.  Considering the pre-selection process we undergo to ensure we have the right participants, they should have done better.  Once we confirmed that the problem was not a computer grading technical error, or a connection error or a transcription error, we had to go back further to when I wrote and set the questions and responses.

On one question, I had defined a wrong answer choice as the correct response, thus everyone answering correctly got a wrong result flag.  This was annoying because the right answer was obvious and apparent.  I had just messed up.

Additionally, two other questions were so subtle, so nuanced, that even I who had set the questions could barely figure them out.  No wonder we were getting some very unhappy messages.  

I remember clearly sitting in my office and setting the questions.  I remember the process was being slow and arduous.  The quiz muse was staying away and I was struggling.  Then I had a rush of ideas and got the questions completed in a hurry.  Clearly the rush was not necessarily lucid.  

Message to self: Hurrying is a bad thing.

A decision was made that we could not let the quiz stand, if possible without inconveniencing the participants.  This was completed through adjusting the auto-marking software and re-evaluating the response.   The marks climbed to exactly the level we anticipated.  Then we informed the participants of the error that had taken place and described the remediation process and told them to recheck their results.  Then I had a discussion with the coordinator on how I would try to avoid making the same mistake of pushing to finish.  I have created a check list that all quiz and examination writers will need to go through before questions can be submitted for final posting.  

Four lessons learned:

 Setting quiz questions without deliberate and sufficient re-check and conformation time is an unnecessary risk taking procedure than increases potential error production. 

Remediation and Correction takes a lot more time than error causation.

Quiz errors hurt program credibility

This simple hurry-up slip has been a TEEM loss event.  The gross cost to us has been 6 hours of emails and discussions and IT labor shared by 3 people, plus we have had to make the check-list and revise our procedure manual.  And then we need to do the preventive thing and re-check Quiz 2 and the final examination to make sure that we (I) didn’t mess them as well.  And to that we have to add my frustration and a whole bunch of participant unhappiness.

Still, having a detection - remediation – correction and prevention process that can and does pick up and analyze (study) mistakes and amend (act) them “expeditiously” makes the point that Quality works.  

Hooray.