Welcome to CPHQ Tutor

Pretesting of US-Specific Competencies Questions: Why the HQCC Got It Wrong

February 2, 2012

This article was first published for members only on the predecessor of the CPHQ Tutor website on February 2, 2012. It was released to the general community on January 1, 2015.

In my previous article, I reported the HQCC’s decision to perform pretesting of the questions on US regulations and accreditation in 2012 and only score these questions from January 1, 2013.

I expect this move to be welcomed by most candidates. But what troubles me is the way in which the pretest items/questions (that address competencies related to US regulations and accreditation) are being pretested.

Allow me to explain.

Members of this site were first informed of the impending changes to the CPHQ exam in mid June 2011. Subsequently, other parties got wind of the news; it has understandably caused a degree of anxiety, especially among international candidates.

Pretesting of the US-specific competencies questions commenced in January 2012 (last month).

The role of pretesting is to validate the questions “as accurate and appropriate before it is included as a measure of candidate competency.” (Page 5, CPHQ Candidate Examination Handbook 2011) This assumes indistinguishability between scored and pretest questions and best effort on the part of candidates:

Pretest Questions on the Examination
In addition to the 125 scored questions, CPHQ examinations also include an additional 15 pretest questions. You will be asked to answer these questions, however, they will not be included in the scored examination result . Pretest questions will be disbursed within the examination, and you will not be able to determine which of the questions are being pretested and which will be included in your score. This is necessary to assure that candidates answer pretest questions in the same manner as they do scored questions . This allows the question to be validated as accurate and appropriate before it is included as a measure of candidate competency.

(Page 5, CPHQ Candidate Examination Handbook 2011. Current as at February 2, 2012.)

Clearly, the key requirements of pretesting are not being met because:

  • Candidates will be able to identify the new (US-specific competencies) questions quite easily because they are not linked to any of the tasks in the existing exam content outline; and
  • Candidates are almost certainly going to answer these questions in a different way as they do scored questions, if at all, because they know that these questions will be unscored. In other words, they are not likely to feel inclined to put in as much effort into answering the identified pretest questions as the scored questions.

Scored questions should be able to help discriminate between candidates who have sufficient competency and those who haven’t. If everyone—both the competent and the less competent candidate— is able to identify the pretest questions, and therefore less likely to give the same amount of effort as the scored questions, it will be impossible for the “question to be validated as accurate and appropriate.”

What the HQCC Should Have Done

The HQCC has included the pretest US-specific competencies questions on the CPHQ exam before updating the exam content outline that is found in the examination handbook. This was probably a mistake.

If the exam content outline had been updated to include competencies in US regulations and accreditation before the introduction of the questions that address these tasks, the situation would have been different. Candidates would not likely be able to tell whether the (US-specific competencies) questions are pretested or scored, provided the HQCC did not inform candidates the exact date when the US-specific competencies questions will begin to be scored. Anyone taking the exam after the date on which the exam content outline was updated would have to assume that questions on US regulations and accreditation are scored and, therefore, give their best effort in answering them. The HQCC could allow, say, 3–4 months of pretesting before scoring eligible questions on the exam.

Unfortunately, the responses to the pretest questions provided by candidates who took their exam from January 2012 to date, if any, will not be valid and are best discarded.

Conclusion

The way in which the HQCC is pretesting questions on US-specific competencies that relate to regulations and accreditation into the CPHQ exam is fraught with risk of bias. However, it is early days yet in the process, and the problem may be easily corrected by masking the dates on which pretesting of those questions begins and ends.

{ 1 trackback }

{ 0 comments… add one now }

Leave a Comment