National clinical assessments

The UK’s General Medical Council is currently considering implementing a national clinical assessment. Allison Barrett investigates the pros and cons of the American system, which is now just over a year old

The clinical skills assessment of the US medical licensing system remains controversial after its inception last June. Some people say that the test is too expensive, redundant, and not easily accessible. Its advocates defend that it creates an even playing field among medical schools and is necessary to maintain the standards of the profession. After one full year of testing, the camps remain divided.

Background

The American system of qualifying doctors requires three sittings of the United States Medical Licensing Exam (USMLE). Step 1, taken after the second year of medical school, is written and tests basic science knowledge. Step 2, taken between third and fourth years of medical school, is part written and part pass or fail clinical skills assessment. Step 3, taken during residency, is written and involves computer based patient cases. After completion of all the exams, the doctor applies for licensure in an individual state.

The National Board of Medical Examiners reconstructed the step 2 last year. It was initially a written only clinical knowledge test. They added the clinical skills component in June 2004. The clinical skills exam is an eight hour test with 11 or 12 standardised patient encounters, each lasting 15 minutes. It is offered at five sites (Philadelphia, Atlanta, Los Angeles, Chicago, and Houston). Medical students graduating in 2005 and beyond are required to complete both portions of the step 2. Foreign medical graduates seeking licensure in the United States must complete the same exams.

For

VOISIN/PHANIE/REX
Say aaaaaarggh...

There are more than 50 independent medical licensing agencies,¹ and 125 allopathic medical schools in the United States.² Each school has its own mission statement and mechan­isms of internal assessment. Some doctors are also highly mobile, receiving education and training in multiple states and schools. High mobility, variability in schools, and the sheer size of the US medical system are important reasons to promote a national assessment, said Peter Scoles, senior vice president for the National Board of Medical Examiners’ assessment programmes. Equally as important is the obligation of the medical community to the public to license only qualified doctors.

“Generally there is acceptance of [the exam] as a concept and of a manifestation of our obligation to the public to demonstrate to the public that our graduates are competent,” said Peter Katsufrakis, national chairman of student affairs for the Association of American Medical Colleges.

The message to schools is clear as well, said Stuart Slavin, associate dean for curriculum at Saint Louis University School of Medicine. “I think that the biggest advantage is that it sends a message to medical schools and students that com­petency in clinical skills is very important, and that performance on computer based exams that focus only on knowledge are not the only or the best measure of what it takes to be a doctor. I think that’s very significant,” said Slavin.

Against

Keeping surgical skills up to scratch

Katsufrakis has mixed feelings. Within his constituency of medical school administrators, some recognise the benefits of the exam but find it challenging to accept on the basis of cost and redundancy, he said.

Those arguments are not new. Since the National Board of Medical Examiners first explored the idea of national clinical skills assessments, it has been unpopular.

To take the clinical skills test costs $975 (£539; a789)—a hefty fee for struggling medical students who sometimes graduate with more than $100000 of debt. But Scoles deflected criticisms of the cost. If the average American medical student spends $250000 on tuition, accomodation, and board, then the $2500 fees for the three USMLE tests is proportionally small, he said.

“The annual percentage increases in medical school tuition have been far greater than the cost of this examination,” said Scoles. “It is true that it is expensive, but this is one small piece of this expense.”

Students also pay the air fare and cost of accommodation at the testing site, and with only five sites in the country, some are far away. The National Board of Medical Examiners has no intention of opening up more testing centres because to do so would add $150 to each student’s testing fee, said Scoles. However, if volume increases, testing capacity will be expanded.

Opponents also argue that the test is redundant. According to Scoles, about 80% to 85% of medical schools require students to take the clinical skills exams to graduate, but surveys indicate that most schools have in their curriculums clinical skills assessments and standardised patients encounters. Some of these tests, called objective structured clinical examinations, have proved to be good predictors of later clinical success.^{3, 4}

But Scoles said he does not think objective structured clinical examinations are sufficient for such an important assessment. Doctors and standardised patients develop relationships with students over the years. Making the transition from mentor to evaluator may be difficult and “simply too much to ask,” he said.

That’s not to say that objective structured clinical examinations are unimportant, he said. “Clinical education is as much a mandate to medical school as the teaching of gross anatomy or physiology or internal medicine. It’s an obligation that schools have and they take it seriously.”

At Saint Louis University, students are assessed by multiple objective structured clinical examinations throughout their education, said Slavin, who directs the school’s medical curriculum.

“I don’t think for Saint Louis University students that [the clinical skills exam] adds a great deal because we have such an extensive system of clinical skills testing, but for schools that don’t provide that level of testing, there may be advantages.”

Last year, only two of the 150 Saint Louis University students who took the USMLE clinical skills exam failed, according to Slavin.

About 30000 US and international students have taken the test so far, according to Scoles. The overall pass rate was 96% for US and Canadian medical students from June 2004 to March 2005.⁵ With a pass rate so high, some question why the test is needed, but Scoles is firm in stating that the purpose of the exam is not to fail students. Katsufrakis agreed.

“I don’t see a conflict between the fact that there’s a 96% pass rate and that there’s a need for the examination. In fact you would hope that the number of individuals taking the exam would pass it,” he said.

To weed out those few “bad” students who fail the exam costs others large amounts of money, however. Ultimately, the test might prevent a handful of physicians from being licensed, but he questioned whether that would affect healthcare delivery. “If we accept the premise that our resources are limited, then would the time and money devoted to this exam be the best use of funds and effort, or would they have a more significant, positive effect on patients’ health if applied in a different manner?”

Leana Wen, president of the American Medical Students Association said the passing rate was somewhat encouraging, however. “In a way it’s good because that means that our ­clinical skills training in medical school is better than expected,” she said.

Worldwide

Canada implemented a national clini­cal skills exam several years before the US. The Medical Council of Canada qualifying exam part II is taken during residency and involves interacting with several standardised patients over the course of the three hour exam. It is offered a few times a year.

The United Kingdom, like most countries, does not use national clinical skills testing. Most schools use standardised patients and objective structured clinical examinations for evaluation. However, the UK testing method may go the way of the US method in the future. The GMC’s education committee is currently consulting on the topic of national testing. In outlining the consultation, the GMC asked whether their current training mechanisms are sufficient to show that graduates are competent.

“A national licensing assessment held at the end of the undergraduate degree would impose more control on the delivery of medical education in order to establish more consistency in standards across medical schools and students. An assessment could foster more transparency and confidence in the medical profession by the public, but it could potentially reduce innovation and flexibility within the system,” said the GMC.⁶

The consultation, which closes in October, will include interviews with students, professors, patients, and administrators. The final proposal with specific guidelines should be issued in the first quarter of 2006, according to Peter Rubin, committee chairman.

Allison Barrett, Second year medical student, University of Boston
Email: ambarret@bu.edu


studentBMJ 2005;13:309-352 September ISSN 0966-6494

1. American Medical Association. Links to state medical boards. Chicago: AMA, 2005. www.ama-assn.org/ama/pub/category/2645.html (accessed 15 Aug 2005).
2. Association of American Medical Colleges. Medical schools. Washington, DC: AAMC. www.aamc.org/medicalschools.htm (accessed 15 Aug 2005).
3. Martin IG, Jolly B. Predictive validity and estimated cut score of an objective structured clinical examination (OSCE) used as an assessment of clinical skills at the end of the first clinical year. Med Educ 2002;36: 418-25.
4. Auewarakul C, Downing SM, Jaturatamrong U, Praditsuwan R. Sources of validity evidence for an internal medicine student evaluation system: an evaluative study of assessment methods. Med Educ 2005;39:276-83.
5. United States Medical Licensing Exam. Step 2 clinical skills (CS): information index. Philadelphia: USMLE. www.usmle.org/step2/Step2CS/Step2Indexes/Step2CS_Scoring.htm (accessed 15 Aug 2005).
6. General Medical Council. Strategic options for undergraduate medical education consultation. London: GMC. www.gmc-uk.org/med_ed/strategic_options.htm (accessed 15 Aug 2005).