John Garrow looks at evidence based medicine and considers who really provides the evidence; Very Senior Clinicians or Omniscient Meta-analysts? He asks whether medical students consider evidence
The case for evidence based medicine is strong. Some 50 years ago, teaching medical students was based on the experience of Very Senior Clinicians who wrote or edited medical textbooks. But if you read a textbook published just 20 years ago, you will find statements that we now know are seriously untrue. If a Very Senior Clinician asserted, in one edition of his or her textbook, that treatment X was the best choice for condition Y, this tended to persist in subsequent editions unless strong evidence to the contrary was discovered. Today, the gurus pronouncing on the efficacy of treatments are no longer senior clinicians, but Omniscient Meta-analysts. These serious minded people may have little personal experience of treating condition Y, but they diligently seek out every randomised controlled trial of available treatments, and by combining all this information may decide that treatment Z is even better than orthodox treatment X. Evidence based medicine involves accepting guidance from meta-analysts, rather than clinicians, which is usually an improvement but is not infallible.
You may want to know if eating lots of fruit and vegetables from early childhood confers protection from malignant disease in late adult life. Or if going through repeated cycles of weight loss and regain carries risks that are avoided if you maintain constant adult weight. Or if it is better for your health to drink three glasses of wine per day or just the one? On such interesting questions the Omniscient Meta-
analysts give a muffled response, because it is difficult to set up a randomised controlled trial to answer them. The available evidence comes from epidemiological associations, and these are often as unreliable witnesses as the clinical experience of Very Senior Clinicians. Worse still is the situation in which there are data from randomised controlled trials, but the trials were so badly designed that the evidence they provide
is almost useless. Evidence based medicine that is based on bad evidence is bad medicine.
So that brings us to the question: who examines evidence? Omniscient Meta-
analysts do, but do not imagine that they always agree about what a combination of all the evidence tells us. For example, when the value of mammography in screening for breast cancer is evaluated, there are some Omniscient Meta-analysts who conclude that it saves lives, but others deny there is any evidence of benefit. The reason for the difference is not in the technique of statistical analysis, but in the weight given to trials that some consider to be sound, but others think are flawed. Editors of peer reviewed research journals, with the help of expert reviewers, evaluate the contribution to current knowledge of papers submitted for publication. When, as a Very Senior Clinician, I became editor of such a journal, I realised that I had no formal training for the task of distinguishing between sound or flawed research protocols--certainly this was not part of the medical curriculum when I was a student.
The winning medical student was James Hopkins (Guy's, King's, and St Thomas's), and the runners up were Samena Chaudry (Birmingham), Miriam Jamal Hanjani, Daniel Marks, Marianna Thomas (all of Royal Free and University College), and Jon Silversides (Belfast). Among the students of nursing (midwifery) the winner was Elaine Sweet, and Harriet Anderson was the runner up (both from King's College).
What of medical undergraduates today, for whom the studentBMJ is published? Last year, HealthWatch, a charity of which I am chairman, was funded to run a competition for undergraduate students of medicine, nursing, or alternative medicine. The task was to evaluate in not more than 600 words four hypothetical research protocols that were obtainable either by post or from our website. We sent notices to 91 academic departments in April 2002, and reminders four weeks before the closing date (31 July 2002). The total prize money on offer was £3000 ($4838; a4612), £500 to the winners in each of the three healthcare categories, and £100 for up to five runners up in each category. We received only 17 entries from medical students, three from nursing students, and one from a student of alternative medicine.
Why were there so few entries? It was not because students do not have access to good teaching material about appraising protocols, because the winning entries were excellent. I suspect that at least part of the reason was that the examinations which students face are not designed to test their ability to appraise protocols, and that students set a low priority on learning skills that will not be tested in examinations. This is understandable, but unfortunate. When they graduate and enter clinical practice these students will be expected to maintain the principles of evidence based medicine, but much of the "evidence" they are shown will be of poor quality. Patients now appear in general practitioner surgeries with material downloaded from the internet that may give misleading information about the efficacy of treatments. Surely it must be an important part of medical training to be able to tell the patient if such information is reliable, and if not, why not?
So, this year, HealthWatch will launch another competition, with a new set of protocols, available from April; the closing date for entries is 31 July. Entering might mean you pick up and practise skills important for your future career, not necessarily for your looming exams. And if you won a prize it would look good on your CV, demonstrate to your tutors and peers your ability critically to appraise research evidence, and the cash might be handy too.
