Commentary

Mona Okasha takes you through this paper and explains what it means and what to look out for

In this paper, the authors set out to estimate the prevalence of epilepsy in the prison population. The method that they chose was to synthesise data that have already been published on this topic. To identify all the previously published papers, they conducted a systematic review. This methodology involves using prespecified, well defined search strategies, typically of more than one electronic database (in this case Medline, Embase, and PsychInfo), as well as other methods (here they used hand searching of relevant journals and searching of bibliographies).

Why is it necessary to be so thorough when searching for papers, you may wonder? It is vital in a systematic review to identify all the previously published papers. Some studies also make an effort to identify unpublished data, or those published in reports or more obscure media, the so called grey literature. If some studies are excluded by the search strategy, it is likely to be those that are in lower profile journals. These tend to be smaller studies and are more likely to have negative findings—if you find interesting positive findings, chances are that youll try to get them published in a high profile journal. So, if negative findings are excluded, the studies you do include will bias the evidence towards a positive association, in this case you may be led to thinking (wrongly) that prisoners have a higher prevalence of epilepsy than the general population.

The authors of this report make reference to publication bias, which is a related issue. If researchers do not attempt to get negative studies published, or if journal editors are more likely to publish positive findings, the overall sense that a reader gleans from the literature is that a positive association is more likely to exist than is really true.

A British epidemiologist, Archie Cochrane, was one of the first to recognise health professionals need for access to reliable, systematic, up to date reviews to allow informed decisions about health care to be made. In 1993 the Cochrane Collaboration was founded. The collaboration publishes systematic reviews covering all areas of health care, in the Cochrane Database of Systematic Reviews. Further details of the activities of the Cochrane Collaboration can be found at www.cochrane.org. These reviews are central to the practice of evidence based medicine, allowing informed, scientifically based decisions to be made in the treatment of patients.

So, given that the authors have done a good systematic review, what did they do with the papers? The numbers of participants in each survey and the number with epilepsy were identified. This allowed calculation of the prevalence (simply the number with epilepsy divided by the total number) in each study. Because of the degree of uncertainty of how well this prevalence may reflect that of the entire prison population, 99% confidence intervals were calculated for each prevalence. These are plotted in the diagram, one row for each individual study. The size of the square representing the prevalence is proportional to the number of individuals in the population studied, so the largest study (the top one) has the biggest square. The width of the 99% confidence interval is shown by the horizontal lines for each study. The bigger the study, the less the degree of imprecision, so the narrower the confidence interval.

To summarise the results from all the studies, the numbers of people with epilepsy was divided by the total number of prisoners across all the studies (23/3, 111=0.7%). This is shown on the diagram as a diamond at the bottom of the figure. The vertical dotted line runs through the centre of this diamond (at 0.7%). The horizontal width of the diamond represents the 99% confidence interval for this combined prevalence. This is much smaller than any of the confidence interval in individual studies, because combining studies increases the precision with which we can estimate prevalence (or other measures of effect).

It is important to note that the method that the authors have used here is a pooled analysis, not a meta-analysis, although the “blob” diagram may be misleading, since these are usually used for meta-analyses. The difference is as follows. Pooled analyses use the original data for each person in each study, and a re-analysis is done, with each person getting the same “weight.” That is, no individual contributes more evidence to the overall result than any other individual. Meta-analyses, on the other hand, use the estimated result (in this case, prevalence) from each study. This means that in a meta-analysis, the unit of analysis is the study, not the individual. In meta-analyses, results are combined by using suitable statistical methods. More weight is given to larger studies, since their estimates are more precise than estimates derived from smaller studies.

In both pooled and meta-analyses, statistical tests to investigate differences in the results between the studies can be performed. The authors report the results of the so called test for heterogeneity as non-significant, P>0.1, which means that there is no evidence to show that the prevalence of epilepsy is significantly different across the different studies.

The authors summarise by noting that the estimated prevalence of epilepsy among prisoners (0.7%) is similar to the prevalence in the general population of young men (quoted as 1%), indicating that there is no good evidence to support a link between epilepsy and criminality. Although the systematic review was thorough and the analysis appropriate, this seems a strong conclusion given the fact that only 23 people across all the studies reported having epilepsy. An alternative study design may be a cohort study, to compare rates of criminality in people with and without epilepsy.