An EBM Primer
John Kellum, MD

Introduction

A variety of reactions are elicited from clinicians when the terms "Evidence-Based Medicine" (EBM) are introduced into a conversation. Most physicians seem to either a. reject it outright as a means for others, who are interested solely in costs, to regulate their practice; or b. dismiss it as something for "those other doctors" who don't know how to or don't want to read the literature. The focus of this monograph will be to dispel many of the myths surrounding EBM (either as an oppressive force or as a panacea) describing both it's potential and its limitations. Furthermore, I shall try to make the case that EBM cannot be successful without the participation, indeed the leadership, of the "front line" practitioners; and that armed with the tools of EBM these clinicians can compel policymakers to accept practice guidelines that protect patients and improve healthcare.

What Is EBM And Why Do We Need It?

There is nothing particularly new about EBM. The idea that the use of principles from epidemiology and biostatistics to improve the care of a given patient is the foundation of clinical epidemiology.1 This field can claim Syndenham, Osler and even Hippocrates as it's early heroes; however, the birth of modern decision analysis was midwifed largely by clinicians like David Sackett, Gordon Guyatt, Brian Haynes and many others. These clinicians sought to apply the basic science of clinical epidemiology to understand better how to care for individual patients. Like other basic sciences (pathology for example), clinical epidemiology has certain techniques which, once learned, can be applied across many fields of clinical science. Instead of the frozen section and the H&E stain, clinical epidemiology utilizes the odds ratio and posterior probability.

There can be little doubt that recent social upheaval in medicine has fueled interest in EBM. Right or wrong, there is belief in medical economics that there exists a resource imbalance. Throughout the last three decades, hospitals have steadily increased charges to patients in order to maintain revenues. The 1990's has then seen a new strategy--the reduction of services. However, the emphasis in the future, and already a growing force, will be on increasing efficency.(2-3) This latter goal will require the sophisticated tools available though EBM. In this way, EBM can be seen as the outgrowth of the "total quality improvement" (TQI) initiatives of this decade. However, EBM is really for clinicians not hospital administrators. Physicians caring for patients remain the target audience for most reviews and editorials written today and clinicians continue to be in control of the vast majority of clinical practice. Unfortunately, this traditional arrangement is being eroded not only by the demands of hospitals, governments and third party payers but also by the sheer amount of information now available to all parties.

Today, there are over 20,000 medical journals and together they publish over two million articles annually. Even if we focus narrowly there is simply too much information available. In 1992 alone, the British Medical Journal and the New England Journal of Medicine together published over 1100 articles; 4400 pages of text. Stacked one on top of the other, their height would exceed 1500 feet!4 Obviously, even the most avid reader cannot possibly keep abreast of every study in his or her field. Accordingly, EBM is the process of finding relevant information in the medical literature to address a specific clinical question. EBM is also the application of the rules of epidemiology and biostatistics to this body of information in order to distinguish evidence from opinion.

EBM Methodology

The majority of methods used in EBM are those used by the fields of epidemiology and biostatistics; and as such are beyond the scope of this discussion. However, certain techniques have been developed specifically for this purpose. Two such techniques are "critical appraisal" and "systematic review". A critically appraised topic or CAT is the back bone of EBM. The purpose of a CAT is to evaluate a given study (or set of studies) using a standardized approach. Studies which address diagnosis, prognosis, etiology, therapy, and cost-effectiveness all have a separate CAT format.1 An example is shown in Table 1 for studies which address therapy. The CAT for these studies asks two questions: a. are the results valid? and b. if so are they clinically useful? Studies which fail to achieve these measures are not generally useful although studies do not necessarily have to fulfill every criteria, depending on the nature of the topic. For example, a study that compared the effectiveness of aspirin vs. placebo for the prevention of stroke would not be expected to include a detailed examination of side effects or a cost:benefit analysis; however a study comparing TPA vs. streptokinase for treatment of stroke would. Alternatively, either study would be "fatally flawed" if it failed in terms of randomization or was not analyzed as "intention to treat".

Table 1. Critical Appraisal Of the Literature

Are the results of the study valid?

• correctly randomized?
• were all the patients accounted for?
• was follow-up complete?
• were patients analyzed according to how they were randomized (i.e. intention to treat)?
• were all people involved in the study blinded?
• were the groups similar at the start?
• were the groups treated equally apart from the experimental intervention?

• how large was the treatment effect?
• how precise was the estimate of the treatment effect?
• are the patients similar to the "norm"?
• were all clinically important outcomes considered?
• was a cost:benefit analysis performed?

Adapted from Sackett et al. 19911

Clinicians often correctly point out that there are both studies for and against many of the treatments they prescribe. A familiar refrain of the busy clinician goes something like this:

"I can show you six articles, three for and three against;
however, in my experience it almost always helps"

Such statements attempt to invalidate the entire body of research in a given area simply because the studies show conflicting results. The conclusion is that there is no "proof" of anything and therefore the clinician's experience is elevated to level above science. While this may be appropriate in many cases particularly when rare diseases are present or when combinations of conditions exist, it is more often the case that when "six articles" exist, science can guide us much more than experience. A striking example of this appears in Figure 1. Although some of the studies are positive and other are negative; a definitive result occurs when the data are pooled. A similar result would have occurred if all of the positive studies were methodologically inferior to the negative ones. Both methods are valid ways of combining the information from multiple trials. The latter requires that some studies can be excluded because of design flaws, while the former requires that the results be pooled in some way.

Figure 1. Cumulative results and number of patients enrolled in trials of intravenous streptokinase for acute myocardial infarction. If a meta-analysis had been done at each date shown, the 95% confidence limits for the odds ratios for effect on mortality would be as shown below each date. Hence, as early as 1973, a positive effect could have been demonstrated, p < 0.05, almost 20 years before the agent came into clinical use. Adapted from Mulrow 4 and other sources.

Combining the results of multiple studies by way of a systematic review is the second major methodology of EBM. A systematic review can be done either by rating individual studies using "levels of evidence" (Table 2) or by pooling data from several, smaller, though high quality, studies. The latter approach is often called a meta-analysis.

Table 2. Levels of Evidence

• Level I
  randomized trials with low false positive (alpha) and low false negative (beta) error (i.e. high power)
• Level II
  randomized trials with high a error or low power
• Level III
  non-randomized concurrent cohort studies
• Level IV
  non-randomized historic cohort studies n Level V case series

Adapted from Cook et al. 1992 (10)

Limitations of EBM

A number of potential barriers to EBM methodology can exist. Not unlike the experience with combining other fields of science, clinical medicine is not easily nor happily married with epidemiology/biostatistics. Many clinicians become physically ill at the mere mention of an incidence density or an analysis of variance. Many more simply lack sufficient experience with these basic sciences to efficiently use them. Furthermore a basic problem in medical training is that physicians are educated as independent thinkers.

A complementary problem exists when epidemiologists, biostatisticians or medical economists try to interpret the medical literature. Studies may be valid from a statistical and design standpoint but still fatally flawed from a medical perspective. This may occur when inappropriate outcome measures are used. For example, the rate of resolution of infection might be an appropriate outcome for studies comparing two types of antibiotics in patients in widely different conditions; however, mortality rates would not appropriate for comparison. Similarly, subtle nuances in the field may be important in understanding what studies can and cannot be combined. For example, some studies on the use of enteral antibiotics for selective decontamination of the GI tract have cultured the stool to assure that virulent organisms were, in fact, suppressed while others have not.6 Pooling the data from all these studies might lead to serious errors in interpretation. Another example would be the evaluation of outcomes in patients with bacteremia without consideration of the source. Finally, even well done studies from both the medical and statistical viewpoints can still be misinterpretted.7-8

Examples of EBM

The literature contains several excellent examples of systematic reviews, many of which have direct relevance to the practice of critical care. For example, The Roundtable Conference of the European Society of Intensive Care Medicine9 used this approach to review treatment options for patients with sepsis. Prior to this, the American College of Chest Physicians use a similar method to develop and justify recommendations for its Antithrombotic Consensus Conference.10 By way of further example, let us consider the use of this approach to review the use of diuretics in the prevention of acute tubular necrosis.(11)

Radiocontrast

Although several forms of therapy have been proposed to treat radiocontrast induced ATN including: saline, furosemide, mannitol, calcium channel blockers, dopamine, atrial natriuretic peptide and theophylline,12 placebo controlled trials testing the effectiveness of any of these therapies are lacking. Virtually all studies have used hydration (usually with 0.45% saline) in addition to the agent being tested and most authors recommend its use. Even then, little comparative data exist for these potential treatments. The exception is a recent study by Solomon and coworkers13 which compared furosemide plus saline to, mannitol plus saline to saline alone; again, 0.45% saline was used. This randomized trial in 78 "high risk" patients found that both diuretic regimens were less effective in preventing ATN than saline alone. Another study by Weinstein et al, found that renal function significantly deteriorated in patients pretreated with furosemide.14 Neither study can actually tell us whether saline was efficacious in preventing radiocontrast induced ATN although comparison to historical benchmarks suggests that the practice is useful. Accordingly, the recommendation to avoid diuretics is based on "Level I" evidence (Table 2); while the recommendation to use 0.45% saline is supported by only "Level IV" evidence. A single, randomized study of sufficient statistical power, showing that saline was not helpful would be that is required to drop this recommendation. Similarly, if the intervention was associated with any morbidity itself it could not be justified on the bases of the available evidence.

Vascular Surgery

In the case of aortic cross-clamping in the repair of abdominal aortic aneurysms, the use of loop diuretics has become routine in many institutions. Nonetheless the evidence in support of this approach is limited. The only controlled study available is by Beall et al. in 1963.(15) This study compared the outcomes of 30 patients who underwent elective abdominal aortic aneurysm repair. Patients were randomized to receive either no preoperative fluid, IV hydration only or IV hydration plus mannitol as required to keep urine output > 60 ml/min. There was no change in renal function or postoperative urine output between the latter two groups. Although this negative study was certainly under powered, it remains the only controlled trial of diuretics in vascular surgery to date. The following year, Powers et al. reported on the outcomes of 104 patients treated with mannitol.16 This uncontrolled study reported that all patients had an increase in urine output and none developed ATN. No recent studies have been done to address this issue and it is unlikely that any will. Hence, the use of mannitol can only be recommended on the basis of Level V evidence and loop diuretics cannot be recommended at all. Table 3 list these recommendation as well as those for the use of loop diuretics to treat ATN.

The use of the systematic review has enormous advantages over the traditional method of "expert review"; Figure 2 illustrates this point.(4) By 1988, 15 studies had been conducted to evaluate the use of prophylactic lidocaine for acute myocardial infarction. While no single study was definitive, pooled data from the nearly 9000 patients conclusively that the practice was useless at best. That same year, recommendations for its use still appeared widely in textbooks and review articles; and CCU house officers (including the author) were still being taught to use it.

Figure 2. Numbers of patients enrolled in randomized controlled trials of prophylactic lidocaine for acute myocardial infarction vs the recommendations of textbooks or review articles on the subject. Even by 1990 after published meta-analyses cleared showed no benefit, there were more recommendations for than against its use. Adapted from Mulrow 4 and other sources.

Conclusion

Clinicians are required to effectively interpret the evidence in their fields. It is much easier to teach a physician how to use the tools of EBM than to teach a non-physician medicine. With clinician leadership, efforts to improve the practice of medicine can succeed and not just from a financial standpoint. EBM is not a panacea; it is only one part of clinical decision making. Experience and consensus will still have a role in modern decision analysis. EBM is no more or less important than the others. Although, it is by definition more objective. Is EBM a tool of clinicians or a leash held by hospital administrators, insurance companies and government bureaucracies? It is likely to be both; however, in the hands of the clinician it can be much more powerful and accurate.

References

1. Clinical Epidemiology: A Basic Science for Clinical Medicine. Sackett DL,

Haynes RB, Tugwell P. Little Brown and Company, Boston, 1985.

2. Berwick DM. Continuous improvement as an ideal in healthcare. N Eng J Med 1989;320:53-56.

3. Ellwood P. Special Report. Shattuck Lecture. Outcomes Management.

N Eng J Med 1988;320:53-56.

4. Mulrow C. Rationale for Systemic Reviews. In Chalmers I, Altman DG (ed) Systematic Reviews , BMJ Publishing Corp, London 1995. Chapter 1, pp 1-8

5. Evidence Based Medicine Working Group. Evidence based Medicine: A new approach to teaching the practice of medicine. JAMA 1992; 268 (17): 2420-2425.

6. Stoutenbeek CP, van Saene HKF. Selective Decontamination of the Digestive Tract. In Pinsky MR, Dhainaut J.F., Artigas A (ed) The Splanchnic Circulation: No longer a silent partner . Springer-Verlag , Berlin 1995; pp 165-174.

7. Kellum JA. Increased Oxygen Delivery for High Risk Surgery. ACP Journal Club, 1994; Nov/Dec, p84.

8. Kellum JA. Increased Oxygen Delivery for High-Risk Surgery: Part II. ACP Journal Club, 1995; Mar/Apr, p 54

9. Sibald WJ, Vincent JL. Roundtable Conference on clinical trials for the treatment of sepsis. Crit Care Med 1995; 23: 394-399.

10. Cook DJ, Guyatt GH, Laupacis A, Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Antithrombotic Therapy Consensus Conference. Chest 1992;102:305-11S.

11. Kellum JA. Diuretics in Acute Renal Failure: Protective or Deleterious. Blood Purif 15: 1997 in press.

12. Barrett BJ, Parfrey PS. Prevention of nephrotoxicity induced by radiocontrast agents. N Engl J Med 1994; 331:1449-50.

13. Solomon R, Werner C, Mann D, D'Elia J, Silva P. Effects of saline, mannitol, and furosemide to prevent acute decreases in renal function induced by radiocontrast agents. N Engl J Med 1994; 331:1416-20.

14. Weinstein JM, Heyman S, Brezis M. Potential deleterious effect of furosemide in radiocontrast nephropathy. Nephron 1992;62:413-5.

15. Beall AC, Holman MR, Morris GC, De Bakey M. Mannitol induced osmotic diuresis during vascular surgery. Renal haemodynamic effects. Arch Surg 1963; 86:34-42.

16. Powers SR, Boba A, Hostnik W, Stein A. Prevention of postoperative acute renal failure with mannitol in 100 cases. Surgery 1964; 35:15-23.