Imaging Wisely: An Introduction to the ACR Appropriateness Criteria and Analysis of its Impact on Internal Medicine Residents

From the 2018 HVPAA National Conference

Mike Cheng (University of Chicago), Andrea Magee (University of Chicago), Carina Yang (University of Chicago), Joyce Tang (University of Chicago)


Inappropriate radiological exam ordering is a large contributor to healthcare waste in the United States. The American College of Radiology (ACR) Appropriateness Criteria (AC) are designed to inform radiological exam ordering practices, however many general internists are unfamiliar with it.


To address the underutilization of the ACR AC and promote evidence-based high-value ordering practices, we developed a curriculum introducing internal medicine residents to the ACR AC and analyzed its impact. Our curriculum is a joint effort between the University of Chicago Internal Medicine and Radiology Departments and has been integrated into the required ambulatory lecture series for internal medicine interns at the University of Chicago in the 2017-2018 academic year. To date, 23 interns have participated.


The two-lecture curriculum includes: (1) an introduction to the ACR AC, its website, and a related mobile app; (2) an assignment to apply the ACR AC to case vignettes through the Radiology-TEACHES online portal (an initiative of the ACR); (3) a group discussion about the vignettes; and (4) a radiologist-led session on the principles of high-value radiological exam ordering. We used pre- and post-intervention surveys to assess for (1) change in knowledge using vignettes from Radiology-TEACHES and (2) change in attitudes related to high-value radiological exam ordering and use of the ACR AC. We used McNemar’s test and paired t-test as appropriate to analyze data.


Pre-intervention, 35% of residents were unaware of the ACR AC and only 26% reported it among their top 3 resources guiding ordering decisions; 52% stated that attending physicians rarely if ever mention radiological exam ordering guidelines. Knowledge scores related to appropriate radiological exam ordering improved from a mean of 59% (SD 15%) correct answers on vignettes pre-intervention to 89% (SD 14%) post-intervention (p<0.0001). Use of ACR AC resources during this “open-book” knowledge assessment increased from 17% of residents pre-intervention to 74% post-intervention (p=0.0003). Post-intervention, residents were more likely to agree that inappropriate ordering of radiological exams is a problem within the field of internal medicine (78% pre vs. 96% post), felt more comfortable with their knowledge of cost of radiological exams (4% pre vs. 74% post) and with discussing cost with patients (9% pre vs. 61% post), and were more likely to value the ACR AC (65% pre vs. 96% post); P-values for all comparisons were <0.05. All residents found the curriculum helpful and nearly all residents thought it should be offered yearly; 91% agreed that they plan to change their ordering practices as a result of the curriculum.


This curriculum was well-received and led to a promising and significant positive shift in knowledge and attitudes towards high-value radiological exam ordering and use of the ACR AC. A large majority of participants believe the curriculum will change their ordering practices. Future assessments will evaluate the durability of these findings over time through follow up surveys and analyze change in ordering practices through electronic medical record chart review.

Implications for the Patient

This is among the few curricula targeting internal medicine residents at a large academic hospital and is unique as it is multidisciplinary and integrates Radiology-TEACHES software, which won the 2017 ABIM Creating Value Challenge. If these findings are durable, this easily-disseminated intervention could have high-value impact on ordering practices.

What are academic medical centers across the country doing to improve healthcare value?

Value improvement guides: Published reviews in JAMA Internal Medicine coauthored by experienced faculty from multiple leading medical centers, with safety outcomes data and an implementation blue print.

Review article detailing 25 labs to refine for high value quality improvement | July 2020

MAVEN campaign: Free 4 year high value care curriculum online.

Join the Alliance! Membership is free with institutional approval and commitment to improving value in your medical center.

Learn more about HVPA on Health Affairs Blog