Rigorous Evaluation in Quality Improvement: Decreasing Vitamin D Screening in an Academic Healthcare System

From the 2018 HVPAA National Conference

James Henderson (Michigan Medicine), Christopher Petrilli (Michigan Medicine), Jenna Keedy (Michigan Medicine), Emily Dibble (Michigan Medicine), Melissa Wei (Michigan Medicine), Julie Prussack (Michigan Medicine), Grant Greenberg (Lehigh Valley Health Network), Eve Kerr (Michigan Medicine)


In response to the Choosing Wisely recommendation to avoid population based screening for Vitamin D deficiency, a multidisciplinary committee at Michigan Medicine developed a consensus guideline for appropriate vitamin D screening that included a list of diagnoses for which it may be appropriate to order Vitamin D testing.


In addition to education efforts to socialize the guideline, the intervention consisted of a point-of-care electronic health record (EHR) alert that displayed the guideline when providers attempted to order vitamin D levels in patients without an appropriate diagnosis. Vitamin D ordering rates began declining around the time the initiative began but prior to educational efforts, which may be at least partially explained as a Hawthorne effect. We evaluated the effectiveness of the EHR alert against this underlying trend by quantifying the impact on vitamin D ordering using a methodologically rigorous interrupted time series approach to compare trends before and after its release. 


Study Design: In this single-center retrospective cohort analysis, we used an interrupted time series approach to assess the value added by the point-of-care EHR intervention beyond that predicted by the previous 12-month trend. We examined intervention effects on both the total number of Vitamin D orders and more specifically on inappropriate tests ordered in healthy individuals (low-value Vitamin D).

Population Studied: Data for this study includes all patients who visited an outpatient clinic within the Michigan Medicine health system between February 2015 and May 2017.  This consists of 254,352 patient encounters and 14,353 vitamin D tests.


For the 12 months after the decision to target Vitamin D testing, the rate of appropriate Vitamin D orders had a monthly percent reduction of 4% (95% CI, 3.9 – 5.0%), while low-value orders reduced by 8% per month (95% CI, 7.0 – 8.8%). In the four months following the EHR alert, the estimated monthly percent reductions were 9% (95% CI, 7.5 – 11.5%) for appropriate orders and 20% (95% CI, 16.2 – 23.9%) for low-value orders. The relative ratio of 0.91 (95% CI .862 – .971) indicates that on a rate-basis the EHR alert had a greater impact on low-value than appropriate orders.


Efforts to develop a consensus guideline around appropriate vitamin D testing and subsequent education were associated with a steady decline in both appropriate and low-value testing.  Following a point-of-care EHR alert, these declines accelerated and the decline in low-value orders accelerated more than the decline in appropriate orders.

Implications for the Patient

Efforts to assess the impact of interventions designed to reduce low-value care can be complicated by the presence of pre-existing trends. In such cases, interrupted time series analyses provide a rigorous assessment framework that allows decision-makers to assess the value and sustainability of intervention components.

What are academic medical centers across the country doing to improve healthcare value?

Value improvement guides: Published reviews in JAMA Internal Medicine coauthored by experienced faculty from multiple leading medical centers, with safety outcomes data and an implementation blue print.

Review article detailing 25 labs to refine for high value quality improvement | July 2020

MAVEN campaign: Free 4 year high value care curriculum online.

Join the Alliance! Membership is free with institutional approval and commitment to improving value in your medical center.

Learn more about HVPA on Health Affairs Blog