Jonathan Marks

Harvard College ‘15

S. Andrew Spooner MD FAAP

Department of Pediatrics, Cincinnati Children’s Hospital Medical Center

Abstract

Background: Computerized Provider Order Entry (CPOE) systems provide clinical decision support (CDS) by alerting care providers when an ordered drug’s dosing does not fall within the range determined by pre-specified dosing rules. Redefining dosing range rules to decrease false-positive alerts currently requires manual identification of incorrect rules.

Objectives: To develop a metric that would identify drugs or classes of drugs with clinically aberrant dosing ranges as potential targets for manual correction and to build an automatic system that empirically modifies dosing rules based on prescriber ordering patterns.

Method: We performed a retrospective analysis of 116,117 distinct drug dosing alerts generated by 521,793 orders placed between June 1st and September 30th, 2010 at Cincinnati Children’s Hospital. We identified the worst-offending dosing rules by comparing relative alert rates and then developed an algorithm to redefine dosing range rules empirically based on the cumulative distribution function (CDF) of dose distributions.

Results: The alert rate of different medications varies greatly with a few excessive outliers that represent clinically-aberrant dosing range rules. There was a clear relationship between the adjusted relative alert rate and the accuracy of the pre-specified dosing ranges. The CDF algorithm produces similar results to manual intervention and reduced the alert rate of the system by 62.51% (0.1394 to 0.0523 average alerts per order).

Conclusions: Automatically determining the worst-offending drugs is more efficient than manual error correction and produces comparable results. Basing dose alert thresholds on empirically prescribed data significantly decreases the alert rate. However, the safety and efficacy of automated dose-defining algorithms have yet to be determined in a clinical setting.

Introduction

Medication errors due to improper drug ordering are both common (Corrigan et sal., 2001 and Bates et al., 1995) and usually preventable (Paton and Wallace, 1997), especially in pediatrics (Jonville et al., 1991). Preventable medication errors and subsequent adverse drug events (ADEs), which are broadly defined as medication-derived injuries (Bates et al., 1996), result in 5-8% of hospital admissions (Beijer and de Blaey, 2002) and have been conservatively estimated to harm 1.5 million patients and cost US$3.5 billion dollars a year (Paton and Wallace, 1997). Children are particularly at risk for medication errors due to pharmacokinetic or pharmacodynamic differences from adults, lower body weights, weight-based dosing, and other factors (Sard et al., 2008 and Kozer et al., 2002). The Institute of Medicine (Kohn et al., 2000), the National Academies of Sciences (Aspden et al., 2007), and the British Department of Health (Smith, 2004), among others, have suggested that medication errors could be reduced through greater implementation of electronic health records and electronic prescription systems.

Computerized Provider Order Entry (CPOE) systems allow care providers to enter prescription information directly into an electronic health record (EHR) while simultaneously sending the order to a pharmacy. As 56-79% of medication errors occur during physician prescribing (Bates et al., 1995 and Kaushal et al., 2001), reducing error during order entry can play a large role in decreasing the frequency of medication error and ADEs. CPOE systems currently reduce error in a variety of ways: by eliminating handwriting errors (Ammenwerth, 2008), by suggesting default doses, which reduces the occurrence of error in commonly-prescribed medications (Gerstle et al., 2007), by largely eliminating calculation errors through automatic dose/weight and dose frequency calculations, and by providing clinical decision support (CDS).

Clinical decision support (CDS) systems provide an extension to CPOE systems that alerts care providers to potential sources of error at the time of ordering. Several types of CDS have been shown to improve patient safety and reduce potential sources of error. These include automatic drug-drug interaction (DDI) checking (Paterno et al., 2009, and Strom et al., 2010), drug-allergy reaction notification (Kuperman et al., 2003), replication of medication alerts, and drug-food interaction checking. However, limited data exists on dose-checking, which alerts providers when the dose for an ordered medication does not fall within pre-specified limits. In pediatrics, especially when compared to adult care, dose-checking is more important than DDI checking. In general, children are on fewer medications and may be more susceptible to harm from dosing errors due to lower body weights (Kozer et al., 2002) and decreased pharmacological tolerance (Sard et al., 2008).

In order to realize the error-reducing qualities of CDS, care providers must see and react to the alerts that are triggered upon order entry. However, several studies (Isaac et al., 2009 and Beccaro et al., 2010) have shown that providers ignore most alerts in both outpatient and inpatient settings. Isaac et al. found DDI checking alert override rates of 90.8% in a study of 2872 ambulatory providers. In a different study of five academic primary care practices, clinicians overrode 91.2% of drug-allergy alerts, and the authors concluded that 36.5% of the alerts were not clinically useful and had no scientific basis (Weingart et al., 2003). Similar rates were seen in Cincinnati Children’s Hospital Medical Setting (CCHMC), where physicians overrode 90.27% (±.004, 95% CI) of dose range checking alerts during the duration of the study with an average of 1.2295 (±.0013, 95% CI) alerts fired per order. Such low alert compliance rates suggest that CDS alerts are generally ineffective and have a high false positive rate, which has led to “alert fatigue” and low confidence in the alert system. Accordingly, clinicians currently forfeit the potential benefit of CDS by ignoring a large proportion of its suggestions.

Manually reducing the number of false-positive drug dosing alerts by individually modifying dose rule thresholds has been shown (Weingart et al., 2003 and Shah et al., 2006) to improve alert acceptance rates due to lower false-positive alert rates. However, this process currently requires a considerable time investment by physicians or pharmacists to determine which dosing rules should be corrected and to determine the proper alert thresholds. The objective of this study was to develop a metric to identify drugs or classes of drugs with clinically-aberrant dosing ranges as potential targets for manual correction. A secondary objective was to determine the feasibility of creating an automatic electronic system that empirically modifies dosing rules based on prescriber ordering patterns.

Methods

Study setting

This study was performed at Cincinnati Children’s Hospital Medical Center (CCHMC), a 523-bed tertiary care academic medical center. CCHMC is a level-1 trauma center comprised of 1,498 active medical staff and 936 faculty members in 2010 with a total of 1,078,798 patient encounters. This study specifically considered dose range checking alerts that fired upon order entry in CCHMC’s CPOE implementation (Epic Summer ’09).

CPOE/EHR implementation

In 2007 CCHMC embarked upon a complete implementation of the Epic EHR (currently using Epic Summer ’09), including administrative systems (scheduling, billing, etc.). Ambulatory clinics went live in phases from 2007 to 2012. Inpatient and perioperative areas went live in January 2010. All prescriptions and drug orders are entered into Epic in all areas where the system is live. The system provides prescribing decision support via Medi-Span (Wolters Kluwers Healthcare). Since CCHMC converted from Siemens Corporation’s CPOE system upon GoLive, the custom rules from that system were applied to Epic, overriding the Medi-Span rules for those dosing subpopulations. Dose range checking alerts are triggered when an order is signed. It is important to note that the system provides suggested dosing levels and occasional inline dosing alerts (e.g. hard stops for excessively high doses). Pharmacists review all orders and see all alerts. Certain Medi-Span rules are filtered for providers but still appear for pharmacists.

Data Collection and Analysis

We performed a retrospective analysis of 116,117 distinct drug dosing alerts generated by 521,793 orders placed between June 1st and September 30th, 2010 in both ambulatory and inpatient, and perioperative settings. Our data set included information on 69,399 patient records and 105,928 dose-range checking rules derived from the Medi-Span CDS system.

Data analysis was performed using Oracle SQL Developer TNS-linked to Oracle Server version 11g. Anonymized patient and order data was derived from a set of Clarity tables that link into Epic. We organized the data into a set of five tables (Patient demographics, encounter data, diagnosis data [derived from problem lists], dose range checking alert data, medication ordering data). We exported the set of Medi-Span dose range checking rules from Wolter Kluwer’s Drug Dosing and Administration Database (DDAD) and, following conversion of this data using string-parsing functions to facilitate comparisons with the data from Clarity, also imported it into the Oracle server for further analysis.

Following data collection, we filtered both alerts and orders to only select dose range checking alerts that fired upon order entry (Figure 1). Following data filtration, 51,478 alerts and 609,650 orders remained for further analysis.

Figure 1. Alert and order filtering process.
Figure 1. Alert and order filtering process.

Limiting the scope of this study to dose range checking alerts reduced both the amount of data and the scope of a variety of data integrity issues related to natural language processing of provider-entered freetext, missing data on patient vitals such as weight or height, and mismatched medication identifiers between the Clarity and Medi-Span data. However, a variety of integrity issues still remained. Current implementations of CPOE/CDS systems are largely incompatible (Spooner and Classen, 2009; Linder et al., 2007) because different systems use varying drug and patient classification schemes as well as using disparate data storage methodologies. While we were able to overcome file type incompatibility by exporting/importing raw data and through similar strategies, transforming the data to force cross-system data compatibility was effort-intensive and likely increased error (Walker et al., 2005). CPOE/CDS developers should continue to make cross-platform compatibility a priority as the current state of incompatibility greatly exacerbated the data integrity challenges faced by this study.

Statistical analysis

Statistical analysis was performed in the open-source statistics program R (version 2.13.0) supplemented with the CRAN packages RODBC, car, lattice, and qcc. Nonparametric kernel density evaluation was generated using a Gaussian kernel on a bandwidth calculated by Silverman’s “rule of thumb” (min{SD,IQR})/(1.34*n^(-1⁄5)) (Silverman, 1986). Multivariate weighted regression analysis was performed using the generalized linear model (Nelder and Wedderburn, 1972) and was weighted on the number of orders per distinct rule ID. All P values are two-tailed and P < 0.05 was considered significant.

RAR and CDF algorithm development

Human evaluation of dosing distributions easily identifies medications or classes of medications that have aberrant dose range checking rules. These distributions often fall into several types: unimodal dosing distributions where the rules capture the center of the distribution but are too conservative in either direction, unimodal distributions where the pre-specified dose range does not capture the center of the distribution, and multimodal distributions where the dosing rules do not account for the complexity of the dosing distribution (Figure 2). However, algorithmic interpretation of these phenomena is difficult because such evaluation is highly qualitative; instead, automatic categorization of successful dose range checking rules requires a quantitative metric.

Figure 2. Examples of dosing distributions. Both Ursodiol 300mg capsules and Acetaminophen 325mg tablets have unimodal dose distributions. The alert thresholds for Ursodiol do not capture the center of the distribution and therefore do not represent typical ordering patterns. The alert thresholds for Acetaminophen demonstrate a different relationship between dose thresholds and provider ac- tivity. While the thresholds do capture the center of the distribution and are therefore more accurate than the thresholds for Ursodiol, the underdose alert threshold remains conservative and generates a large proportion of the alerts for that dose range checking rule.
Figure 2. Examples of dosing distributions. Both Ursodiol 300mg capsules and Acetaminophen 325mg tablets have unimodal dose distributions. The alert thresholds for Ursodiol do not capture the center of the distribution and therefore do not represent typical ordering patterns. The alert thresholds for Acetaminophen demonstrate a different relationship between dose thresholds and provider ac- tivity. While the thresholds do capture the center of the distribution and are therefore more accurate than the thresholds for Ursodiol, the underdose alert threshold remains conservative and generates a large proportion of the alerts for that dose range checking rule.

To develop this metric, we hypothesized that the degree of incompatibility between the pre-specified dose range rules and the empiric ordering patterns could be explained by the alert rate (the number of visible alerts per attempted order) per medication rule. However, because the Epic EHR system does not track attempted orders but rather only successful orders, and because the system records both filtered and viewed alerts, we adjusted the counts of alerts and orders to more accurately reflect the actual relative alert rate per medication rule (Equation 1). Additionally, Medi-Span dosing rules are based on several different factors: patient age, patient weight, administration route, encounter diagnosis, and creatinine clearance levels. We restricted alert and order populations to fit the rule (e.g. by selecting alerts/orders for patients under the age of 12, weighing between 25 and 50 kg, and having diabetes) and calculated relative alert rates (RAR) per medication rule, per medication irrespective of rules, and per pharmaceutical and therapeutic classes.

After we developed the RAR metric to determine which dose range checking rules did not reflect empiric provider ordering patterns, we developed an algorithm to redefine the alert thresholds automatically without manual intervention. The low alert acceptance rates seen both at CCHMC and other institutions show that current CDS systems are both flawed and largely ignored regardless of alert severity (Isaac et al., 2009; Beccaro et al., 2010; Weingart et al., 2003). Additionally, previous attempts to manually “fine-tune” dosing thresholds based on “standard dosing” without addressing provider prescribing patterns failed to greatly reduce alert acceptance rates (Stutman et al., 2007). Therefore, instead of basing alert threshold values on the pre-determined values from Medi-Span, we hypothesized that assigning alert thresholds based solely on provider activity would generate clinically reasonable dose ranges, reducing both the alert rate and the false-positive alert rate, and could eventually improve alert acceptance rates.

In developing this automatic algorithm, we assumed that provider-prescribing patterns generally represent recommended dosing strategies far better than most CDS systems. Accordingly, any order that results in a medication error or an ADE should fall outside of the normal dosing ranges of a particular medication in a specified weight class, age class, or other division. By basing dose alert thresholds on dosing distributions, one can visually find an “inflection point” in the distribution where the number of orders decreases rapidly, marking the highest/lowest typical dosing amount. For example, at ~16 mg/kg of acetaminophen in Figure 2, the dosing distribution density falls rapidly, which shows that providers place few orders above that dosing level. Attempting to place an order with a large deviation from this value might suggest a potential order error and should result in a dose-range checking alert.

To find this point quantitatively, we calculated the probability density function (pdf) of the dosing distribution and then used this data to create a cumulative distribution (cdf) of the dosing distribution (Equation 2, 3). The result of this function gives the percent of orders that would trigger an alert if the alert threshold were placed at a given level. After selecting a pre-specified alert rate (UEmax and LEmax for overdose and underdose error rates, respectively), the algorithm calculates overdose and underdose alert thresholds (amax and bmax). For the greatest accuracy, UEmax + LEmax should be equal to the total level of prescribing error in the system (the actual rate of medication error). As determining the population rate of medication error requires extensive chart review, for the purpose of this study we used the estimate of 5.7 medications errors per order found by Kaushal et al. Bates et al. (1995) found a similar rate of 5.3 errors per 100 orders in a separate study of adult inpatients, which provides additional support that population rates of medication errors are approximately 5 errors per 100 orders at major healthcare centers. Once the algorithm calculated amax and bmax using the estimate of 5.7 errors per 100 orders, we calculated what the relative alert rate would have been if the alert thresholds had been set to the calculated values. In effect, instead of assigning pre-specified alert thresholds for all medications and alerting doctors when their orders exceed these limits, the CDF algorithm dynamically changes based on prescribing patterns and alerts doctors when their orders fall above or below a specified percentile of the orders in the system.

Equations 1-3
Equations 1-3

Results

Improving relative alert rates is feasible

Alert rates and override rates vary greatly both among different medications and among different pharmaceutical classes. While most medications had total alert rates (daily and single dose range checking alerts) between 3 and 20 alerts per 100 orders (RAR IQR 0.033 – 0.201), the distribution of relative alert rates is strongly right-skewed with a few excessive outliers (Figure 3a). The few extreme outliers (where the relative alert rate is higher than 0.1815 alerts per order) suggest potentially aberrant dose range checking rules and account for 20,125 of the 49,043 alerts considered in this study (41.04%), which suggests that improving a subset of dose range checking rules could have a large impact on the alert rate for the entire system. The results for the subset of medications with high relative alert rates are summarized in Table 1.

Figure 3a. Distribution of relative alert rates per medication. A non-parametric kernel density distribution of relative alert rates for the top 50% of medications by order volume (317 medications, 96.4% of all alerts). The distribution is quite right-skewed for single, daily, and total orders relative alert rates. 55 medications are outliers with alert rates exceeding .1815. Where RAR > 1, an average of more than one alert fired per order (for example, both single dose and daily dose alerts can fire simultaneously).
Figure 3a. Distribution of relative alert rates per medication. A non-parametric kernel density distribution of relative alert rates for the top 50% of medications by order volume (317 medications, 96.4% of all alerts). The distribution is quite right-skewed for single, daily, and total orders relative alert rates. 55 medications are outliers with alert rates exceeding .1815. Where RAR > 1, an average of more than one alert fired per order (for example, both single dose and daily dose alerts can fire simultaneously).

Table 1. Relative alert rates per medication summary (selected medications ordered by total relative alert rate description).

Medication Name No. Discrete Alerts No. Orders Relative Alert Rates
Single-Dose Daily-Dose Total Single-Dose (CI) Daily-Dose (CI) Total (CI)
Ursodiol300mg PO Capsules 250 210 294 186 1.344 (±.098) 1.193 (±.071) 1.581 (±.138)
Polymyxin B-Trimethoprim10000-0.1 Unit/mL 46 523 530 559 .0823 (±.023) 2.1612 (±.2) .9481 (±.018)
Chlorothiazide250mg/5mL PO Suspension 16 234 238 300 .0533 (±.025) .8211 (±.045) .7933 (±.046)
Sulfamethoxazole-Trimethoprim200-40mg/5mL 1264 1987 2185 2770 .4563 (±.019) .9562 (±.009) .7888 (±.015)
Clindamycin Palmitate HCl75mg/5mL POw 1049 874 1060 1353 .7753 (±.022) .76 (±.025) .7834 (±.022)
Tacrolimus1mg PO Capsules 302 29 302 476 .6345 (±.043) .0653 (±.023) .6345 (±.043)
Amoxicillin-Pot Clavulanate400-57mg/5mL PO 663 548 682 1084 .6116 (±.029) .6524 (±.032) .6292 (±.029)
Penicillin V Potassium250mg/5mL PO Solution 57 22 76 124 .4597 (±.088) .1913 (±.072) .6129 (±.086)
Methadone HCl1mg/mL IV Solution 227 0 227 377 .6021 (±.049) 0 (±0) .6021 (±.049)
Furosemide10mg/mL Solution 55 67 121 2479 .0222 (±.006) .0959 (±.022) .0488 (±.008)
Ondansetron HCl2mg/mL Solution 13 14 19 471 .0276 (±.015) .068 (±.034) .0403 (±.018)
Ranitidine HCl150mg PO Tablets 4 9 11 533 .0075 (±.007) .0192 (±.012) .0206 (±.012)
Amoxicillin875mg PO Tablets 2 3 5 268 .0075 (±.01) .0119 (±.013) .0187 (±.016)
Clindamycin HCl300mg PO Capsules 8 9 10 760 .0105 (±.007) .0134 (±.009) .0132 (±.008)
Diazepam5mg PO Tablets 2 1 3 282 .0071 (±.01) .0042 (±.008) .0106 (±.012)
Epoetin Alfa10000 Unit/mL Solution 5 2 5 541 .0092 (±.008) .1176 (±.153) .0092 (±.008)
Melatonin3mg PO Tablets 5 5 5 1014 .0049 (±.004) .0064 (±.006) .0049 (±.004)
Albuterol Sulfate(5mg/mL) 0.5% 1 21 22 5046 .0002 (±0) .015 (±.006) .0044 (±.002)

The relative alert rate and total alert count per pharmaceutical class and per therapeutic class exhibit a similar right-skewed distribution, with a few classes accounting for a large proportion of the total dose alerts for the system (Table 2). By improving only four pharmaceutical classes or a single therapeutic class (analgesics and anesthetics), one could address approximately 50% of all the alerts in this study (51.27% and 47.12%, respectively) (Figure 3b). This again shows the feasibility of improving alert rates across the entire system. Additionally, while one might expect certain classes, such as neuromuscular blockers, to have a high relative alert rate due to the severity of misusing these types of medications, other lower-severity classes such as anti-infective agents or non-narcotic analgesics also exhibit high relative alert rates that again suggest aberrant dosing rules.

Figure 3b. Pareto chart of alerts per therapeutic class. Most alerts are derived from a small number of pharmaceutical and therapeutic classes, which shows that improving the rules of certain classes can have a large impact on the system as a whole. By improving only five therapeutic classes one could account for more than 75% of all alerts system wide.
Figure 3b. Pareto chart of alerts per therapeutic class.
Most alerts are derived from a small number of pharmaceutical and therapeutic classes, which shows that improving the rules of certain classes can have a large impact on the system as a whole. By improving only five therapeutic classes one could account for more than 75% of all alerts system wide.

Table 2. Relative alert rates per pharmaceutical class summary (20 classes with the highest number of alerts).

Class Name No. Distinct Single-Dose Alerts (% of total) No. Single-Dose Orders (% of total) Relative Alert Rate Total (CI)
Analgesics-narcotic 8706 (25.87%) 46322 (6.73%) .1879 (±.0036)
Analgesics-nonnarcotic 4680 (13.91%) 46403 (6.74%) .1009 (±.0027)
Misc. antiinfectives 3867 (11.49%) 16233 (2.36%) .2382 (±.0066)
Anticonvulsant 2620 (7.79%) 15188 (2.21%) .1725 (±.006)
Antiasthmatic 2187 (6.5%) 39988 (5.81%) .0547 (±.0022)
Penicillins 1837 (5.46%) 12720 (1.85%) .1444 (±.0061)
Antihistamines 1709 (5.08%) 20211 (2.94%) .0846 (±.0038)
Laxatives 1554 (4.62%) 11750 (1.71%) .1323 (±.0061)
Antianxiety agents 1341 (3.99%) 7801 (1.13%) .1719 (±.0084)
Hypnotics 1240 (3.69%) 12161 (1.77%) .102 (±.0054)
Local anesthetics-parenteral 1064 (3.16%) 20771 (3.02%) .0512 (±.003)
Corticosteroids 1057 (3.14%) 17905 (2.6%) .059 (±.0035)
Ulcer drugs 1008 (3%) 10849 (1.58%) .0929 (±.0055)
Anti-rheumatic 1006 (2.99%) 32215 (4.68%) .0312 (±.0019)
Assorted Classes 939 (2.79%) 3418 (.5%) .2747 (±.015)
Ophthalmic 901 (2.68%) 10758 (1.56%) .0838 (±.0052)
Stimulants 721 (2.14%) 11377 (1.65%) .0634 (±.0045)
Gastrointestinal Agents 708 (2.1%) 4259 (.62%) .1662 (±.0112)
Cephalosporins 670 (1.99%) 14316 (2.08%) .0468 (±.0035)
Antidepressants 653 (1.94%) 7245 (1.05%) .0901 (±.0066)

The RAR metric correctly identifies incorrect dose range checking rules

The objective of the RAR metric is to quantitatively identify dose range checking rules that do not accurately represent empiric provider prescribing patterns. As demonstrated in Figure 2, manual evaluation of dosing distributions easily identifies aberrant dose range checking rules but algorithmic interpretation of such qualitative data is difficult. Manual evaluation of the dosing distributions with high, low, and moderate relative alert rates shows a marked association between the relative alert rate and the accuracy of the pre-specified Medi-Span dose range checking rules (Figure 4). High-RAR dose range checking rules typically have alert thresholds that do not capture the peak of the dose distribution. This is the case for both Ursodiol and Hydromorphone HCl in Figure 4. Mid-range RAR dose range checking rules either have alert thresholds that are slightly too strict or have thresholds that fail to capture the complexity of a distribution. The upper threshold for acetaminophen is correct; the lower threshold could be decreased, which demonstrates a slightly strict rule that could be modified. However, the maximum alert threshold for Methylphenidate does not account for the peak of orders near 70 mg and therefore alerts providers whenever one orders in that range. Low-RAR dose range checking rules are generally too broad. For both Ranitidine HCl and Prednisolone Sodium Phosphate, the acceptable dosing range could be decreased to better capture atypical dosing levels.

Figure 4. Correlation between RAR and Medi-Span Alert threshold accuracy.The relative alert rate per medication is a good metric for determining the accuracy of its dose range checking rules. High-RAR dose range checking rules, such as those for Ursodiol and Hydromorphone in this selection, have dose thresholds that are too conservative, generate many alerts, and do not accurately represent typical dosing patterns. Low-RAR rules (Ranitidine HCl and Prednisolone Sodium Phosphate) have dose thresholds that are generally too liberal and do not alert providers frequently enough. Mid-range RAR dose range checking rules fall somewhere in the middle, with some thresholds (such as the maximum threshold for Acetaminophen) that accurately represent dosing patterns and with others that fail to capture the complexity of the dose distribution (i.e. the maximum dose threshold for Methylphenidate HCl does not account for doses at a 72mg level and therefore fires an alert for an unnecessarily large proportion of orders). NB: there is no minimum threshold for Methyphenidate HCl dosage in the Medi-Span system.
Figure 4. Correlation between RAR and Medi-Span Alert threshold accuracy.The relative alert rate per medication is a good metric for determining the accuracy of its dose range checking rules. High-RAR dose range checking rules, such as those for Ursodiol and Hydromorphone in this selection, have dose thresholds that are too conservative, generate many alerts, and do not accurately represent typical dosing patterns. Low-RAR rules (Ranitidine HCl and Prednisolone Sodium Phosphate) have dose thresholds that are generally too liberal and do not alert providers frequently enough. Mid-range RAR dose range checking rules fall somewhere in the middle, with some thresholds (such as the maximum threshold for Acetaminophen) that accurately represent dosing patterns and with others that fail to capture the complexity of the dose distribution (i.e. the maximum dose threshold for Methylphenidate HCl does not account for doses at a 72mg level and therefore fires an alert for an unnecessarily large proportion of orders). NB: there is no minimum threshold for Methyphenidate HCl dosage in the Medi-Span system.

The CDF algorithm accurately adjusts dosing alert thresholds

The CDF algorithm greatly and significantly decreases the RAR for medications with strict dose range checking rules and increases the RAR for medications whose alert thresholds were too liberal (Table 3). Following the application of the CDF algorithm, the expected number of alerts fired per order dropped by 62.51% (from 0.1394 to 0.0523, p < 0.001). In addition, the calculated values for the new alert thresholds are similar to those of manual intervention. For example, when considering solely provider prescribing activity for Ursodiol 300 mg PO for patients between the ages of 11 and 98 (Figure 4), a human observer would suggest that the dosing thresholds should fall at approximately 3 and 11 mg/kg to accurately capture the dosing distribution. In this case, the CDF algorithm suggested threshold values of 2.9667 – 10.2401 mg/kg, which closely mirrors the results of manual intervention. As another example, the calculated thresholds for Acetaminophen 325mg tablets (CDF: 6.2 – 15.96 mg/kg, Medi-Span: 10 – 15 mg/kg) are not exceedingly different than the Medi-Span values, which was expected due to the lower RAR for this dose range checking rule.

The percent difference between the original Medi-Span alert thresholds and the CDF calculated thresholds varies linearly with the relative alert rate for that alert threshold (the proportion of alerts fired by orders with doses above/below the over/underdose threshold) (Figure 5 and Table 4). For medications with excessively high relative alert rates, the CDF algorithm greatly changed the alert thresholds either by decreasing the minimum dose or by increasing the maximum dose. The alert thresholds for low-alert-rate medications also changed greatly as the CDF algorithm increased the minimum or decreased the maximum to better express the prescribing patterns of providers. In general, the greater the relative alert rate before threshold adjustment, the larger change the CDF algorithm makes in the thresholds. Therefore, the relative alert rate before adjustment predicts the amount of change in the thresholds after adjustment. This linear dependence shows that the CDF algorithm improves both excessively strict and liberal dose range checking rules and increases or decreases alert thresholds as expected.

Figure 5. Relative alert rate and CDF dependency. The individual relative alert rates for the alert thresholds vary linearly with the percent difference between the original Medi-Span threshold limit and the calculated CDF limit. As the RAR of the overdose thresholds increases, the amount of difference between the thresholds before and after adjustment also increases. The exact opposite trend is seen for underdose limits—as the alert rate increases, the percent difference between the Medi-Span and CDF limit becomes more negative. Both of these trends show that when alert rates are higher, the CDF algorithm changes the Medi- Span alert thresholds to a greater degree.
Figure 5. Relative alert rate and CDF dependency. The individual relative alert rates for the alert thresholds vary linearly with the percent difference between the original Medi-Span threshold limit and the calculated CDF limit. As the RAR of the overdose thresholds increases, the amount of difference between the thresholds before and after adjustment also increases. The exact opposite trend is seen for underdose limits—as the alert rate increases, the percent difference between the Medi-Span and CDF limit becomes more negative. Both of these trends show that when alert rates are higher, the CDF algorithm changes the Medi- Span alert thresholds to a greater degree.

Table 3: Summary of threshold changes per dose range checking rule.

Single Dose
Medication Name Age Range No. Orders RAR Before RAR After
(% Change)
Medi-Span Thresholds Calculated Thresholds
Acetaminophen 325mg Tablets 0 – 12 1521 0.2249 .0544 (-75.8%) 10 – 15 mg/kg 6.85 – 15.8 mg/kg
Acetaminophen 325mg Tablets 12 – 99 5460 0.0636 .0442 (-30.5%) 325 – 650 mg 259.71 – 955.54 mg
Diphenhydramine HCl12.5mg/5mL Elixir 0 – 12 4541 0.2506 .0491 (-80.4%) .5 – 1 mg/kg .35 – 1.01 mg/kg
Hydromorphone HCl2mg/mL IJ Solution 0 – 12 434 1.0392 .0516 (-95%) .015 – .015 mg/kg .01 – .2 mg/kg
Hydromorphone HCl2mg/mL IJ Solution 12 – 99 771 0.3671 .0504 (-86.3%) 1 – 4 mg .26 – 3.02 mg
Levetiracetam 100mg/mL Solution 0 – 12 625 0.3152 .0535 (-83%) 5 – 20 mg/kg 4.59 – 45.93 mg/kg
Lorazepam 2mg/mL IJ Solution 12 – 99 622 0.082 .0424 (-48.3%) 1 – 4 mg .38 – 5.24 mg
Prednisolone Sodium Phosphate15mg/5mL Po Solution 0 – 12 5459 0.0048 .0577 (1101.2%) .025 – 2 mg/kg .17 – 1.71 mg/kg
Ranitidine HCl 75mg/5mL Syrup .04 – 12 1100 0.0118 .0355 (200.4%) 1 – 2.5 mg/kg 1 – 3.26 mg/kg
Rocuronium Bromide10mg/mL IV Solution 0 – 99 1114 0.3851 .0574 (-85.1%) .3 – 1.5 mg/kg .39 – 1.51 mg/kg
Tacrolimus 1mg Capsules 0 – 12 213 0.723 .0578 (-92%) .12 – .18 mg/kg .04 – .3 mg/kg
Ursodiol 300mg Capsules 12 – 99 110 2.1364 .0709 (-96.7%) 2 – 5 mg/kg 3.5 – 10.08 mg/kg

Table 4: Generalized linear model results. Daily underdose values not reported due to few data in the selection of medications used for regression analysis.

Single Overdose Daily Overdose Single Underdose
Intercept -0.096 (± 0.265) P =0.486 -0.2687 (± 0.325) P = 0.118 -0.31499 (± 0.071) P < .001
Threshold RAR 4.10601 (± 1.849) P < .001 2.3616 (± 1.931) P = .0247 -4.09885 (± 1.492) P < .001
Correlation coefficient r = 0.521 r = 0.222 r = -0.367

Discussion

Implementation feasibility

Creating a system that automatically identifies worst-offending dose range checking rules is both possible and feasible. In this study, the RAR metric appropriately identified aberrant dosing thresholds and produced results that are comparable to manual intervention. Additionally, due to the quantitative nature of the RAR metric, one can easily identify potential targets for manual intervention amongst a large population of medications and also compare rule effectiveness across pharmaceutical classes. There are few patient safety implications that would restrict the use of such a system, for the RAR metric operates only on an infrastructural/administrative level and would not change clinical practice without active human intervention. Therefore, due to the large time-saving benefits and low cost of using an automated system to find aberrant dose range checking rules, many practices with a similar CPOE/EHR infrastructure could benefit from this system.

Using an automated algorithm to define dose alert thresholds on empiric prescribing data instead of on pre-specified dose range checking rules produces similar results to manual intervention, automatically readjusts threshold values as prescribing activity changes, and significantly decreases the alert rate for medications with excessively strict dose range checking rules while increasing the alert rate for medications with liberal rules. However, as the CDF algorithm would have immediate impact on clinical practice, there are a few limitations that must be considered before utilizing similar algorithms as more than an administrative tool. The CDF algorithm is highly dependent on large order volume and continuous dosing distributions. While it accurately sets alert thresholds for medications with many orders (100 or more in the study period in the particular patient/weight/age subclass), it cannot always accurately calculate thresholds for less-used medications (approximately fewer than 50 orders in a subclass). Additionally, as the CDF algorithm bases its age-, weight-, and condition-based rules on the corpus of Medi-Span rules, it cannot determine whether dose range checking rules should be different for separate age groups and other divisions; rather, it readjusts the dosing thresholds supplied by Medi-Span. (For example, the CDF algorithm cannot determine whether 12-year-olds should be treated differently than 18-year-olds. It merely calculates new dosing thresholds for the age classes or weight classes pre-supplied by Medi-Span.) However, both of these issues are non-fundamental and potentially resolvable. By increasing the amount of data considered by the CDF algorithm, either using an expanded date range or by considering additional clinical practices, the limited data could easily be reduced or eliminated. One could also reduce the CDF algorithm’s dependency on the Medi-Span rules through a pairing methodology. Instead of assigning rules based on age ranges or weight classes, the CDF algorithm could dynamically suggest dosing ranges based on similar cases (for example, if one is treating a 50 kg 11-year-old with allergic rhinitis, the CDF algorithm could calculate suggested dose ranges by considering only the population of 50 kg 11-year-olds with similar clinical characteristics). This approach would again require greatly expanded datasets and enhanced computational infrastructure, but it would completely eliminate the dependency on Medi-Span rules and would base all alerting decisions on empiric prescribing data. As these algorithmic improvements have yet to be developed, however, the results of the CDF algorithm corroborated by a human observer likely will serve best as an advisory tool during manual alert threshold modification.

Study limitations and sources of error

There are several data-based limitations to this study. First, all analyses were based on data derived from a single pediatric hospital and thus the specific clinical results may not be generalizable to a larger population. Missing data and limited sample sizes likely account for a large part of the statistical error in this study: for example, 93,367 of 703,017 orders (13.28%) did not have data on patient vitals (weight, height, etc.) and therefore were not considered in this study.

Utilizing the CDF algorithm assumes that the medication error rate is known and does not vary across medications or pharmaceutical classes. While it is possible to estimate error rates on a system-wide scale, it is more difficult to accurately capture medication error rates for a particular medication or class of medications again due to limited datasets. Further research into error rates will improve the CDF algorithm by giving it more accurate actual error rate estimates.

Besides the data-derived limitations of the CDF and RAR algorithms, there is a more fundamental methodology-based concern that must be addressed prior to implementing either system on a wide scale. As Van der Sijs et al. argues in “Turning off frequently overridden drug alerts: limited opportunities for doing it safely” (2008), great care must be taken whenever one turns off alerts of any kind because turning off alerts can lead to adverse patient outcomes (though the Van der Sijs et al. study only considered DDI alerts). While most alerts are simply ignored by providers due to elevated false-positive rates, reducing the alert rate for its own sake without considering clinical practice may have severe patient safety implications. While false-positive alerts irritate care providers and reduce the effectiveness of CDS, false-negative alerts, i.e., not firing an alert when one should fire, have graver patient safety implications. Therefore, while the potential efficiency benefits of both the RAR and CDF algorithms are great, the clinical efficacy and safety of such interventions must be considered prior to broad implementation.

Conclusion

Creating a system to automatically determine worst-offending dose range checking rules requires far less manpower investment than manual error correction and produces results that are comparable to manual intervention while correctly reflecting the accuracy of the original pre-specified dosing range rules.

Defining dose alert thresholds automatically based on empiric prescribing data significantly decreases the alert rate of a CDS system and dynamically changes based on prescribing patterns, which may increase the quality of CDS alerts. However, despite the efficiency gains of automatic threshold redefinition, the safety and efficacy of automated dose-defining algorithms have yet to be determined in a clinical setting.

References

Corrigan JM, Donaldson MS, Kohn LT, eds. (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press Washington, DC.

Bates DW, Boyle DL, Vander Vliet MB et al. (1995). Incidence of adverse drug events and medication errors. Implications for prevention. ADE Prevention Study Group. JAMA, 274, 29-34.

Paton J, Wallace J. (2007). Medication errors. Lancet, 349, 959-60.

Jonville APE, Autret E, Bavoux F et al. (1991). Characteristics of medication errors in pediatrics. DICP, 25, 1113-8.

Bates, DW. (1996). Medication errors. How common are they and what can be done to prevent them? Drug Saf, 15, 303-310.

Beijer HJM, de Blaey CJ. (2002). Hospitalizations caused by adverse drug reactions (ADR): a meta-analysis of observational studies. Pharm World Sci., 24, 46-54.

Sard BE, Walsh KE, Doros G, et al. (2008). Retrospective evaluation of a computerized physician order entry adaptation to prevent prescribing errors in a pediatric emergency department. Pediatrics, 122, 782-787.

Kozer E, Scolnik D, Keays T, et al. (2002). Large errors in the dosing of medications for children. N Engl J Med, 346, 1175-1176.

Kohn LT, Corrigan JM, Donaldson MS. (2000). To Err is Human—Building a Safer Health System. Washington, DC: Institute of Medicine, National Academy Press.

Aspden P, Wolcott JA, Bootman JL, Cronenwett LR. (2007). Preventing Medication Errors: Quality Chasm Series. National Academies Press Washington, DC.

Smith J. (2004). Building a Safer National Health System for Patients: Improving Medication Safety. London: Department of Health.

Kaushal R, Bates DW, Landrigan C, et al. (2001). Medication Errors and Adverse Drug Events in Pediatric Inpatients. JAMA, 285, 2114-2120.

Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. (2008). The effect of electronic prescribing on medication errors and adverse drug effects: a systematic review. JAMIA, 15, 585-600.

Gerstle RS, Lehmann CU, the Council on Clinical Information Technology. (2007). Electron prescribing systems in pediatrics: the rationale and functionality requirements. Pediatrics, 119, 1413-1422.

Paterno MD, Maviglia SM, Gorman PN, et al. (2009). Tiering drug-drug interaction alerts by severity increases compliance rates. JAMIA, 16, 40-46.

Strom BL, Schinnar R, Bilker W, et al. (2010). Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co-prescribing as a test case. J Am Med Inform Assoc, 17, 411-415.

Kuperman G, Marston E, Paterno M, et al. (2003). Creating an enterprise-wide allergy repository at Partners HealthCare System. AMIA Annu Symp Proc 2003, 376–380.

Isaac T, Weissman, JS, Davis RB, et al. (2009). Overrides of medication alerts in ambulatory care. Arch Intern Med, 169, 305-311.

Beccaro MA, Villanueva R, Knudson KM, et al. (2010). Decision support alerts for medication ordering in a computerized provider order entry (CPOE) system. Appl Clin Inf, 1, 346-362.

Weingart SN, Toth M, Sands D, et al. (2003). Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med, 163, 2625-2628.

Shah NR, Seger AC, Seger DL, et al. (2006). Improving Acceptance of Computerized Prescribing Alerts in Ambulatory Care. J Am Med Inform Assoc, 13, 5–11.

Wolter Kluwer Corporation, Indianapolis, IA

Spooner SA, Classen DC. (2009). Data Standards and Improvement of Quality and Safety in Child Health Care. Pediatrics, 123 (Supplement), S74 -S79

Linder JA, Ma J, Bates DW, et al. (2007). Electronic Health Record Use and the Quality of Ambulatory Care in the United States. Arch Intern Med, 167, 1400-1405.

Walker J, Pan E, Johnston D, et al. (2005). The value of health care information exchange and interoperability. Health Affairs, Suppl Web Exclusives, W5-10-W5-18

R Development Core Team (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org/.

Brian Ripley (from 1999 to Oct 2002) and Michael Lapsley (2010). RODBC: ODBC Database Access. R package version 1.3-2. http://CRAN.R-project.org/package=RODBC

John Fox and Sanford Weisberg (2011). An {R} Companion to Applied Regression, Second Edition. Thousand Oaks CA: Sage.

Sarkar, Deepayan (2008) Lattice: Multivariate Data Visualization with R. Springer, New York. ISBN 978-0-387-75968-5

Scrucca, L. (2004). qcc: an R package for quality control charting and statistical process control. R News 4/1, 11-17.

Silverman, B. W. (1986) Density Estimation. London: Chapman & Hall, 48.

Nelder J, Wedderburn R. (1972). Generalized Linear Models. Journal of the Royal Statistical Society, 135, 370–384.

Stutman HR, Fineman R, Meyer K, Jones D. (2007). Optimizing the acceptance of medication-based alerts by physicians during CPOE implementation in a community hospital environment. AMIA Annu Symp Proc, 2007, 701–705.

Bates DW, Boyle DL, Vander Vliet MB, et al. (1995). Relationship between medication errors and adverse drug events. J Gen Intern Med, 10, 199-205.

Van der Sijs H, Aarts J, van Gelder T, et al. (2008). Turning off frequently overridden drug alerts: limited opportunities for doing it safely. JAMIA, 15, 439-448.

 

Comments:

NO COMMENTS

LEAVE A REPLY