Every quarter, HQI receives a large volume of discharge records (as part of the Hospital Quality Improvement Platform) and safety reports (as part of the Collaborative Healthcare Patient Safety Organization, or CHPSO) from hospitals.
To make use of this timely data, we designed and implemented sentinel signal detection systems that automatically detect abnormal changes in the incidence of diagnosis codes and terms’ document frequencies (i.e., signals).
Moving forward, when these systems detect sudden increases in disease incidence or document frequency, HQI/CHPSO staff will investigate and alert member hospitals of these emerging conditions so they can act to stop the spread of disease or harmful conditions and decrease the risk of additional harm to patients.
Hospital Quality Improvement Platform
For the Hospital Quality Improvement Platform sentinel signal detection system, historical Office of Statewide Health Planning and Development discharges (OSHPD) from 2016 to 2018 were used as the baseline for developing the signal detection system. The system actively monitors the monthly representative frequency (proportion) among all patient discharge records of primary diagnosis codes truncated to the first three digits (1,911 unique three-digit ICD-10 code categories).
The analytic method used for detecting signals is Auto-Regressive Integrated Moving Average (ARIMA) time series analysis with simultaneous outlier detection and re-estimation based on the work of Chen and Liu (1993). Additive outliers, level shifts, temporary changes, and seasonal level shifts are all considered to be signals. An overall “signal” outlier score is calculated for each diagnosis code category to highlight diagnosis codes with the most aberrant activity for review by HQI staff.
While the first implementation of the Hospital Quality Improvement Platform sentinel signal detection system has been validated by demonstrating that it correctly identifies known outbreaks such as the acute Hepatitis A outbreak in Los Angeles and San Diego that occurred in 2017 (Figure 1), its real application is for identifying emerging diagnoses and conditions during the current year. As hospital enrollment in HQI’s quality analytics platform increases, the volume of current-year data available for detecting signals will also grow, which will provide even greater statistical power for HQI to more quickly detect changes in signals during the current year.
Future enhancements planned for the Hospital Quality Improvement Platform sentinel signal detection system will make it more powerful and flexible. For example, the system will eventually identify signals for combinations of ICD-10 category codes, include all diagnosis codes on the records, and include methods to identify changes in both rare code incidence and ICD-10 code by age interactions.
CHPSO
For the CHPSO sentinel signal detection system, patient safety reports with event discovery dates ranging from 2015 to present were used as the baseline for developing the signal detection system. To identify emerging topics in patient safety reports, the system monitors document frequency of term/phrase among all the reports using ARIMA models similar to the outlier detection method mentioned above. Data processing of the CHPSO patient safety reports is much more complex because the raw data consist of free-text reports. Due to the fact that these reports contain terms that have similar meanings, a neural network trained model is first used to create a list of terms with similar meanings or potential typos, which are aggregated for purposes of determining terms’ document frequencies and monitored for signals as a group. The system also highlights newly occurring terms. For example, “covid19” is highlighted since it was never mentioned before 2020.
While the first version of the CHPSO signal detection system holds great promise, there are many aspects planned for improvements. For example, given that the number of terms to check for signals is massive, and there are many terms that can likely be excluded from monitoring because they may not provide useful information (e.g. terms such as “consider” or “ask”), the future version will use a more efficient method to extract only useful terms for signal detection.
In addition, because terms can have different meanings, single term monitoring may be misleading without understanding the context. Therefore, the Word Sense algorithm — which distinguishes word meanings by context — will be explored for use. Finally, the information provided by single term is limited. For example, if a signal for the term “delay” is detected, HQI investigators need to review all the associated free-text reports to know what kind of “delay” occurred. In the future, this context information will be provided using artificial intelligence, thereby removing the need to review hundreds of thousands of patient safety reports.