The Inner Workings of Machine Learning: An exploration into a predictive model’s decisions

By David Simon, Ph.D., Research Data Scientist

Predicting a patient’s risk for adverse outcomes is an important part of delivering personalized care and improving the lives of patients. In order to achieve these goals, axialHealthcare has developed a number of machine learning models that quantify patient risk.

An exciting example is our machine learning model that estimates a patient’s risk for a future diagnosis of opioid use disorder (OUD) using numerous descriptions of a patient’s life. Although the majority of these factors are based on a patient’s health care history, some are demographic measurements like the location where a patient lives. While we believe this type of information is a critical step for developing accurate predictive models, we were interested in understanding how the model utilizes these factors to make better decisions about patient risk.

The Shapley Value
Understanding the inner workings of complex machine learning models can be a difficult task, resulting in them often being referred to as “black boxes.” To answer questions about how our model makes decisions, we used an exciting new method called the Shapley value or SHAP (SHapley Additive ExPlanations) [1,2] to “unpack” our model’s internal workings for a small group of patients. SHAP provides individualized explanations for how much a specific factor contributes to a model’s output. This not only lets us explore how important a factor is but allows us to understand how important it is for every unique patient.

SHAP in Action
We used SHAP to investigate how our model incorporates information about the opioid prescribing rate in the county a patient lives in. In the figure below, you’ll see that as the opioid prescribing rate in a county increases, the patients’ SHAP Value decreases in a step-like pattern. With negative values associated with increased risk, this indicates that patients living in counties with higher opioid prescribing rates are considered to be at higher risk for a future diagnosis of OUD by our model. The step-like pattern occurs because only a relatively small number of decisions within our model are dedicated to this factor.


It’s important to note that our model doesn’t only consider where a patient lives. As you may expect, the impact of where you live on your health varies greatly with other factors, such as age, which is why our model also captures these types of interactions to improve the risk estimations.

For example, in the figure below, we contrasted young adults aged 18-25 (gray dots) with older adults aged 60-65 (red dots). Here we can see that not only does the opioid prescribing rate in a patient’s county matter, but it matters more for older adults than for young adults. This is illustrated by the majority of red dots (older adults) at any given county opioid prescription rate being further from a SHAP value of zero than the black dots (younger adults), on average. This means that for older adults, our model thinks living in a low prescription rate county presents less risk and living in a high prescription rate county adds more risk for a future OUD diagnosis.


Including information about where a patient lives not only greatly improves the quality of the predictive model, but also provides new insights. In this case, unpacking the “black box” model enabled us to observe interactions between where someone lives and their age, which therefore helped us further refine our models by including more information that might capture the influences of age-community interactions.

Patient Intervention Insights
Knowing about these interactions might also help when deciding what types of interventions are appropriate for a patient. For example, our model suggests that community-based approaches for reducing the risk of developing OUD might be most effective for older adults living in counties with a low opioid prescription rate. According to our model, this is because older individuals are particularly responsive to their communities’ characteristics. Therefore, increasing their community interaction when they reside in a positive community setting makes sense during interventions.

Exploring our model with SHAP not only gives us information and confidence about how our model makes decisions, but it also creates new opportunities by understanding what exactly is contributing to an individual patient’s risk. With this individualized information we can do a better job of implementing precision medicine to best help patients.

[1] Lundberg S., Su-In L., A unified Approach to Interpreting Model Predictions. 2017.
[2] Lundberg S., Erion, G., Su-In L., Consistent Individualized Feature Attribution for Tree Ensembles. 2018

Leave a Comment

Scroll to Top