The Fault in Prediction
The technological landscape has changed considerably over the last two decades, but the underlying principles of prediction remain the same. P-values are still p-values, odds ratios are still odds ratios, and historical performance is still relevant when thinking about future results.
My predictive analytics journey began in 2001 at Indiana University’s Kelley School of Business. Decision support modeling was the hot topic in the MBA program, and I was not going to be left behind. I spent two years learning everything I could about data mining, binary logistic regression and neural networks. In 2001, the ideas of big data and cloud computing were theoretical at best.
Fast forward 17 years, and I find myself in a predictive analytics playground. Data in the healthcare field is abundant, and data at Baptist Health is no exception. Our data is threaded with such elements as doctor visits, hospital admissions, emergency room visits, chronic conditions, and prescription medications. Even though the data is fairly straightforward, it is also fairly intertwined. This interconnectedness sometimes makes delivering on value-based care feel as if we are cracking the Enigma code. Even so, the work is important, and it is absolutely necessary if we expect to bend the healthcare cost curve in the future.
This past year, I was asked to join the code-breaking effort. Armed with a dataset of 50,000 Medicare patient records, we set out to find a subset of patients who were most likely to experience hospital admissions and emergency room visits in the future. Once we identified the subset, we would put actions in place to reduce the frequency of these costly episodes.
As long as there are delays between the time it takes to collect enough data and the time it takes to use the data, there will always be accountability gaps
Our first predictive model was promising, but it still needed refinement. Over the course of a few days, the team worked diligently to add and subtract factors – checking results along the way – until we arrived at a final model. Starting with a list of 50,000 patients, we pared it to 631. The next step was to involve our care teams. A group of highly experienced care advisors would learn everything they could about these 631 patients, and conduct an outreach campaign. Outreach would include patient education, medication reconciliation, assistance with appointments, and much-needed emotional support. It is what happened next however, that caught me off guard and forever changed the way I think about prediction.
Of the 631 patients we had identified as high-risk, 75patients were already deceased. As it turns out, our model was using data that was more than a year old. Seventy-five patients could not wait for us to figure out our algorithms. Seventy-five families had lost their loved ones. Although the predictive model seemed to work quite well, my heart was broken. I had discovered the fault in prediction.
The fault in prediction is not in the data we use. The fault in prediction is not in the technologies we employ. The fault in prediction lies in the delays. As long as there are delays between the time it takes to collect enough data and the time it takes to use the data, there will always be accountability gaps. In a different field, this defect is not as obvious. It is one thing for delayed actions to result in lost sales. It is a completely different story when we are talking about lives. So how might we reduce the delays in our predictive modeling efforts? I certainly do not have all of the answers, but I do have a few suggestions that may help the next wave of predictive analytic professionals.
1) Defend yourself against information bias
Information bias is a cognitive bias that pushes us to seek additional information even when it does not alter the story nor change our actions. In corporate America, it is easy to fall into this trap. We want to be 100 percent certain that our decisions are the right ones, but there is no such thing as 100 percent certainty in predictive modeling. Gather your data and go.
2) Think of predictive modeling as a continuous cycle
Predictive modeling is about testing, failing, and failing better the next time. We fear failure when we see it as the opposite of success, but this could not be further from the truth. If you are not failing, you are not taking enough risks, and you are not pushing the limits of what you can do. Get comfortable with a plan-do-check-act mentality. Predictive modeling is not a point-in-time exercise. Continuous improvement is the key.
3) Don’t forget the human element
Whatever industry you are in, never lose sight of the information behind your data. Data may come disguised as numbers in a spreadsheet, but it can represent real experiences and very real life events. Remember the human element in your data and use it as fuel when you are working long hours or perhaps feeling stuck.
As my journey continues, I promise to heed my own advice. I promise to lead the charge against the fault in prediction. There are 75 more patients out there who just might need our help. There are 75 more families who are counting on it.