Predictive modeling is an essential element of population health analytics. When using the Johns Hopkins ACG System or building your own predictive models, there are four key components to consider: performance, updates, data availability and relevance.

Performance

When it comes to the performance of a predictive model (typically regression based), it is important to understand how well the model “fits” the data. The most common way this is measured is by the R-squared (R²) statistic. The value is always between 0 and 1 and indicates how much variance there is around the average. Examples of this include linear regressions for continuous variables such as cost, or logistic regression for a binary outcome, such as an event (hospitalization).

While a high R² statistic indicates a well performing model, if you focus too much on perfecting your R² value, you run the risk of removing the very thing you need to measure: variation. This is called “overfitting” a model, and can lead to a model which can’t be replicated. It’s especially important to avoid this in health data analytics, because naturally occurring variation in a population is expected.

Fortunately, the ACG system has a long history of validated, well performing predictive models that take these statistical considerations into account in order to ensure users see a wholistic view of their populations.

Updates

Most predictive models will experience some degree of performance decay over time, as patterns of medical treatment change, coding and diagnostic shifts occur, and new codes are put into practice. The ACG System offers periodic updates to ensure models remain accurate, incorporating new pharmacy and medical-side codes on a quarterly basis to account for newly released medications and procedure codes. The ACG System also updates the internal predictive models with regular frequency and undergoes local calibrations to ensure applicability to local markets, ensuring that your models are always up-to-date.

Data Availability

ACG System models utilize data regarding diagnosis, pharmacy and prior cost information; however, you may have a situation where you’re using pharmacy or diagnostic data that is incomplete or missing. Luckily, the ACG system has model variants in place to adapt to available data, including shortened time periods, pharmacy-only data, and diagnostic-data-only. For example, if no pharmacy data is available, the system will recognize this and use only a diagnosis-based predictive model in order to improve accuracy of the results.

Relevance

Many predictive models are developed and validated on a singular system or singular population datasets. But can a model in one region perform as well in another without rigorous testing? Can model results in a commercial population be easily replicated in a Medicare population? One advantage of the ACG System is development and testing in large national databases, local calibrations and multiple independent validations in countries all over the world. The ACG System’s models apply across many populations, age groups, and regions, making them relevant in almost any context.

Ultimately, when using a predictive model, paying attention to performance, updates, data availability and relevance will pay off. And with the ACG System, you can rest assured that your models will be high-performing, up-to-date, tailored to your specific data and relevant to your population.