Area under the curve and risk assessment

This month, Greg was finally released from prison. In fact, many people in his county were released from prison this month, some of them under parole supervision. Unfortunately, some of the released individuals will commit at least one crime in the upcoming several years. Many people relapse into criminal behavior because the factors that initially facilitated their crimes often persist. Additionally, it can be quite challenging to reintegrate into society after prison.

Although, three years later, some former prisoners will reoffend and end up in prison again, the majority will desist from crime. Policy makers and practitioners would like to know in advance who among released prisoners are likely to reoffend, so they can keep a closer eye on them and provide additional support if necessary, such as access to substance abuse treatment programs or vocational training.

But how can they do it, and how can they do it effectively?

Greg Year 1 Year 2 Year 3

Risk assessment

The main way to predict potential future reoffending is through the use of some form of risk assessment, such as standardised risk assessment instruments. These instruments evaluate various risk factors and protective factors that comprise an individual’s characteristics and circumstances.

Risk assessment High risk Likely to reoffend Low risk Not likely to reoffend

Many different instruments exist, and they have slightly different abilities to predict potential future reoffending. The ability of a risk assessment instrument to predict future outcomes is called predictive validity, and one of the measures of predictive validity is the AUC ROC curve.

AUC has several technical interpretations, but our goal is to create a simple, actionable intuition behind it.

Area Under the Curve (AUC) intuition

You can watch the video accompanying this text on YouTube ->

Imagine that we implemented a risk assessment instrument and measured the reoffending risk of Greg, using a risk score that ranges from 0 to 100. They scored somewhere around 30. Then we measured the risk in 10 more individuals, who each received different scores, ranging from 5 to 90.

We let these people go about their lives and checked on them after 3 years.

Different AUC corresponds to different degree of separation

AUC = 1.0
AUC = 1.0
AUC = 0.9

As you can see, in first three scenarios, some of released prisoners reoffended, and all offenders are among the top scorers. This perfect separation between reoffenders and desisters corresponds to an AUC of 1. It means that our hypothetical risk assessment tool can perfectly distinguish one group of individuals from the other. The number of people who reoffended and their scores don’t matter, as long as they are clearly grouped, the AUC is 1.

AUC = 0.8