python - Make variation on accuracy score to incorporate ordinality

Question

For a classification problem with ordinal data (classes are : 1,2,3,4,5) I'm looking for a variation on the sklearn function "accuracy_score". Specifically, I want to create a new performance metric that includes predicitions within one notch from the actual value as correct classifications. I.e. if an observation has actual value 3, then predictions of 2, 3, and 4 should be seen as being correct. Example:

y_test = [1,3,3,3,5]
y_pred = [5,4,3,2,1]
accuracy_score(y_test,y_pred) # 0.2

And my required output would be:

accuracy_score_within1(y_test,y_pred) # 0.6

How can I do this?

Thanks in advance!

P.S. I realiser that these extra "correct" values are of course not actually correct, but this metric gives inside into whether my model predicts close to the actual value which can allready be very useful for my purposes.

question from:https://stackoverflow.com/questions/65872754/make-variation-on-accuracy-score-to-incorporate-ordinality

Answer 1

I propose you this custom function:

import numpy as np

def accuracy_score_within1(y_test, y_pred):
    # Compute the absolute difference between each term
    diff = abs(np.array(y_test) - np.array(y_pred))
    # If the difference is less or equal than 1 replace it by 1 else 0
    res = [1 if i <= 1 else 0 for i in diff]
    # The score is the mean of this resulting list
    return np.mean(res)

Test:

y_test = [1,3,3,3,5]
y_pred = [5,4,3,2,1]

accuracy_score_within1(y_test, y_pred)

Output:

0.6