Core Protocol

Evaluation Metrics

The Evaluation Protocol forms the mathematical foundation of FairnessAudit. We calculate exact disparity metrics between protected and unprotected groups in your dataset.

1Demographic Parity

Measures whether the positive prediction rate is independent of the sensitive attribute. A perfectly fair model under this metric will predict the positive outcome at the exact same rate for all groups.

P(Ŷ = 1 | A = 0) = P(Ŷ = 1 | A = 1)

When to use: Best for situations where historical bias in the training data is assumed, and you want to ensure equal representation of outcomes (e.g., hiring rates).

2Equalized Odds

Requires that both the true positive rate (TPR) and false positive rate (FPR) are equal across all groups. This ensures that the model is equally accurate for both the privileged and unprivileged classes.

P(Ŷ = 1 | Y = y, A = 0) = P(Ŷ = 1 | Y = y, A = 1), y ∈ {0, 1}

Note: Equalized Odds is harder to satisfy than Demographic Parity but often preferred when predicting an outcome tied to a ground truth (e.g., criminal recidivism).