aif360.metrics
.MDSSClassificationMetric
- class aif360.metrics.MDSSClassificationMetric(dataset: BinaryLabelDataset, classified_dataset: BinaryLabelDataset, scoring: str | ScoringFunction = 'Bernoulli', unprivileged_groups: dict = None, privileged_groups: dict = None, **kwargs)[source]
Bias subset scanning is proposed as a technique to identify bias in predictive models using subset scanning [1].
This class is a wrapper for the bias scan scoring and scanning methods that uses the ClassificationMetric abstraction.
References
- Parameters:
dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
scoring (str or ScoringFunction) – One of ‘Bernoulli’ (parametric), or ‘BerkJones’ (non-parametric) or subclass of
aif360.metrics.mdss.ScoringFunctions.ScoringFunction
. Defaults to Bernoulli.privileged_groups (list(dict)) – Privileged groups. Format is a list of
dicts
where the keys areprotected_attribute_names
and the values are values inprotected_attributes
. Eachdict
element describes a single group. See examples for more details.unprivileged_groups (list(dict)) – Unprivileged groups in the same format as
privileged_groups
.
Methods
accuracy
\(ACC = (TP + TN)/(P + N)\).
average_abs_odds_difference
Average of absolute difference in FPR and TPR for unprivileged and privileged groups:
average_odds_difference
Average of difference in FPR and TPR for unprivileged and privileged groups:
average_predictive_value_difference
Average of difference in PPV and FOR for unprivileged and privileged groups:
base_rate
Compute the base rate, \(Pr(Y = 1) = P/(P+N)\), optionally conditioned on protected attributes.
between_all_groups_coefficient_of_variation
The between-group coefficient of variation is the square root of two times the
between_all_groups_generalized_entropy_index()
with \(\alpha = 2\).between_all_groups_generalized_entropy_index
Between-group generalized entropy index that uses all combinations of groups based on
self.dataset.protected_attributes
.between_all_groups_theil_index
The between-group Theil index is the
between_all_groups_generalized_entropy_index()
with \(\alpha = 1\).between_group_coefficient_of_variation
The between-group coefficient of variation is the square root of two times the
between_group_generalized_entropy_index()
with \(\alpha = 2\).between_group_generalized_entropy_index
Between-group generalized entropy index that uses
self.privileged_groups
andself.unprivileged_groups
as the only two groups.between_group_theil_index
The between-group Theil index is the
between_group_generalized_entropy_index()
with \(\alpha = 1\).binary_confusion_matrix
Compute the number of true/false positives/negatives, optionally conditioned on protected attributes.
coefficient_of_variation
The coefficient of variation is the square root of two times the
generalized_entropy_index()
with \(\alpha = 2\).consistency
Individual fairness metric from [1]_ that measures how similar the labels are for similar instances.
difference
Compute difference of the metric for unprivileged and privileged groups.
differential_fairness_bias_amplification
Bias amplification is the difference in smoothed EDF between the classifier and the original dataset.
disparate_impact
equal_opportunity_difference
Alias of
true_positive_rate_difference()
.equalized_odds_difference
Greater of the absolute difference in FPR and TPR for unprivileged and privileged groups
error_rate
\(ERR = (FP + FN)/(P + N)\)
error_rate_difference
Difference in error rates for unprivileged and privileged groups, \(ERR_{D = \text{unprivileged}} - ERR_{D = \text{privileged}}\).
error_rate_ratio
Ratio of error rates for unprivileged and privileged groups, \(\frac{ERR_{D = \text{unprivileged}}}{ERR_{D = \text{privileged}}}\).
false_discovery_rate
\(FDR = FP/(TP + FP)\)
false_discovery_rate_difference
\(FDR_{D = \text{unprivileged}} - FDR_{D = \text{privileged}}\)
false_discovery_rate_ratio
\(\frac{FDR_{D = \text{unprivileged}}}{FDR_{D = \text{privileged}}}\)
false_negative_rate
\(FNR = FN/P\)
false_negative_rate_difference
\(FNR_{D = \text{unprivileged}} - FNR_{D = \text{privileged}}\)
false_negative_rate_ratio
\(\frac{FNR_{D = \text{unprivileged}}}{FNR_{D = \text{privileged}}}\)
false_omission_rate
\(FOR = FN/(TN + FN)\)
false_omission_rate_difference
\(FOR_{D = \text{unprivileged}} - FOR_{D = \text{privileged}}\)
false_omission_rate_ratio
\(\frac{FOR_{D = \text{unprivileged}}}{FOR_{D = \text{privileged}}}\)
false_positive_rate
\(FPR = FP/N\)
false_positive_rate_difference
\(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}\)
false_positive_rate_ratio
\(\frac{FPR_{D = \text{unprivileged}}}{FPR_{D = \text{privileged}}}\)
generalized_binary_confusion_matrix
Compute the number of generalized true/false positives/negatives, optionally conditioned on protected attributes.
generalized_entropy_index
Generalized entropy index is proposed as a unified individual and group fairness measure in [3]_.
generalized_equalized_odds_difference
Greater of the absolute difference in generalized FPR and generalized TPR for unprivileged and privileged groups
generalized_false_negative_rate
\(GFNR = GFN/P\)
generalized_false_positive_rate
\(GFPR = GFP/N\)
generalized_true_negative_rate
\(GTNR = GTN/N\)
generalized_true_positive_rate
Return the ratio of generalized true positives to positive examples in the dataset, \(GTPR = GTP/P\), optionally conditioned on protected attributes.
mean_difference
Alias of
statistical_parity_difference()
.negative_predictive_value
\(NPV = TN/(TN + FN)\)
num_false_negatives
\(FN = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
num_false_positives
\(FP = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\)
num_generalized_false_negatives
Return the generalized number of false negatives, \(GFN\), the weighted sum of 1 - predicted scores where true labels are 'favorable', optionally conditioned on protected attributes.
num_generalized_false_positives
Return the generalized number of false positives, \(GFP\), the weighted sum of predicted scores where true labels are 'unfavorable', optionally conditioned on protected attributes.
num_generalized_true_negatives
Return the generalized number of true negatives, \(GTN\), the weighted sum of 1 - predicted scores where true labels are 'unfavorable', optionally conditioned on protected attributes.
num_generalized_true_positives
Return the generalized number of true positives, \(GTP\), the weighted sum of predicted scores where true labels are 'favorable', optionally conditioned on protected attributes.
num_instances
Compute the number of instances, \(n\), in the dataset conditioned on protected attributes if necessary.
num_negatives
Compute the number of negatives, \(N = \sum_{i=1}^n \mathbb{1}[y_i = 0]\), optionally conditioned on protected attributes.
num_positives
Compute the number of positives, \(P = \sum_{i=1}^n \mathbb{1}[y_i = 1]\), optionally conditioned on protected attributes.
num_pred_negatives
\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
num_pred_positives
\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{favorable}]\)
num_true_negatives
\(TN = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
num_true_positives
Return the number of instances in the dataset where both the predicted and true labels are 'favorable', \(TP = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\), optionally conditioned on protected attributes.
performance_measures
Compute various performance measures on the dataset, optionally conditioned on protected attributes.
positive_predictive_value
\(PPV = TP/(TP + FP)\)
power
Alias of
num_true_positives()
.precision
Alias of
positive_predictive_value()
.ratio
Compute ratio of the metric for unprivileged and privileged groups.
recall
Alias of
true_positive_rate()
.rich_subgroup
Audit dataset with respect to rich subgroups defined by linear thresholds of sensitive attributes
Compute the bias score for a prespecified group of records.
selection_rate
\(Pr(\hat{Y} = \text{favorable})\)
sensitivity
Alias of
true_positive_rate()
.smoothed_empirical_differential_fairness
Smoothed EDF from [#foulds18]_.
specificity
Alias of
true_negative_rate()
.statistical_parity_difference
theil_index
The Theil index is the
generalized_entropy_index()
with \(\alpha = 1\).true_negative_rate
\(TNR = TN/N\)
true_positive_rate
Return the ratio of true positives to positive examples in the dataset, \(TPR = TP/P\), optionally conditioned on protected attributes.
true_positive_rate_difference
\(TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}\)
- __init__(dataset: BinaryLabelDataset, classified_dataset: BinaryLabelDataset, scoring: str | ScoringFunction = 'Bernoulli', unprivileged_groups: dict = None, privileged_groups: dict = None, **kwargs)[source]
- Parameters:
dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
scoring (str or ScoringFunction) – One of ‘Bernoulli’ (parametric), or ‘BerkJones’ (non-parametric) or subclass of
aif360.metrics.mdss.ScoringFunctions.ScoringFunction
. Defaults to Bernoulli.privileged_groups (list(dict)) – Privileged groups. Format is a list of
dicts
where the keys areprotected_attribute_names
and the values are values inprotected_attributes
. Eachdict
element describes a single group. See examples for more details.unprivileged_groups (list(dict)) – Unprivileged groups in the same format as
privileged_groups
.
- score_groups(privileged=True, penalty=1e-17)[source]
Compute the bias score for a prespecified group of records.
- Parameters:
privileged (bool) – Flag for which direction to scan: privileged (
True
) implies negative (observed worse than predicted outcomes) while unprivileged (False
) implies positive (observed better than predicted outcomes).- Returns:
float –
- Bias score for the given group.
The higher the score, the evidence for bias.