aif360.metrics
.ClassificationMetric
- class aif360.metrics.ClassificationMetric(dataset, classified_dataset, unprivileged_groups=None, privileged_groups=None)[source]
Class for computing metrics based on two BinaryLabelDatasets.
The first dataset is the original one and the second is the output of the classification transformer (or similar).
- Parameters:
dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
privileged_groups (list(dict)) – Privileged groups. Format is a list of
dicts
where the keys areprotected_attribute_names
and the values are values inprotected_attributes
. Eachdict
element describes a single group. See examples for more details.unprivileged_groups (list(dict)) – Unprivileged groups in the same format as
privileged_groups
.
- Raises:
TypeError –
dataset
andclassified_dataset
must beBinaryLabelDataset
types.
Methods
\(ACC = (TP + TN)/(P + N)\).
Average of absolute difference in FPR and TPR for unprivileged and privileged groups:
Average of difference in FPR and TPR for unprivileged and privileged groups:
Average of difference in PPV and FOR for unprivileged and privileged groups:
base_rate
Compute the base rate, \(Pr(Y = 1) = P/(P+N)\), optionally conditioned on protected attributes.
The between-group coefficient of variation is the square root of two times the
between_all_groups_generalized_entropy_index()
with \(\alpha = 2\).Between-group generalized entropy index that uses all combinations of groups based on
self.dataset.protected_attributes
.The between-group Theil index is the
between_all_groups_generalized_entropy_index()
with \(\alpha = 1\).The between-group coefficient of variation is the square root of two times the
between_group_generalized_entropy_index()
with \(\alpha = 2\).Between-group generalized entropy index that uses
self.privileged_groups
andself.unprivileged_groups
as the only two groups.The between-group Theil index is the
between_group_generalized_entropy_index()
with \(\alpha = 1\).Compute the number of true/false positives/negatives, optionally conditioned on protected attributes.
The coefficient of variation is the square root of two times the
generalized_entropy_index()
with \(\alpha = 2\).consistency
Individual fairness metric from [1]_ that measures how similar the labels are for similar instances.
difference
Compute difference of the metric for unprivileged and privileged groups.
Bias amplification is the difference in smoothed EDF between the classifier and the original dataset.
Alias of
true_positive_rate_difference()
.Greater of the absolute difference in FPR and TPR for unprivileged and privileged groups
\(ERR = (FP + FN)/(P + N)\)
Difference in error rates for unprivileged and privileged groups, \(ERR_{D = \text{unprivileged}} - ERR_{D = \text{privileged}}\).
Ratio of error rates for unprivileged and privileged groups, \(\frac{ERR_{D = \text{unprivileged}}}{ERR_{D = \text{privileged}}}\).
\(FDR = FP/(TP + FP)\)
\(FDR_{D = \text{unprivileged}} - FDR_{D = \text{privileged}}\)
\(\frac{FDR_{D = \text{unprivileged}}}{FDR_{D = \text{privileged}}}\)
\(FNR = FN/P\)
\(FNR_{D = \text{unprivileged}} - FNR_{D = \text{privileged}}\)
\(\frac{FNR_{D = \text{unprivileged}}}{FNR_{D = \text{privileged}}}\)
\(FOR = FN/(TN + FN)\)
\(FOR_{D = \text{unprivileged}} - FOR_{D = \text{privileged}}\)
\(\frac{FOR_{D = \text{unprivileged}}}{FOR_{D = \text{privileged}}}\)
\(FPR = FP/N\)
\(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}\)
\(\frac{FPR_{D = \text{unprivileged}}}{FPR_{D = \text{privileged}}}\)
Compute the number of generalized true/false positives/negatives, optionally conditioned on protected attributes.
Generalized entropy index is proposed as a unified individual and group fairness measure in [3].
Greater of the absolute difference in generalized FPR and generalized TPR for unprivileged and privileged groups
\(GFNR = GFN/P\)
\(GFPR = GFP/N\)
\(GTNR = GTN/N\)
Return the ratio of generalized true positives to positive examples in the dataset, \(GTPR = GTP/P\), optionally conditioned on protected attributes.
mean_difference
Alias of
statistical_parity_difference()
.\(NPV = TN/(TN + FN)\)
\(FN = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
\(FP = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\)
Return the generalized number of false negatives, \(GFN\), the weighted sum of 1 - predicted scores where true labels are 'favorable', optionally conditioned on protected attributes.
Return the generalized number of false positives, \(GFP\), the weighted sum of predicted scores where true labels are 'unfavorable', optionally conditioned on protected attributes.
Return the generalized number of true negatives, \(GTN\), the weighted sum of 1 - predicted scores where true labels are 'unfavorable', optionally conditioned on protected attributes.
Return the generalized number of true positives, \(GTP\), the weighted sum of predicted scores where true labels are 'favorable', optionally conditioned on protected attributes.
num_instances
Compute the number of instances, \(n\), in the dataset conditioned on protected attributes if necessary.
num_negatives
Compute the number of negatives, \(N = \sum_{i=1}^n \mathbb{1}[y_i = 0]\), optionally conditioned on protected attributes.
num_positives
Compute the number of positives, \(P = \sum_{i=1}^n \mathbb{1}[y_i = 1]\), optionally conditioned on protected attributes.
\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{favorable}]\)
\(TN = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
Return the number of instances in the dataset where both the predicted and true labels are 'favorable', \(TP = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\), optionally conditioned on protected attributes.
Compute various performance measures on the dataset, optionally conditioned on protected attributes.
\(PPV = TP/(TP + FP)\)
Alias of
num_true_positives()
.Alias of
positive_predictive_value()
.ratio
Compute ratio of the metric for unprivileged and privileged groups.
Alias of
true_positive_rate()
.rich_subgroup
Audit dataset with respect to rich subgroups defined by linear thresholds of sensitive attributes
\(Pr(\hat{Y} = \text{favorable})\)
Alias of
true_positive_rate()
.smoothed_empirical_differential_fairness
Smoothed EDF from [#foulds18]_.
Alias of
true_negative_rate()
.The Theil index is the
generalized_entropy_index()
with \(\alpha = 1\).\(TNR = TN/N\)
Return the ratio of true positives to positive examples in the dataset, \(TPR = TP/P\), optionally conditioned on protected attributes.
\(TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}\)
- __init__(dataset, classified_dataset, unprivileged_groups=None, privileged_groups=None)[source]
- Parameters:
dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
privileged_groups (list(dict)) – Privileged groups. Format is a list of
dicts
where the keys areprotected_attribute_names
and the values are values inprotected_attributes
. Eachdict
element describes a single group. See examples for more details.unprivileged_groups (list(dict)) – Unprivileged groups in the same format as
privileged_groups
.
- Raises:
TypeError –
dataset
andclassified_dataset
must beBinaryLabelDataset
types.
- accuracy(privileged=None)[source]
\(ACC = (TP + TN)/(P + N)\).
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- average_abs_odds_difference()[source]
Average of absolute difference in FPR and TPR for unprivileged and privileged groups:
\[\tfrac{1}{2}\left[|FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}| + |TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}|\right]\]A value of 0 indicates equality of odds.
- average_odds_difference()[source]
Average of difference in FPR and TPR for unprivileged and privileged groups:
\[\tfrac{1}{2}\left[(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}) + (TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}))\right]\]A value of 0 indicates equality of odds.
- average_predictive_value_difference()[source]
Average of difference in PPV and FOR for unprivileged and privileged groups:
\[\tfrac{1}{2}\left[(PPV_{D = \text{unprivileged}} - PPV_{D = \text{privileged}}) + (FOR_{D = \text{unprivileged}} - FOR_{D = \text{privileged}}))\right]\]A value of 0 indicates equality of chance of success.
- between_all_groups_coefficient_of_variation()[source]
The between-group coefficient of variation is the square root of two times the
between_all_groups_generalized_entropy_index()
with \(\alpha = 2\).
- between_all_groups_generalized_entropy_index(alpha=2)[source]
Between-group generalized entropy index that uses all combinations of groups based on
self.dataset.protected_attributes
. See_between_group_generalized_entropy_index()
.- Parameters:
alpha (int) – See
generalized_entropy_index()
.
- between_all_groups_theil_index()[source]
The between-group Theil index is the
between_all_groups_generalized_entropy_index()
with \(\alpha = 1\).
- between_group_coefficient_of_variation()[source]
The between-group coefficient of variation is the square root of two times the
between_group_generalized_entropy_index()
with \(\alpha = 2\).
- between_group_generalized_entropy_index(alpha=2)[source]
Between-group generalized entropy index that uses
self.privileged_groups
andself.unprivileged_groups
as the only two groups. See_between_group_generalized_entropy_index()
.- Parameters:
alpha (int) – See
generalized_entropy_index()
.
- between_group_theil_index()[source]
The between-group Theil index is the
between_group_generalized_entropy_index()
with \(\alpha = 1\).
- binary_confusion_matrix(privileged=None)[source]
Compute the number of true/false positives/negatives, optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Returns:
dict – Number of true positives, false positives, true negatives, false negatives (optionally conditioned).
- coefficient_of_variation()[source]
The coefficient of variation is the square root of two times the
generalized_entropy_index()
with \(\alpha = 2\).
- differential_fairness_bias_amplification(concentration=1.0)[source]
Bias amplification is the difference in smoothed EDF between the classifier and the original dataset. Positive values mean the bias increased due to the classifier.
- Parameters:
concentration (float, optional) – Concentration parameter for Dirichlet smoothing. Must be non-negative.
- disparate_impact()[source]
- \[\frac{Pr(\hat{Y} = 1 | D = \text{unprivileged})} {Pr(\hat{Y} = 1 | D = \text{privileged})}\]
- equal_opportunity_difference()[source]
Alias of
true_positive_rate_difference()
.
- equalized_odds_difference()[source]
Greater of the absolute difference in FPR and TPR for unprivileged and privileged groups
- error_rate(privileged=None)[source]
\(ERR = (FP + FN)/(P + N)\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- error_rate_difference()[source]
Difference in error rates for unprivileged and privileged groups, \(ERR_{D = \text{unprivileged}} - ERR_{D = \text{privileged}}\).
- error_rate_ratio()[source]
Ratio of error rates for unprivileged and privileged groups, \(\frac{ERR_{D = \text{unprivileged}}}{ERR_{D = \text{privileged}}}\).
- false_discovery_rate(privileged=None)[source]
\(FDR = FP/(TP + FP)\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- false_discovery_rate_difference()[source]
\(FDR_{D = \text{unprivileged}} - FDR_{D = \text{privileged}}\)
- false_discovery_rate_ratio()[source]
\(\frac{FDR_{D = \text{unprivileged}}}{FDR_{D = \text{privileged}}}\)
- false_negative_rate(privileged=None)[source]
\(FNR = FN/P\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- false_negative_rate_difference()[source]
\(FNR_{D = \text{unprivileged}} - FNR_{D = \text{privileged}}\)
- false_negative_rate_ratio()[source]
\(\frac{FNR_{D = \text{unprivileged}}}{FNR_{D = \text{privileged}}}\)
- false_omission_rate(privileged=None)[source]
\(FOR = FN/(TN + FN)\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- false_omission_rate_difference()[source]
\(FOR_{D = \text{unprivileged}} - FOR_{D = \text{privileged}}\)
- false_omission_rate_ratio()[source]
\(\frac{FOR_{D = \text{unprivileged}}}{FOR_{D = \text{privileged}}}\)
- false_positive_rate(privileged=None)[source]
\(FPR = FP/N\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- false_positive_rate_difference()[source]
\(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}\)
- false_positive_rate_ratio()[source]
\(\frac{FPR_{D = \text{unprivileged}}}{FPR_{D = \text{privileged}}}\)
- generalized_binary_confusion_matrix(privileged=None)[source]
Compute the number of generalized true/false positives/negatives, optionally conditioned on protected attributes. Generalized counts are based on scores and not on the hard predictions.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Returns:
dict – Number of generalized true positives, generalized false positives, generalized true negatives, generalized false negatives (optionally conditioned).
- generalized_entropy_index(alpha=2)[source]
Generalized entropy index is proposed as a unified individual and group fairness measure in [3]. With \(b_i = \hat{y}_i - y_i + 1\):
\[\begin{split}\mathcal{E}(\alpha) = \begin{cases} \frac{1}{n \alpha (\alpha-1)}\sum_{i=1}^n\left[\left(\frac{b_i}{\mu}\right)^\alpha - 1\right],& \alpha \ne 0, 1,\\ \frac{1}{n}\sum_{i=1}^n\frac{b_{i}}{\mu}\ln\frac{b_{i}}{\mu},& \alpha=1,\\ -\frac{1}{n}\sum_{i=1}^n\ln\frac{b_{i}}{\mu},& \alpha=0. \end{cases}\end{split}\]- Parameters:
alpha (int) – Parameter that regulates the weight given to distances between values at different parts of the distribution.
References
- generalized_equalized_odds_difference()[source]
Greater of the absolute difference in generalized FPR and generalized TPR for unprivileged and privileged groups
- generalized_false_negative_rate(privileged=None)[source]
\(GFNR = GFN/P\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- generalized_false_positive_rate(privileged=None)[source]
\(GFPR = GFP/N\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- generalized_true_negative_rate(privileged=None)[source]
\(GTNR = GTN/N\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- generalized_true_positive_rate(privileged=None)[source]
Return the ratio of generalized true positives to positive examples in the dataset, \(GTPR = GTP/P\), optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- negative_predictive_value(privileged=None)[source]
\(NPV = TN/(TN + FN)\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_false_negatives(privileged=None)[source]
\(FN = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_false_positives(privileged=None)[source]
\(FP = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_generalized_false_negatives(privileged=None)[source]
Return the generalized number of false negatives, \(GFN\), the weighted sum of 1 - predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_generalized_false_positives(privileged=None)[source]
Return the generalized number of false positives, \(GFP\), the weighted sum of predicted scores where true labels are ‘unfavorable’, optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be must be provided at initialization to condition on them.
- num_generalized_true_negatives(privileged=None)[source]
Return the generalized number of true negatives, \(GTN\), the weighted sum of 1 - predicted scores where true labels are ‘unfavorable’, optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_generalized_true_positives(privileged=None)[source]
Return the generalized number of true positives, \(GTP\), the weighted sum of predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_pred_negatives(privileged=None)[source]
\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_pred_positives(privileged=None)[source]
\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{favorable}]\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_true_negatives(privileged=None)[source]
\(TN = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- num_true_positives(privileged=None)[source]
Return the number of instances in the dataset where both the predicted and true labels are ‘favorable’, \(TP = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\), optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- performance_measures(privileged=None)[source]
Compute various performance measures on the dataset, optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Returns:
dict – True positive rate, true negative rate, false positive rate, false negative rate, positive predictive value, negative predictive value, false discover rate, false omission rate, and accuracy (optionally conditioned).
- positive_predictive_value(privileged=None)[source]
\(PPV = TP/(TP + FP)\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- power(privileged=None)[source]
Alias of
num_true_positives()
.
- precision(privileged=None)[source]
Alias of
positive_predictive_value()
.
- recall(privileged=None)[source]
Alias of
true_positive_rate()
.
- selection_rate(privileged=None)[source]
\(Pr(\hat{Y} = \text{favorable})\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- sensitivity(privileged=None)[source]
Alias of
true_positive_rate()
.
- specificity(privileged=None)[source]
Alias of
true_negative_rate()
.
- statistical_parity_difference()[source]
- \[Pr(\hat{Y} = 1 | D = \text{unprivileged}) - Pr(\hat{Y} = 1 | D = \text{privileged})\]
- theil_index()[source]
The Theil index is the
generalized_entropy_index()
with \(\alpha = 1\).
- true_negative_rate(privileged=None)[source]
\(TNR = TN/N\)
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.
- true_positive_rate(privileged=None)[source]
Return the ratio of true positives to positive examples in the dataset, \(TPR = TP/P\), optionally conditioned on protected attributes.
- Parameters:
privileged (bool, optional) – Boolean prescribing whether to condition this metric on the
privileged_groups
, ifTrue
, or theunprivileged_groups
, ifFalse
. Defaults toNone
meaning this metric is computed over the entire dataset.- Raises:
AttributeError –
privileged_groups
orunprivileged_groups
must be provided at initialization to condition on them.