`aif360.metrics`¶

Dataset Metric¶

class aif360.metrics.DatasetMetric(dataset, unprivileged_groups=None, privileged_groups=None)[source]¶

Class for computing metrics based on one StructuredDataset.

Parameters:

dataset (StructuredDataset) – A StructuredDataset.
privileged_groups (list(dict)) – Privileged groups. Format is a list of dicts where the keys are protected_attribute_names and the values are values in protected_attributes. Each dict element describes a single group. See examples for more details.
unprivileged_groups (list(dict)) – Unprivileged groups in the same format as privileged_groups.

Raises:

TypeError – dataset must be a StructuredDataset type.
ValueError – privileged_groups and unprivileged_groups must be disjoint.

Examples

>>> from aif360.datasets import GermanDataset
>>> german = GermanDataset()
>>> u = [{'sex': 1, 'age': 1}, {'sex': 0}]
>>> p = [{'sex': 1, 'age': 0}]
>>> dm = DatasetMetric(german, unprivileged_groups=u, privileged_groups=p)

num_instances(privileged=None)[source]¶

Compute the number of instances, \(n\), in the dataset conditioned on protected attributes if necessary.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be must be provided at initialization to condition on them.

Binary Label Dataset Metric¶

class aif360.metrics.BinaryLabelDatasetMetric(dataset, unprivileged_groups=None, privileged_groups=None)[source]¶

Class for computing metrics based on a single BinaryLabelDataset.

Parameters:

dataset (BinaryLabelDataset) – A BinaryLabelDataset.
privileged_groups (list(dict)) – Privileged groups. Format is a list of dicts where the keys are protected_attribute_names and the values are values in protected_attributes. Each dict element describes a single group. See examples for more details.
unprivileged_groups (list(dict)) – Unprivileged groups in the same format as privileged_groups.

Raises:

TypeError – dataset must be a BinaryLabelDataset type.

base_rate(privileged=None)[source]¶

Compute the base rate, \(Pr(Y = 1) = P/(P+N)\), optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Returns:	Base rate (optionally conditioned).
Return type:	float

consistency(n_neighbors=5)[source]¶

Individual fairness metric from [1] that measures how similar the labels are for similar instances.

\[1 - \frac{1}{n\cdot\text{n_neighbors}}\sum_{i=1}^n |\hat{y}_i - \sum_{j\in\mathcal{N}_{\text{n_neighbors}}(x_i)} \hat{y}_j|\]

Parameters:	n_neighbors (int, optional) – Number of neighbors for the knn computation.

References

[1]	R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, “Learning Fair Representations,” International Conference on Machine Learning, 2013.

disparate_impact()[source]¶: \[\frac{Pr(Y = 1 | D = \text{unprivileged})} {Pr(Y = 1 | D = \text{privileged})}\]

mean_difference()[source]¶: Alias of statistical_parity_difference().

num_negatives(privileged=None)[source]¶

Compute the number of negatives, \(N = \sum_{i=1}^n \mathbb{1}[y_i = 0]\), optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be must be provided at initialization to condition on them.

num_positives(privileged=None)[source]¶

Compute the number of positives, \(P = \sum_{i=1}^n \mathbb{1}[y_i = 1]\), optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be must be provided at initialization to condition on them.

statistical_parity_difference()[source]¶: \[Pr(Y = 1 | D = \text{unprivileged}) - Pr(Y = 1 | D = \text{privileged})\]

Classification Metric¶

class aif360.metrics.ClassificationMetric(dataset, classified_dataset, unprivileged_groups=None, privileged_groups=None)[source]¶

Class for computing metrics based on two BinaryLabelDatasets.

The first dataset is the original one and the second is the output of the classification transformer (or similar).

Parameters:

dataset (BinaryLabelDataset) – Dataset containing ground-truth labels.
classified_dataset (BinaryLabelDataset) – Dataset containing predictions.
privileged_groups (list(dict)) – Privileged groups. Format is a list of dicts where the keys are protected_attribute_names and the values are values in protected_attributes. Each dict element describes a single group. See examples for more details.
unprivileged_groups (list(dict)) – Unprivileged groups in the same format as privileged_groups.

Raises:

TypeError – dataset and classified_dataset must be BinaryLabelDataset types.

_between_group_generalized_entropy_index(groups, alpha=2)[source]¶

Between-group generalized entropy index is proposed as a group fairness measure in [2] and is one of two terms that the generalized entropy index decomposes to.

Parameters:	groups (list) – A list of groups over which to calculate this metric. Groups should be disjoint. By default, this will use the privileged_groups and unprivileged_groups as the only two groups. alpha (int) – See `generalized_entropy_index()`.

References

[2]

T. Speicher, H. Heidari, N. Grgic-Hlaca, K. P. Gummadi, A. Singla, A. Weller, and M. B. Zafar, “A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual and Group Unfairness via Inequality Indices,” ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2018.

accuracy(privileged=None)[source]¶

\(ACC = (TP + TN)/(P + N)\).

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

average_abs_odds_difference()[source]¶

Average of absolute difference in FPR and TPR for unprivileged and privileged groups:

\[\tfrac{1}{2}\left[|FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}| + |TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}|\right]\]

A value of 0 indicates equality of odds.

average_odds_difference()[source]¶

Average of difference in FPR and TPR for unprivileged and privileged groups:

\[\tfrac{1}{2}\left[(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}) + (TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}))\right]\]

A value of 0 indicates equality of odds.

between_all_groups_coefficient_of_variation()[source]¶: The between-group coefficient of variation is two times the square root of the between_all_groups_generalized_entropy_index() with \(\alpha = 2\).

between_all_groups_generalized_entropy_index(alpha=2)[source]¶

Between-group generalized entropy index that uses all combinations of groups based on self.dataset.protected_attributes. See _between_group_generalized_entropy_index().

Parameters:	alpha (int) – See `generalized_entropy_index()`.

between_all_groups_theil_index()[source]¶: The between-group Theil index is the between_all_groups_generalized_entropy_index() with \(\alpha = 1\).

between_group_coefficient_of_variation()[source]¶: The between-group coefficient of variation is two times the square root of the between_group_generalized_entropy_index() with \(\alpha = 2\).

between_group_generalized_entropy_index(alpha=2)[source]¶

Between-group generalized entropy index that uses self.privileged_groups and self.unprivileged_groups as the only two groups. See _between_group_generalized_entropy_index().

Parameters:	alpha (int) – See `generalized_entropy_index()`.

between_group_theil_index()[source]¶: The between-group Theil index is the between_group_generalized_entropy_index() with \(\alpha = 1\).

binary_confusion_matrix(privileged=None)[source]¶

Compute the number of true/false positives/negatives, optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Returns:	Number of true positives, false positives, true negatives, false negatives (optionally conditioned).
Return type:	dict

coefficient_of_variation()[source]¶: The coefficient of variation is two times the square root of the generalized_entropy_index() with \(\alpha = 2\).

disparate_impact()[source]¶: \[\frac{Pr(\hat{Y} = 1 | D = \text{unprivileged})} {Pr(\hat{Y} = 1 | D = \text{privileged})}\]

equal_opportunity_difference()[source]¶: Alias of true_positive_rate_difference().

error_rate(privileged=None)[source]¶

\(ERR = (FP + FN)/(P + N)\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

error_rate_difference()[source]¶: Difference in error rates for unprivileged and privileged groups, \(ERR_{D = \text{unprivileged}} - ERR_{D = \text{privileged}}\).

error_rate_ratio()[source]¶: Ratio of error rates for unprivileged and privileged groups, \(\frac{ERR_{D = \text{unprivileged}}}{ERR_{D = \text{privileged}}}\).

false_discovery_rate(privileged=None)[source]¶

\(FDR = FP/(TP + FP)\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

false_discovery_rate_difference()[source]¶: \(FDR_{D = \text{unprivileged}} - FDR_{D = \text{privileged}}\)

false_discovery_rate_ratio()[source]¶: \(\frac{FDR_{D = \text{unprivileged}}}{FDR_{D = \text{privileged}}}\)

false_negative_rate(privileged=None)[source]¶

\(FNR = FN/P\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

false_negative_rate_difference()[source]¶: \(FNR_{D = \text{unprivileged}} - FNR_{D = \text{privileged}}\)

false_negative_rate_ratio()[source]¶: \(\frac{FNR_{D = \text{unprivileged}}}{FNR_{D = \text{privileged}}}\)

false_omission_rate(privileged=None)[source]¶

\(FOR = FN/(TN + FN)\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

false_omission_rate_difference()[source]¶: \(FOR_{D = \text{unprivileged}} - FOR_{D = \text{privileged}}\)

false_omission_rate_ratio()[source]¶: \(\frac{FOR_{D = \text{unprivileged}}}{FOR_{D = \text{privileged}}}\)

false_positive_rate(privileged=None)[source]¶

\(FPR = FP/N\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

false_positive_rate_difference()[source]¶: \(FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}\)

false_positive_rate_ratio()[source]¶: \(\frac{FPR_{D = \text{unprivileged}}}{FPR_{D = \text{privileged}}}\)

generalized_binary_confusion_matrix(privileged=None)[source]¶

Compute the number of generalized true/false positives/negatives, optionally conditioned on protected attributes. Generalized counts are based on scores and not on the hard predictions.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Returns:	Number of generalized true positives, generalized false positives, generalized true negatives, generalized false negatives (optionally conditioned).
Return type:	dict

generalized_entropy_index(alpha=2)[source]¶

Generalized entropy index is proposed as a unified individual and group fairness measure in [3]. With \(b_i = \hat{y}_i - y_i + 1\):

\[\begin{split}\mathcal{E}(\alpha) = \begin{cases} \frac{1}{n \alpha (\alpha-1)}\sum_{i=1}^n\left[\left(\frac{b_i}{\mu}\right)^\alpha - 1\right],& \alpha \ne 0, 1,\\ \frac{1}{n}\sum_{i=1}^n\frac{b_{i}}{\mu}\ln\frac{b_{i}}{\mu},& \alpha=1,\\ -\frac{1}{n}\sum_{i=1}^n\ln\frac{b_{i}}{\mu},& \alpha=0. \end{cases}\end{split}\]

Parameters:	alpha (int) – Parameter that regulates the weight given to distances between values at different parts of the distribution.

References

[3]

T. Speicher, H. Heidari, N. Grgic-Hlaca, K. P. Gummadi, A. Singla, A. Weller, and M. B. Zafar, “A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual and Group Unfairness via Inequality Indices,” ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2018.

generalized_false_negative_rate(privileged=None)[source]¶

\(GFNR = GFN/P\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

generalized_false_positive_rate(privileged=None)[source]¶

\(GFPR = GFP/N\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

generalized_true_negative_rate(privileged=None)[source]¶

\(GTNR = GTN/N\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

generalized_true_positive_rate(privileged=None)[source]¶

Return the ratio of generalized true positives to positive examples in the dataset, \(GTPR = GTP/P\), optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

negative_predictive_value(privileged=None)[source]¶

\(NPV = TN/(TN + FN)\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_false_negatives(privileged=None)[source]¶

\(FN = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_false_positives(privileged=None)[source]¶

\(FP = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_generalized_false_negatives(privileged=None)[source]¶

Return the generalized number of false negatives, \(GFN\), the weighted sum of predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_generalized_false_positives(privileged=None)[source]¶

Return the generalized number of false positives, \(GFP\), the weighted sum of predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be must be provided at initialization to condition on them.

num_generalized_true_negatives(privileged=None)[source]¶

Return the generalized number of true negatives, \(GTN\), the weighted sum of predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_generalized_true_positives(privileged=None)[source]¶

Return the generalized number of true positives, \(GTP\), the weighted sum of predicted scores where true labels are ‘favorable’, optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_pred_negatives(privileged=None)[source]¶

\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{unfavorable}]\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_pred_positives(privileged=None)[source]¶

\(\sum_{i=1}^n \mathbb{1}[\hat{y}_i = \text{favorable}]\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_true_negatives(privileged=None)[source]¶

\(TN = \sum_{i=1}^n \mathbb{1}[y_i = \text{unfavorable}]\mathbb{1}[\hat{y}_i = \text{unfavorable}]\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

num_true_positives(privileged=None)[source]¶

Return the number of instances in the dataset where both the predicted and true labels are ‘favorable’, \(TP = \sum_{i=1}^n \mathbb{1}[y_i = \text{favorable}]\mathbb{1}[\hat{y}_i = \text{favorable}]\), optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

performance_measures(privileged=None)[source]¶

Compute various performance measures on the dataset, optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Returns:	True positive rate, true negative rate, false positive rate, false negative rate, positive predictive value, negative predictive value, false discover rate, false omission rate, and accuracy (optionally conditioned).
Return type:	dict

positive_predictive_value(privileged=None)[source]¶

\(PPV = TP/(TP + FP)\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

power(privileged=None)[source]¶: Alias of num_true_positives().

precision(privileged=None)[source]¶: Alias of positive_predictive_value().

recall(privileged=None)[source]¶: Alias of true_positive_rate().

selection_rate(privileged=None)[source]¶

\(Pr(\hat{Y} = \text{favorable})\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

sensitivity(privileged=None)[source]¶: Alias of true_positive_rate().

specificity(privileged=None)[source]¶: Alias of true_negative_rate().

statistical_parity_difference()[source]¶: \[Pr(\hat{Y} = 1 | D = \text{unprivileged}) - Pr(\hat{Y} = 1 | D = \text{privileged})\]

theil_index()[source]¶: The Theil index is the generalized_entropy_index() with \(\alpha = 1\).

true_negative_rate(privileged=None)[source]¶

\(TNR = TN/N\)

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

true_positive_rate(privileged=None)[source]¶

Return the ratio of true positives to positive examples in the dataset, \(TPR = TP/P\), optionally conditioned on protected attributes.

Parameters:	privileged (bool, optional) – Boolean prescribing whether to condition this metric on the privileged_groups, if True, or the unprivileged_groups, if False. Defaults to None meaning this metric is computed over the entire dataset.
Raises:	`AttributeError` – privileged_groups or unprivileged_groups must be provided at initialization to condition on them.

true_positive_rate_difference()[source]¶: \(TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}\)

Sample Distortion Metric¶

class aif360.metrics.SampleDistortionMetric(dataset, distorted_dataset, unprivileged_groups=None, privileged_groups=None)[source]¶

Class for computing metrics based on two StructuredDatasets.

Parameters:

dataset (StructuredDataset) – A StructuredDataset.
distorted_dataset (StructuredDataset) – A StructuredDataset.
privileged_groups (list(dict)) – Privileged groups. Format is a list of dicts where the keys are protected_attribute_names and the values are values in protected_attributes. Each dict element describes a single group. See examples for more details.
unprivileged_groups (list(dict)) – Unprivileged groups in the same format as privileged_groups.

Raises:

TypeError – dataset and distorted_dataset must be StructuredDataset types.

euclidean_distance(privileged=None, returned=False)[source]¶: Compute the average Euclidean distance between the samples from the two datasets.

mahalanobis_distance(privileged=None, returned=False)[source]¶: Compute the average Mahalanobis distance between the samples from the two datasets.

manhattan_distance(privileged=None, returned=False)[source]¶: Compute the average Manhattan distance between the samples from the two datasets.

mean_euclidean_distance_difference(privileged=None)[source]¶: Difference of the averages.

mean_euclidean_distance_ratio(privileged=None)[source]¶: Ratio of the averages.

mean_mahalanobis_distance_difference(privileged=None)[source]¶: Difference of the averages.

mean_mahalanobis_distance_ratio(privileged=None)[source]¶: Ratio of the averages.

mean_manhattan_distance_difference(privileged=None)[source]¶: Difference of the averages.

mean_manhattan_distance_ratio(privileged=None)[source]¶: Ratio of the averages.

Utility Functions¶

This is the helper script for implementing metrics.

aif360.metrics.utils.compute_boolean_conditioning_vector(X, feature_names, condition=None)[source]¶

Compute the boolean conditioning vector.

Parameters:	X (numpy.ndarray) – Dataset features feature_names (list) – Names of the features. condition (list(dict)) – Specifies the subset of instances we want to use. Format is a list of dicts where the keys are feature_names and the values are values in X. Elements in the list are clauses joined with OR operators while key-value pairs in each dict are joined with AND operators. See examples for more details. If None, the condition specifies the entire set of instances, X.
Returns:	Boolean conditioning vector. Shape is [n] where n is X.shape[0]. Values are True if the corresponding row satisfies the condition and False otherwise.
Return type:	numpy.ndarray(bool)

Examples

>>> condition = [{'sex': 1, 'age': 1}, {'sex': 0}]

This corresponds to (sex == 1 AND age == 1) OR (sex == 0).

aif360.metrics.utils.compute_distance(X_orig, X_distort, X_prot, feature_names, dist_fun, condition=None)[source]¶

Compute the distance element-wise for two sets of vectors.

Parameters:

X_orig (numpy.ndarray) – Original features.
X_distort (numpy.ndarray) – Distorted features. Shape must match X_orig.
X_prot (numpy.ndarray) – Protected attributes (used to compute condition). Should be same for both original and distorted.
feature_names (list) – Names of the protected features.
dist_fun (function) – Function which returns the distance (float) between two 1-D arrays (e.g. scipy.spatial.distance.euclidean()).
condition (list(dict)) – Same format as compute_boolean_conditioning_vector().

Returns:

Element-wise distances (1-D).
Condition vector (1-D).

Return type:

(numpy.ndarray(numpy.float64), numpy.ndarray(bool))

aif360.metrics.utils.compute_num_TF_PN(X, y_true, y_pred, w, feature_names, favorable_label, unfavorable_label, condition=None)[source]¶

Compute the number of true/false positives/negatives optionally conditioned on protected attributes.

Parameters:

X (numpy.ndarray) – Dataset features.
y_true (numpy.ndarray) – True label vector.
y_pred (numpy.ndarray) – Predicted label vector.
w (numpy.ndarray) – Instance weight vector - the true and predicted datasets are supposed to have same instance level weights.
feature_names (list) – names of the features.
favorable_label (float) – Value of favorable/positive label.
unfavorable_label (float) – Value of unfavorable/negative label.
condition (list(dict)) – Same format as compute_boolean_conditioning_vector().

Returns:

Number of positives/negatives (optionally conditioned).

aif360.metrics.utils.compute_num_gen_TF_PN(X, y_true, y_score, w, feature_names, favorable_label, unfavorable_label, condition=None)[source]¶

Compute the number of generalized true/false positives/negatives optionally conditioned on protected attributes. Generalized counts are based on scores and not on the hard predictions.

Parameters:

X (numpy.ndarray) – Dataset features.
y_true (numpy.ndarray) – True label vector.
y_score (numpy.ndarray) – Predicted score vector. Values range from 0 to 1. 0 implies prediction for unfavorable label and 1 implies prediction for favorable label.
w (numpy.ndarray) – Instance weight vector - the true and predicted datasets are supposed to have same instance level weights.
feature_names (list) – names of the features.
favorable_label (float) – Value of favorable/positive label.
unfavorable_label (float) – Value of unfavorable/negative label.
condition (list(dict)) – Same format as compute_boolean_conditioning_vector().

Returns:

Number of positives/negatives (optionally conditioned).

aif360.metrics.utils.compute_num_instances(X, w, feature_names, condition=None)[source]¶

Compute the number of instances, \(n\), conditioned on the protected attribute(s).

Parameters:	X (numpy.ndarray) – Dataset features. w (numpy.ndarray) – Instance weight vector. feature_names (list) – Names of the features. condition (list(dict)) – Same format as `compute_boolean_conditioning_vector()`.
Returns:	Number of instances (optionally conditioned).
Return type:	int

aif360.metrics.utils.compute_num_pos_neg(X, y, w, feature_names, label, condition=None)[source]¶

Compute the number of positives, \(P\), or negatives, \(N\), optionally conditioned on protected attributes.

Parameters:	X (numpy.ndarray) – Dataset features. y (numpy.ndarray) – Label vector. w (numpy.ndarray) – Instance weight vector. feature_names (list) – Names of the features. label (float) – Value of label (unfavorable/positive or unfavorable/negative). condition (list(dict)) – Same format as `compute_boolean_conditioning_vector()`.
Returns:	Number of positives/negatives (optionally conditioned)
Return type:	int

aif360.metrics¶

Dataset Metric¶

Binary Label Dataset Metric¶

Classification Metric¶

Sample Distortion Metric¶

Utility Functions¶

`aif360.metrics`¶