uq_keras_utils module

class uq_keras_utils.AbstentionAdapt_Callback(acc_monitor, abs_monitor, alpha0, init_abs_epoch=4, alpha_scale_factor=0.8, min_abs_acc=0.9, max_abs_frac=0.4, acc_gain=5.0, abs_gain=1.0)

Bases: keras.callbacks.Callback

This callback is used to adapt the parameter alpha in the abstention loss. The parameter alpha (weight of the abstention term in the abstention loss) is increased or decreased adaptively during the training run. It is decreased if the current abstention accuracy is less than the minimum accuracy set or increased if the current abstention fraction is greater than the maximum fraction set. The abstention accuracy metric to use must be specified as the ‘acc_monitor’ argument in the initialization of the callback. It could be: the global abstention accuracy (abstention_acc), the abstention accuracy over the ith class (acc_class_i), etc. The abstention metric to use must be specified as the ‘abs_monitor’ argument in the initialization of the callback. It should be the metric that computes the fraction of samples for which the model is abstaining (abstention). The factor alpha is modified if the current abstention accuracy is less than the minimum accuracy set or if the current abstention fraction is greater than the maximum fraction set. Thresholds for minimum and maximum correction factors are computed and the correction over alpha is not allowed to be less or greater than them, respectively, to avoid huge swings in the abstention loss evolution.

on_epoch_end(epoch, logs=None)

Updates the weight of abstention term on epoch end. :param epoch: Current epoch in training. :type epoch: integer :param logs: Metrics stored during current keras training. :type logs: keras logs

class uq_keras_utils.Contamination_Callback(x, y, a_max=0.99)

Bases: keras.callbacks.Callback

This callback is used to update the parameters of the contamination model. This functionality follows the EM algorithm: in the E-step latent variables are updated and in the M-step global variables are updated. The global variables correspond to ‘a’ (probability of membership to normal class), ‘sigmaSQ’ (variance of normal class) and ‘gammaSQ’ (scale of Cauchy class, modeling outliers). The latent variables correspond to ‘T_k’ (the first column corresponds to the probability of membership to the normal distribution, while the second column corresponds to the probability of membership to the Cauchy distribution i.e. outlier).

on_epoch_end(epoch, logs={})

Updates the parameters of the distributions in the contamination model on epoch end. The parameters updated are: ‘a’ for the global weight of the membership to the normal distribution, ‘sigmaSQ’ for the variance of the normal distribution and ‘gammaSQ’ for the scale of the Cauchy distribution of outliers. The latent variables are updated as well: ‘T_k’ describing in the first column the probability of membership to normal distribution and in the second column probability of membership to the Cauchy distribution i.e. outlier. Stores evolution of global parameters (a, sigmaSQ and gammaSQ).

Parameters

• epoch (integer) – Current epoch in training.

• logs (keras logs) – Metrics stored during current keras training.

uq_keras_utils.abstention_acc_class_i_metric(nb_classes, class_i)

Function to estimate accuracy over the class i prediction after removing the samples where the model is abstaining.

Parameters

• nb_classes (int or ndarray) – Integer or numpy array defining indices of the abstention class

• class_i (int) – Index of the class to estimate accuracy after removing abstention samples

uq_keras_utils.abstention_acc_metric(nb_classes)

Abstained accuracy:

Function to estimate accuracy over the predicted samples after removing the samples where the model is abstaining.

Parameters

nb_classes (int or ndarray) – Integer or numpy array defining indices of the abstention class

uq_keras_utils.abstention_class_i_metric(nb_classes, class_i)

Function to estimate fraction of the samples where the model is abstaining in class i.

Parameters

• nb_classes (int or ndarray) – Integer or numpy array defining indices of the abstention class

• class_i (int) – Index of the class to estimate accuracy

uq_keras_utils.abstention_loss(alpha, mask)

Function to compute abstention loss.

It is composed by two terms: (i) original loss of the multiclass classification problem, (ii) cost associated to the abstaining samples.

Parameters

• alpha (Keras variable) – Weight of abstention term in cost function

• mask (ndarray) – Numpy array to use as mask for abstention: it is 1 on the output associated to the abstention class and 0 otherwise

uq_keras_utils.abstention_metric(nb_classes)

Function to estimate fraction of the samples where the model is abstaining.

Parameters

nb_classes (int or ndarray) – Integer or numpy array defining indices of the abstention class

uq_keras_utils.acc_class_i_metric(class_i)

Function to estimate accuracy over the ith class prediction.

This estimation is global (i.e. abstaining samples are not removed)

Parameters

class_i (int) – Index of the class to estimate accuracy

uq_keras_utils.add_index_to_output(y_train)

This function adds a column to the training output to store the indices of the corresponding samples in the training set.

Parameters

y_train (ndarray) – Numpy array of the output in the training set

uq_keras_utils.add_model_output(modelIn, mode=None, num_add=None, activation=None)

This function modifies the last dense layer in the passed keras model. The modification includes adding units and optionally changing the activation function.

Parameters

• modelIn (keras model) – Keras model to be modified.

• mode (string) – Mode to modify the layer. It could be: ‘abstain’ for adding an arbitrary number of units for the abstention optimization strategy. ‘qtl’ for quantile regression which needs the outputs to be tripled. ‘het’ for heteroscedastic regression which needs the outputs to be doubled.

• num_add (integer) – Number of units to add. This only applies to the ‘abstain’ mode.

• activation (string) – String with keras specification of activation function (e.g. ‘relu’, ‘sigomid’, ‘softmax’, etc.)

Returns

modelOut (keras model) – Keras model after last dense layer has been modified as specified. If there is no mode specified it returns the same model. If the mode is not one of ‘abstain’, ‘qtl’ or ‘het’ an exception is raised.

uq_keras_utils.contamination_loss(nout, T_k, a, sigmaSQ, gammaSQ)

Function to compute contamination loss. It is composed by two terms: (i) the loss with respect to the normal distribution that models the distribution of the training data samples, (ii) the loss with respect to the Cauchy distribution that models the distribution of the outlier samples. Note that the evaluation of this contamination loss function does not make sense for any data different to the training set. This is because latent variables are only defined for samples in the training set.

Parameters

• nout (int) – Number of outputs without uq augmentation (in the contamination model the augmentation corresponds to the data index in training).

• T_k (Keras tensor) – Tensor containing latent variables (probability of membership to normal and Cauchy distributions) for each of the samples in the training set. (Validation data is usually augmented too to be able to run training with validation set, however loss in validation should not be used as a criterion for early stopping training since the latent variables are defined for the training only, and thus, are not valid when used in combination with data different from training).

• a (Keras variable) – Probability of belonging to the normal distribution

• sigmaSQ (Keras variable) – Variance estimated for the normal distribution

• gammaSQ (Keras variable) – Scale estimated for the Cauchy distribution

uq_keras_utils.heteroscedastic_loss(nout)

This function computes the heteroscedastic loss for the heteroscedastic model. Both mean and standard deviation predictions are taken into account.

Parameters

nout (int) – Number of outputs without uq augmentation

uq_keras_utils.mae_contamination_metric(nout)

This function computes the mean absolute error (mae) for the contamination model. The mae is computed over the prediction. Therefore, the augmentation for the index variable is ignored.

Parameters

nout (int) – Number of outputs without uq augmentation (in the contamination model the augmentation corresponds to the data index in training).

uq_keras_utils.mae_heteroscedastic_metric(nout)

This function computes the mean absolute error (mae) for the heteroscedastic model. The mae is computed over the prediction of the mean and the standard deviation prediction is not taken into account.

Parameters

nout (int) – Number of outputs without uq augmentation

uq_keras_utils.meanS_heteroscedastic_metric(nout)

This function computes the mean log of the variance (log S) for the heteroscedastic model. The mean log is computed over the standard deviation prediction and the mean prediction is not taken into account.

Parameters

nout (int) – Number of outputs without uq augmentation

uq_keras_utils.modify_labels(numclasses_out, ytrain, ytest, yval=None)

This function generates a categorical representation with a class added for indicating abstention.

Parameters

• numclasses_out (integer) – Original number of classes + 1 abstention class

• ytrain (ndarray) – Numpy array of the classes (labels) in the training set

• ytest (ndarray) – Numpy array of the classes (labels) in the testing set

• yval (ndarray) – Numpy array of the classes (labels) in the validation set

uq_keras_utils.mse_contamination_metric(nout)

This function computes the mean squared error (mse) for the contamination model. The mse is computed over the prediction. Therefore, the augmentation for the index variable is ignored.

Parameters

nout (int) – Number of outputs without uq augmentation (in the contamination model the augmentation corresponds to the data index in training).

uq_keras_utils.mse_heteroscedastic_metric(nout)

This function computes the mean squared error (mse) for the heteroscedastic model. The mse is computed over the prediction of the mean and the standard deviation prediction is not taken into account.

Parameters

nout (int) – Number of outputs without uq augmentation

uq_keras_utils.quantile_loss(quantile, y_true, y_pred)

This function computes the quantile loss for a given quantile fraction.

Parameters

• quantile (float in (0, 1)) – Quantile fraction to compute the loss.

• y_true (Keras tensor) – Keras tensor including the ground truth

• y_pred (Keras tensor) – Keras tensor including the predictions of a quantile model.

uq_keras_utils.quantile_metric(nout, index, quantile)

This function computes the quantile metric for a given quantile and corresponding output index. This is provided as a metric to track evolution while training.

Parameters

• nout (int) – Number of outputs without uq augmentation

• index (int) – Index of output corresponding to the given quantile.

• quantile (float in (0, 1)) – Fraction corresponding to the quantile

uq_keras_utils.r2_contamination_metric(nout)

This function computes the r2 for the contamination model. The r2 is computed over the prediction. Therefore, the augmentation for the index variable is ignored.

Parameters

nout (int) – Number of outputs without uq augmentation (in the contamination model the augmentation corresponds to the data index in training).

uq_keras_utils.r2_heteroscedastic_metric(nout)

This function computes the r2 for the heteroscedastic model. The r2 is computed over the prediction of the mean and the standard deviation prediction is not taken into account.

Parameters

nout (int) – Number of outputs without uq augmentation

uq_keras_utils.sparse_abstention_acc_metric(nb_classes)

Abstained accuracy:

Function to estimate accuracy over the predicted samples after removing the samples where the model is abstaining. Assumes y_true is not one-hot encoded.

Parameters

nb_classes (int or ndarray) – Integer or numpy array defining indices of the abstention class

uq_keras_utils.sparse_abstention_loss(alpha, mask)

Function to compute abstention loss.

It is composed by two terms: (i) original loss of the multiclass classification problem, (ii) cost associated to the abstaining samples. Assumes y_true is not one-hot encoded.

Parameters

• alpha (Keras variable) – Weight of abstention term in cost function

• mask (ndarray) – Numpy array to use as mask for abstention: it is 1 on the output associated to the abstention class and 0 otherwise

uq_keras_utils.triple_quantile_loss(nout, lowquantile, highquantile)

This function computes the quantile loss for the median and low and high quantiles. The median is given twice the weight of the other components.

Parameters

• nout (int) – Number of outputs without uq augmentation

• lowquantile (float in (0, 1)) – Fraction corresponding to the low quantile

• highquantile (float in (0, 1)) – Fraction corresponding to the high quantile