core.model#

This module contains the class implementing the full BulkDGD model (core.model.BulkDGG).

class bulkdgd.core.model.BulkDGD(latent_dim: int, latent_options: dict[str, object], decoder_options: dict[str, object], latent_type: str = 'tgmm', genes_txt_file: str | None = None, scaling_factor: str = 'mean', dtype: str = 'float32', device: str = 'cpu')#

Class implementing the BulkDGD model.

static rescale_pred_means(df_pred_means: DataFrame, df_pred_r_values: DataFrame) → DataFrame#

Rescale the means of the negative binomials modeling the genes’ counts.

Parameters:
df_pred_meanspandas.DataFrame

A data frame containing the predicted scaled means of the negative binomials modeling the genes’ counts.

Here, each row contains the scaled mean for a given representation/sample, and the columns contain either the values of the scaled means or additional information.

The columns containing the scaled means must be named after the corresponding genes’ Ensembl IDs.

df_pred_r_valuespandas.DataFrame

A data frame containing the predicted r-values of the negative binomials modeling the genes’ counts.

Here, each row contains the r-value for a given representation/sample, and the columns contain either the r-values or additional information.

The columns containing the r-values must be named after the corresponding genes’ Ensembl IDs.

Returns:
df_scaledpandas.DataFrame

A data frame containing the predicted means.

It contains the same columns of the df_pred_means data frame, in the same order they appear in the df_pred_means data frame.

However, the values in the columns containing the predicted means are scaled back by the corresponding r-values.

__init__(latent_dim: int, latent_options: dict[str, object], decoder_options: dict[str, object], latent_type: str = 'tgmm', genes_txt_file: str | None = None, scaling_factor: str = 'mean', dtype: str = 'float32', device: str = 'cpu') → None#

Initialize an instance of the class.

The model is initialized on the CPU. To move the model to another device, modify the device property.

Parameters:
latent_dimint

The dimensionality of the latent space.

latent_typestr, {"lgmm", "tgmm"}, "tgmm"

The type of the latent space to use.

The available options are:

latent_optionsdict

The options for setting up the latent space.

For the available options, refer to the Configuration for creating an instance of the BulkDGD model page.

decoder_optionsdict

The options for setting up the decoder.

For the available options, refer to the Configuration for creating an instance of the BulkDGD model page.

genes_txt_filestr

A plain text file containing the Ensembl IDs of the genes included in the model.

Training data will be checked to ensure counts are reported for all genes.

The number of output units in the decoder is initialized from the number of genes found in this file.

scaling_factorstr, {"mean", "median"}, "mean"

How to compute the scaling factor of a sample - the number the decoder’s predicted means are multiplied by to put them on the scale of the sample’s own counts.

The mean is not robust to the few genes that take a large and variable share of a library: in GTEx the thirteen mitochondrial genes take 14.49% of the reads, and their share runs from 0.10% to 90.85% between samples, which moves the mean by up to a factor of ten. The median is moved by at most 3.7% by the same genes. See bulkdgd.core.dataclasses.GeneExpressionDataset.

This belongs to the model and not to a single run of it. The median is about a third of the mean, and the decoder’s output is fitted against whichever the model was trained with, so a model trained with one and used with the other has its predicted means wrong by about a factor of three - without failing. It is therefore written in the model’s configuration file, so that finding representations, imputing, and the differential expression analysis all read the same value the training did.

dtypestr, {"float32", "float64"}, "float32"

The precision the model’s parameters are built in.

This is not a preference that can be applied afterwards. A module’s parameters are made in whatever torch’s default dtype is at the moment the module is constructed, and load_state_dict copies a checkpoint INTO the parameters that are already there, casting as it goes - so a float64 checkpoint read into a model built in float32 gives a float32 decoder, and nothing says so.

The Gaussian mixture does not go quietly, which is the only piece of luck in it: tgmm keeps the tensors it is handed rather than copying into its own, so it stays float64 while the decoder becomes float32, and the first matrix multiply of the two raises mat1 and mat2 must have the same dtype.

Torch’s default dtype is put back to what it was once the model is built: it is global, and a model asked for in double is not a reason for the rest of the program to be in double.

devicestr, "cpu"

The device where the model will be initialized. The model is initialized on the CPU by default.

get_probability_density(df_rep: DataFrame) → DataFrame#

Given a set of representations, get the probability density of each component of the Gaussian mixture model for each representation and the representation(s) having the maximum probability density for each component.

Parameters:
df_reppandas.DataFrame

A data frame containing the representations.

Returns:
df_prob_reppandas.DataFrame

A data frame containing the probability densities for each representation, together with an indication of what the maximum probability density found is and for which component it is found.

df_prob_comppandas.DataFrame

A data frame containing, for each component, the representation(s) having the maximum probability density for the component, together with the probability density for that(those) representation(s).

get_representations(df_samples: DataFrame, config_rep: dict[str, object], get_saliency_map: bool = False, genes_mask: Tensor | None = None) → tuple[DataFrame, DataFrame, DataFrame | None, DataFrame]#

Find the best representations for a set of samples.

Parameters:
df_samplespandas.DataFrame

A data frame containing the samples.

genes_masktorch.Tensor, optional

A 2D tensor of 1.0 and 0.0, one row per sample and one column per gene, saying which of a sample’s genes were MEASURED. If not passed, all of them were, which is what a whole transcriptome is.

Most callers want impute() instead, which builds the mask from the missing values of the data frame and returns the imputed counts. This argument is here for the caller who wants the representation itself.

config_repdict

A dictionary of options for the optimization(s). It varies according to the selected method.

The supported options for all available methods can be found here.

get_saliency_mapbool, optional

Whether to also compute and return the saliency maps showing the importance of each latent dimension for each gene’s expression based on the obtained representations. Default: False.

Returns:
df_reppandas.DataFrame

A data frame containing the representations.

Here, each row contains a representation and the columns contain either the values of the representations’ along the latent space’s dimensions or additional information about the input samples found in the input data frame. Columns containing additional information, if present in the input data frame, will appear last in the data frame.

df_pred_meanspandas.DataFrame

A data frame containing the predicted means of the distributions modelling the genes’ counts for the representations found.

Here, each row contains the predicted means for a given representation, and the columns contain either the mean of a distribution or additional information about the input samples found in the input data frame. Columns containing additional information, if present in the input data frame, will appear last in the data frame.

If the genes counts are modelled using negative binomial distributions, the predicted means are scaled by the corresponding distributions’ r-values.

df_pred_r_valuespandas.DataFrame, optional

A data frame containing the predicted r-values of the negative binomials for the representations found, if the genes’ counts are modelled by negative binomial distributions

Here, each row contains the predicted r-values for a given representation, and the columns contain either the r-value of a negative binomial or additional information about the input samples found in the input data frame. Columns containing additional information, if present in the input data frame, will appear last in the data frame.

df_pred_r_values is None if the genes’ counts are modelled by Poisson distributions.

df_timepandas.DataFrame

A data frame containing data about the CPU and wall clock time used by each epoch (and backpropagation step within each epoch) in each optimization step.

Here, each row represents an epoch of an optimization step, and the columns contain data about the platform where the calculation was run, the number of CPU threads used by the computation, and the CPU and wall clock time used by the entire epoch and by the backpropagation step run inside it.

df_saliency_mappandas.DataFrame, optional

A data frame containing the gradients indicating the importance of each latent dimension for each gene’s expression.

Here, each row is a gene (indexed by ENSG naming) and columns correspond to each latent dimension. Returned as an element of a tuple uniquely when get_saliency_map is True.

impute(df_samples: DataFrame, config_rep: dict[str, object], genes_measured: list[str] | None = None, quantiles: tuple[float, float] = (0.025, 0.975)) → tuple[DataFrame, DataFrame, DataFrame, DataFrame, DataFrame]#

Predict the counts of the genes a sample does not have.

A sample only part of whose transcriptome was read - a gene panel, a targeted assay - is given, the genes that were read are used to find its representation, and the decoder is asked for the genes that were not.

THE MISSING GENES MUST BE MISSING, and not zero. A gene whose count is numpy.nan was never measured; a gene whose count is 0 was measured and found to be silent. They are different facts, and the difference is the whole of this method: a zero is evidence, and the model will move a sample’s representation in order to explain it. Hand it a panel with the unmeasured genes written as zeros and it will conclude that the sample has switched off nine tenths of its transcriptome - and it will conclude it about the genes that WERE measured too, because the representation is one thing and it is fitted to all of them at once.

So the unmeasured genes are taken out of the likelihood altogether, which for a factorized likelihood is exactly what conditioning on the genes that were measured means, and the scaling factor of the negative binomials - which is otherwise the mean count over ALL of the genes, and would be deflated by every gene that is not there - is estimated from the measured genes alone.

Parameters:
df_samplespandas.DataFrame

The samples, with numpy.nan in the genes that were not measured.

A column of genes may be missing from the data frame entirely; it is taken to have been measured in no sample.

config_repdict

The options for finding the representations, as for get_representations().

genes_measuredlist, optional

The genes that were measured, if it is more convenient to say so than to write numpy.nan everywhere else. Every other gene is taken to be unmeasured in every sample.

quantilestuple, optional

The quantiles of the predicted negative binomial to report, which give the imputed count an interval and not only a point. Default: the central 95%.

Returns:
df_imputedpandas.DataFrame

The expected count of every gene of every sample, on that sample’s own scale - the mean of the negative binomial the model predicts for it.

The genes that WERE measured are in it as well, and are the model’s expectation for them rather than what was observed. The two should agree, and where they do not is worth looking at: it is the same quantity a differential expression analysis reports.

df_lower, df_upperpandas.DataFrame

The quantiles of the predicted distribution. An imputed count without them is a number without an error bar, and a gene the model is uncertain of looks exactly like a gene it is sure of.

df_pred_r_valuespandas.DataFrame

The r-values of those negative binomials.

With df_imputed, they are the whole predicted distribution of every gene - not a point and an interval, but the thing the point and the interval were taken from. It is what is needed to ask where an observed count LANDS in what the model expected, which is the question a differential expression analysis asks, and the question by which an imputation is honestly judged.

df_reppandas.DataFrame

The representations found from the measured genes.

train(df_samples: DataFrame, names_train: list, names_test: list, config_train: dict[str, object], gmm_pth_file: str = 'gmm.pth', dec_pth_file: str = 'dec.pth', pathways: DataFrame | None = None, labels_train: object | None = None, labels_test: object | None = None) → tuple[tuple[DataFrame, DataFrame], tuple[DataFrame, DataFrame], tuple[DataFrame, DataFrame] | None, DataFrame, tuple[DataFrame, DataFrame] | None, DataFrame]#

Train the model.

Parameters:
df_samplespandas.DataFrame

A data frame containing the samples.

Each row should contain a unique sample, and each column should either contain a gene’s expression for that sample (if the column is named after the gene’s Ensembl ID) or additional information about the sample.

names_trainlist

A list of the names of the training samples, which should be a subset of the names of the samples in the input data frame.

names_testlist

A list of the names of the test samples, which should be a subset of the names of the samples in the input data frame.

config_traindict

A dictionary of options for the training.

gmm_pth_filestr, "gmm.pth"

The .pth file where to save the GMM’s trained parameters (means of the components, weights of the components, and log-variance of the components).

dec_pth_filestr, "dec.pth"

The .pth file where to save the decoder’s trained parameters (weights and biases).

pathwaysdict, optional

A dictionary where the keys are pathway names and the values are lists of genes’ Ensembl IDs belonging to each pathway.

It is needed if save_pathways_saliency_maps_epoch is set to True.

labels_trainnumpy.ndarray, optional

The ground-truth labels for the training samples.

labels_testnumpy.ndarray, optional

The ground-truth labels for the test samples.

Returns:
dfs_reptuple

A tuple (df_rep_train, df_rep_test) with the optimized latent representations for training and testing samples.

dfs_pred_meanstuple

A tuple (df_pred_means_train, df_pred_means_test) with the predicted decoder means for training and testing samples.

dfs_pred_r_valuesNone or pandas.DataFrame or tuple

The predicted r-values, depending on the output module:

df_losspandas.DataFrame

A data frame containing the losses calculated during training.

dfs_metricsNone or tuple

The per-epoch metrics rows for training and testing samples, depending on whether the user requested to calculate metrics during training:

  • None if the user did not request to calculate metrics during training.

  • A tuple (df_metrics_train, df_metrics_test) of data frames, where each data frame contains the metrics calculated for the training or test samples in a given epoch, if the user requested to calculate metrics during training.

df_timepandas.DataFrame

A data frame containing the training-time metrics.

property decoder#

The decoder.

property device#

The device where the model is.

property genes#

The genes the model knows, in the order the decoder emits them.

property latent#

The latent space.

property scaling_factor#

How the scaling factor of a sample is computed - either "mean" or "median".

It is a property of the model: the decoder’s output is fitted against it, so everything done with the model afterwards has to use the one it was trained with.