# `bulkdgd_reduction_*`

These commands allow the user to perform dimensionality reduction analyses on sets of data points (such as latent representations generated by the bulkdgd model).

bulkdgd provides five dedicated executables:

* `bulkdgd_reduction_pca` performs a principal component analysis (PCA).
* `bulkdgd_reduction_kpca` performs a kernel principal component analysis (KPCA).
* `bulkdgd_reduction_mds` performs a multidimensional scaling analysis (MDS).
* `bulkdgd_reduction_tsne` performs a t-distributed stochastic neighbor embedding analysis (t-SNE).
* `bulkdgd_reduction_umap` performs a UMAP analysis.

All executables expect a CSV file containing a data frame with the data points as input, where each row represents a data point. Each column should contain either the data point values along a dimension or additional information about the data points.

The executables produce three outputs:

* A CSV file containing a data frame with the results of the dimensionality reduction.
* A PKL file containing the fitted model used to perform the dimensionality reduction.
* A file containing a scatter plot of the results of the dimensionality reduction.

The executables can also take a configuration file as input, specifying the plot's aesthetics and output format. If not provided, the default configuration file for the selected executable is used to generate the plot. These default configuration files can be found in `bulkdgd/configs/plotting` and are named after the analyses they refer to.

## Parallelization

The command can be run in parallel over different inputs in different directories by using the `-ds`, `--dirs` option. The directories may be specified either by name (if they are in the current working directory) or their absolute or relative path.

* If `-ds dir1 path/to/dir2`, the program will be run in parallel in each directory using the input and configuration files in it. The names of the input files may be provided using the `-id`/`--input-data`, `-im`/`--input-model`, and `-ic`/`--input-columns` options, and the names of the configuration files may be provided using the `-cd`/`--config-file-dim-red` and `-cp`/`--config-file-plot` options. The output files and the log file for each run will be saved in the corresponding directory and named according to the file names provided in the `-oa`/`--output-analysis`, `-om`/`--output-model`, `-op`/`--output-plot`, and `-lf`/`--log-file` options.

* If `-ds file.txt`, `file.txt` the file is expected to contain a newline-separated list of either names of directories in the working directory or absolute/relative paths to directories. The names of the input files may be provided using the `-id`/`--input-data`, `-im`/`--input-model`, and `-ic`/`--input-columns` options, and the names of the configuration files may be provided using the `-cd`/`--config-file-dim-red` and `-cp`/`--config-file-plot` options. The output files and the log file for each run will be saved in the corresponding directory and named according to the file names provided in the `-oa`/`--output-analysis`, `-om`/`--output-model`, `-op`/`--output-plot`, and `-lf`/`--log-file` options. `file.txt` can, for instance, look like this:

    .. code-block::

       dir1
       dir2
       absolute/path/to/dir3
       ..relative/path/to/dir4
       ...
       ...

## Command lines

```
bulkdgd_reduction_pca [-h] -id INPUT_DATA [-im INPUT_MODEL] [-ic INPUT_COLUMNS] [-oa OUTPUT_ANALYSIS] [-om OUTPUT_MODEL] [-op OUTPUT_PLOT] [-cd CONFIG_FILE_DIM_RED] [-cp CONFIG_FILE_PLOT] [-fp FILL_POS_INF] [-fn FILL_NEG_INF] [-gc GROUPS_COLUMN] [-gr GROUPS] [-pg] [-d WORK_DIR] [-lf LOG_FILE] [-lc] [-v] [-vv] [-p] [-n N_PROC] [-ds DIRS [DIRS ...]]
```

```
bulkdgd_reduction_kpca [-h] -id INPUT_DATA [-im INPUT_MODEL] [-ic INPUT_COLUMNS] [-oa OUTPUT_ANALYSIS] [-om OUTPUT_MODEL] [-op OUTPUT_PLOT] [-cd CONFIG_FILE_DIM_RED] [-cp CONFIG_FILE_PLOT] [-fp FILL_POS_INF] [-fn FILL_NEG_INF] [-gc GROUPS_COLUMN] [-gr GROUPS] [-pg] [-d WORK_DIR] [-lf LOG_FILE] [-lc] [-v] [-vv] [-p] [-n N_PROC] [-ds DIRS [DIRS ...]]
```

```
bulkdgd_reduction_mds [-h] -id INPUT_DATA [-im INPUT_MODEL] [-ic INPUT_COLUMNS] [-oa OUTPUT_ANALYSIS] [-om OUTPUT_MODEL] [-op OUTPUT_PLOT] [-cd CONFIG_FILE_DIM_RED] [-cp CONFIG_FILE_PLOT] [-fp FILL_POS_INF] [-fn FILL_NEG_INF] [-gc GROUPS_COLUMN] [-gr GROUPS] [-pg] [-d WORK_DIR] [-lf LOG_FILE] [-lc] [-v] [-vv] [-p] [-n N_PROC] [-ds DIRS [DIRS ...]]
```

```
bulkdgd_reduction_tsne [-h] -id INPUT_DATA [-im INPUT_MODEL] [-ic INPUT_COLUMNS] [-oa OUTPUT_ANALYSIS] [-om OUTPUT_MODEL] [-op OUTPUT_PLOT] [-cd CONFIG_FILE_DIM_RED] [-cp CONFIG_FILE_PLOT] [-fp FILL_POS_INF] [-fn FILL_NEG_INF] [-gc GROUPS_COLUMN] [-gr GROUPS] [-pg] [-d WORK_DIR] [-lf LOG_FILE] [-lc] [-v] [-vv] [-p] [-n N_PROC] [-ds DIRS [DIRS ...]]
```

```
bulkdgd_reduction_umap [-h] -id INPUT_DATA [-im INPUT_MODEL] [-ic INPUT_COLUMNS] [-oa OUTPUT_ANALYSIS] [-om OUTPUT_MODEL] [-op OUTPUT_PLOT] [-cd CONFIG_FILE_DIM_RED] [-cp CONFIG_FILE_PLOT] [-fp FILL_POS_INF] [-fn FILL_NEG_INF] [-gc GROUPS_COLUMN] [-gr GROUPS] [-pg] [-d WORK_DIR] [-lf LOG_FILE] [-lc] [-v] [-vv] [-p] [-n N_PROC] [-ds DIRS [DIRS ...]]
```

## Options (all executables)

### Help options

| Option         | Description                     |
| -------------- | ------------------------------- |
| `-h`, `--help` | Show the help message and exit. |

### Input options

| Option                   | Description                                                  |
| ------------------------ | ------------------------------------------------------------ |
| `-id`, `--input-data`    | The input CSV file containing the data frame with the data points. |
| `-im`, `--input-model`   | The input PKL file containing the fitted model on which to project the new data points. |
| `-ic`, `--input-columns` | A comma-separated list of columns or a string representing a pattern matching the columns of interest. These will be the columns considered when performing the dimensionality reduction analysis. By default, all columns are considered. |

### Output files

| Option                     | Description                                                  |
| -------------------------- | ------------------------------------------------------------ |
| `-oa`, `--output-analysis` | The name of the output CSV file containing the results of the dimensionality reduction analysis. The default file name changes according to the executable used. |
| `-om`, `--output-model`    | The name of the output pickle file containing the fitted model used to perform the dimensionality reduction analysis. The default file name changes according to the executable used. |
| `-op`, `--output-plot`     | The name of the output file containing the plot displaying the results of the dimensionality reduction analysis. The default file name changes according to the executable used. The file format and, therefore, extension are inferred from the `output` section of the configuration file for plotting. |

### Configuration files

| Option                        | Description                                                  |
| ----------------------------- | ------------------------------------------------------------ |
| `-cd`, `--config-file-dim-red` | The YAML configuration file specifying the options for the dimensionality reduction analysis. If it is a name without an extension, it is assumed to be the name of a configuration file in `$INSTALLDIR/bulkdgd/configs/dimensionality_reduction`. If not provided, the default configuration file for the analysis performed by the selected executable will be used. |
| `-cp`, `--config-file-plot`   | The YAML configuration file specifying the plot's aesthetics and output format. If it is a name without an extension, it is assumed to be the name of a configuration file in `$INSTALLDIR/bulkdgd/configs/plotting`. If not provided, the default configuration file for the analysis performed by the selected executable will be used. |

### Pre-processing options

| Option                  | Description                                                  |
| ----------------------- | ------------------------------------------------------------ |
| `-fp`, `--fill-pos-inf` | Replace all positive infinite values with the given value before performing the dimensionality reduction analysis. |
| `-fn`, `--fill-neg-inf` | Replace all negative infinite values with the given value before performing the dimensionality reduction analysis. |

### Plotting options

| Option                       | Description                                                  |
| ---------------------------- | ------------------------------------------------------------ |
| `-gc`, `--groups-column`     | The name/index of the column in the input data frame containing the groups by which the samples will be colored in the output plot. By default, the program assumes that no such column is present. |
| `-gr`, `--groups`            | A comma-separated list of groups whose data points should be plotted. By default, all groups found in the `-gc`, `--groups-column` column, if passed, will be included in the plot. Data points not belonging to these groups will not be included. However, you can use the `-pg`, `--plot-other-groups` option to plot them using different aesthetics compared to the groups of interest. |
| `-pg`, `--plot-other-groups` | Whether to plot data points from the groups not included in the `-gr`, `--groups` list. The aesthetics to plot these data points should also be defined in the configuration file for plotting. |

### Working directory options

| Option             | Description                                                  |
| ------------------ | ------------------------------------------------------------ |
| `-d`, `--work-dir` | The working directory. The default is the current working directory. |

### Logging options

| Option                    | Description                                                  |
| ------------------------- | ------------------------------------------------------------ |
| `-lf`, `--log-file`       | The name of the log file. The default file name depends on the selected executable. |
| `-lc`, `--log-console`    | Show log messages also on the console. |
| `-v`, `--logging-verbose` | Enable verbose logging (INFO level). |
| `-vv`, `--logging-debug`  | Enable maximally verbose logging for debugging purposes (DEBUG level). |

### Parallelization options

| Option                | Description                                                  |
| --------------------- | ------------------------------------------------------------ |
| `-p`, `--parallelize` | Whether to run the command in parallel. |
| `-n`, `--n-proc`      | The number of processes to start. The default number of processes started is 1. |
| `-ds`, `--dirs`       | The directories containing the input/configuration files. It can be either a list of names or paths, a pattern that the names or paths match, or a plain text file containing the names of or the paths to the directories. If names are given, the directories are assumed to be inside the working directory. If paths are given, they are assumed to be relative to the working directory. |

## Example

```
bulkdgd_reduction_umap -id representations.csv -ic "0,1,...,29" -gc tissue
```

This performs a UMAP analysis on the first 30 columns (columns `0` to `29`, for instance the latent space's dimensions) of `representations.csv`, coloring the resulting scatter plot by the values found in the `tissue` column. Replacing `bulkdgd_reduction_umap` with any of the other four executables (`bulkdgd_reduction_pca`, `_kpca`, `_mds`, `_tsne`) performs the corresponding analysis instead, using the same options.