reproducibility - seeding a run#

Seed the generators the package draws from, so that a run can be repeated.

bulkdgd.reproducibility.set_seeds(seed: int, deterministic: bool = False) → dict#

Seed every generator a run of the package draws from.

A model is built before it is trained, and the building is already random: the decoder’s weights are drawn, the Gaussian mixture model’s components are placed, and the representations are initialized. So this has to be called before the model is constructed, and not before it is trained - by then the decoder’s weights have already been drawn from an unseeded generator.

What draws from what:

  • The decoder’s weights come from torch’s global generator, through torch.nn.Linear’s own initialization.

  • The representations of the training and the test samples are initialized with torch.randn(), from the same generator.

  • The noise added to the representations during the training comes from it as well.

  • The data loader shuffles the training samples with it, unless it is given a generator of its own.

  • The Gaussian mixture model places its components with it, when its random_state is left unset. When it is set, tgmm calls torch.manual_seed() itself, which reseeds the global generator for everything that comes after it - so it is better to leave random_state unset and to seed here, once.

  • numpy and random are seeded too. Little is drawn from them, but ‘little’ is not ‘nothing’, and a generator that is not seeded is a generator that makes a run unrepeatable.

Parameters:
seedint

The seed.

deterministicbool, False

Whether to also ask torch for deterministic algorithms.

Seeding makes a run repeat itself only if the operations it runs are themselves deterministic, and several are not: an operation that sums with atomics on a GPU adds its terms in whatever order the threads finish, and floating-point addition is not associative, so the same seed gives a slightly different number. This turns those operations into their deterministic versions where they have one, and makes them raise where they do not.

It is off by default because it is slower, and because an operation without a deterministic version raises rather than falls back - which is the right behaviour when reproducibility is what is being asked for, and the wrong one when it is not.

Returns:
seedsdict

What was seeded and with what, to be recorded with the run’s results. A seed that is not written down is a seed nobody has.

bulkdgd.reproducibility.get_seeds_state() → dict#

Get the state of the generators, as a set of numbers that can be written down.

This is not the state itself - the state of a Mersenne twister is 624 words, and writing it into a results file helps nobody. It is enough to tell two runs apart, and to notice that a run which was supposed to have been seeded was not.

Returns:
statedict

A number per generator.

Seeding has to happen before the model is built#

bulkdgd.reproducibility.set_seeds() must be called before bulkdgd.core.model.BulkDGD is constructed, and not merely before it is trained.

Constructing the model is already random. torch.nn.Linear draws its weights in its own __init__, through reset_parameters, so by the time BulkDGD(...) returns, every weight of the decoder is a number that has already been drawn from whatever state the generator happened to be in. Seeding afterwards seeds nothing that has already happened, and a run seeded that way looks seeded - the log says so - while reproducing nothing.

import bulkdgd
from bulkdgd.core.model import BulkDGD

# Right: the decoder's weights are drawn from a seeded generator.
bulkdgd.set_seeds(42)
model = BulkDGD(**config_model)

# Wrong: the weights were drawn before the seed was set.
model = BulkDGD(**config_model)
bulkdgd.set_seeds(42)

Building a model also consumes the generator’s stream, so anything drawn afterwards - the representations’ initialization, the order of the batches, the noise added during training - continues from where the decoder left off. This is why the seed is set once, at the top, rather than before each thing that draws.

The same applies to torch.set_default_dtype(): a model built before the default dtype is changed is built in the old one.

What is seeded#

bulkdgd.reproducibility.set_seeds() seeds torch’s global generator (the decoder’s weights, the representations’ initialization, the training noise, and the data loader’s shuffling), the generators of every GPU, numpy’s global generator (which is what a random_state of None falls back on in anything scikit-learn-shaped), and the standard library’s random.

It returns a record of what it seeded, which is meant to be written down beside the run’s results. A seed that is set and not recorded is a seed nobody has.

A note on the Gaussian mixture model’s random_state#

"random_state" in the model’s "latent_options" (see Configuration for creating an instance of the BulkDGD model) reaches further than its name suggests when the latent space is a tgmm: that implementation calls torch.manual_seed() with it when it places the mixture’s components, which reseeds the global generator for everything that comes after. It is therefore usually better to leave it unset and to call bulkdgd.reproducibility.set_seeds() once, which makes the placement reproducible along with everything else.

What seeding does not fix#

A seeded run repeats only if the operations it runs are themselves deterministic, and several are not. A sum reduced with atomics on a GPU adds its terms in whatever order its threads finish in, and floating-point addition is not associative, so the same numbers can add up to slightly different numbers. Training is chaotic: a difference in the last bits of an early epoch is a different model many epochs later.

deterministic = True asks torch for the deterministic version of those operations, and makes it raise where there is not one. It is off by default because it is slower, and because raising is the right behaviour only when reproducibility is what is being asked for.

Note that CUBLAS_WORKSPACE_CONFIG has to be set in the environment before cuBLAS is initialized - which is to say before the first matrix multiplication, and not merely before bulkdgd.reproducibility.set_seeds() is called. bulkdgd.reproducibility.set_seeds() sets it if it is not already set, which is too late if something has already used cuBLAS. Set it in the environment of the job to be sure of it.