Random seeds and reproducible results in PyTorch

Many a time, while building and evaluating neural network based models in PyTorch, I found that the results can be sensitive to the seed values used. This becomes a problem when you have multiple components in your model and you are trying to analyze the effects of adding or deleting those components in your final model. For a fair comparison, one needs to make sure that the performance difference that one observes by adding or deleting these components is not influenced by the changes in the seed values. In this article, I will talk about random seeds and their effects and how to obtain reproducible results in PyTorch.

Look at the following snippet of code

>>> import torch
>>> torch.rand(1,3)
tensor([[0.5056, 0.5815, 0.5133]])
>>> torch.rand(1,3)
tensor([[0.7061, 0.9229, 0.7290]])
>>> torch.rand(1,3)
tensor([[0.2745, 0.9491, 0.1396]])

Every time we run the same command (rand) with the same arguments (1,3), we get a different set of numbers. This is the expected behavior since we are asking the system to give us ‘random’ numbers and not deterministic numbers and the state of the random number generator (RNG) changes with every call to the function. To check the state of the RNG, we can use the command ‘torch.get_rng_state()’. So, if we check this command in between every call to the function, we can see that the state of the RNG has changed.

>>> import torch
>>> t1 = torch.get_rng_state()
>>> torch.rand(1,3)
tensor([[0.5817, 0.8917, 0.7106]])
>>> t2 = torch.get_rng_state()
>>> torch.equal(t1,t2)
False

Now, let us see if it is possible to obtain a deterministic set of random numbers (this itself is an oxymoron, but we need to understand how to do this in order to get a better sense of the influence of random numbers.). In order to do so, we need to ‘set’ the seed.

>>> import torch
>>> random_seed = 1
>>> torch.manual_seed(random_seed)
<torch._C.Generator object at 0x7ff97e751890>
>>> torch.rand(1,3)
tensor([[0.7576, 0.2793, 0.4031]])
>>> torch.manual_seed(random_seed)
<torch._C.Generator object at 0x7ff97e751890>
>>> torch.rand(1,3)
tensor([[0.7576, 0.2793, 0.4031]])

We can see that the function ‘rand’ has produced exactly the same numbers. This is because we set the seed to the same value in between both the function calls. Also, we have the function ‘torch.set_rng_state()’ to set the RNG to a per-determined state (can be a state obtained using ‘torch.get_rng_state()).

Now, lets look at how the seeds can affect the neural network layer initialization.

>>> import torch
>>> w = torch.empty(3,5) # an empty weight matrix
>>> torch.nn.init.xavier_normal_(w)
tensor([[ 0.2818,  0.5715,  0.4295,  0.3528, -0.1703],
        [-0.6360, -0.5974,  0.0125, -0.3813,  0.6985],
        [-0.1623,  0.1439,  0.5289,  0.4810,  0.1967]])
>>> torch.nn.init.xavier_normal_(w)
tensor([[ 0.5661, -0.2702, -1.1051,  1.0565, -0.0020],
        [ 0.6900, -0.6753,  0.1727,  0.2523,  0.9107],
        [-0.0907, -0.4757,  0.2029, -0.7582,  0.3661]])
>>> torch.nn.init.xavier_normal_(w)
tensor([[ 1.1410, -0.6040,  0.5560,  1.1087, -0.2134],
        [ 0.5073, -0.0910,  0.3091,  0.0196,  0.4631],
        [-0.4798,  0.1759, -0.2020,  0.8792, -0.2073]])
>>> random_seed = 1
>>> torch.manual_seed(random_seed)
<torch._C.Generator object at 0x7ff97e751890>
>>> torch.nn.init.xavier_normal_(w)
tensor([[ 0.3307,  0.1335,  0.0308,  0.3107, -0.2260],
        [-0.0831, -0.7614,  0.1908, -0.5138, -0.2815],
        [-0.4461, -0.0291, -0.0978, -0.4828,  0.2112]])
>>> torch.manual_seed(random_seed)
<torch._C.Generator object at 0x7ff97e751890>
>>> torch.nn.init.xavier_normal_(w)
tensor([[ 0.3307,  0.1335,  0.0308,  0.3107, -0.2260],
        [-0.0831, -0.7614,  0.1908, -0.5138, -0.2815],
        [-0.4461, -0.0291, -0.0978, -0.4828,  0.2112]])

In PyTorch, we have several options for layer initialization (https://pytorch.org/docs/stable/nn.init.html) and I took Xavier Normal as an example. Here, we can see that the initial weight values obtained depends on the state of the RNG, which is why when I set the seed before every call to the init function, I can get the same initial weight values.

Now that we have seen the effects of seed and the state of random number generator, we can look at how to obtain reproducible results in PyTorch. The following code snippet is a standard one that people use to obtain reproducible results in PyTorch.

>>> import torch
>>> random_seed = 1 # or any of your favorite number 
>>> torch.manual_seed(random_seed)
>>> torch.cuda.manual_seed(random_seed)
>>> torch.backends.cudnn.deterministic = True
>>> torch.backends.cudnn.benchmark = False
>>> import numpy as np
>>> np.random.seed(random_seed)

You can read more about it in https://pytorch.org/docs/stable/notes/randomness.html .