How to load model pytorch

Saving and loading models for inference in PyTorch¶

Introduction¶

Saving the model’s state_dict with the torch.save() function will give you the most flexibility for restoring the model later. This is the recommended method for saving models, because it is only really necessary to save the trained model’s learned parameters. When saving and loading an entire model, you save the entire module using Python’s pickle module. Using this approach yields the most intuitive syntax and involves the least amount of code. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors. In this recipe, we will explore both ways on how to save and load models for inference.

Setup¶

Before we begin, we need to install torch if it isn’t already available.

Steps¶

Import all necessary libraries for loading our data

Define and intialize the neural network

Initialize the optimizer

Save and load the model via state_dict

Save and load the entire model

1. Import necessary libraries for loading our data¶

2. Define and intialize the neural network¶

For sake of example, we will create a neural network for training images. To learn more see the Defining a Neural Network recipe.

3. Initialize the optimizer¶

We will use SGD with momentum.

4. Save and load the model via state_dict ¶

Remember too, that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

5. Save and load entire model¶

Now let’s try the same thing with the entire model.

Again here, remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference.

Congratulations! You have successfully saved and load models for inference in PyTorch.

Learn More¶

Take a look at these other recipes to continue your learning:

Total running time of the script: ( 0 minutes 0.000 seconds)

Saving/Loading your model in PyTorch

Working on a Deep Learning project usually takes time, and there are many things to tweak and change over time.

Whether you “babysit” your model while training or you leave it and go do something else, It’s always a good practice to save checkpoints of your model for many reasons.

In this article, I’ll be showing you What you need to save in a checkpoint, where, and How to do it in PyTorch.

1- Best State

The ultimate goal of any learning project is to find the best model, the one the fits just right to the training set and generalizes well, So it makes sense you check every iteration if the model achieves a better score on your own metric and save it if so.

2- Latest State

Like we said, Training a model takes time.
And you may need to pause the training for any reason and continue training later without having to start over.

Also it’s possible that you lose connection to the working environment.
So you may need to save the latest state of your model every epoch of training, for you to be able to load it later and continue from where you were.

If you are working on a hosted environment it’s always better to save the model in cloud storage, to be easier for you later to load your model without having to upload it which would take time because the models are usually of big size.

Also if you plan to deploy your model in an app on the web, Saving in cloud is better because it allows you to make tweaks and changes, put your model to test and perform faster iterations.

Saving and loading a model in PyTorch is very easy and straight forward.
It’s as simple as this:

A checkpoint is a python dictionary that typically includes the following:

1- The network structure: input and output sizes and Hidden layers to be able to reconstruct the model at loading time.

2- The model state dict : includes parameters of the network layers that is learned during training, you get it by calling this method on your model instance.
model.state_dict()

3- The optimizer state dict : In case you are saving the latest checkpoint to continue training later, you need to save the optimizer’s state as well.
you get it by calling this method on an optimizer’s instance.
optimizer.state_dict()

4- Additional info: You may need to store additional info, like number of epochs and your class to index mapping in your checkpoint.

Loading is as simple as saving

1- Reconstruct the model from the structure saved in the checkpoint.

2- Load the state dict to the model.

3- Freeze the parameters and enter evaluation mode if you are loading the model for inference.

Try it yourself now!

Looks easy right? Why don’t you try it yourself now!
Here’s a link to a Kaggle kernel, a notebook that walks you into creating, training, saving and loading your model.
Fork it and run it yourself and see how easy it really is!

Saving and loading models across devices in PyTorch¶

There may be instances where you want to save and load your neural networks across different devices.

Introduction¶

Saving and loading models across devices is relatively straightforward using PyTorch. In this recipe, we will experiment with saving and loading models across CPUs and GPUs.

Setup¶

In order for every code block to run properly in this recipe, you must first change the runtime to “GPU” or higher. Once you do, we need to install torch if it isn’t already available.

Steps¶

Import all necessary libraries for loading our data

Define and intialize the neural network

Save on a GPU, load on a CPU

Save on a GPU, load on a GPU

Save on a CPU, load on a GPU

Saving and loading DataParallel models

1. Import necessary libraries for loading our data¶

2. Define and intialize the neural network¶

For sake of example, we will create a neural network for training images. To learn more see the Defining a Neural Network recipe.

3. Save on GPU, Load on CPU¶

When loading a model on a CPU that was trained with a GPU, pass torch.device(‘cpu’) to the map_location argument in the torch.load() function.

In this case, the storages underlying the tensors are dynamically remapped to the CPU device using the map_location argument.

4. Save on GPU, Load on GPU¶

5. Save on CPU, Load on GPU¶

Be sure to call model.to(torch.device(‘cuda’)) to convert the model’s parameter tensors to CUDA tensors.

6. Saving torch.nn.DataParallel Models¶

torch.nn.DataParallel is a model wrapper that enables parallel GPU utilization.

Congratulations! You have successfully saved and loaded models across devices in PyTorch.

Learn More¶

Take a look at these other recipes to continue your learning:

Total running time of the script: ( 0 minutes 0.000 seconds)

Saving and Loading Models¶

This document provides solutions to a variety of use cases regarding the saving and loading of PyTorch models. Feel free to read the whole document, or just skip to the code you need for a desired use case.

When it comes to saving and loading models, there are three core functions to be familiar with:

torch.save: Saves a serialized object to disk. This function uses Python’s pickle utility for serialization. Models, tensors, and dictionaries of all kinds of objects can be saved using this function.

torch.load: Uses pickle’s unpickling facilities to deserialize pickled object files to memory. This function also facilitates the device to load the data into (see Saving & Loading Model Across Devices).

torch.nn.Module.load_state_dict: Loads a model’s parameter dictionary using a deserialized state_dict. For more information on state_dict, see What is a state_dict?.

Contents:

In PyTorch, the learnable parameters (i.e. weights and biases) of an torch.nn.Module model are contained in the model’s parameters (accessed with model.parameters() ). A state_dict is simply a Python dictionary object that maps each layer to its parameter tensor. Note that only layers with learnable parameters (convolutional layers, linear layers, etc.) and registered buffers (batchnorm’s running_mean) have entries in the model’s state_dict. Optimizer objects ( torch.optim ) also have a state_dict, which contains information about the optimizer’s state, as well as the hyperparameters used.

Because state_dict objects are Python dictionaries, they can be easily saved, updated, altered, and restored, adding a great deal of modularity to PyTorch models and optimizers.

Example:¶

Let’s take a look at the state_dict from the simple model used in the Training a classifier tutorial.

Output:

Saving & Loading Model for Inference¶

Save/Load state_dict (Recommended)¶

Save:

Load:

When saving a model for inference, it is only necessary to save the trained model’s learned parameters. Saving the model’s state_dict with the torch.save() function will give you the most flexibility for restoring the model later, which is why it is the recommended method for saving models.

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

If you only plan to keep the best performing model (according to the acquired validation loss), don’t forget that best_model_state = model.state_dict() returns a reference to the state and not its copy! You must serialize best_model_state or use best_model_state = deepcopy(model.state_dict()) otherwise your best best_model_state will keep getting updated by the subsequent training iterations. As a result, the final model state will be the state of the overfitted model.

Save/Load Entire Model¶

Save:

Load:

This save/load process uses the most intuitive syntax and involves the least amount of code. Saving a model in this way will save the entire module using Python’s pickle module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

Export/Load Model in TorchScript Format¶

One common way to do inference with a trained model is to use TorchScript, an intermediate representation of a PyTorch model that can be run in Python as well as in a high performance environment like C++. TorchScript is actually the recommended model format for scaled inference and deployment.

Using the TorchScript format, you will be able to load the exported model and run inference without defining the model class.

Export:

Load:

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

For more information on TorchScript, feel free to visit the dedicated tutorials. You will get familiar with the tracing conversion and learn how to run a TorchScript module in a C++ environment.

Saving and Loading Models¶

This document provides solutions to a variety of use cases regarding the saving and loading of PyTorch models. Feel free to read the whole document, or just skip to the code you need for a desired use case.

When it comes to saving and loading models, there are three core functions to be familiar with:

torch.save: Saves a serialized object to disk. This function uses Python’s pickle utility for serialization. Models, tensors, and dictionaries of all kinds of objects can be saved using this function.

torch.load: Uses pickle’s unpickling facilities to deserialize pickled object files to memory. This function also facilitates the device to load the data into (see Saving & Loading Model Across Devices).

torch.nn.Module.load_state_dict: Loads a model’s parameter dictionary using a deserialized state_dict. For more information on state_dict, see What is a state_dict?.

Contents:

In PyTorch, the learnable parameters (i.e. weights and biases) of an torch.nn.Module model are contained in the model’s parameters (accessed with model.parameters() ). A state_dict is simply a Python dictionary object that maps each layer to its parameter tensor. Note that only layers with learnable parameters (convolutional layers, linear layers, etc.) and registered buffers (batchnorm’s running_mean) have entries in the model’s state_dict. Optimizer objects ( torch.optim ) also have a state_dict, which contains information about the optimizer’s state, as well as the hyperparameters used.

Because state_dict objects are Python dictionaries, they can be easily saved, updated, altered, and restored, adding a great deal of modularity to PyTorch models and optimizers.

Example:¶

Let’s take a look at the state_dict from the simple model used in the Training a classifier tutorial.

Output:

Saving & Loading Model for Inference¶

Save/Load state_dict (Recommended)¶

Save:

Load:

When saving a model for inference, it is only necessary to save the trained model’s learned parameters. Saving the model’s state_dict with the torch.save() function will give you the most flexibility for restoring the model later, which is why it is the recommended method for saving models.

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

If you only plan to keep the best performing model (according to the acquired validation loss), don’t forget that best_model_state = model.state_dict() returns a reference to the state and not its copy! You must serialize best_model_state or use best_model_state = deepcopy(model.state_dict()) otherwise your best best_model_state will keep getting updated by the subsequent training iterations. As a result, the final model state will be the state of the overfitted model.

Save/Load Entire Model¶

Save:

Load:

This save/load process uses the most intuitive syntax and involves the least amount of code. Saving a model in this way will save the entire module using Python’s pickle module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

Export/Load Model in TorchScript Format¶

One common way to do inference with a trained model is to use TorchScript, an intermediate representation of a PyTorch model that can be run in Python as well as in a high performance environment like C++. TorchScript is actually the recommended model format for scaled inference and deployment.

Using the TorchScript format, you will be able to load the exported model and run inference without defining the model class.

Export:

Load:

Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

For more information on TorchScript, feel free to visit the dedicated tutorials. You will get familiar with the tracing conversion and learn how to run a TorchScript module in a C++ environment.