PyTorch

Introduction

PyTorch is another deep learning library that's is actually a fork of Chainer(Deep learning library completely on python) with the capabilities of torch. Basically it's the facebook solution to merge torch with python.

Some advantages

  • Easy to Debug and understand the code

  • Has as many type of layers as Torch (Unpool, CONV 1,2,3D, LSTM, Grus)

  • Lot's of loss functions

  • Can be considered as a Numpy extension to GPUs

  • Faster than others "define-by-run" libraries, like chainer and dynet

  • Allow to build networks which structure is dependent on the computation itself (Useful on reinforcement learning)

PyTorch Components

Package

Description

torch

Numpy like library with GPU support

torch.autograd

Give differentiation support for all torch ops

torch.nn

Neural network library integrated with autograd

torch.optim

Optimization for torch.nn (ADAM, SGD, RMSPROP, etc...)

torch.multiprocessing

Memory sharing between tensors

torch.utils

DataLoader, Training and other utility functions

torch.legacy

Old code ported from Torch

How it differs from Tensorflow/Theano

The major difference from Tensorflow is that PyTorch methodology is considered "define-by-run" while Tensorflow is considered "defined-and-run", so on PyTorch you can for instance change your model on run-time, debug easily with any python debugger, while tensorflow has always a graph definition/build. You can consider tensorflow as a more production tool while PyTorch is more a research tool.

The Basics:

Here we will see how to create tensors, and do some manipulation:

import torch
import numpy as np

# Create a tensor on torch
a = torch.rand(3, 3)

# Create a matrix on numpy and conver to PyTorch
b_npy = np.array([[1,2,3],[4,5,6],[7,8,9]])
# Convert from numpy to torch
b = torch.from_numpy(b_npy)

print(a)
print(b)

# Get a specific element
print(b[1,1])

# Get a range of elements
print(b[1:None,1:None])

# Set elements on array
a[1:None,1:None] = 0
print(a)

Create tensors filled with some value

import torch

a = torch.ones(2,3)
b = torch.zeros(3,2)
print(a)
print(b)

Now we will do some computation on the GPU

import torch
import numpy as np

# Define tensors on the GPU
a = torch.rand(2, 3).cuda()
b = torch.rand(2, 3).cuda()

# Define some operation (will execute on the GPU)
c = (a + b) * 2

# Print "c" contents and shape(size)
print(c)
print(c.size())

Autograd and variables

The Autograd on PyTorch is the component responsible to do the backpropagation, as on Tensorflow you only need to define the forward propagation. PyTorch autograd looks a lot like TensorFlow: in both frameworks we define a computational graph, and use automatic differentiation to compute gradients.

We just need to wrap tensors with Variable objects, a Variable represents a node in a computational graph. They are not like tensorflow placeholders, on PyTorch you place the values directly on the model. Again to include a tensor on the graph wrap it with a variable.

Consider the following simple graph:

import torch
from torch.autograd import Variable

# Define scalar a=2, b=3
a = Variable(torch.ones(1, 1) * 2, requires_grad=True)
b = Variable(torch.ones(1, 1) * 3, requires_grad=True)
c = Variable(torch.ones(1, 1) * 4, requires_grad=True)

# Define the function "out" having 2 parameters a,b
out = (a*b)+c
#c = torch.mul(a,b)+c
print('Value out:',out)

# Do the backpropagation
out.backward()

# Get dout/da (Derivative of out w.r.t to a)
print('Derivative of out w.r.t to a:',a.grad)
print('Derivative of out w.r.t to b:',b.grad)
print('Derivative of out w.r.t to c:',c.grad)

Complete example

Here we mix the concepts and show how to train a MNIST dataset using CNN

# Import libraries
import torch
from torch.autograd import Variable
import torchvision.datasets as dsets
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F

# Hyper Parameters
num_epochs = 5
batch_size = 50
learning_rate = 0.001

# MNIST Dataset
train_dataset = dsets.MNIST(root='../data/',
                            train=True, 
                            transform=transforms.ToTensor(),
                            download=True)

test_dataset = dsets.MNIST(root='../data/',
                           train=False, 
                           transform=transforms.ToTensor())


# Data Loader (Input Pipeline)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size, 
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size, 
                                          shuffle=False)


# CNN Model (2 conv layer) nn.Module is the base class to all neural networks
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.fc = nn.Linear(7*7*32, 10)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out

cnn = CNN()
cnn.cuda()
print(cnn)

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(cnn.parameters(), lr=learning_rate)

# Train the Model
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = Variable(images)
        labels = Variable(labels)

        images, labels = images.cuda(), labels.cuda()

        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = cnn(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        if (i+1) % 500 == 0:
            print ('Epoch [%d/%d], Iter [%d/%d] Loss: %.4f' 
                   %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.data[0]))


# Test the Model
cnn.eval()  # Change model to 'eval' mode (BN uses moving mean/var).
correct = 0
total = 0
for images, labels in test_loader:
    images = Variable(images)
    images, labels = images.cuda(), labels.cuda()
    outputs = cnn(images)
    _, predicted = torch.max(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels).sum()

print('Test Accuracy of the model on the 10000 test images: %d %%' % (100 * correct / total))

# Save the Trained Model
torch.save(cnn.state_dict(), 'cnn.pkl')

References:

PreviousSkFlowNextTransfer Learning

Last updated