On this book I stressed out the importance of knowing how to write your own deep learning/artificial intelligence library. But is also very important specially while researching some topic, to understand the most common libraries. This chapter will teach the basics on Torch, but before that we're going also to learn Lua.

Lua was first created to be used on embedded systems, the idea was to have a simple cross-platform and fast language. One the main features of Lua is it's easy integration with C/C++.

Lua was originally designed in 1993 as a language for extending software applications to meet the increasing demand for customization at the time.

This extension means that you could have a large C/C++ program and, some parts in Lua where you could easily change without the need to recompile everything.

Torch is a scientific computing framework based on Lua with CPU and GPU backends. You can imagine like a Numpy but with CPU and GPU implementation. Some nice features:

Efficient linear algebra functions with GPU support

Neural Network package, with automatic differentiation (No need to backpropagate manually)

Multi-GPU support

Bellow we have some simple examples on Lua just to have some contact with the language.

print("Hello World") -- First thing, note that there is no main...--[[This is how we do a multi-line commenton lua, to execute a lua program just use...lua someFile.lua]]

someVar = "Leonardo"io.write("Size of variable is ", #someVar, " and it's value is: \"", someVar, "\"\n")-- Variables on lua are dynamically typed...someVar = 10; -- You can use ";" to end a statementio.write("Now it's value is:\"", someVar, "\"")

The language offer those basic types:

Numbers(Float)

string

boolean

table

print(type(someVar))someVar = 'Leonardo' -- Strings can use use simple quotesprint(type(someVar))someVar = trueprint(type(someVar))someVar = {1,2,"leo",true}print(type(someVar))

Normally we will rely on Torch, but Lua has some math support as well.

io.write("5 + 3 = ", 5+3, "\n")io.write("5 - 3 = ", 5-3, "\n")io.write("5 * 3 = ", 5*3, "\n")io.write("5 / 3 = ", 5/3, "\n")io.write("5.2 % 3 = ", 5%3, "\n")-- Generate random number between 0 and 1io.write("math.random() : ", math.random(0,3), "\n")-- Print float to 10 decimalsprint(string.format("Pi = %.10f", math.pi))

5 + 3 = 85 - 3 = 25 * 3 = 155 / 3 = 1.66666666666675.2 % 3 = 2math.random() : 2Pi = 3.1415926536

The lua statement to include other lua files is the "require", as usual it is used to add some library

require 'image'pedestrian = image.load('./pedestrianSign.png')itorch.image(pedestrian)

Just the simple if-then-else. Lua does not have switch statement.

age = 17if age < 16 thenprint(string.format("You are still a kid with %d years old\n", age))elseif (age == 34) or (age==35) thenprint("Getting old leo...")elseprint("Hi young man")endâ€‹-- Lua does not have ternary operatorscanVote = age > 18 and true or false -- canVote=true if age>18 else canVote=falseio.write("Can I vote: ", tostring(canVote))

Lua have while, repeat and for loops. For loops has also a "for-each" extension to iterate on tables.

i = 1while (i <= 10) doio.write(i,"\n")i = i+1if i==4 then break endend

-- Initial value, end value, increment at each loop...for i=1,3,1 doio.write(i,"\n")end

-- Create a table which is a list of items like an arraysomeTable = {"January", "February", "March", "April",10}â€‹-- Iterate on table monthsfor keyVar, valueVar in pairs(someTable) doio.write(valueVar, "(key=", keyVar, "), ")end

January(key=1), February(key=2), March(key=3), April(key=4), 10(key=5),

Defining functions in Lua is quite easy, it's syntax reminds matlab.

-- Function definitionfunction getSum(a,b)return a+bendâ€‹-- Call functionprint(string.format("5 + 2 = %d", getSum(5,2)))

On Lua we use tables for everything else (ie: Lists, Dictionaries, Classes, etc...)

-- tablesdict = {a = 1, b = 2, c = 3}list = {10,20,30}â€‹-- two prints that display the same valueprint(dict.a)print(dict["a"])-- Tables start with 1 (Like matlab)print(list[1])â€‹-- You can also add functions on tablestab = {1,3,4}-- Add function sum to table tabfunction tab.sum ()c = 0for i=1,#tab doc = c + tab[i]endreturn cendâ€‹print(tab:sum()) -- displays 8 (the colon is used for calling methods)-- tab:sum() means tab.sum(tab)print(tab.sum()) -- On this case it will also workprint(tab)

111088{1 : 12 : 33 : 4sum : function: 0x4035ede8}

Lua does not support directly OOP, but you can emulate all it's main functionalities (Inheritance, Encapsulation) with tables and metatables

Metatable tutorial: Used to override operations (metamethods) on tables.

--[[â€‹Create a class "Animal" with properties:height,weight,name,soundand methods: new,getInfo,saySomethingâ€‹]]â€‹-- Define the defaults for our tableAnimal = {height = 0, weight = 0, name = "No Name", sound = "No Sound"}â€‹-- Constructorfunction Animal:new (height, weight, name, sound)-- Set a empty metatable to the table Animal (Crazy whay to create an instance)setmetatable({}, Animal)-- Self is a reference to this table instanceself.height = heightself.weight = weightself.name = nameself.sound = soundreturn selfendâ€‹-- Some methodfunction Animal:getInfo()animalStr = string.format("%s weighs %.1f kg, is %.1fm in tall", self.name, self.weight, self.height)return animalStrendâ€‹function Animal:saySomething()print(self.sound)end

-- Create an Animalflop = Animal:new(1,10.5,"Flop","Auau")print(flop.name) -- same as flop["name"]print(flop:getInfo()) -- same as flop.getInfo(flop)print(flop:saySomething())â€‹-- Other way to say the samethingprint(flop["name"])print(flop.getInfo(flop))â€‹-- Type of our objectprint(type(flop))

FlopFlop weighs 10.5 kg, is 1.0m in tallAuauâ€‹FlopFlop weighs 10.5 kg, is 1.0m in talltable

-- Open a file to writefile = io.open("./output.txt", "w")â€‹-- Copy the content of the file input.txt to test.txtfor line in io.lines("./input.txt") doprint(line)file:write(string.format("%s from input (At output)\n", line)) -- write on fileendâ€‹file:close()

Line 1 at inputLine 2 at input

local t = os.execute("ls")print(t)local catResult = os.execute("cat output.txt")print(catResult)

FirstContactTorch.ipynbinput.txtLuaLanguage.ipynboutput.txtpedestrianSign.pngplot.pngtrueâ€‹â€‹Line 1 at input from input (At output)Line 2 at input from input (At output)true

On this section we're going to see how to do simple operations with Torch, more complex stuff will be dealt latter.

One of the torch objectives is to give some matlab functionality, an usefull cheetsheat can be found here:

-- Include torch libraryrequire 'torch'; -- Like matlab the ";" also avoid echo the outputâ€‹-- Create a 2x4 matrixm = torch.Tensor({{1, 2, 3, 4}, {5, 6, 7, 8}})print(m)â€‹-- Get element at second row and third collumnprint(m[{2,3}])

1 2 3 45 6 7 8[torch.DoubleTensor of size 2x4]â€‹7

-- Define some Matrix/Tensorsa = torch.Tensor(5,3) -- construct a 5x3 matrix/tensor, uninitializeda = torch.rand(5,3) -- Create a 5x3 matrix/tensor with random valuesb=torch.rand(3,4) -- Create a 3x4 matrix/tensor with random valuesâ€‹-- You can also fill a matrix with values (On this case with zeros)allZeros = torch.Tensor(2,2):fill(0)print(allZeros)â€‹-- Matrix multiplcation and it's syntax variantsc = a*bc = torch.mm(a,b)print(c)d=torch.Tensor(5,4)d:mm(a,b) -- store the result of a*b in câ€‹-- Transpose a matrixm_trans = m:t()print(m_trans)

0 00 0[torch.DoubleTensor of size 2x2]â€‹0.8259 0.6816 0.3766 0.70481.3681 0.9438 0.7739 1.16531.2885 0.9653 0.5573 0.98081.2556 0.8850 0.5501 0.91421.8468 1.3579 0.7680 1.3500[torch.DoubleTensor of size 5x4]â€‹1 52 63 74 8[torch.DoubleTensor of size 4x2]

-- Include torch (cuda) libraryrequire 'cutorch'â€‹-- Move arrays to GPU (and convert it's types to cuda types)a = a:cuda()b = b:cuda()d = d:cuda()â€‹-- Same multiplication just different syntaxc = a*bd:mm(a,b)â€‹print(c)

1.1058 0.6183 1.0518 0.74510.5932 0.8015 0.9441 0.54770.4915 0.8143 0.9613 0.43450.1699 0.6697 0.6656 0.25000.6525 0.6174 0.8894 0.4446[torch.CudaTensor of size 5x4]

Plot = require 'itorch.Plot'â€‹-- Give me 10 random numberslocal y = torch.randn(10)â€‹-- Get 1d tensor from 0 to 9 (10 elements)local x = torch.range(0, 9)Plot():line(x, y,'red' ,'Sinus Wave'):title('Simple Plot'):draw()

require "nn"â€‹-- make a multi-layer perceptronmlp = nn.Sequential();-- 2 inputs, one output 1 hidden layer with 20 neuronsinputs = 2; outputs = 1; hiddenUnits = 20;â€‹-- Mount the modelmlp:add(nn.Linear(inputs, hiddenUnits))mlp:add(nn.Tanh())mlp:add(nn.Linear(hiddenUnits, outputs))

On torch the loss function is called criterion, as on this case we're dealling with a binary classification, we will choose the Mean Squared Error criterion

criterion_MSE = nn.MSECriterion()

Here we're going to back-propagate our model to get the output related to the loss gradient $dout$ then use gradient descent to update the parameters.

for i = 1,2500 do-- random samplelocal input= torch.randn(2); -- normally distributed example in 2dlocal output= torch.Tensor(1);-- Create XOR lables on the fly....if input[1] * input[2] > 0 thenoutput[1] = -1elseoutput[1] = 1endâ€‹-- Feed to the model (with current set of weights), then calculate a losscriterion_MSE:forward(mlp:forward(input), output)â€‹-- Reset the current gradients before backpropagate (Always do)mlp:zeroGradParameters()-- Backpropagate the loss to the hidden layermlp:backward(input, criterion_MSE:backward(mlp.output, output))-- Update parameters(Gradient descent) with alpha=0.01mlp:updateParameters(0.01)end

x = torch.Tensor(2)x[1] = 0.5; x[2] = 0.5; print(mlp:forward(x)) -- 0 XOR 0 = 0 (negative)x[1] = 0.5; x[2] = -0.5; print(mlp:forward(x)) -- 0 XOR 1 = 1 (positive)x[1] = -0.5; x[2] = 0.5; print(mlp:forward(x)) -- 1 XOR 0 = 1 (positive)x[1] = -0.5; x[2] = -0.5; print(mlp:forward(x)) -- 1 XOR 1 = 0 (negative)

-0.8257[torch.DoubleTensor of size 1]â€‹0.6519[torch.DoubleTensor of size 1]â€‹0.4468[torch.DoubleTensor of size 1]â€‹-0.7814[torch.DoubleTensor of size 1]

Torch provides a standard way to optimize any function with respect to some parameters. In our case, our function will be the loss of our network, given an input, and a set of weights. The goal of training a neural net is to optimize the weights to give the lowest loss over our training set of input data. So, we are going to use optim to minimize the loss with respect to the weights, over our training set.

-- Create a dataset (128 elements)batchSize = 128batchInputs = torch.Tensor(batchSize, inputs)batchLabels = torch.DoubleTensor(batchSize)â€‹for i=1,batchSize dolocal input = torch.randn(2) -- normally distributed example in 2dlocal label = 1if input[1]*input[2]>0 then -- calculate label for XOR functionlabel = -1;endbatchInputs[i]:copy(input)batchLabels[i] = labelend

-- Get flatten parameters (Needed to use optim)params, gradParams = mlp:getParameters()-- Learning parametersoptimState = {learningRate=0.01}

require 'optim'â€‹for epoch=1,200 do-- local function we give to optim-- it takes current weights as input, and outputs the loss-- and the gradient of the loss with respect to the weights-- gradParams is calculated implicitly by calling 'backward',-- because the model's weight and bias gradient tensors-- are simply views onto gradParamslocal function feval(params)gradParams:zero()â€‹local outputs = mlp:forward(batchInputs)local loss = criterion_MSE:forward(outputs, batchLabels)local dloss_doutput = criterion_MSE:backward(outputs, batchLabels)mlp:backward(batchInputs, dloss_doutput)return loss,gradParamsendoptim.sgd(feval, params, optimState)end

x = torch.Tensor(2)x[1] = 0.5; x[2] = 0.5; print(mlp:forward(x)) -- 0 XOR 0 = 0 (negative)x[1] = 0.5; x[2] = -0.5; print(mlp:forward(x)) -- 0 XOR 1 = 1 (positive)x[1] = -0.5; x[2] = 0.5; print(mlp:forward(x)) -- 1 XOR 0 = 1 (positive)x[1] = -0.5; x[2] = -0.5; print(mlp:forward(x)) -- 1 XOR 1 = 0 (negative)

-0.6607[torch.DoubleTensor of size 1]â€‹0.5321[torch.DoubleTensor of size 1]â€‹0.8285[torch.DoubleTensor of size 1]â€‹-0.7458[torch.DoubleTensor of size 1]

â€‹