On this chapter we will learn about the batch norm layer. Previously we said that feature scaling make the job of the gradient descent easier. Now we will extend this idea and normalize the activation of every Fully Connected layer or Convolution layer during training. This also means that while we're training we will select an batch calculate it's mean and standard deviation.

You can think that the batch-norm will be some kind of adaptive (or learnable) pre-processing block with trainable parameters. Which also means that we need to back-propagate them.

The original batch-norm paper can be found here.

Here is the list of advantages of using Batch-Norm:

Improves gradient flow, used on very deep models (Resnet need this)

Allow higher learning rates

Reduce dependency on initialization

Gives some kind of regularization (Even make Dropout less important but keep using it)

As a rule of thumb if you use Dropout+BatchNorm you don't need L2 regularization

It basically force your activations (Conv,FC ouputs) to be unit standard deviation and zero mean.

To each learning batch of data we apply the following normalization.

â€‹$\Large \hat{x}^{(k)}=\frac{x^{(k)}-E[x^{(k)}]}{ \sqrt{VAR[x^{(k)}]} }$â€‹

The output of the batch norm layer, has the $\gamma, \beta$ are parameters. Those parameters will be learned to best represent your activations. Those parameters allows a learnable (scale and shift) factor $\Large y^{k}=\gamma^{k}.\hat{x}^{(k)}+\beta^{k}$â€‹

Now summarizing the operations:

Here, $\epsilon$ is a small number, 1e-5.

The batch norm layer is used after linear layers (ie: FC, conv), and before the non-linear layers (relu). There is actually 2 batch norm implementations one for FC layer and the other for conv layers (Spatial batch-norm). The good news is that the Spatial batch norm just calls the normal batch-norm after some reshapes.

At prediction time that batch norm works differently. The mean/std are not computed based on the batch. Instead, we need to build a estimate during training of the mean/std of the whole dataset(population) for each batch norm layer on your model.

One approach to estimating the population mean and variance during training is to use an exponential moving average.

â€‹$S_t=\alpha.S_{t-1}+(1-\alpha).Y_t$â€‹

Where: ${S_t,S_{t-1}}$: Current and previous estimation $(\alpha)$: Represents the degree of weighting decrease, a constant smoothing factor between 0 and 1 $Y_t$: Current value (could be mean or std) that we're trying to estimate

Normally when we implement this layer we have some kind of flag that detects if we're on training or testing.

As reference we can find some tutorials with Tensorflow or manually on python.

As mentioned earlier we need to know how to backpropagate on the batch-norm layer, first as we did with other layers we need to create the computation graph. After this step we need to calculate the derivative of each node with respect to it's inputs.

In order to find the partial derivatives on back-propagation is better to visualize the algorithm as a computation graph:

By inspecting this graph we have some new nodes ($\frac{1}{N}.\sum\limits_{i=1}^N X(i)$, $x^2$, $\sqrt(x-\epsilon)$, $\frac{1}{x}$). To simplify things you can use Wolfram alpha to find the derivatives. For backpropagate other nodes refer to the Back-propagation chapterâ€‹

â€‹Block 1/xâ€‹

In other words:

$$\Large\frac{\partial(\frac{1}{x})}{\partial x}=-\frac{1}{x^2} \therefore dx=-\frac{1}{x_{cache}^2}.dout

Where: $x_{cache}$: the cached (or saved) input from the forward propagation. $dout$: the previous block gradient $\epsilon$: Some small number 0.00005

â€‹Block x^2â€‹

In other words:

$$\Large\frac{\partial{x^2}}{\partial x}=2.x \therefore dx=2.x_{cache}.dout

$$

Like the SUM block this block will copy the input gradient dout equally to all it's inputs. So for all elements in X we will divide by N and multiply by dout.

function [activations] = ForwardPropagation(obj, input, weights, bias)obj.previousInput = input;% Tensor format (rows,cols,channels, batch) on matlab% Get batch sizelenSizeActivations = length(size(input));[~,D] = size(input);if (lenSizeActivations < 3)N = size(input,1);elseN = size(input,ndims(input));endâ€‹% Initialize for the first time running_mean and running_varif isempty(obj.running_mean)obj.running_mean = zeros(1,D);obj.running_var = zeros(1,D);endâ€‹if (obj.isTraining)% Step1: Calculate mean on the batchmu = (1/N) * sum(input,1);â€‹% Step2: Subtract the mean from each columnobj.xmu = input - repmat(mu,N,1);â€‹% Step3: Calculate denominatorsq = obj.xmu .^ 2;â€‹% Step4: Calculate varianceobj.var = (1/N) * sum(sq,1);â€‹% Step5: add eps for numerical stability, then sqrtobj.sqrtvar = sqrt(obj.var + obj.eps);â€‹% Step6: Invert the square rootobj.ivar = 1./obj.sqrtvar;â€‹%Step7: Do normalizationobj.xhat = obj.xmu .* repmat(obj.ivar,N,1);â€‹%Step8: Nor the two transformation stepsgammax = repmat(weights,N,1) .* obj.xhat;â€‹% Step9: Adjust with bias (Batchnorm output)activations = gammax + repmat(bias,N,1);â€‹% Calculate running mean and variance to be used latter on% predictionobj.running_mean = (obj.momentum .* obj.running_mean) + (1.0 - obj.momentum) * mu;obj.running_var = (obj.momentum .* obj.running_var) + (1.0 - obj.momentum) .* obj.var;elsexbar = (input - repmat(obj.running_mean,N,1)) ./ repmat(sqrt(obj.running_var + obj.eps),N,1);activations = (repmat(weights,N,1) .* xbar) + repmat(bias,N,1);endâ€‹% Store stuff for backpropagationobj.activations = activations;obj.weights = weights;obj.biases = bias;end

function [gradient] = BackwardPropagation(obj, dout)dout = dout.input;lenSizeActivations = length(size(obj.previousInput));[~,D] = size(obj.previousInput);if (lenSizeActivations < 3)N = size(obj.previousInput,1);elseN = size(obj.previousInput,ndims(obj.previousInput));endâ€‹% Step9:dbeta = sum(dout, 1);dgammax = dout;â€‹% Step8:dgamma = sum(dgammax.*obj.xhat, 1);dxhat = dgammax .* repmat(obj.weights,N,1);â€‹% Step7:divar = sum(dxhat.* obj.xmu, 1);dxmu1 = dxhat .* repmat(obj.ivar,N,1);â€‹% Step6:dsqrtvar = -1 ./ (obj.sqrtvar.^2) .* divar;â€‹% Step 5:dvar = 0.5 * 1 ./sqrt(obj.var+obj.eps) .* dsqrtvar;â€‹% Step 4:dsq = 1 ./ N * ones(N,D) .* repmat(dvar,N,1);â€‹% Step 3:dxmu2 = 2 .* obj.xmu .* dsq;â€‹% Step 2:dx1 = (dxmu1 + dxmu2);dmu = -1 .* sum(dxmu1+dxmu2, 1);â€‹% Step 1:dx2 = 1. /N .* ones(N,D) .* repmat(dmu,N,1);â€‹gradient.input = dx1+dx2;gradient.weight = dgamma;gradient.bias = dbeta;end

As mentioned before the spatial batchnorm is used between CONV and Relu layers. To implement the spatial batchnorm we just call the normal batchnorm but with the input reshaped and permuted. Bellow we present the matlab version of forward and backward propagation of the spatial batchnorm.

% It's just a call to the normal batchnorm but with some% permute/reshape on the input signalfunction [activations] = ForwardPropagation(obj, input, weights, bias)obj.previousInput = input;[H,W,C,N] = size(input);â€‹% Permute the dimensions to the following format% (cols, channel, rows, batch)% On python was: x.transpose((0,2,3,1))% Python tensor format:% (batch(0), channel(1), rows(2), cols(3))% Matlab tensor format:% (rows(1), cols(2), channel(3), batch(4))inputTransposed = permute(input,[2,3,1,4]);â€‹% Flat the input (On python the reshape is row-major)inputFlat = reshape_row_major(inputTransposed,[(numel(inputTransposed) / C),C]);â€‹% Call the forward propagation of normal batchnormactivations = obj.normalBatchNorm.ForwardPropagation(inputFlat, weights, bias);â€‹% Reshape/transpose back the signal, on python was (N,H,W,C)activations_reshape = reshape_row_major(activations, [W,C,H,N]);% On python was transpose(0,3,1,2)activations = permute(activations_reshape,[3 1 2 4]);â€‹% Store stuff for backpropagationobj.activations = activations;obj.weights = weights;obj.biases = bias;end

Now for the backpropagation we just reshape and permute again.

function [gradient] = BackwardPropagation(obj, dout)% Observe that we use the same reshape/permutes from forward% propagationdout = dout.input;[H,W,C,N] = size(dout);% On python was: x.transpose((0,2,3,1))dout_transp = permute(dout,[2,3,1,4]);â€‹% Flat the inputdout_flat = reshape_row_major(dout_transp,[(numel(dout_transp) / C),C]);â€‹% Call the backward propagation of normal batchnormgradDout.input = dout_flat;gradient = obj.normalBatchNorm.BackwardPropagation(gradDout);â€‹% Reshape/transpose back the signal, on python was (N,H,W,C)gradient.input = reshape_row_major(gradient.input, [W,C,H,N]);% On python was transpose(0,3,1,2)gradient.input = permute(gradient.input,[3 1 2 4]);â€‹end

Next chapter we will learn about how to optimize our model weights.