# Batch Norm layer

## Introduction

On this chapter we will learn about the batch norm layer. Previously we said that feature scaling make the job of the gradient descent easier. Now we will extend this idea and normalize the activation of every Fully Connected layer or Convolution layer during training. This also means that while we're training we will select an batch calculate it's mean and standard deviation.

You can think that the batch-norm will be some kind of adaptive (or learnable) pre-processing block with trainable parameters. Which also means that we need to back-propagate them.

The original batch-norm paper can be found here.

Here is the list of advantages of using Batch-Norm:

Improves gradient flow, used on very deep models (Resnet need this)

Allow higher learning rates

Reduce dependency on initialization

Gives some kind of regularization (Even make Dropout less important but keep using it)

As a rule of thumb if you use Dropout+BatchNorm you don't need L2 regularization

It basically force your activations (Conv,FC ouputs) to be unit standard deviation and zero mean.

To each learning batch of data we apply the following normalization.

$\Large \hat{x}^{(k)}=\frac{x^{(k)}-E[x^{(k)}]}{ \sqrt{VAR[x^{(k)}]} }$

The output of the batch norm layer, has the $\gamma, \beta$ are parameters. Those parameters will be learned to best represent your activations. Those parameters allows a learnable (scale and shift) factor $\Large y^{k}=\gamma^{k}.\hat{x}^{(k)}+\beta^{k}$

Now summarizing the operations:

Here, $\epsilon$ is a small number, 1e-5.

## Where to use the Batch-Norm layer

The batch norm layer is used after linear layers (ie: FC, conv), and before the non-linear layers (relu). There is actually 2 batch norm implementations one for FC layer and the other for conv layers (Spatial batch-norm). The good news is that the Spatial batch norm just calls the normal batch-norm after some reshapes.

## Test time

At prediction time that batch norm works differently. The mean/std are not computed based on the batch. Instead, we need to build a estimate during training of the mean/std of the whole dataset(population) for each batch norm layer on your model.

One approach to estimating the population mean and variance during training is to use an exponential moving average.

$S_t=\alpha.S_{t-1}+(1-\alpha).Y_t$

Where: ${S_t,S_{t-1}}$: Current and previous estimation $(\alpha)$: Represents the degree of weighting decrease, a constant smoothing factor between 0 and 1 $Y_t$: Current value (could be mean or std) that we're trying to estimate

Normally when we implement this layer we have some kind of flag that detects if we're on training or testing.

As reference we can find some tutorials with Tensorflow or manually on python.

## Backpropagation

As mentioned earlier we need to know how to backpropagate on the batch-norm layer, first as we did with other layers we need to create the computation graph. After this step we need to calculate the derivative of each node with respect to it's inputs.

### Computation Graph

In order to find the partial derivatives on back-propagation is better to visualize the algorithm as a computation graph:

### New nodes

By inspecting this graph we have some new nodes ($\frac{1}{N}.\sum\limits_{i=1}^N X(i)$, $x^2$, $\sqrt(x-\epsilon)$, $\frac{1}{x}$). To simplify things you can use Wolfram alpha to find the derivatives. For backpropagate other nodes refer to the Back-propagation chapter

In other words:

$$\Large\frac{\partial(\frac{1}{x})}{\partial x}=-\frac{1}{x^2} \therefore dx=-\frac{1}{x_{cache}^2}.dout

Where: $x_{cache}$: the cached (or saved) input from the forward propagation. $dout$: the previous block gradient $\epsilon$: Some small number 0.00005

In other words:

$$\Large\frac{\partial{x^2}}{\partial x}=2.x \therefore dx=2.x_{cache}.dout

$$

#### Block Summation

Like the SUM block this block will copy the input gradient dout equally to all it's inputs. So for all elements in X we will divide by N and multiply by dout.

## Implementation

### Python Forward Propagation

### Python Backward Propagation

### Matlab version forward propagation

### Matlab version backward propagation

### Spatial batchnorm

As mentioned before the spatial batchnorm is used between CONV and Relu layers. To implement the spatial batchnorm we just call the normal batchnorm but with the input reshaped and permuted. Bellow we present the matlab version of forward and backward propagation of the spatial batchnorm.

Now for the backpropagation we just reshape and permute again.

## References

## Next Chapter

Next chapter we will learn about how to optimize our model weights.

Last updated