Deep Learning Libraries
Discussion, and some examples on the most common deep learning libraries:
- Caffe
- Torch
- TensorFlow
- Theano
- CNTK
One of the most basic characteristic of caffe is that is easy to train simple non recurrent models.
Most cool features:
- Good Performance, allows training with multiple GPUs
- Implementation for CPU and GPU
- Source code is easy to read
- Allow layer definition in Python
- Has bidings for Python and Matlab
- Allows network definition with text language (No need to write code)
- Fast dataset access through LMDB
- Allows network vizualization
- Has web interface (Digits)
- Blob: Used to store data and diffs(Derivatives)
- Layer: Some operation that transform a bottom blob(input) to top blobs(outputs)
- Net: Set of connected layers
- Solver: Call Net forward and backward propagation, update weights using gradient methods (Gradient descent, SGD, adagrad, etc...)
path/to/image/1.jpg [label]
Here a logistic regression classifier. Imagine as a neural network with one layer and a sigmoid (cross-entropy softmax) non-linearity.
- Need to write C++ / Cuda code for new layers
- Bad to write protofiles for big networks (Resnet, googlenet)
- Bad to experience new architectures (Mainstream version does not support Fast RCNN)
Really good for research, the problem is that use a new language called Lua.
- Flexible
- Very easy source code
- Easy biding with C/C++
- Web interface (Digits)
- New language Lua
- Difficult to load data from directories
- No Matlab bidings
- Less Plug and play than caffe
- Not easy for RNN
- More manual
- No matlab biding
- Slower than other frameworks
- No much pre-trained models
- Flexible
- Good for RNN
- Allow distributed training
- Tensorboard for signal visualization
- Python Numpy
- Not much pre-trained models
- No Support for new object detection features (Ex Roi pooling)
- No support for datasets like Caffe
- Slower than Caffe for single GPU training
- Flexible
- Good for RNN
- Allows distributed training
- No visualization
- Any error CNTK crash
- No simple source code to read
- New language (ndl) to describe networks
- No current matlab or python bindings
- For research use Torch or Tensorflow (Last option Theano)
- For training convnets or use pre-trained models use Caffe
- Get features from known model (Alexnet, Googlenet, Vgg): Use caffe
- Fine tune known models (Alexnet, Googlenet, Vgg): Use Caffe
- Image Captioning: Torch or Tensorflow
- Segmentation: Caffe, Torch
- Object Detection: Caffe with python layers, Torch (More work)
- Language Modelling: Torch, Theano
- Implement Bath Norm: Torch, Theano or Tensorflow
Normally Tensorflow can be used in all cased that torch can, but if you need to understand what a specific layer does, or if you need to create a new layer, use torch instead of tensorflow. Torch is preferable on those cases, because the layer source code is more easy to read in torch.
On the next chapter we will discuss Distributed Learning
Last modified 3yr ago