Deep Learning

some general deep learning techniques

Survey Papers / Books

• Deep Learning Book [deeplearningbook.org]

• Deep learning [Nature]

Resources

Courses

• Coursera Deep Learning Specialization by Andrew Ng

Frameworks

• TensorFlow / Keras

• PyTorch / fastai

• mxnet / GLUON

• hardly used now (2019.11)

  • Caffe

  • Caffe2 (now part of PyTorch)

  • Torch (Lua)

  • Matlab / Octave

Models

Optimizer

• Stochastic Gradient Descent (SGD)

• RMSProp

• Momentum

• AdaGrad

• Adam [1412.6980]

• AdamW [1711.05101]

• AdaBound [1902.09843]

• AMSGrad [1904.09237]

• RAdam [LiyuanLucasLiu/RAdam] [1908.03265]

Normalization

• Batch Normalization [1502.03167]

• Weight Normalization [1602.07868]

• Layer Normalization [1607.06450]

• Instance Normalization [1607.08022]

• Spectral Normalization [1802.05957]

• Group Normalization [1803.08494]

• Switchable Normalization [1806.10779]

Weight Initialization

• Xavier Initialization [JMLR'10]

• He Initialization [1502.01852]

Activation Function

• Sigmoid

• Tanh

• ReLU

• Leaky ReLU

• Swish [1710.05941]

• Mish [digantamisra98/Mish] [1908.08681]