Deep learning is a branch of machine learning based on representation of data with complex representations at a high level of abstraction. These representations are achieved by a sequence of trained non-linear transformations. Deep learning methods have been successfully applied in many important artificial intelligence fields such as computer vision, natural language processing, speech and audio understanding as well as in bioinformatics. This course introduces the most important deep discriminative and generative models with a special focus on practical implementations. Part one introduces key elements of classical feed-forward neural networks and overviews basic building blocks, regularization techniques and learning procedures which are specific for deep models. Part two considers deep convolutional models and illustrates their application in image classification and natural language processing. Part three considers sequence modelling with deep recurrent models and illustrates applications in natural language processing. Finally, part four is devoted to generative deep models and their applications in vision and text representation. All concepts are followed with examples and exercies in a modern dynamic language (e.g. Python, Lua, Julia). Most exercises shall be implemented in a suitable deep learning application framework (e.g. Tensorflow and Torch).
- Explain advantages of deep learning with respect to the alternative machine learning approaches.
- Distinguish techniques which enable successful training of deep models.
- Explain application fields of deep discriminative and generative models.
- Distinguish kinds of deep models which are appropriate in supervised, semi-supervised and unsupervised applications.
- Apply deep learning techniques in understanding of images and text.
- Analyze and evaluate the performance of deep models.
- Design deep models in a high-level programming language.
Forms of Teaching
13 lectures, three hours each.Laboratory Work
Two exercises in each half-semester.Consultations
After a prior e-mail arrangement.Seminars
Students may earn bonus credits by presenting a technical seminar.
|Type||Threshold||Percent of Grade||Threshold||Percent of Grade|
|Laboratory Exercises||40 %||20 %||40 %||0 %|
|Mid Term Exam: Written||0 %||40 %||0 %|
|Final Exam: Written||0 %||40 %|
|Exam: Written||50 %||80 %|
|Exam: Oral||20 %|
Week by Week Schedule
- Motivation for deep learning. Partial differentiation of a composition of vector functions. Logistic regression. Backprop. Multiclass logistic regression. Basics of Python and Numpy. Problem solving.
- Introduction to deep learning: model, loss, optimization, classification, regression, capacity, parsimony, regularization, bias and variance, hyper-parameters, stochastic gradient descent, curse od dimensionality, compositionality principle, data representations.
- Discriminative fully-connected feed-forward models. Loss. Non-linear activation. Universal approximation. Loss gradients. Backprop training. Computational graph. Evaluation and training in Tensorflow. Problem solving.
- Convolutional models. Pooling layers. Loss gradients. Backprop training. Fully convolutional networks. Principles for flexible implementation. Problem solving.
- Challenges in learning deep models: saddle points, multiple minima, unsuitable initialization, vanishing and exploding gradients, choice of hiperparameters, poor generalization.
- Techniques for learning deep models. Training with momentum. Accelarated gradient. Adaptive momentum (ADAM). Data normalization. Fine tuning. Problem solving.
- Regularization. Parameter norm penalty. Data generation. Noise introduction. Early stopping. Parameter sharing. Bagging. Dropout. Problem solving, preparation for the mid-exam.
- Mid-term exam.
- Mid-term exam.
- Convolutional architectures for image and video understanding and natural language processing. Deep metric learning. Outcomes of deep learning.
- Sequence modelling. Recurrent and bidirectional recurrent models. Applications in natural language processing.
- Training recurrent models (BPTT). Deep recurrent models. Long short-term memory cell. Sequence translation. Attention.
- Boltzmann machines. Restricted Boltzmann machines. Markov random fields. Contrastive divergence. Cascaded Boltzmann machines. Deep belief networks.
- Deep generative models. Regularization. Convolutional autoencoders. Variational autoencoders. Adversarial models.
- Problem solving, preparation for the final exam.
Computer Science (profile)Specialization Course (2. semester)
Michael Nielsen (2015.), Neural Networks and Deep Learning, Determination press
Nikhil Buduma (2016.), Fundamentals of Deep Learning, O'Reilly Media
Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2017.), Deep learning, MIT Press
L1 English Level
8 Laboratory exercises
76 Very Good