Prof Milan Vojnovic COL5.05

This course is available on the MSc in Applicable Mathematics, MSc in Applied Social Data Science, MSc in Data Science, MSc in Geographic Data Science, MSc in Health Data Science, MSc in Management of Information Systems and Digital Innovation, MSc in Operations Research & Analytics, MSc in Quantitative Methods for Risk Management, MSc in Statistics, MSc in Statistics (Financial Statistics), MSc in Statistics (Financial Statistics) (LSE and Fudan), MSc in Statistics (Financial Statistics) (Research), MSc in Statistics (Research), MSc in Statistics (Social Statistics) and MSc in Statistics (Social Statistics) (Research). This course is available with permission as an outside option to students on other programmes where regulations permit.

MSc Data Science students will be given priority for enrollment in this course.

The course requires some mathematics, in particular some use of vectors and some calculus. Basic knowledge of computer programming is expected, mainly Python.

This course is about deep learning, covering fundamental concepts of deep learning and neural networks, design of neural network architectures, optimisation methods for training neural networks, and neural networks design for particular purposes such as image recognition, sequence modelling, natural language processing and generative models. The course will cover the following topics:

1. Introduction – course overview
2. Introduction to neural networks – single-layer networks, linear discriminant functions, XOR problem, perceptron, multi-layer perceptron, perceptron learning criteria, perceptron learning algorithm, feedforward neural network architecture
3. Optimisation algorithms – empirical loss function minimisation, gradient descent algorithm, stochastic gradient descent algorithm
4. Advanced optimisation algorithms – adaptive learning rates, momentum, backpropagation, dropout
5. Convolutional neural networks (CNNs) – principles and basic operations of convolutional neural networks, LeNet example
6. Modern convolutional neural networks – understanding principles of some modern CNN architectures, including AlexNet, VGGNet, NiN, GoogLeNet, ResNet, and DenseNet
7. Recurrent neural networks (RNNs) – RNN models, training RNNs, gated RNNs, GRU, LSTM, Deep RNNs, bidirectional RNNs, vector to sequence models using RNNs
8. Sequence to sequence models  – machine translation tasks, encoder-decoder architecture, attention mechanisms, transformer
9. Autoencoders – introduction to autoencoders, linear factor models, PCA and probabilistic PCA, sparse coding, autoencoders, variational autoencoders
10. Generative adversarial networks (GANs) – introduction to GANs, GAN architecture and training, Wasserstein GANs, Wasserstain GANs with gradient penalty

20 hours of lectures and 15 hours of classes in the LT.

This course will be delivered through a combination of classes, and lectures and Q&A sessions totalling a minimum of 35 hours across Lent Term. This course includes a reading week in Week 6 of Lent Term.

Students will be expected to produce 8 problem sets in the LT.

• Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, https://www.deeplearningbook.org/
• Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola, Dive into Deep Learning, https://d2l.ai/
• TensorFlow – An end-to-end open source machine learning platform, https://www.tensorflow.org/

Project (80%) in the LT.
Continuous assessment (10%) in the MT Week 4.
Continuous assessment (10%) in the MT Week 9.

Two of the problem sets submitted by students weekly will be assessed (20% in total). Each problem set will have an individual mark of 10% and submission will be required in LT Weeks 4 and 9. In addition, there will be a take-home exam (80%) in the form of a group project in which they will demonstrate their ability to develop and evaluate neural network algorithms for solving a prediction or classification task of their choice.