Module aims
What does fitting a machine learning model have to do with copying a file over a computer network? The answer, you may be surprised to hear, is quite a lot. The fundamental principles of information theory were originally devised to study optimal communication over information channels, but have since found widespread applications across areas of data science and computing. This module will introduce the basics of information theory as devised by Claude Shannon in 1948, and then delve into its deep connections with statistics and machine learning. Through mathematical and computational exercises, the module presents information as a unifying theory linking computing, statistics, and geomety, and provides crucial theoretical background for students wishing to pursue a career in data science or machine learning.
Learning outcomes
Upon successful completion of this module you will be able to:
– Explain the key properties of information metrics in terms of communication principles.
– Calculate these metrics in common probability distributions.
– Compare different metrics and coding schemes on real-world data.
– Explain the mathematical connections between data transmission and model fitting.
– Design principled data analysis plans with appropriate statistical criteria.
Module syllabus
The module will cover a variety of topics in information theory and statistics, including:
– Entropy and the source coding theorem.
– Mutual information and the channel coding theorem.
– Simple coding schemes, such as Huffman and Hamming codes.
– Introduction to information geometry and projections.
– Connections between information theory and maximum likelihood estimation.
– Recent trends in multivariate information theory for data analysis.