Some advanced topics and theories related to statistics and machine learning are taught. More specifically, a nonparametric method called kernel method, statistical properties of training and prediction errors, generalization error bound using Rademacher complexity, and recent deep learning algorithms are taught.
[Objectives] Statistical science and machine learning are disciplines in which useful information is extracted from data to aid
prediction and decision making. Students will learn methodology not simply as knowledge but also learning the background theory including the validity of those methods to promote understanding the essence. Students will broadly apply all kinds of techniques to a variety of problems, learning to construct new techniques on one's own.
[Topics] Students in this course will learn several of statistical science's more advanced techniques, based on their connection to various application fields. We will focus in particular on the connection with machine learning, introducing central topics from both statistical science and machine learning.
machine learning, statistics, kernel methods, prediction error, Rademacher complexity, deep learning
✔ Specialist skills | Intercultural skills | Communication skills | ✔ Critical thinking skills | Practical and/or problem-solving skills |
Lectures are given using black board and slides.
Course schedule | Required learning | |
---|---|---|
Class 1 | Regression analysis and kernel methods | Learn the problem setup of regression analysis. Understand the statistical modeling with kernels, regularization, and kernel ridge regressions in regression analysis. |
Class 2 | Kernel methods I: positive definite kernels | Review the positive definite matrix. Understand the definition of positive definite kernels, and learn some properties of kernel functions and examples. |
Class 3 | Kernel methods II: reproducing property, representor theorem, etc. | Understand reproducing property, representor theorem, etc used in statistical learning with kernels. |
Class 4 | Spline smoothing and kernel methods | Learn the relationship between spline smoothing methods and kernel methods. |
Class 5 | Classification analysis and kernel methods: support vector machine | Learn kernel-based support vector machine for classification problems. |
Class 6 | Inequalities in Probability Theory. | Review the probability theory and understand some probabilistic inequalities used in machine learning. |
Class 7 | Problem setup of statistical learning theory | Understand the problem setup of statistical learning theory. Learn the concepts of hypothesis class, training errors, prediction errors, Bayes errors and Bayes rules. |
Class 8 | Prediction error bound for finite hypothesis class | Learn the derivation of the prediction error for statistical learning with finite hypothesis classes. |
Class 9 | Rademacher Complexity | Learn Rademacher complexity that measures the complexity of hypothesis classes. |
Class 10 | Uniform law of Large Numbers and Rademacher Complexity | Learn uniform law of large numbers (ULLN) that is an extension of the law of large numbers. Understand how Rademacher Complexity relates to ULLN. |
Class 11 | Generalization Error bound of Learning Algorithms. | Understand the derivation of the generalization error bound for various learning algorithms. |
Class 12 | Deep Learning | Learn learning algorithms for deep neural networks. |
Class 13 | Generative Adversarial Networks(GAN) | Learn the algorithm and statistical properties of Generative Adversarial Networks(GAN) that is used as the generative model for images. |
Class 14 | Summary | Summarize this course. |
To enhance effective learning, students are encouraged to spend approximately 100 minutes preparing for class and another 100 minutes reviewing class content afterwards (including assignments) for each class.
They should do so by referring to textbooks and other course material.
Unspecified.
Course materials are provided during class.
Reference book: Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014.
Evaluated by report submission.
It is preferred that students know the basics of statistics and probability theory.