Expressivity of Neural Networks
Type
Master's thesis / Bachelor's thesis
Prerequisites
- Proficiency with Python and deep learning frameworks (TensorFlow or PyTorch)
- Preferably Functional Analysis background
Description
Deep Learning models have achieved tremendous success in many application areas. This also raises interesting theoretical questions such as: Why can Deep Learning models adapt to widely different tasks? Are we able to describe their expressive power? A longstanding result in this direction is the Universal Approximator Theorem, which states that neural networks with only two layers can, under weak conditions, can approximate any continuous function on a compact set up to arbitrary precision. However, this is only a starting point to the above questions. A common goal is to relate the approximation accuracy of neural networks to their complexity. It turns out that for many well-known classical function spaces, one can derive upper and lower complexity bounds in terms of the number of weights, neurons, and layers. Another exciting line of research tries to explain the benefit of deep neural networks over shallow ones. Yet another approach studies how neural architectural design choices influence expressivity. These examples are among the many interesting and open research questions within the field of expressivity of neural networks.
References
- Ingo Gühring, Mones Raslan, Gitta Kutyniok, Expressivity of Deep Neural Networks. (https://arxiv.org/abs/2007.04759)
- Patrick Kidger and Terry Lyons, Universal approximation with deep narrow networks. (https://arxiv.org/abs/1905.08539)
- Ingo Gühring, Gitta Kutyniok, and Philipp Petersen, Error bounds for approximations with deep
- ReLU neural networks in W^{s,p} norms. (https://arxiv.org/abs/1902.07896)
- D. Yarotsky. Universal approximations of invariant maps by neural networks. (https://arxiv.org/abs/1804.10306)