Neural Networks are a widely used mathematical model for solving problems in Machine Learning. While their success in applications is indisputable, their theoretical understanding remains limited. Advancing our theoretical understanding of Neural Networks is a key challenge for the scientific community, and mathematicians have an important role to play. In this talk, we explore the functional analytic framework underlying machine learning models, ranging from kernel methods to deep learning. While kernel methods are naturally associated with function spaces that are reproducing kernel Hilbert spaces (RKHSs), we show that neural networks can be characterized in terms of suitable reproducing kernel Banach spaces (RKBSs). We begin by defining these spaces via the infinite-width limit of neural networks, interpreting the integration over neurons as a Riesz–Markov duality. In this setting, the spaces are equipped with a total variation norm. We then present an alternative description of RKBSs, where the Riesz–Markov duality is replaced by the Kantorovich–Rubinstein duality, establishing an interesting connection to optimal transport. Once we have introduced these hypothesis spaces of infinite-width neural networks, we study the regularized empirical risk minimization (ERM) problem associated with them. In particular, we prove the existence of minimizers that take the form of finite-width neural networks, thereby demonstrating that the proposed spaces of integral functions serve as models for the finite-width architectures typically optimized in practice.