Principal Component Analysis (PCA) is a fundamental statistical technique with broad applications in machine learning, biology, quantitative finance, and actuarial science. For example, in actuarial science, PCA can be used to reduce the dimensionality of yield curve data, thereby summarizing key information in high-dimensional datasets while facilitating their exploration and interpretation for predicting future asset prices and economic movements. Statistically, PCA arises as the Maximum Likelihood (ML) solution to the Probabilistic PCA (PPCA) model in the limit of vanishing Gaussian noise. Despite its widespread use, a strong theoretical understanding of the ML solution to the PPCA model was missing, primarily because the model is non-identifiable. In this thesis, we propose a general framework, inspired by ideas from quotient topology, to address such non-identifiability issues arising in statistics. However, PPCA with Gaussian noise is well known to be sensitive to extreme observations, or outliers, which can lead to misleading conclusions. We subsequently propose a robust Bayesian alternative to PPCA that preserves its main advantages while offering improved reliability in the presence of outliers. Our robust Bayesian PCA (BPCA) model replaces Gaussian noise with heavy-tailed distributions, such as the Student’s t distribution, in order to limit the influence of outliers by assigning non-negligible probabilities to tail events. Building on prior work in robust linear regression, we theoretically show that proposed robust BPCA achieves robustness, accurately estimating model parameters under reasonable assumptions on the number of outliers. Here, robustness refers to a bounded or vanishing impact on parameter estimates as outliers become more extreme. We support our theoretical findings with numerical experiments and use the Coordinate Ascent Variational Inference (CAVI) algorithm to compute robust estimators, thus providing an end-to-end solution. For completeness, we also provide a comparative study of variational inference and Gibbs sampling in this context. Finally, using tools from dynamical systems and leveraging recent advances in the variational inference literature, we establish convergence guarantees for CAVI applied to BPCA with Gaussian errors.

Exponential random graph models are flexible and allow analyzing relations between any type of objects. Unfortunately, they come with a large number of nuisances. Sampling random graphs is impossible. It left no choice except to use methods such as Metropolis-Hastings. Approximating the maximum likelihood estimator is not trivial, and employing not well- known tools are needed. In Bayesian statistics, a normal law approximation is the closest we can get from the posterior law. In this monograph, we first show how to sample random graphs since it is required for all other techniques. After that, a well-developed study on the maximum likelihood estimator is done. Finally, we show how to approximate the posterior law. A great effort has been made to clarify the Markov chain Monte Carlo approximation of the maximum likelihood estimator. We exhaustively enumerate the conditions for the existence and the uniqueness of the estimator and its approximation. We then bring im- provements to existing algorithms for better reflecting the theory and strengthen robustness. Lastly, we show the asymptotic behaviour of this approximation. Previous studies simply applied the technique without verifying all needed conditions. We pass through all those conditions, and apply modifications when needed.

The EM algorithm Dempster et al. (1977) yields a sequence of estimators that converges to the maximum likelihood estimator for missing data models whose maximum likelihood estimator is not directly tractable. The EM algorithm is remarkable given its numerous applications in statistical learning. However, it may suffer from its computational cost. Cappé and Moulines (2009) proposed an online version of the algorithm in models whose likelihood belongs to the exponential family that provides an upgrade in computational efficiency in large data sets. However, the conditional expected value of the sufficient statistic is often intractable for complex models and/or when the missing data is of a high dimension. In those cases, it is replaced by an estimator. Many questions then arise naturally: do the convergence results pertaining to the initial estimator hold when the expected value is substituted by an estimator? In particular, does the asymptotic normality property remain in this case? How does the variance of the estimator of the expected value affect the asymptotic variance of the EM estimator? Are Monte-Carlo and MCMC estimators suitable in this situation? Could variance reduction tools such as control variates provide variance relief? These questions will be tackled by the means of examples containing latent data models. This master’s thesis’ main contributions are the presentation of a unified framework for stochastic approximation EM algorithms, an illustration of the impact that the estimation of the conditional expected value has on the variance and the introduction of online EM algorithms which reduce the additional variance stemming from the estimation of the conditional expected value.

Markov chain Monte Carlo (MCMC) methods commonly use chains that respect the detailed balance condition. These chains are called reversible. Most of the theory developed for MCMC evolves around those particular chains. Peskun (1973) and Tierney (1998) provided useful theorems on the ordering of the asymptotic variances for two estimators produced by two different reversible chains. In this thesis, we are interested in non-reversible chains, which are chains that don’t respect the detailed balance condition. We present algorithms that simulate non-reversible chains, mainly the Guided Random Walk (GRW) by Gustafson (1998) and the Discrete Bouncy Particle Sampler (DBPS) by Sherlock and Thiery (2017). For both algorithms, we compare the asymptotic variance of estimators with the ones produced by the Metropolis- Hastings algorithm. We present a recent theoretical framework introduced by Andrieu and Livingstone (2019) and their analysis of the GRW. We then show that the DBPS is part of this framework and present an analysis on the asymptotic variance of estimators. Their main theorem can provide an ordering of the asymptotic variances of two estimators resulting from nonreversible chains. We show that an estimator could have a lower asymptotic variance by adding propositions to the DBPS. We then present empirical results of a modified DBPS. Through the thesis we will mostly be interested in chains that are produced by deterministic proposals. We show a general construction of the delayed rejection algorithm using deterministic proposals and one possible equivalent for non-reversible chains.