In this thesis, we are interested in a geometric way of looking at the Lasso method in the context of linear regression. The Lasso is a method that simultaneously estimates the coefficients associated with the predictors and selects the important predictors to explain the response variable. The coefficients are calculated using computational algorithms. Despite its virtues, the Lasso method is forced to select at most n variables when we are in highdimensional contexts (p > n). Moreover, in a group of correlated variables, the Lasso selects a variable “at random”, without caring about the choice of the variable. To address these two problems, we turn to the Linear Lasso. The response vector is then seen as the focal point of the space and all other explanatory variables vectors orbit around the response vector. The angles formed between the response vector and the explanatory variables are assumed to be fixed, and will be used as a basis for constructing the method. The information contained in the explanatory variables is projected onto the response vector. The theory of normal linear models allows us to use ordinary least squares (OLS) for the coefficients of the Linear Lasso. The Linear Lasso (LL) is performed in two steps. First, variables are dropped from the model based on their correlation with the response variable; the number of variables dropped (or ordered) in this step depends on a tuning parameter γ. Then, an exclusion criterion based on the variance of the distribution of the response variable is introduced to remove (or order) the remaining variables. A repeated cross-validation guides us in the choice of the final model. Simulations are presented to study the algorithm for different values of the tuning parameter γ. Comparisons are made between the Linear Lasso and competing methods in small dimensions (Ridge, Lasso, SCAD, etc.). Improvements in the implementation of the method are suggested, for example the use of the 1se rule allowing us to obtain more parsimonious models. An implementation of the LL algorithm is provided in the function R entitled linlasso available at https://github.com/yanwatts/linlasso.

Hidden Markov Models (HMMs) are popular tools to interpret, model and forecast financial data. In these models, the return dynamics on a financial asset evolve according to a non-observed variable, a Markov chain, which generally represents the volatility of the asset. This volatility is notoriously difficult to reproduce with statistical models as it is very persistent in time. HMMs allow the volatility to vary according to the states of a Markov chain. Historically, these models are estimated with a very small number of regimes (states), because the number of parameters to be estimated grows quickly with the number of regimes and the optimization becomes difficult. The objective of this thesis is to propose a general framework to construct HMMs with a richer state space and a higher level of volatility persistence. In the first part, this thesis studies a general class of high-dimensional HMMs, called factorial HMMs, and derives its theoretical properties. In these models, the volatility is linked to a high-dimensional Markov chain built by multiplying lower-dimensional Markov chains, called components. We discuss how previously proposed models based on two-dimensional components adhere to the factorial HMM framework. Furthermore, we propose a new process---the Multifractal Discrete Stochastic Volatility (MDSV) process---which generalizes existing factorial HMMs to dimensions larger than two. The particular parametrization of the MDSV model allows for enough flexibility to reproduce different decay rates of the autocorrelation function, akin to those observed on financial data. A framework is also proposed to model financial log-returns and realized variances, either separately or jointly. An empirical analysis on 31 financial indices reveals that the MDSV model outperforms the realized EGARCH model in terms of fitting and forecasting performance. Our MDSV model requires us to pre-specify the number of components and assumes that there is no uncertainty on that number. In the second part of the thesis, we propose the infinite Factorial Hidden Markov Volatility (iFHMV) model as part of a Bayesian framework to let the data drive the selection of the number of components and take into account the uncertainty related to the number of components in the fitting and forecasting procedure. We also develop an algorithm inspired by the Indian Buffet Process (IBP) to estimate the iFHMV model on financial log-returns. Empirical analyses on two financial indices and two stocks show that the iFHMV model outperforms popular benchmarks in terms of forecasting performance.

This memoir aims at introducing adaptation within the Multiple-Try Metropolis (MTM) algorithms which are a special case of the Markov chain Monte Carlo (MCMC) methods. The MCMC methods, along with their adaptive and multiple-try extensions, are thoroughly explored (both in their possible variations and in their theoretical properties) in order to firmly anchor the study of the proposed adaptive Multiple-Try Metropolis (aMTM) algorithm. Moreover, some existing results on the properties of MTM algorithms are generalized to enable more general results about the aMTM algorithm. The ergodicity of the algorithm is then established using well known results of Roberts and Rosenthal (2007), Andrieu and Moulines (2006) and Craiu et al. (2015) and its empirical performance is studied through a series of simulation experiments. The aMTM algorithm achieves notably better performance than simpler samplers (non-adaptive or single-try) when applied to distributions that are multimodal or that exhibit complex geometry. Finally, many variations of the algorithm are proposed and compared to identify settings that are particularly more efficient. An implementation of the algorithm is provided in a R package called aMTM available at https://github.com/fontaine618/aMTM.

In this master's thesis, we are interested in a new class of informed proposal distributions for Metropolis-Hastings algorithms. These new proposals, called balanced proposals, are obtained by adding information about the target density to an uninformed proposal distribution. A Markov chain generated by a balanced proposal is reversible with respect to the target density without the need for an acceptance probability in two extreme cases: the local case, where the proposal variance tends to zero, and the global case, where it tends to infinity. The balanced proposals need to be approximated to be used in practice. We show that the local case leads to the Metropolis-adjusted Langevin algorithm (MALA), while the global case leads to a small modification of the MALA. These results are used to create a new algorithm that generalizes the MALA by adding a new parameter. Depending on the value of this parameter, the new algorithm will use a locally balanced proposal, a globally balanced proposal, or an interpolation between these two cases. We then study the optimal choice for this parameter as a function of the dimension of the target distribution under two regimes: the asymptotic regime and a finite-dimensional regime. Simulations are presented to illustrate the theoretical results. Finally, we apply the new algorithm to a Bayesian logistic regression problem and compare its efficiency to existing algorithms. The results are satisfying on a theoretical and computational standpoint.

Sampling from multimodal distributions using MCMC methods is an ongoing challenge. Most adaptive algorithms aim to find a good compromise between asymptotic efficiency and actual computation time. In this masters thesis, we propose an improvement to the regional adaptive(RAPT) MCMC algorithm of Craiu et al. (2009), that consists in adding a constrained adaptive partitionning of the sample space. Precisely, the adaptive partition is defined as the hyperplane that is orthogonal to the line joining the cumulative sample means in each region, and which passes through the equidistant point between those two means, according to the Mahalanobis distance. We thus obtain an algorithm that is robust to the choice of the initial partition, at a mimimal computational cost. The workings of the algorithm and its variants are detailed and set in the general framework of adaptive MCMC. A review of some widely used regional adaptation algorithms is also included. The ergodicity of the generated sampler is proved using the sufficient ergodicity conditions of Roberts et Rosenthal (2007). Graphical illustrations of the adaptive partitionning are presented. Our approach is then compared to the RAPT and RAPTOR algorithms through many examples inspired by MCMC litterature. Our new algorithm yields satisfactory results and is in some cases the best choice among those considered, especially when taking into account the computational effort.

Model selection and parameter estimation are two main aspects of statistical analysis. This thesis discusses these aspects from a Bayesian point of view via three papers. The first one deals with a computational procedure, named the reversible jump algorithm, that allows to simultaneously select models and estimate parameters. This sampler being difficult to tune in practice, we aim at providing guidelines to users for an optimal implementation. An optimally tuned sampler corresponds to a sampler that generates Markov chains that optimally explore their state space. Our goal is achieved through the optimisation of a stochastic process that corresponds to the limit (in distribution) of the sequence of stochastic processes engendered by the algorithm. In the second paper, a strategy leading to robust estimation of the parameters of a linear regression model in presence of outliers is presented. The strategy is to make assumptions that are more adapted to the eventual presence of outliers, compared with the traditional model assuming normality of errors. This normality assumption is indeed replaced by a super heavy-tailed distribution assumption. Robustness, which is represented by the convergence of the posterior distribution of the parameters (based on the whole sample) towards that arising from the nonoutliers only, is guaranteed when the number of outliers does not exceed a given threshold. Finally, the results presented in the first two papers are combined to introduce a Bayesian robust principal component regression approach that involves model selection in the prediction process. The characteristics of this approach contribute to increase the accuracy of the predictions produced.

In this thesis, we present a new Metropolis-Hastings algorithm whose proposal distribution has been designed to successfully estimate bimodal target distributions. This sampler may be seen as a variant of the usual random walk Metropolis sampler in which we propose large candidate steps at random times. The goal of these large candidate steps is to leave the actual mode of the target distribution in order to find the second one. We then state and prove a weak convergence result stipulating that if we let the dimension of the target distribution increase to infinity, the Markov chain yielded by the algorithm converges to a certain stochastic process that is almost everywhere continuous. The theoretical result is in the flavour of Roberts et al. (1997), while the method of proof is similar to that found in Bédard (2006). We propose a strategy for optimally parameterizing our new sampler. This strategy aims at optimizing local exploration of the target modes, while correctly estimating the relative weight of each mode. As is traditionally done in the statistical literature, our approach consists of optimizing the limiting process rather than the finite-dimensional Markov chain. Finally, we illustrate our method via numerical examples on some target distributions, one of which violates the regularity conditions of the theoretical result.

Markov Chain Monte Carlo (MCMC) algorithms are methods that are used for sampling from probability distributions. These tools are based on the path of a Markov chain whose stationary distribution is the distribution to be sampled. Given their relative ease of application, they are one of the most popular approaches in the statistical community, especially in Bayesian analysis. These methods are very popular for sampling from complex and/or high dimensional probability distributions. Since the appearance of the first MCMC method in 1953 (the Metropolis algorithm, see [10]), the interest for these methods, as well as the range of algorithms available, continue to increase from one year to another. Although the Metropolis-Hastings algorithm (see [8]) can be considered as one of the most general Markov chain Monte Carlo algorithms, it is also one of the easiest to understand and explain, making it an ideal algorithm for beginners. As such, it has been studied by several researchers. The multiple-try Metropolis (MTM) algorithm , proposed by [9], is considered as one interesting development in this field, but unfortunately its implementation is quite expensive (in terms of time). Recently, a new algorithm was developed by [1]. This method is named the revisited multiple-try Metropolis algorithm (MTM revisited), which is obtained by expressing the MTM method as a Metropolis-Hastings algorithm on an extended space. The objective of this work is to first present MCMC methods, and subsequently study and analyze the Metropolis-Hastings and standard MTM algorithms to allow readers a better perspective on the implementation of these methods. A second objective is to explore the opportunities and disadvantages of the revisited MTM algorithm to see if it meets the expectations of the statistical community. We finally attempt to fight the sedentarity of the revisited MTM algorithm, which leads to a new algorithm. The latter performs efficiently when the number of generated candidates in a given iteration is small, but the performance of this new algorithm then deteriorates as the number of candidates in a given iteration increases.

Financial stocks are often modeled by stochastic differential equations (SDEs). These equations could describe the behavior of the underlying asset as well as some of the model's parameters. For example, the Heston (1993) model, which is a stochastic volatility model, describes the behavior of the stock and the variance of the latter. The Heston model is very interesting since it has semi-closed formulas for some derivatives, and it is quite realistic. However, many simulation schemes for this model have problems when the Feller (1951) condition is violated. In this thesis, we introduce new simulation schemes to simulate price paths using the Heston model. These new algorithms are based on Broadie and Kaya's (2006) method. In order to increase the speed of the exact scheme of Broadie and Kaya, we use, among other things, Markov chains Monte Carlo (MCMC) algorithms and some well-chosen approximations. In our first algorithm, we modify the second step of the Broadie and Kaya's method in order to get faster schemes. Instead of using the second-order Newton method coupled with the inversion approach, we use a Metropolis-Hastings algorithm. The second algorithm is a small improvement of our latter scheme. Instead of using the real integrated variance over time p.d.f., we use Smith's (2007) approximation. This helps us decrease the dimension of our problem (from three to two). Our last algorithm is not based on MCMC methods. However, we still try to speed up the second step of Broadie and Kaya. In order to achieve this, we use a moment-matched gamma random variable. According to Stewart et al. (2007), it is possible to approximate a complex gamma convolution (somewhat near the representation given by Glasserman and Kim (2008) when T-t is close to zero) by a gamma distribution.

Markov Chain Monte Carlo algorithms (MCMC) have become popular tools for sampling from complex and/or high dimensional probability distributions. Given their relative ease of implementation, these methods are frequently used in various scientific areas, particularly in Statistics and Bayesian analysis. The volume of such methods has risen considerably since the first MCMC algorithm described in 1953 and this area of research remains extremely active. A new MCMC algorithm using a directional adjustment has recently been described by Bédard et al. (IJSS, 9:2008) and some of its properties remain unknown. The objective of this thesis is to attempt determining the impact of a key parameter on the global performance of the algorithm. Moreover, another aim is to compare this new method to existing MCMC algorithms in order to evaluate its performance in a relative fashion.