University of Palermo
Which network? Sparse, structured graphical models for dynamic networks
We propose a sparse dynamic graphical Gaussian model with L1 penalty of structured precision matrix that does model selection and parameter stimation simultaneously in the Gaussian graph model. We employ an L1 penalty on the off diagonal elements of the precision matrix. This is similar to the idea of the glasso (2009 Friedman et al.). The L1 penalty encourages sparsity and at the same time gives shrinkage estimates. In addition, we can model arbitrary, locally additive models for the precision matrix, while explicitly ensure that the estimator of the concentration matrix is positive defined. This is achieved via an efficient semi-definite programming solver SDPT3.
The Open University, UK
Computational Information Geometry: Geometry of Model Choice
Our project uses computational information geometry to develop diagnostic tools to help understand sensitivity to model choice by building an appropriate perturbation space for a given inference problem. Focused on generalised linear models, the long-term aim is to engage with statistical practice via appropriate R software encapsulating the underlying geometry, whose details the user may ignore. This poster exploits the universality of the extended multinomial model to examine the interaction between prior considerations (scientific and otherwise), model choice and final inference. The novel concept of an approximate cut plays a key role. Amongst other features, the perturbation space thereby constructed allows us to expose the differences in inferences resulting from different models, albeit that they may be equally well supported by the data. A couple of examples are used for illustration.
Serena Arima and Luca Tardella
Sapienza University of Rome
Improved harmonic mean estimator for phylogenetic model evidence
Phylogenetic studies aim at constructing a phylogenetic tree which describes the evolutionary relationships between species. Several methods for phylogenetic tree reconstruction have been suggested in literature: we will deal with phylogeny reconstruction methods based on stochastic models in a Bayesian framework. These models, called substitution models, describe the change of nucleotide in the DNA sequences as a Markov process with four state (Adenine, Guanine, Cytosine and Thymine): a transition matrix, called substitution matrix, defines the probability change from the nucleotide i to j and completely specifies the process. We propose some transformations of the target distribution which reduce the variance of in the improved method to phylogenetic data: using simulated data, we compare the IDR estimates with those obtained with the GHM method, which is the most widely used model comparison tool since it is automatically implemented in most phylogenetic softwares. For a fixed topology, the IDR method produces more precise and robust estimates of the GHM method. Moreover, the estimate the marginal likelihood when both substitution model parameters and topology are not fixed.
Devon K. Barrow and Sven F. Crone
Dynamic vs. Ex post model selection for Forecasting an Empirical Evaluation of Bagging and Boosting to Model Selection and Combination on Time Series data
In forecasting a large number of time series, research remains inconclusive on how to best deal with model selection. We provide evidence on the relative performance of dynamic ensemble methods of Bagging and two Boosting variants in comparison to ex‐post approaches of conventional model selection and forecast combination. All algorithms employ a neural network baselearner of multilayer perceptrons, to facilitate comparisons. The experimental design follows the best‐practices for rigorous empirical forecasting evaluations, comparing multiple‐step ahead predictions across a large number of 111 monthly industry time series from the NN3 forecasting competition, across multiple rolling time origins. The relative performance is assessed in comparison to established statistical benchmark methods of the Random Walk, Seasonal Random Walk and Exponential Smoothing, using robust error measures (sMAPE) and across multiple data conditions of time series length (long vs. short), and time series pattern (seasonal vs. non‐seasonal). The results shown that boosting and bagging consistently improves forecasting accuracy; AdaBoost.R2 outperforms AdaBoost.RT while both are outperformed by bagging which appears less sensitive to noise and outliers. All three methods show gains over ensemble averaging and model selection.
Laura Bonnett, Tony Marson, Paula Williamson, and Catrin Tudur-Smith
University of Liverpool
Handling Variables with Every Entry Missing in the Context of External Validation of a Prognostic Model
There are many available strategies for handling missing values within covariates. These include the simple deletion approaches of complete case analysis and available case analysis and all case approaches of analysing the missing data as a separate category, single imputation and multiple imputation. More complex approaches are based on the use of maximum likelihood estimation. We propose that several of these strategies may be extended to handle missing covariates within datasets and therefore undertook a simulation study to test the suitability of these extended strategies.
Thomas Deckers and Christoph Hanck
University of Groningen
Variable Selection via Multiple Testing with an Application to Growth Econometrics
In long regressions many variables are simultaneously tested for significance. We propose to control the false discovery rate (FDR)---the expected value of the number of falsely rejected hypotheses divided by the overall number of rejections---so as not to erroneously declare variables significant in this multiple testing situation only because of the large number of tests performed. Doing so, we provide a simple and fast way to robustly select variables in long regressions. In an extensive Monte Carlo study we compare different FDR controlling techniques concerning their performance with varying degrees of dependence among the regressors. As an illustration we investigate cross-sectional GDP growth regressions where typically a large set of candidate variables exists. We provide a comparison of the variables identified as important with those found using other model selection strategies.
University of Leuven
Applications of transdimensional MCMC using the product space method
In the transdimensional Markov Chain Monte Carlo framework, a comprehensive model is formulated, consisting of the models at comparison at one hand, and a model index on the other hand. This model index, a discrete parameter to be estimated, is sampled within the standard MCMC framework and its sampled values allow the models at comparison to be activated in turn. Of primary interest are the prior and posterior distributions of this model index, as it gives an approximation of the Bayes factor. Alongside of the widely used reversible jump MCMC methodology (Green, 1995), another promising transdimensional MCMC method was introduced by Carlin and Chib (1995), often referred to as the product space method. In this technique, a Gibbs sampler is formulated for the comprehensive model, i.e., full conditional distributions for the model index and the parameters in the two models under consideration are derived. The dimension matching between models is obtained by formulating linking densities or “pseudopriors”. We highlight and discuss some of the practical issues that come about when using the product space method and apply this technique to study applications in cognitive psychology and emotion dynamics.
Testing (In)equality Constrained Models Using Fractional Bayes Methodology
In applied fields, such as the social and medical sciences, researchers often have expectations that can be formulated as a set of models that contain inequality and equality constraints between the parameters of interest. By allowing inequality constraints between the parameters of interest (in addition to equality constraints), researchers are able to formulate models that correspond precisely to their scientific expectations. Consequently, when selecting a model from a set of (in)equality constrained models of interest, researchers receive a direct answer as to whether their expectations are supported by the data. In this poster, I explore how the fractional Bayes factor can be modified such that it can be used for testing (in)equality constrained models. The selection criterion is specified such that model fit and complexity are optimally balanced. Furthermore, the outcome can simply be obtained from a posterior sample under the unconstrained model using a noninformative improper prior. This last property is especially useful when it is desirable to avoid subjective or ad hoc prior specification.
Vahid Nassiri and Ignace Loris
Model Selection in Least Absolute Deviation Regression using L1-penalization
The least squares approach is used widely in fitting many different models, specially linear models. As one may know, the minimizer of the least squares cost function is the maximizer of the likelihood function in the Gaussian case. Therefore, if we face with a noise which shows some heavy-tails, i.e. in presence of some outliers, then the least squares approach is no longer useful. In this poster first we try to justify that with heavy-tail errors the least absolute deviation works better than least squares and then we study and compare an iterative (see Chambolle and Pock, 2010) and a non-iterative algorithm (see e.g. Fuchs 2009 or Li and Zhu 2008) to solve the problem. We also provide some modifications.
Mohammad Lutfor Rahman
Queen Mary, University of London
Bayesian Model Selection in Multi-stratum Designs
Model selection is a process of estimating the performance of different models in order to choose the best approximate one. A few of the criteria that are used in model selection are Akaike information criterion (AIC), Mallow’s Cp, minimum description length, cross validation. For Bayesian model comparison the used criteria are deviance information criterion (DIC), Bayesian information criterion (BIC), and Bayes factor. Mixed models are used to analyze multi-stratum designs as each stratum may have random effects on the responses. The current study was motivated by a polypropylene experiment conducted by four Belgian companies. Goos and Gilmour (2010) analyzed the polypropylene data by likelihood-based methods. As the number of whole plots were not sufficiently large Bayesian analysis was recommended by them to avoid misleading inferences possibly arising in likelihood-based methods. For the polypropylene experiment data, we are trying in Bayesian approach to select the linear mixed models that describe the categorical responses best. We have used manual forward selection method and judged the models by DIC values. In Bayesian method implemented by MCMC techniques (computed by WinBUGS 14), we obtained complex models which were differing slightly with the simplified models obtained by Goos and Gilmour (2010).
Stichting Wetenschappelijk Onderzoek Verkeersveiligheid
Crash prediction models
Crash prediction models are commonly used in road safety research to determine the relation between the number of crashes and road characteristics (including the amount of traffic). In developing these models several assumptions are made, such as the stochastic distribution of road crashes and the road characteristics to be included in the model. The poster will discuss some of these assumptions and explain the advantages and disadvantages of them.
Emma Suckling and Leonard A. Smith
London School of Economics
Big models are wrong, little models are wrong: Which models are worth paying to look at? A Case study of Global Mean Temperature
Large simulation models (GCMs) are expensive to construct and interpret, while the statistical assumptions underlying them assume “small” data-based models are more transparent. Models which “capture the physics” are expected to outperform data-based models in extrapolation; does today’s “best available” simulation model do so? A framework is presented to justify the time and cost of using complicated models by demonstrating in-sample skill (ideally value) as a function of lead time.
Structural uncertainty in health economics
My research concerns uncertainty in health economic evaluations, with application to a comparison of four diagnostic tests for coronary artery disease. A clinical trial provided up to seven years of data to estimate expected costs and benefits of each test strategy, but a long-term model is necessary to incorporate all available evidence and estimate the lifetime impact of the decision on public health. I will discuss the development of the long-term multi-state transition model and the many sources of uncertainty and potential bias in it. Given a particular model structure, health economic models are subject to uncertainty about the values of parameters, such as the sensitivity and specificity of screening tests. But "structural" uncertainties in these models are harder to handle. These include "statistical" model uncertainties such as covariate selections or shapes of survival curves, but uncertainty also arises due to the choice of states in transition models representing the clinical history, or the choice of which data are judged to be relevant to the problem. Accounting for model uncertainty then involves a combination of applied judgement and formal, quantitative model assessment. Formally we are interested in Bayesian model choice in the "M-open" situation where none of the models under consideration are assumed to represent the true clinical and economic process.
National University of Ireland, Galway
A New Approach to Choosing Between Models: The Dragnet Test
In statistical analysis the substitution of a maximum likelihood estimator of a parameter ῶ for the true, but unknown, parameter ω of a model for given data, is so commonplace that possible consequences of the fact that the values of ω and ῶ may differ considerably are often overlooked. The criteria we adopt for the dragnet test are basically those of the standard Cox test for non-nested models: we reject the null hypothesis if the observed log likelihood ratio is inconsistent with what would be expected if the null hypothesis were true, rejection being possible both towards and away from the alternative hypothesis. We then reverse the hypotheses, and repeat the procedure. This results in two p-values, one for each null hypothesis, from which we may classify the models, with possibly inconclusive results. Fixing of parameters enables Cox's method to be extended to nested or overlapping models. Keywords: Model Discrimination, Hypothesis Testing, Cox's test, (non-)nested models.
Rabobank Nederland and VU University
On the validation of risk models: practices and challenges
According to international regulation, every bank should have an independent department validating the risk models used. Such a model validation consists of a variety of activities that contribute to an assessment of these risk models. Moreover, model validation is a continuous process and assesses both the quantitative aspects as well as the qualitative aspects (e.g., usage, implementation, embedding). This talk presents the challenges banks face when developing and implementing risk models (e.g., limited data, complex environment and the need of estimating high quantiles of loss distributions). Subsequently, attention is paid to the role validation teams play in preventing that the wrong model is selected.