<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<rss version="2.0">
  <channel>
    <title>Model Evaluation and Testing</title>
    <link>http://repub.eur.nl/res/concept/jel-C52/</link>
    <description>Recent publications classified by JEL Code C52</description>
    <language>en</language>
    <image>
      <url>http://repub.eur.nl/static-eur/img/logo.png</url>
      <title>RePub, Erasmus University Rotterdam</title>
      <link>http://repub.eur.nl</link>
    </image>
    <item>
      <title>Comparing the Accuracy of Copula-
Based Multivariate Density Forecasts in
Selected Regions of Support (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/39848/</link>
      <pubDate>2013-04-19T00:00:00Z</pubDate>
      <description>
        
        This paper develops a testing framework for comparing the predictive accuracy of copula-based multivariate density forecasts, focusing on a specific part of the joint distribution. The test is framed in the context of the Kullback-Leibler Information Criterion, but using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties in small samples. In an empirical application to daily exchange rate returns we find evidence that the dependence structure varies with the sign and magnitude of returns, such that different parametric copula models achieve superior forecasting performance in different regions of the support. Our analysis highlights the importance of allowing for lower and upper tail dependence for accurate forecasting of common extreme appreciation and depreciation of different currencies.


      </description>
      <author>Diks, C.G.H.</author> <author>Panchenko, V.</author> <author>Sokolinskiy, O.</author> <author>Dijk, D.J.C. van</author>
    </item> <item>
      <title>Evidence on Features of a DSGE Business Cycle Model from Bayesian Model Averaging
 (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/32101/</link>
      <pubDate>2012-03-20T00:00:00Z</pubDate>
      <description>
        
        The empirical support for features of a Dynamic Stochastic General Equilibrium model with two technology shocks is valuated using Bayesian model averaging over vector autoregressions. The model features include equilibria, restrictions on long-run responses, a structural break of unknown date and a range of lags and deterministic processes. We find support for a number of features implied by the economic model and the evidence suggests a break in the entire model structure around 1984 after which technology shocks appear to account for all stochastic trends. Business cycle volatility seems more due to investment specific technology shocks than neutral technology shocks.


      </description>
      <author>Strachan, R.W.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>Measuring and Predicting Heterogeneous Recessions (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/26863/</link>
      <pubDate>2011-10-01T00:00:00Z</pubDate>
      <description>
        
        This paper examines whether the Conference Board's Leading Economic Index (LEI) can be used for modeling and forecasting a more refined business cycle classification beyond the usual distinction between expansions and contractions. Univariate Markov-switching models for monthly coincident variables and the LEI show that a three regime model is more appropriate than a model with only two regimes. Interestingly, the third regime captures `severe recessions' contrasting the conventional view that the additional third regime represents a 'recovery' phase. This is confirmed by means of Markov-switching vector autoregressive models that allow for phase shifts between the cyclical regimes of LEI and industrial production. Results indicate that a three regime model with a severe recession phase describes the cyclical dynamics in these series better than a two regime model (with only recession and expansion regimes) and a three regime model with a recovery phase. T he timing of the third regime mostly corresponds with periods of substantial credit squeezes and dramatic increases in the default spread as in the recent recession of 2007-2009. These findings provide empirical evidence for the theory of 'financial accelerator'. The severe recession regime of the LEI leads that of IP by 6.5 months whereas for mild recessions this lead time increases to one year.
      </description>
      <author>Cakmakli, C.</author> <author>Paap, R.</author> <author>Dijk, D.J.C. van</author>
    </item> <item>
      <title>Backtesting Value-at-Risk using Forecasts for Multiple Horizons, a Comment on the Forecast Rationality Tests of A.J. Patton and A. Timmermann (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/26505/</link>
      <pubDate>2011-09-01T00:00:00Z</pubDate>
      <description>
        
        Patton and Timmermann (2011, 'Forecast Rationality Tests Based on Multi-Horizon Bounds', Journal of Business &amp; Economic Statistics, forthcoming) propose a set of useful tests for forecast rationality or optimality under squared error loss, including an easily implemented test based on a regression that only involves (long-horizon and short-horizon) forecasts and no observations on the target variable. We propose an extension, a simulation-based procedure that takes into account the presence of errors in parameter estimates. This procedure can also be applied in the field of 'backtesting' models for Value-at-Risk. Applications to simple AR and ARCH time series models show that its power in detecting certain misspecifications is larger than the power of well-known tests for correct Unconditional Coverage and Conditional Coverage.
      </description>
      <author>Hoogerheide, L.F.</author> <author>Ravazzolo, F.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>Ranking Multivariate GARCH Models by Problem Dimension: An Empirical Evaluation (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/23582/</link>
      <pubDate>2011-05-31T00:00:00Z</pubDate>
      <description>
        
        In the last 15 years, several Multivariate GARCH (MGARCH) models have appeared in
the literature. Recent research has begun to examine MGARCH specifications in terms of their out-of-sample forecasting performance. In this paper, we provide an empirical comparison of a set of models, namely BEKK, DCC, Corrected DCC (cDCC) of Aeilli (2008), CCC, Exponentially Weighted Moving Average, and covariance shrinking, using historical data of 89 US equities. Our methods follow part of the approach described in Patton and Sheppard (2009), and the paper contributes to the literature in several directions. First, we consider a wide range of models, including the recent cDCC model and covariance shrinking. Second, we use a range of tests and approaches for direct and indirect model comparison, including the Weighted Likelihood Ratio test of Amisano and Giacomini (2007). Third, we examine how the model rankings are influenced by the cross-sectional dimension of the problem.
      </description>
      <author>Caporin, M.</author> <author>McAleer, M.J.</author>
    </item> <item>
      <title>Divergent Priors and well Behaved Bayes Factors (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/22334/</link>
      <pubDate>2011-01-01T00:00:00Z</pubDate>
      <description>
        
        Divergent priors are improper when defined on unbounded supports. Bartlett's paradox has been taken to imply that using improper priors results in ill-defined Bayes factors, preventing model comparison by posterior probabilities. However many improper priors have attractive properties that econometricians may wish to access and at the same time conduct model comparison. We present a method of computing well defined Bayes factors with divergent priors by setting rules on the rate of diffusion of prior certainty. The method is exact; no approximations are used. As a further result, we demonstrate that exceptions to Bartlett's paradox exist. That is, we show it is possible to construct improper priors that result in well defined Bayes factors. One important improper prior, the Shrinkage prior due to Stein (1956), is one such example. This example highlights pathologies with the resulting Bayes factors in such cases, and a simple solution is presented to this problem. A simple Monte Carlo experiment demonstrates the applicability of the approach developed in this paper.
      </description>
      <author>Strachan, R.W.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>Stock Index Returns' Density Prediction using GARCH Models: Frequentist or Bayesian Estimation? (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/22344/</link>
      <pubDate>2011-01-01T00:00:00Z</pubDate>
      <description>
        
        Using well-known GARCH models for density prediction of daily S&amp;P 500 and Nikkei 225 index returns, a comparison is provided between frequentist and Bayesian estimation. No significant difference is found between the qualities of the forecasts of the whole density, whereas the Bayesian approach exhibits significantly better left-tail forecast accuracy.
      </description>
      <author>Hoogerheide, L.F.</author> <author>Ardia, D.</author> <author>Corre, N.</author>
    </item> <item>
      <title>Modeling and Estimation of Synchronization in Multistate Markov-Switching Models (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/22327/</link>
      <pubDate>2010-12-01T00:00:00Z</pubDate>
      <description>
        
        This paper develops a Markov-Switching vector autoregressive model that allows for imperfect synchronization of cyclical regimes in multiple variables, due to phase shifts of a single common cycle. The model has three key features: (i) the amount of phase shift can be different across regimes (as well as across variables), (ii) it allows the cycle to consist of any number of regimes J is larger than or equal to 2, and (iii) it allows for regime-dependent volatilities and correlations. In an empirical application to monthly returns on size-based stock portfolios, a three-regime model with asymmetric phase shifts and regime-dependent heteroscedasticity is found to characterize the joint distribution of returns most adequately. While large- and small-cap portfolios switch contemporaneously into boom and crash regimes, the large-cap portfolio leads the small-cap portfolio for switches to a moderate regime by a month.
      </description>
      <author>Cakmakli, C.</author> <author>Paap, R.</author> <author>Dijk, D.J.C. van</author>
    </item> <item>
      <title>A Comparative Study of Monte Carlo Methods for Efficient Evaluation of Marginal Likelihoods (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/19830/</link>
      <pubDate>2010-06-01T00:00:00Z</pubDate>
      <description>
        
        Strategic choices for efficient and accurate evaluation of marginal likelihoods by means of Monte Carlo simulation methods are studied for the case of highly non-elliptical posterior distributions. A comparative analysis is presented of possible advantages and limitations of different simulation techniques; of possible choices of candidate distributions and choices of target or warped target distributions; and finally of numerical standard errors. The importance of a robust and flexible estimation strategy is demonstrated where the complete posterior distribution is explored. Given an appropriately yet quickly tuned adaptive candidate, straightforward importance sampling provides a computationally efficient estimator of the marginal likelihood (and a reliable and easily computed corresponding numerical standard error) in the cases investigated in this paper, which include a non-linear regression model and a mixture GARCH model. Warping the posterior density can lead to a further gain in efficiency, but it is more important that the posterior kernel is appropriately wrapped by the candidate distribution than that is warped.
      </description>
      <author>Ardia, D.</author> <author>Basturk, N.</author> <author>Hoogerheide, L.F.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>Evidence on a Real Business Cycle Model with Neutral and Investment-Specific Technology Shocks using Bayesian Model Averaging (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/19511/</link>
      <pubDate>2010-05-17T00:00:00Z</pubDate>
      <description>
        
        The empirical support for a real business cycle model with two technology shocks is evaluated using a Bayesian model averaging procedure. This procedure makes use of a finite mixture of many models within the class of vector autoregressive (VAR) processes. The linear VAR model is extended to permit cointegration, a range of deterministic processes, equilibrium restrictions and restrictions on long-run responses to technology shocks. We find support for a number of the features implied by the real business cycle model. For example, restricting long run responses to identify technology shocks has reasonable support and important implications for the short run responses to these shocks. Further, there is evidence that savings and investment ratios form stable relationships, but technology shocks do not account for all stochastic trends in our system. There is uncertainty as to the most appropriate model for our data, with thirteen models receiving similar support, and the model or model set used has signficant implications for the results obtained.
      </description>
      <author>Strachan, R.W.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>Ranking multivariate GARCH models by problem dimension (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/19447/</link>
      <pubDate>2010-05-11T00:00:00Z</pubDate>
      <description>
        
        In the last 15 years, several Multivariate GARCH (MGARCH) models have appeared in the literature. The two most widely known and used are the Scalar BEKK model of Engle and Kroner (1995) and Ding and Engle (2001), and the DCC model of Engle (2002). Some recent research has begun to examine MGARCH specifications in terms of their out-of-sample forecasting performance. In this paper, we provide an empirical comparison of a set of MGARCH models, namely BEKK, DCC, Corrected DCC (cDCC) of Aeilli (2008), CCC of Bollerslev (1990), Exponentially Weighted Moving Average, and covariance shrinking of Ledoit and Wolf (2004), using the historical data of 89 US equities. Our methods follow some of the approach described in Patton and Sheppard (2009), and contribute to the literature in several directions. First, we consider a wide range of models, including the recent cDCC model and covariance shrinking. Second, we use a range of tests and approaches for direct and indirect model comparison, including the Weighted Likelihood Ratio test of Amisano and Giacomini (2007). Third, we examine how the model rankings are influenced by the cross-sectional dimension of the problem.
      </description>
      <author>Caporin, M.</author> <author>McAleer, M.J.</author>
    </item> <item>
      <title>Threshold, news impact surfaces and dynamic asymmetric multivariate GARCH (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/19452/</link>
      <pubDate>2010-05-11T00:00:00Z</pubDate>
      <description>
        
        DAMGARCH is a new model that extends the VARMA-GARCH model of Ling and McAleer (2003) by introducing multiple thresholds and time-dependent structure in the asymmetry of the conditional variances. Analytical expressions for the news impact surface implied by the new model are also presented. DAMGARCH models the shocks affecting the conditional variances on the basis of an underlying multivariate distribution. It is possible to model explicitly asset-specific shocks and common innovations by partitioning the multivariate density support. This paper presents the model structure, describes the implementation issues, and provides the conditions for the existence of a unique stationary solution, and for consistency and asymptotic normality of the quasi-maximum likelihood estimators. The paper also presents an empirical example to highlight the usefulness of the new model.
      </description>
      <author>Caporin, M.</author> <author>McAleer, M.J.</author>
    </item> <item>
      <title>Seasonality in Revisions of Macroeconomic Data (Article)</title>
      <link>http://repub.eur.nl/res/pub/23954/</link>
      <pubDate>2010-04-01T00:00:00Z</pubDate>
      <description>
        
        We analyze the revision history of quarterly and monthly (seasonally unadjusted) macroeconomic variables for the Netherlands, Ireland, Luxemburg and the United States, where we focus on the degree of deterministic seasonality in these revisions. We document that the data show most deterministic seasonality in the final revision. The first-release data and the in-between revisions show a variety of seasonal patterns. The consequences of these findings for the interpretation and modeling of macroeconomic data are discussed.

      </description>
      <author>Franses, Ph.H.B.F.</author> <author>Segers, R.</author>
    </item> <item>
      <title>Evaluating Macroeconomic Forecast: A Review of Some Recent Developments (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/18604/</link>
      <pubDate>2010-03-30T00:00:00Z</pubDate>
      <description>
        
        Macroeconomic forecasts are frequently produced, published, discussed and used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyse some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC) and the ECB, are based on econometric model forecasts as well as on human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes non-standard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model, the other forecast, and intuition; and (iii) the two forecasts are generated from two distinct combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the Federal Reserve Board and the FOMC on inflation, unemployment and real GDP growth
      </description>
      <author>Franses, Ph.H.B.F.</author> <author>McAleer, M.J.</author> <author>Legerstee, R.</author>
    </item> <item>
      <title>A Mathematical Analysis of the Long-run Behavior of Genetic Algorithms for Social Modeling (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/15181/</link>
      <pubDate>2009-03-09T00:00:00Z</pubDate>
      <description>
        
        We present a mathematical analysis of the long-run behavior of genetic algorithms that are used for modeling social phenomena. The analysis relies on commonly used mathematical techniques in evolutionary game theory. Assuming a positive but infinitely small mutation rate, we derive results that can be used to calculate the exact long-run behavior of a genetic algorithm. 
Using these results, the need to rely on computer simulations can be avoided. We also show that if the mutation rate is infinitely small the crossover rate has no effect on the long-run behavior of a genetic algorithm. To demonstrate the usefulness of our mathematical analysis, we replicate a well-known study by Axelrod in which a genetic algorithm is used to model the evolution of strategies in iterated prisoner’s dilemmas. The theoretically predicted long-run behavior of the genetic algorithm turns out to be in perfect agreement with the long-run behavior observed in computer simulations. Also, in line with our theoretically informed expectations, computer simulations indicate that the crossover rate has virtually no long-run effect. Some general new insights into the behavior of genetic algorithms in the prisoner’s dilemma context are provided as well.
      </description>
      <author>Waltman, L.R.</author> <author>Eck, N.J.P. van</author>
    </item> <item>
      <title>To Bridge, to Warp or to Wrap? (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/14945/</link>
      <pubDate>2009-02-01T00:00:00Z</pubDate>
      <description>
        
        Important choices for efficient and accurate evaluation of marginal likelihoods by means of Monte Carlo simulation methods are studied for the case of highly non-elliptical posterior distributions. We focus on the situation where one makes use of importance sampling or the independence chain Metropolis-Hastings algorithm for posterior analysis. A comparative analysis is presented of possible advantages and limitations of different simulation techniques; of possible choices of candidate distributions and choices of target or warped target distributions; and finally of numerical standard errors. The importance of a robust and flexible estimation strategy is demonstrated where the complete posterior distribution is explored. In this respect, the adaptive mixture of Student-t distributions of Hoogerheide et al.(2007) works particularly well. Given an appropriately yet quickly tuned candidate, straightforward importance sampling provides the most efficient estimator of the marginal likelihood in the cases investigated in this paper, which include a non-linear regression model of Ritter and Tanner (1992) and a conditional normal distribution of Gelman and Meng (1991). A poor choice of candidate density may lead to a huge loss of efficiency where the numerical standard error may be highly unreliable.
      </description>
      <author>Ardia, D.</author> <author>Hoogerheide, L.F.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>Out-of-sample Comparison of Copula Specifications in Multivariate Density Forecasts (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/14052/</link>
      <pubDate>2008-11-01T00:00:00Z</pubDate>
      <description>
        
        We introduce a statistical test for comparing the predictive accuracy of competing copula specifications in multivariate density forecasts, based on the Kullback-Leibler Information Criterion (KLIC). The test is valid under general conditions: in particular it allows for parameter estimation uncertainty and for the copulas to be nested or non-nested. Monte Carlo simulations demonstrate that the proposed test has satisfactory size and power properties in finite samples. Applying the test to daily exchange rate returns of several major currencies against the US dollar we find that the Student's t copula is favored over Gaussian, Gumbel and Clayton copulas. This suggests that these exchange rate returns are characterized by symmetric tail dependence.
      </description>
      <author>Diks, C.G.H.</author> <author>Panchenko, V.</author> <author>Dijk, D.J.C. van</author>
    </item> <item>
      <title>Bayesian Averaging over Many Dynamic Model Structures with Evidence on the Great Ratios and Liquidity Trap Risk (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/14049/</link>
      <pubDate>2008-10-02T00:00:00Z</pubDate>
      <description>
        
        A Bayesian model averaging procedure is presented that makes use of a finite mixture of many model structures within the class of vector autoregressive (VAR) processes. It is applied to two empirical issues. First, stability of the Great Ratios in U.S. macro-economic time series is investigated, together with the effect of permanent shocks on business cycles. Second, the linear VAR model is extended to include a smooth transition function in a (monetary) equation and stochastic volatility in the disturbances. The risk of a liquidity trap in the U.S.A. and Japan is evaluated. Although this risk found to be reasonably high, we find only mild evidence that the monetary policy transmission mechanism is different and that central banks consider the expected cost of a liquidity trap in policy setting. Posterior probabilities of different models are evaluated using Markov chain Monte Carlo techniques.
      </description>
      <author>Strachan, R.W.</author> <author>Dijk, H.K. van</author>
    </item> <item>
      <title>A Simple Test for GARCH against a Stochastic Volatility Model (Article)</title>
      <link>http://repub.eur.nl/res/pub/13212/</link>
      <pubDate>2008-06-01T00:00:00Z</pubDate>
      <description>
        
        GARCH models and Stochastic Volatility (SV) models can both be used to describe unobserved volatility in asset returns. We consider the issue of testing a GARCH model against an SV model. For that purpose, we propose a new and parsimonious GARCH-t model with an additional restricted moving average term, which can capture SV model properties. We discuss model representation, parameter estimation, and our simple test for model selection. Furthermore, we derive the theoretical moments and the autocorrelation function of our new model. We illustrate our model and test for nine daily stock-return series.
      </description>
      <author>Franses, Ph.H.B.F.</author> <author>Leij, M.J. van der</author> <author>Paap, R.</author>
    </item> <item>
      <title>Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails (Research Paper)</title>
      <link>http://repub.eur.nl/res/pub/13975/</link>
      <pubDate>2008-05-19T00:00:00Z</pubDate>
      <description>
        
        We propose new scoring rules based on partial likelihood for assessing the relative out-of-sample predictive accuracy of competing density forecasts over a specific region of interest, such as the left tail in financial risk management. By construction, existing scoring rules based on weighted likelihood or censored normal likelihood favor density forecasts with more probability mass in the given region, rendering predictive accuracy tests biased towards such densities. Our novel partial likelihood-based scoring rules do not suffer from this problem, as illustrated by means of Monte Carlo simulations and an empirical application to daily S&amp;P 500 index returns.
      </description>
      <author>Diks, C.G.H.</author> <author>Panchenko, V.</author> <author>Dijk, D.J.C. van</author>
    </item>
  </channel>
</rss>