http://repub.eur.nl/res/aut/5559/ List of Publications en http://repub.eur.nl/static-eur/img/logo.png http://repub.eur.nl http://repub.eur.nl/res/pub/38706/ 2013-01-01T00:00:00Z We determine whether statistical model forecasts of SKU level sales data can be improved by formally including past expert knowledge in the model as additional variables. Upon analyzing various forecasts in a large database, using various models, forecast samples and accuracy measures, we demonstrate that experts' knowledge, on average, apparently is not associated with variables which are systematically omitted from the statistical models. We also find that the formal inclusion of past judgment can be helpful in cases when the model performs poorly. This can lead to an improved interaction between models and experts, and we discuss the design features of a forecasting support system. http://repub.eur.nl/res/pub/37653/ 2012-09-19T00:00:00Z Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC. http://repub.eur.nl/res/pub/32244/ 2012-05-10T00:00:00Z This thesis is about forecasting situations which involve econometric models and expert intuition. The first three chapters are about what it is that experts do when they adjust statistical model forecasts and what might improve that adjustment behavior. It is investigated how expert forecasts are related to model forecasts, how this potential relation is influenced by other factors and how it influences forecast accuracy, how feedback influences forecasting behavior and accuracy and which loss function is associated with experts’ forecasts. The final chapter focuses on how to make use in an optimal way of multiple forecasts produced by multiple experts for one and the same event. It is found that potential disagreement amongst forecasters can have predictive value, especially when used in Markov regime-switching models. http://repub.eur.nl/res/pub/32410/ 2012-05-01T00:00:00Z http://repub.eur.nl/res/pub/31226/ 2011-12-15T00:00:00Z We propose a new and simple methodology to estimate the loss function associated with experts' forecasts. Under the assumption of conditional normality of the data and the forecast distribution, the asymmetry parameter of the lin-lin and linex loss function can easily be estimated using a linear regression. This regression also provides an estimate for potential systematic bias in the forecasts of the expert. The residuals of the regression are the input for a test for the validity of the normality assumption. We apply our approach to a large data set of SKU-level sales forecasts made by experts and we compare the outcomes with those for statistical model-based forecasts of the same sales data. We find substantial evidence for asymmetry in the loss functions of the experts, with underprediction penalized more than overprediction. http://repub.eur.nl/res/pub/30685/ 2011-12-15T00:00:00Z We propose a new and simple methodology to estimate the loss function associated with experts' forecasts. Under the assumption of conditional normality of the data and the forecast distribution, the asymmetry parameter of the lin-lin and linex loss function can easily be estimated using a linear regression. This regression also provides an estimate for potential systematic bias in the forecasts of the expert. The residuals of the regression are the input for a test for the validity of the normality assumption. We apply our approach to a large data set of SKU-level sales forecasts made by experts and we compare the outcomes with those for statistical model-based forecasts of the same sales data. We find substantial evidence for asymmetry in the loss functions of the experts, with underprediction penalized more than overprediction. http://repub.eur.nl/res/pub/26526/ 2011-09-30T00:00:00Z Experts can rely on statistical model forecasts when creating their own forecasts. Usually it is not known what experts actually do. In this paper we focus on three questions, which we try to answer given the availability of expert forecasts and model forecasts. First, is the expert forecast related to the model forecast and how? Second, how is this potential relation influenced by other factors? Third, how does this relation influence forecast accuracy? We propose a new and innovative two-level Hierarchical Bayes model to answer these questions. We apply our proposed methodology to a large data set of forecasts and realizations of SKU-level sales data from a pharmaceutical company. We find that expert forecasts can depend on model forecasts in a variety of ways. Average sales levels, sales volatility, and the forecast horizon influence this dependence. We also demonstrate that theoretical implications of expert behavior on forecast accuracy are reflected in the empirical data. http://repub.eur.nl/res/pub/26660/ 2011-09-30T00:00:00Z Experts can rely on statistical model forecasts when creating their own forecasts. Usually it is not known what experts actually do. In this paper we focus on three questions, which we try to answer given the availability of expert forecasts and model forecasts. First, is the expert forecast related to the model forecast and how? Second, how is this potential relation influenced by other factors? Third, how does this relation influence forecast accuracy? We propose a new and innovative two-level Hierarchical Bayes model to answer these questions. We apply our proposed methodology to a large data set of forecasts and realizations of SKU-level sales data from a pharmaceutical company. We find that expert forecasts can depend on model forecasts in a variety of ways. Average sales levels, sales volatility, and the forecast horizon influence this dependence. We also demonstrate that theoretical implications of expert behavior on forecast accuracy are reflected in the empirical data. http://repub.eur.nl/res/pub/26506/ 2011-09-26T00:00:00Z We analyze the behavior of experts who quote forecasts for monthly SKU-level sales data where we compare data before and after the moment that experts received different kinds of feedback on their behavior. We have data for 21 experts located in as many countries who make SKU-level forecasts for a variety of pharmaceutical products for October 2006 to September 2007. We study the behavior of the experts by comparing their forecasts with those from an automated statistical program, and we report the forecast accuracy over these 12 months. In September 2007 these experts were given feedback on their behavior and they received a training at the headquarters' office, where specific attention was given to the ins and outs of the statistical program. Next, we study the behavior of the experts for the 3 months after the training session, that is, October 2007 to December 2007. Our main conclusion is that in the second period the experts' forecasts deviated lesser from the statistical forecasts and that their accuracy improved substantially. http://repub.eur.nl/res/pub/26656/ 2011-09-22T00:00:00Z We analyze the behavior of experts who quote forecasts for monthly SKU-level sales data where we compare data before and after the moment that experts received different kinds of feedback on their behavior. We have data for 21 experts located in as many countries who make SKUlevel forecasts for a variety of pharmaceutical products for October 2006 to September 2007. We study the behavior of the experts by comparing their forecasts with those from an automated statistical program, and we report the forecast accuracy over these 12 months. In September 2007 these experts were given feedback on their behavior and they received a training at the headquarters’ office, where specific attention was given to the ins and outs of the statistical program. Next, we study the behavior of the experts for the 3 months after the training session, that is, October 2007 to December 2007. Our main conclusion is that in the second period the experts’ forecasts deviated less from the statistical forecasts and that their accuracy improved substantially. http://repub.eur.nl/res/pub/23711/ 2011-03-01T00:00:00Z Experts (managers) may have domain-specific knowledge that is not included in a statistical model and that can improve short-run and long-run forecasts of SKU-level sales data. While one-step-ahead forecasts address the conditional mean of the variable, model-based forecasts for longer horizons have a tendency to convert to the unconditional mean of a time series variable. Analyzing a large database concerning pharmaceutical sales forecasts for various products and adjusted by a range of experts, we examine whether the forecast horizon has an impact on what experts do and on how good they are once they adjust model-based forecasts. For this, we use regression-based methods and we obtain five innovative results. First, all horizons experience managerial intervention of forecasts. Second, the horizon that is most relevant to the managers shows greater overweighting of the expert adjustment. Third, for all horizons the expert adjusted forecasts have less accuracy than pure model-based forecasts, with distant horizons having the least deterioration. Fourth, when expert-adjusted forecasts are significantly better, they are best at those distant horizons. Fifth, when expert adjustment is down-weighted, expert forecast accuracy increases. http://repub.eur.nl/res/pub/23715/ 2011-03-01T00:00:00Z Abstract We study the performance of SKU-level sales forecasts which linearly combine statistical model forecasts and expert forecasts. Using a large and unique database containing model forecasts for monthly sales of various pharmaceutical products and forecasts given by about 50 experts, we document that a linear combination of those forecasts usually is most accurate. Correlating the weights of the expert forecasts in these linear combinations with the experts’ experience and behaviour shows that more experience and modest deviation from model forecasts gives most weight of the expert forecast. When the rate of bracketing increases, we notice a convergence to equal weights. We show that these results are robust across 12 different forecast horizons. http://repub.eur.nl/res/pub/20744/ 2010-09-22T00:00:00Z Forecasts from various experts are often used in macroeconomic forecasting models. Usually the focus is on the mean or median of the survey data. In the present study we adopt a different perspective on the survey data as we examine the predictive power of disagreement amongst forecasters. The premise is that this variable could signal upcoming structural or temporal changes in an economic process or in the predictive power of the survey forecasts. In our empirical work, we examine a variety of macroeconomic variables, and we use different measurements for the degree of disagreement, together with measures for location of the survey data and autoregressive components. Forecasts from simple linear models and forecasts from Markov regime-switching models with constant and with time-varying transition probabilities are constructed in real-time and compared on forecast accuracy. We find that disagreement has predictive power indeed and that this variable can be used to improve forecasts when used in Markov regime-switching models. http://repub.eur.nl/res/pub/20753/ 2010-09-02T00:00:00Z Forecasts from various experts are often used in macroeconomic forecasting models. Usually the focus is on the mean or median of the survey data. In the present study we adopt a different perspective on the survey data as we examine the predictive power of disagreement amongst forecasters. The premise is that this variable could signal upcoming structural or temporal changes in an economic process or in the predictive power of the survey forecasts. In our empirical work, we examine a variety of macroeconomic variables, and we use different measurements for the degree of disagreement, together with measures for location of the survey data and autoregressive components. Forecasts from simple linear models and forecasts from Markov regime-switching models with constant and with time-varying transition probabilities are constructed in real-time and compared on forecast accuracy. We find that disagreement has predictive power indeed and that this variable can be used to improve forecasts when used in Markov regime-switching models. http://repub.eur.nl/res/pub/20234/ 2010-07-01T00:00:00Z This paper unifies two methodologies for multi-step forecasting from autoregressive time series models. The first is covered in most of the traditional time series literature and it uses short-horizon forecasts to compute longer-horizon forecasts, while the estimation method minimizes one-step-ahead forecast errors. The second methodology considers direct multi-step estimation and forecasting. In this paper, we show that both approaches are special (boundary) cases of a technique called partial least squares (PLS) when this technique is applied to an autoregression. We outline this methodology and show how it unifies the other two. We also illustrate the practical relevance of the resultant PLS autoregression for 17 quarterly, seasonally adjusted, industrial production series. Our main findings are that both boundary models can be improved by including factors indicated from the PLS technique. http://repub.eur.nl/res/pub/19985/ 2010-04-01T00:00:00Z Model-based SKU-level forecasts are often adjusted by experts. In this paper we propose a statistical methodology to test whether these expert forecasts improve on model forecasts. Application of the methodology to a very large database concerning experts in 35 countries who adjust SKU-level forecasts for pharmaceutical products in seven distinct categories leads to the general conclusion that expert forecasts are equally good at best, but are more often worse than model-based forecasts. We explore whether this is due to experts putting too much weight on their contribution, and this indeed turns out to be the case. http://repub.eur.nl/res/pub/18604/ 2010-03-30T00:00:00Z Macroeconomic forecasts are frequently produced, published, discussed and used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyse some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC) and the ECB, are based on econometric model forecasts as well as on human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes non-standard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model, the other forecast, and intuition; and (iii) the two forecasts are generated from two distinct combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the Federal Reserve Board and the FOMC on inflation, unemployment and real GDP growth http://repub.eur.nl/res/pub/16372/ 2009-08-01T00:00:00Z Expert opinion is an opinion given by an expert, and it can have significant value in forecasting key policy variables in economics and finance. Expert forecasts can either be expert opinions, or forecasts based on an econometric model. An expert forecast that is based on an econometric model is replicable, and can be defined as a replicable expert forecast (REF), whereas an expert opinion that is not based on an econometric model can be defined as a non-replicable expert forecast (Non-REF). Both replicable and non-replicable expert forecasts may be made available by an expert regarding a policy variable of interest. In this paper we develop a model to generate replicable expert forecasts, and compare REF with Non-REF. A method is presented to compare REF and Non-REF using efficient estimation methods, and a direct test of expertise on expert opinion is given. The latter serves the purpose of investigating whether expert adjustment improves the model-based forecasts. Illustrations for forecasting pharmaceutical SKUs, where the econometric model is of (variations of) the ARIMA type, show the relevance of the new methodology proposed in the paper. In particular, experts possess significant expertise, and expert forecasts are significant in explaining actual sales. http://repub.eur.nl/res/pub/23714/ 2009-03-26T00:00:00Z We study the performance of SKU-level sales forecasts which linearly combine statistical model forecasts and expert forecasts. Using a large and unique database containing model forecasts for monthly sales of various pharmaceutical products and forecasts given by about fifty experts, we document that a linear combination of those forecasts usually is most accurate. Correlating the weights of the expert forecasts in these linear combinations with the experts’ experience and behaviour shows that more experience and modest deviation from model forecasts gives most weight of the expert forecast. When the rate of bracketing increases, we notice a convergence to equal weights. We show that these results are robust across twelve different forecast horizons. http://repub.eur.nl/res/pub/19417/ 2009-03-01T00:00:00Z This paper reports on the Wald test for a1=a2=0 in the regression model where κ is estimated using nonlinear least squares. As this situation is not standard we provide critical values for further use. An illustration to quarterly GDP in the Netherlands is given. A power study shows that choosing inappropriate starting values for κ leads to a quick loss of power. http://repub.eur.nl/res/pub/20170/ 2009-01-01T00:00:00Z The recent literature on expert adjustment of model-based forecasts at the SKU level suggests that such adjustments occur quite frequently. Second, over-optimism of experts is found to cause adjustments to be upwards more often than downwards. We analyze a unique database containing one-step-ahead model-based forecasts adjusted by many experts, who are located in 37 countries, and are making forecasts for pharmaceutical products within 7 distinct categories. Our results are consistent with earlier findings that the experts make frequent adjustments and that these tend to be upward. Next, and this is new to the literature, we document the fact that expert adjustment itself is largely predictable, where the weight of a forecaster’s own earlier adjustment is about three times as large as the weight of past model-based forecast errors. We also show that expert adjustment is not independent of the model-based forecasts, and we argue that this affects the way we should evaluate the contribution of expert adjustment to the overall forecast quality. http://repub.eur.nl/res/pub/13980/ 2008-12-01T00:00:00Z The primary purpose of the paper is to answer the following two questions regarding the performance of the influential Federal Open Market Committee (FOMC) of the Federal Reserve System, in comparison with the forecasts contained in the “Greenbooks” of the professional staff of the Board of Governors: Does the FOMC have expertise, and can it forecast better than the staff? The FOMC forecasts that are analyzed in practice are non-replicable forecasts. In order to evaluate such forecasts, this paper develops a model to generate replicable FOMC forecasts, and compares the staff forecasts, non-replicable FOMC forecasts, and replicable FOMC forecasts, considers optimal forecasts and efficient estimation methods, and presents a direct test of FOMC expertise on non-replicable FOMC forecasts. The empirical analysis of Romer and Romer (2008) is re-examined to evaluate whether their criticisms of the FOMC’s forecasting performance should be accepted unreservedly, or might be open to alternative interpretations. http://repub.eur.nl/res/pub/13902/ 2008-11-24T00:00:00Z Expert opinion is an opinion given by an expert, and it can have significant value in forecasting key policy variables in economics and finance. Expert forecasts can either be expert opinions, or forecasts based on an econometric model. An expert forecast that is based on an econometric model is replicable, and can be defined as a replicable expert forecast (REF), whereas an expert opinion that is not based on an econometric model can be defined as a non-replicable expert forecast (Non-REF). Both replicable and non-replicable expert forecasts may be made available by an expert regarding a policy variable of interest. In this paper we develop a model to generate replicable expert forecasts, and compare REF with Non-REF. A method is presented to compare REF and Non-REF using efficient estimation methods, and a direct test of expertise on expert opinion is given. The latter serves the purpose of investigating whether expert adjustment improves the model-based forecasts. Illustrations for forecasting pharmaceutical SKUs, where the econometric model is of (variations of) the ARIMA type, show the relevance of the new methodology proposed in the paper. In particular, experts possess significant expertise, and expert forecasts are significant in explaining actual sales. http://repub.eur.nl/res/pub/10769/ 2007-12-06T00:00:00Z We study the performance of sales forecasts which linearly combine model-based forecasts and expert forecasts. Using a unique and very large database containing monthly model-based forecasts for many pharmaceutical products and forecasts given by thirty-seven different experts, we document that a combination almost always is most accurate. When correlating the specific weights in these "best" linear combinations with experts' experience and behaviour, we find that more experience is beneficial for forecasts for nearby horizons. And, when the rate of bracketing increases the relative weights converge to a 50%-50% distribution, when there is some slight variation across forecasts horizons. http://repub.eur.nl/res/pub/10876/ 2007-12-03T00:00:00Z Experts may have domain-specific knowledge that is not included in a statistical model and that can improve forecasts. While one-step-ahead forecasts address the conditional mean of the variable, model-based forecasts for longer horizons have a tendency to convert to the unconditional mean of a time series variable. This suggests that added expertise could be most beneficial to forecast quality for immediate horizons (as very recent events are not in the model) and for further away horizons (as they may miss foreseen trends), and less so for intermediate horizons. Relying on a huge database concerning pharmaceutical sales forecasts for various products and adjusted by a range of experts, we examine and verify this and other conjectures. We also document that the forecast horizon that is the most important for supply chain management here entails the heaviest adjustment by the experts. Unfortunately, that additional adjustment harms forecast accuracy. http://repub.eur.nl/res/pub/10565/ 2007-10-16T00:00:00Z Experts frequently adjust statistical model-based forecasts. Sometimes this leads to higher forecast accuracy, but expert forecasts can also be dramatically worse. We explore the potential drivers of the relevance and quality of experts' added knowledge. For that purpose, we examine a very large database covering monthly forecasts for pharmaceutical products in seven categories concerning thirty-five countries. The extensive results lead to two main outcomes which are (1) that more balance between model and expert leads to more relevance of the added value of the expert and (2) that smaller-sized adjustments lead to higher quality, although sometimes very large adjustments can be beneficial too. In general, too much input of the expert leads to a deterioration of the quality of the final forecast. http://repub.eur.nl/res/pub/10557/ 2007-10-01T00:00:00Z We perform a large-scale empirical analysis of the question whether model-based forecasts can be improved by adding expert knowledge. We consider a huge database of a pharmaceutical company where the head office uses a statistical model to generate monthly sales forecasts at various horizons for various products in seven categories across thirty-five countries and where local managers can modify those model-based forecasts. To sensibly compare realizations and forecasts we develop a useful statistical methodology. Our main finding is that on average the model-based forecasts are about equally good with or without added expertise. We examine the possibility that the expert puts too much weight on his or her own contribution and we obtain strong evidence that this is the case. http://repub.eur.nl/res/pub/10469/ 2007-08-30T00:00:00Z It frequently occurs that experts adjust forecasts from statistical models. There is some evidence that such adjusted forecasts can lead to substantially better performance. Little is known about competence and confidence effects in what these experts do. Theoretical and experimental results in the decision-making literature suggest that those effects should well exist for experts' adjustment. We examine this possibility for a unique data set concerning managers in thirty-seven countries who adjust statistical model-based forecasts delivered by the headquarters of a large pharmaceutical firm. Our literature-consistent findings are that older and younger managers adjust more than middle-aged managers and that a female manager adjusts less, except in the case where she has more experience. http://repub.eur.nl/res/pub/10468/ 2007-08-07T00:00:00Z Experts often add domain knowledge to model-based forecasts while aiming to reduce forecast errors. Indeed, there is some empirical evidence that expert-adjusted forecasts improve forecast quality. However, surprisingly little is known about what experts actually do. Based on a large and detailed database concerning monthly pharmaceutical sales forecasts, we examine whether expert adjustment is predictable. We find substantial evidence that the size, the relative size and even the sign of such adjustment show positive-valued dynamics. The main drivers of current expert adjustments are past adjustment and past model-based forecast errors. Our findings are also that experts' adjustment may suffer from double counting and that trust in the statistical model is not large. An implication is that models may need improvement. Also, experts need to focus on other variables than past sales data when adjusting model-based forecasts. Finally, the method to evaluate the quality of experts' adjustment needs to be modified. http://repub.eur.nl/res/pub/8526/ 2007-01-31T00:00:00Z This paper reports on simulation results for the Wald test for 1=2=0 in the regression model for the case  is known and for the case where  has to be estimated using nonlinear least squares (NLS). This last situation is not standard, and we therefore provide critical values for further use. A power study shows that choosing inappropriate starting values for  leads to a quick loss of power.