Journal of Macroeconomics 24 (2002) 599606 www.elsevier.

com/locate/econbase

Comments on A vector error-correction forecasting model of the US economy

1. Synopsis of the paper with remarks The authors are to be applauded for their eorts in this paper. They deftly undertake to outline and implement a practical empirical methodology for forecasting various macroeconomic variables including the CPI, the GDP price index, real M1, the federal funds rate, the yield on 10-year government bonds and real GDP. This is all done within the framework of a vector error-correction (VEC) model; a model that allows for the incorporation of long-run equilibrium relationships. One of the many important features of this paper is that a systematic approach to testing for cointegration among macroeconomic variables when some of the cointegrating vectors are known a priori is implemented, including great ratios and interest rate spreads, for example. This feature of the paper is important. There is a large body of literature on the forecasting performance of VEC models versus other more parsimonious forecasting models, and the evidence is largely mixed on the potential usefulness of VEC models (see e.g. Homan and Rasche, 1995; Lin and Tsay, 1996, and the references cited therein). One feature of much of this literature, however, is that cointegrating vectors and space ranks are usually estimated. While this may seem an innocuous thing to do, it is widely known that cointegrating space rank estimates, for example, are very sensitive to the number of variables in a system, and to the number of lags used in the posited underlying vector autoregression model. The following two experiments serve to illustrate this point, and underscore the importance of such issues when comparing forecasting models. 1 The rst experiment is designed to study the importance of cointegration vector rank and parameter estimation error on forecasts from VEC models. In particular,

E-mail address: nswanson@econ.rutgers.edu (N.R. Swanson). The following discussion is a summary of some of the results reported in Bachmeier and Swanson (2001).
1

0164-0704/02/$ - see front matter 2002 Published by Elsevier Science Inc. PII: S 0 1 6 4 - 0 7 0 4 ( 0 2 ) 0 0 0 6 8 - X

600

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606

5000 samples of data were generated using the following data generating process (DGP): DYt a3 b3 DYt1 c3 Zt1 3t ; 1

where Yt pt ; mt ; qt 0 , D is the rst dierence operator, 3t $ IN0; R3 , with R3 a 3 3 matrix, and Zt1 dYt1 , with d is an r 3 matrix of cointegration vectors, r is the rank of the cointegrating space (which is either 0, 1, or 2), and a3 , b3 , c3 , and R3 are parameters estimated using historical US data. 2 In all of our comparisons, data are generated with one lag of Yt , cointegrating rank, r, equal to unity, and d either estimated from historical US data or set equal to 1; 1; 1. We estimate the parameters of (1) using four dierent sample periods: the entire sample, covering 1959:11999:4; the period prior to the well known monetarist experiment, covering 1959:1 1979:3; the period 1979:41989:3; and the period 1989:41999:4, which corresponds to the empirical work in the previous section. Given data generated according to (1), two prediction models are estimated, including: (i) versions of (1) where r and d are estimated, corresponding to the estimated VEC model; (iii) versions of (1) where r 0 is imposed, corresponding to a VAR in dierences model. Note that we have generated the data according to a VEC model in all cases, so that we should expect the estimated VEC prediction model to perform well, assuming that coecients are estimated with suciently little parameter estimation error, for example. Results from this experiment are gathered in Table 1. The results vary across the dierent DGPs, but two patterns emerge. First, for small samples (T 100), imprecise estimates of the cointegrating vector parameters and rank generally prevent the VEC model forecasts from dominating the VAR in dierences forecasts, and in many cases the VAR in dierences model even forecasts more accurately. Second, as the sample size grows, the VEC model forecasts begin to dominate more often, and for some DGPs the VEC model almost always forecasts better for T 500. The second Monte Carlo experiment is designed to show that parsimonious time series models will often forecast better than more heavily parameterized, but correctly specied rival models, likely due to parameter estimation error. Specically, we generate data according to two DGPs. The rst DGP is a second-order autoregressive (AR(2)) process: Dpt a1 b1 Dpt1 c1 Dpt2 1t ; 2 where pt and D are dened as above, so that Dpt is the percentage change in the price level from period t 1 to period t, 1t $ IN0; r2 , and a1 , b1 , c1 , and r2 are pa1 1 rameters estimated using historical US data for the period 1959:11999:4. The second DGP is a rst-order vector autoregressive (VAR(1)) process: DYt a2 b2 DYt1 2t ; 3

The data used are the price level, Pt , which is the gross domestic product implicit price deator, gross domestic product, Qt , in chained 1996 dollars, and seasonally adjusted M2. Lowercase mnemonics denote the use of the natural logarithms of these variables.

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606 Table 1 Monte Carlo resultsDM statistics for comparison of estimated VEC and dierences VAR Sample T P 1=3T A B C P 1=2T A 0.10 0.01 0.00 0.15 0.10 0.10 0.14 0.11 0.09 0.21 0.14 0.11 B 0.33 0.06 0.00 0.46 0.46 0.49 0.43 0.44 0.45 0.55 0.43 0.35 C 0.68 0.29 0.05 0.80 0.83 0.84 0.78 0.81 0.82 0.85 0.77 0.67 P 2=3T A 0.09 0.01 0.00 0.13 0.09 0.10 0.12 0.10 0.09 0.21 0.12 0.09 B 0.32 0.06 0.00 0.44 0.44 0.49 0.41 0.43 0.44 0.57 0.44 0.34 C

601

I. Cointegration vector (1,)1,1) used in true DGP 1959:11999:4 100 0.11 0.34 0.68 250 0.01 0.08 0.36 500 0.00 0.01 0.11 1959:11979:3 100 250 500 100 250 500 100 250 500 0.15 0.12 0.11 0.16 0.12 0.10 0.21 0.16 0.13 0.47 0.46 0.49 0.44 0.46 0.46 0.54 0.44 0.37 0.80 0.83 0.84 0.77 0.82 0.83 0.83 0.77 0.67

0.68 0.27 0.03 0.79 0.83 0.85 0.77 0.81 0.82 0.87 0.79 0.66

1979:41989:3

1989:41999:4

II. Estimated cointegration vector 1959:11999:4 100 0.07 250 0.00 500 0.00 1959:11979:3 100 250 500 100 250 500 100 250 500 0.04 0.01 0.06 0.20 0.17 0.15 0.26 0.19 0.18

used in true DGP 0.26 0.60 0.07 0.04 0.26 0.00 0.01 0.09 0.00 0.16 0.06 0.16 0.52 0.54 0.48 0.68 0.61 0.53 0.45 0.21 0.31 0.81 0.84 0.82 0.94 0.91 0.84 0.03 0.01 0.03 0.18 0.18 0.14 0.29 0.21 0.17

0.25 0.03 0.00 0.13 0.03 0.09 0.51 0.54 0.51 0.73 0.65 0.55

0.59 0.18 0.04 0.41 0.12 0.23 0.81 0.85 0.83 0.95 0.93 0.86

0.07 0.00 0.00 0.03 0.00 0.01 0.16 0.17 0.13 0.34 0.23 0.18

0.24 0.02 0.00 0.12 0.01 0.04 0.49 0.56 0.51 0.77 0.71 0.60

0.59 0.14 0.02 0.41 0.07 0.13 0.81 0.86 0.84 0.97 0.96 0.89

1979:41989:3

1989:41999:4

Notes: A refers to percentage of cases in 5000 replications where the DM statistic was less than or equal to 1, assuming an MSE loss function. B refers to percentage of cases where the DM statistic was less than or equal to 0. C refers to the percentage of cases where the DM statistic was less than or equal to 1. A negative DM statistic implies the VAR in dierences model performed better.

where Yt pt ; mt ; qt , with pt , mt , qt and D dened as above, 2t $ IN0; R2 , with R2 a 3 3 matrix, and a2 , b2 , and R2 are parameters estimated using historical US data for the period 1959:11999:4. Given these DGPs, 5000 samples of varying lengths (T 164, which corresponds to the actual sample size used in the empirical work above, and T 300, 500) were generated. For each sample generated from the DGP given in Eq. (2), both AR(1) and AR(2) models were tted, and one-step ahead forecasts were compared using the Diebold and Mariano (DM: 1995) test of equal predictive accuracy. Although the AR(2) model is correctly specied, it requires the estimation of an additional parameter beyond that of the AR(1) model, so that it is not clear which model will forecast better, out-of-sample. For each sample generated

602

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606

according to DGP (3), one-step ahead forecasts are compared for the dierences random walk and VAR in dierences models analyzed in the previous section. Again, even though the VAR in dierences model is correctly specied, there is no reason to expect that it will forecast better than the dierences random walk model, as the lag length and several other parameters need to be estimated for the VAR in dierences model. As a nal metric for assessing the importance of parameter estimation error, true model forecasts, for which the model parameters are imposed a priori to be equal to their true values, rather than estimated, are also included for all of the comparisons. Table 2 shows the percentage of times the DM test was able to reject the null hypothesis that the AR(1) and AR(2) models forecast equally well, given that the DGP is an AR(2) model. The gure shows results for two comparisons, where the AR(1) model forecasts are compared to those of an AR(2) model for which the coecients
Table 2 Monte Carlo resultspower of test of H0 : AR(1) model forecasts as well as AR(2) model (DGP is an AR(2) model) Comparison AR(2) model OOS period P 2=3T Sample size, T 164 300 500 164 300 500 P 1=2T 164 300 500 164 300 500 P 1=3T 164 300 500 164 300 500 Power 0.20 0.34 0.51 0.38 0.47 0.60 0.18 0.29 0.42 0.31 0.38 0.50 0.20 0.28 0.38 0.29 0.34 0.43

True model

AR(2) model

True model

AR(2) model

True model

Notes: The last column of numerical entries shows the power of the DM test to determine whether an AR(2) model forecasts signicantly better, one-step ahead, than an AR(1) model, under MSE loss. The DGP is an AR(2) model, with parameters estimated using historical US data for the period 1959:11999:4. Power of the test indicates the percentage of times in 5000 replications that the predictive ability test rejected equal forecast accuracy of AR(1) and AR(2) models at a signicance level of 95%, where critical values are taken from McCracken (1999). AR(2) model refers to comparison of the AR(1) model forecasts with AR(2) model forecasts, where the parameters of both models are estimated. True model refers to comparison of the AR(1) model forecasts with AR(2) model forecasts, where the parameters of the AR(2) model are imposed to be equal to their true values.

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606

603

are estimated (the AR(2) model comparisons), and also where the AR(1) model forecasts are compared to those of an AR(2) model where the true coecients are imposed rather than estimated (the True model) comparisons. We see that for samples of size 164, the power is never more than 20%. This means that in practice we would have mistakenly concluded that an AR(1) model is the correct specication 80% of the time. As expected, the power of the test increases with the sample size, but is never more than 51% when the AR(2) model parameters are estimated, even for samples of size 500. Table 3 has related results for two comparisons. In the rst (the VAR model comparisons), dierences random walk model forecasts are compared to VAR in differences forecasts, with estimated lag lengths and coecients. In the second (the True model comparisons), dierences random walk model forecasts are compared to forecasts from a rst-order VAR model where the coecients are imposed to be equal to their true values rather than estimated. The results depend on the specication, but when the VAR parameters are estimated, the random walk model almost always does better. In fact, for all of the congurations, the DM statistics are never greater than 1.96, but are often less than )1.96, with negative DM statistics implying that the random walk model forecasts better. On the other hand, for the True model comparisons, very few of the DM statistics are negative, and the percentage of DM statistics greater than 1.96 is greater than 80% for all but three cases. In nearly all cases,

Table 3 Monte Carlo resultsforecast comparison of VAR and random walk models (DGP is a VAR model) Comparison VAR model OOS period P 2=3T Sample size, T 164 300 500 164 300 500 P 1=2T 164 300 500 164 300 500 P 1=3T 164 300 500 164 300 500 DM 6 1.96 0.60 0.86 0.97 0.00 0.00 0.00 0.47 0.76 0.93 0.00 0.00 0.00 0.36 0.57 0.82 0.00 0.00 0.00 DM 6 0 0.98 1.00 1.00 0.00 0.00 0.00 0.97 1.00 1.00 0.00 0.00 0.00 0.95 0.99 1.00 0.01 0.00 0.00 DM 6 1:96 1.00 1.00 1.00 0.17 0.03 0.00 1.00 1.00 1.00 0.32 0.08 0.01 1.00 1.00 1.00 0.47 0.24 0.06

True model

VAR model

True model

VAR model

True model

Notes: See notes to Table 2.

604

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606

then, a VAR model where the lag length and coecients are known a priori will forecast better than a random walk model, but when the lag length and coecients need to be estimated, the random walk model forecasts better. The above experiments serve to illustrate the importance of parameter estimation error when forming forecasting models, and to some extent account for the evidence in the literature suggesting that VEC model do not make particularly good forecasting models. In their paper, Anderson, Homan and Rasche take the surprisingly novel approach of carefully specifying cointegrating restrictions through the use of economic theory rather than simply estimating them. They then proceed to show using a number of useful tests, including that of Horvath and Watson (1995), that the cointegrating restrictions that they imposed from theoretical considerations are consistent with the data. Further, they proceed to establish that predictions from their VEC are largely more accurate than naiive random-walk alternatives. In addition, their predictions compare favorably with those made from professional forecasters. This is a great feat given the previous performance of VEC models, and the authors correctly stress that their results are largely due to their proper and careful implementation of an estimation strategy that combines both theory and empirical estimation techniques, a lesson that we should all learn, and a lesson which the above experimental results suggest is important. Interestingly, the authors also nd that comparing individual variable predictions from their VEC models with those made by random walk models sometimes yields evidence in favor of the simple and parsimonious random walk models. This is a result that is also supported by the Monte Carlo experiments discussed above, and does not detract at all from the important lessons to be learned from their paper.

2. Other remarks The authors deftly handle the issue of integratedness and cointegratedness in their paper. For example, they stress that evidence is mixed concerning whether or not to model ination as I(1) or I(0), and they stress that an assumption that ination is I(1) suggests that price levels are I(2); another result for which empirical evidence to date has been rather mixed. 3 As mentioned above, they also carry out an exhaustive set of cointegration tests in establishing that their a priori cointegrating restrictions agree with the data. Another useful feature of applied methodology that is perhaps worth mentioning in this context is the use of complexity penalized likelihood criteria and/or t- and F-test statistics for choosing the lags in a unit root regression or vector autoregression model. These tools allow one to undertake data dependent lag selection; and given the sensitivity of empirical results of this sort to lag length, are potentially of use to the applied practitioner (see e.g. Ng and Perron, 1995).

In an interesting recent paper, Granger (1995) points out that the very use of terminology like I(0) and I(1) is tantamount to assuming a linear modeling framework, and may not be appropriate in a more general nonlinear framework (see also Corradi et al., 2000).

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606

605

The authors use a variety of interesting approaches for comparing the forecast accuracy of their competing models, including the system-based test statistic of Clements and Hendry (1993). Additional useful test statistics for comparing forecasts are also perhaps worth mentioning, including those for pairwise forecast model comparison (e.g. Corradi et al., 2001; Diebold and Mariano, 1995; West, 1996); those for the comparison of multiple alternative prediction models with a benchmark null model (e.g. White, 2000); and those for the comparison of a potentially innite number of alternative models with a benchmark null model (Corradi and Swanson, 2002). One of the important features of a number of the tests discussed in these papers is that parameter estimation error is properly accounted for in the construction of appropriate test critical values. Another is that when multiple models are compared, the impact of sequential test bias caused by the incorrect sequential application of pairwise tests are used is potentially serious. The authors touch on the important issue of the use of real-time data in forecasting applications, and their points deserve to be reiterated. One of the rst in a long line of recent papers in the area of real-time data (a topic which has been examined since at least Morgenstern (1963)) is the paper by Diebold and Rudebusch (1991). As Diebold and Rudebusch point out for the case of the composite leading index (CLI), using revised data for investigating predictive ability can provide a distorted picture since the CLI is regularly subjected to ex post redenitions to strengthen its historical link to output. The same criticism can be levelled at monetary aggregates, for example, which are also occasionally subjected to redenitions, presumably in part to improve their historical link with output. Redenitions aside, monetary and output aggregates are also continually revised because of incomplete data collection and seasonal factor adjustments, for example (see e.g. Amato and Swanson, 2001, and the references cited therein). The combination of redenitions and revisions suggests that much care needs to be taken when forming rolling prediction sequences; at least whenever these predictions are to be compared with those truly made in real-time by professional forecasters, agencies, etc. When all predictions being compared are based on time series models, though, the importance of using real-time data is no longer as clear-cut.

3. Concluding remark The paper lays the groundwork for an empirical methodology useful not only for constructing reasonable predictions using VEC models, but also for carrying out experiments in which multiple alternative forecasting models are compared. Additionally, interesting new evidence supporting the usefulness of theoretically inspired cointegrating restrictions in forecasting models is presented.

References
Amato, J., Swanson, N.R., 2001. The real-time predictive content of money for output. Journal of Monetary Economics 48, 324.

606

N.R. Swanson / Journal of Macroeconomics 24 (2002) 599606

Bachmeier, L., Swanson, N.R., 2001. Predicting ination: does the quantity theory help. Working Paper, Purdue University. Clements, M.P., Hendry, D.F., 1993. On the limitations of comparing mean square forecast errors. Journal of Forecasting 12, 617637. Corradi, V., Swanson, N.R., Olivetti, C., 2001. Predictive ability with cointegrated variables. Journal of Econometrics 104, 315358. Corradi, V., Swanson, N.R., White, H., 2000. Testing for stationary ergodicity and for comovement between nonlinear discrete time Markov processes. Journal of Econometrics 96, 3973. Corradi, V., Swanson, N.R., 2002. A consistent test for nonlinear out-of-sample predictive accuracy. Journal of Econometrics 110(2), 353381. Diebold, F.X., Mariano, R.S., 1995. Comparing predictive accuracy. Journal of Business and Economic Statistics 13, 253263. Diebold, F.X., Rudebusch, G.D., 1991. Forecasting output with the composite leading index: A real time analysis. Journal of the American Statistical Association 86, 603610. Granger, C.W.J., 1995. Modeling nonlinear relationships between extended memory variables. Econometrica 63, 265279. Homan, D.L., Rasche, R.H., 1996. Assessing forecast performance in a cointegrated system, Journal of Applied Econometrics 11, 495517. Horvath, M.T.K., Watson, M.W., 1995. Testing for cointegration when some cointegrating vectors and known. Econometric Theory 1, 9841014. Lin, J.-H., Tsay, R.S., 1996. Co-integration constraint and forecasting: An empirical examination, Journal of Applied Econometrics 11, 519538. McCracken, M.W., 1999. Asymptotics for out of sample tests of causality. Working Paper, Louisiana State University. Morgenstern, O., 1963. On the accuracy of economic observations, second ed. Princeton University Press, Princeton, NJ. Ng, S., Perron, P., 1995. Unit root tests in ARMA models with data dependent methods for the truncation lag. Journal of the American Statistical Association 90, 268281. West, K., 1996. Asymptotic inference about predictive ability. Econometrica 64, 10671084. White, H., 2000. A reality check for data snooping. Econometrica 68, 10971126.