date: Wed, 04 Jul 2007 16:21:48 +0100 from: david.parker@metoffice.gov.uk subject: Re: Fwd: RE: IPCC Table 3.2 to: "Jones, Phil" Phil Thanks for both. Steve McIntyre has ashed for the software I used for calculating the DW statistic so I will send him just the relevant lines. I use a printout of the Chapter 3 pdf! Regards David On Wed, 2007-07-04 at 14:03 +0100, Phil Jones wrote: > David, > Thanks. Tim thinks he could get the procedures to work, > but won't bother as your explanation is fine. I'll paraphrase it > and send off. I doubt though that he will be happy. He will > probably say we didn't give enough explanation. I'll > be cc'ing to the same list as earlier. > > Thanks again. > > It would be nice to get the CUP book. I keep opening the pdf > version of the chapter! > > Cheers > Phil > > > At 12:06 04/07/2007, you wrote: > >Phil > > > >The DW statistic was done on the residuals after removing the AR1 > >persistence as modelled by the restricted maximum likelihood software. > >That is why the values were close to 2. This procedure is correct > >because the restricted maximum likelihood software widens the error-bars > >to take account of AR1; the DW is a test to see whether any further > >widening is needed, and the results show that it isn't. Steve McIntyre > >probably used the residuals unadjusted for AR1. > > > >I attach our software and 2 sample series. I don't think "R" is used > >widely here. The DW coding is at lines 78-91 of the plot routine > >(p_plot_glm_out_annual.pro). > > > >I hope this helps > > > >Regards > > > >David > > > > > >On Wed, 2007-07-04 at 09:08 +0100, Phil Jones wrote: > > > > > > David, > > > Any idea what McIntyre is doing wrong? Maybe check the pv-wave > > > code and if possible send to me. It is a bit like IDL so I can > > > probably follow. Maybe there is a routine in PV-Wave for DW. > > > I've calculated DW on numerous occasions over the past 30 years > > > for the following: > > > > > > 1. In the riverflow reconstruction work. Here I was reconstructing > > > monthly > > > flows from rainfall. In the southeastern parts of Britain where there > > > is > > > a high groundwater component, rainfall errors propagate for up to a > > > year. > > > Here I got some low DWs with values between 0.5 and 1.5. > > > > > > 2. In reconstruction from paleo data we calculated DW and > > > occasionally > > > got values down to the range 1.2 to 1.5. > > > > > > 90% of the time the values were between 1.8 and 2.4 which is where > > > ours > > > are in the Tables. > > > > > > I've never been able to get values down to the 0.27 and 0.49 that > > > McIntyre > > > is talking about. > > > > > > Tim Osborn thinks he would be able to run your pv-wave script here, > > > so > > > if you can send this with one or two of the series - need the > > > residuals, > > > we can check things out. > > > > > > Also looking at his R script, I don't think DW can have a p-value as > > > such, > > > as it has the two critical levels I was talking about in my response. > > > > > > What I think you have calculated DW based upon is the residuals, > > > so the difference between the original series and the fitted line. > > > > > > If anyone at MOHC is adept at R, it would be very nice to point out > > > what is > > > wrong with his script !!! > > > > > > Cheers > > > Phil > > > > > > > > > > > > > X-YMail-OSG: > > > > Lbmzx_0VM1nx78zXgGstXLvBNj_1bKKWu3u3EfY0YVl7XZyftROTuk6Cnr.ZURgyAw-- > > > > From: "Steve McIntyre" > > > > To: "'Phil Jones'" > > > > Subject: RE: IPCC Table 3.2 > > > > Date: Tue, 3 Jul 2007 13:03:18 -0400 > > > > X-Mailer: Microsoft Outlook, Build 10.0.2627 > > > > X-UEA-Spam-Score: 0.1 > > > > X-UEA-Spam-Level: / > > > > X-UEA-Spam-Flag: NO > > > > > > > > Dear Phil, thanks for the prompt reply, but there are a number of > > > > points that remain very unclear. I am extremely familiar with the > > > > Durbin-Watson statistic as it is familiar to all econometricians. I > > > > use the R language which has a convenient Durbin-Watson test in the > > > > lmtest package (the dwtest function.) When I ran a Durbin-Watson > > > > test on residuals from fitting a trend, I obtained a Durbin-Watson > > > > statistic of 0.49 for an OLS-fitted trend to the HadSST2 series > > > > presently online (over 1850-2005).When I re-fitted a trend line > > > > using the reported slope of 0.038 deg C/decade, I obtained an even > > > > lower Durbin-Watson statistic of 0.27. For this same situation, > > > > you reported a DW statistic of 2.2. I've attached a script in R. > > > > You say that"We used the lag-1 autocorrelations to calculate the > > > > reduced number of degrees of freedom of the residuals." In order > > > > to get a Durbin-Watson statistic of 2.2, you must have done > > > > something to the data that is not a typical procedure and which is > > > > not explained in Diggle. The best guess that I could come up with > > > > as to what you did was that you might have fitted an AR1 arima model > > > > to the trend residuals and then calculated a DW statistic for the > > > > residuals for the arima-fit. However, this is just speculation and > > > > there is no clue in AR4 as to what was done. > > > > > > > > BTW the usual interpretation of the DW test in econometrics is as a > > > > test for first-order autocorrelation, so the exact meaning of using > > > > a DW test "after allowing for first-order serial correlation" is by > > > > no means obvious. Again, if you can direct me to an article > > > > describing the exact procedure that you used together with its > > > > statistical properties, I'd appreciate it. > > > > > > > > Regards, Steve McIntyre > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Phil Jones [mailto:p.jones@uea.ac.uk] > > > > Sent: Monday, July 02, 2007 9:59 AM > > > > To: Steve McIntyre > > > > Cc: Palmer Dave Mr (LIB); david.parker@metoffice.gov.uk; > > > > Kevin Trenberth; Susan Solomon; Martin Manning > > > > Subject: Re: IPCC Table 3.2 > > > > > > > > > > > > Steve, > > > > The Durbin-Watson statistics were in an earlier draft of > > > > the chapter. They were > > > > removed simply for space reasons, as none were significant. > > > > As you can see, > > > > we also removed the lag-1 autcorrelations as well. > > > > > > > > REML comes from Diggle et al 1999 (section 4.5 pp > > > > 64-68). This reference is > > > > given at the end of the chapter. The page numbers refer to > > > > the 1999 edition of the book. > > > > There is a later one available on Amazon, so the page > > > > numbers may differ in that > > > > edition. David Parker programmed the calculations of all > > > > the trends. As far as I > > > > know he didn't do this with any specific statistical > > > > packages. He likely used PV-WAVE > > > > which the Hadley Centre used for almost all their analysis > > > > work. The use of REML > > > > is discussed in Appendix 3.A. > > > > > > > > DW is very simple to calculate. We used the lag-1 > > > > autocorrelations to calculate the > > > > reduced number of degrees of freedom of the residuals. This > > > > number was used with > > > > the DW statistic to estimate the significance. Basically, > > > > any DW value above > > > > about 1.8 is not significant. DW Tables are in some > > > > statistics books. There should > > > > be two significance values (for any DW value and N, here > > > > the effective number > > > > of degrees of freedom). For the lower of these, values > > > > below would be significant. > > > > Values above the upper are not significant. For values in > > > > between nothing can > > > > be said. We were always above the upper value. For random > > > > numbers, the > > > > DW statistic should return a value about 2. There are > > > > different sets of Tables for > > > > different significance levels (1%, 5% etc). We used 5%, > > > > which is generally > > > > the one given in text books. > > > > > > > > Any statistical package would likely not use the reduced > > > > number of degrees > > > > of freedom (reduced based on the lag-1 autocorrelation) > > > > when giving the > > > > significance of DW. Using the reduction to the degrees of > > > > freedom makes the > > > > test harder to pass. > > > > > > > > Phil > > > > > > > > > > > > At 17:36 29/06/2007, Steve McIntyre wrote: > > > > Dear Phil, > > > > In Table 3.2 of IPCC AR4, you refer to Durbin-Watson > > > > statistics for various trend calculations, but do not show > > > > them. Could you please provide me with these statistics. > > > > > > > > I am unfamiliar with any prior use of the Durbin-Watson > > > > statistic ´after allowing for first-order serial > > > > correlationĦ. Could you please provide me your statistical > > > > reference showing how one calculates a Durbin-Watson > > > > statistic ´after allowing for first-order serial > > > > correlationĦ and giving significance levels for the > > > > statistic ´after allowing for first-order serial > > > > correlationĦ. > > > > > > > > Could you please identify the statistical packages used in > > > > your calculation of REML trends and Durbin-Watson > > > > statistics? > > > > > > > > Would it be correct to say that (1) fitted a trend to the > > > > various series; (2) fitted an AR1 arima model to the > > > > residuals from (1)? (3) carried out a Durbin-Watson test on > > > > the residuals from (2)? > > > > > > > > Where applicable, these requests are made under FOI > > > > provisions. > > > > > > > > Thank you for your attention, Steve McIntyre > > > > > > > > > > > > > > > > > Prof. Phil Jones > > > Climatic Research Unit Telephone +44 (0) 1603 592090 > > > School of Environmental Sciences Fax +44 (0) 1603 507784 > > > University of East Anglia > > > Norwich Email p.jones@uea.ac.uk > > > NR4 7TJ > > > UK > > > > > ---------------------------------------------------------------------------- > > > > > > > > >Prof. Phil Jones > >Climatic Research Unit Telephone +44 (0) 1603 592090 > >School of Environmental Sciences Fax +44 (0) 1603 507784 > >University of East Anglia > >Norwich Email p.jones@uea.ac.uk > >NR4 7TJ > >UK > >---------------------------------------------------------------------------- > > > > > >-- > >David Parker Met Office Hadley Centre FitzRoy Road EXETER EX1 3PB UK > >E-mail: david.parker@metoffice.gov.uk > >Tel: +44-1392-886649 Fax: +44-1392-885681 http://www.metoffice.gov.uk > > > > > > > > > > > > > > > > > > > > Prof. Phil Jones > Climatic Research Unit Telephone +44 (0) 1603 592090 > School of Environmental Sciences Fax +44 (0) 1603 507784 > University of East Anglia > Norwich Email p.jones@uea.ac.uk > NR4 7TJ > UK > ---------------------------------------------------------------------------- -- David Parker Met Office Hadley Centre FitzRoy Road EXETER EX1 3PB UK E-mail: david.parker@metoffice.gov.uk Tel: +44-1392-886649 Fax: +44-1392-885681 http://www.metoffice.gov.uk