date: Thu, 10 Jul 2008 14:18:35 -0700 from: Ben Santer subject: [Fwd: Re: [Fwd: JOC-08-0098.R1 - Decision on Manuscript]] to: Professor Glenn McGregor Dear Glenn, I thought you might be interested in this email exchange with Francis Zwiers. It's directly relevant to the third criticism raised by Reviewer 2. With best regards, Ben ---------------------------------------------------------------------------- Benjamin D. Santer Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory P.O. Box 808, Mail Stop L-103 Livermore, CA 94550, U.S.A. Tel: (925) 422-3840 FAX: (925) 422-7675 email: santer1@llnl.gov ---------------------------------------------------------------------------- X-Account-Key: account1 Return-Path: Received: from mail-1.llnl.gov ([unix socket]) by mail-1.llnl.gov (Cyrus v2.2.12) with LMTPA; Thu, 10 Jul 2008 13:08:08 -0700 Received: from nspiron-2.llnl.gov (nspiron-2.llnl.gov [128.115.41.82]) by mail-1.llnl.gov (8.13.1/8.12.3/LLNL evision: 1.7 $) with ESMTP id m6AK864P023034 for ; Thu, 10 Jul 2008 13:08:07 -0700 X-Attachments: None X-IronPort-AV: E=McAfee;i="5200,2160,5336"; a="21284881" X-IronPort-AV: E=Sophos;i="4.30,340,1212390000"; d="scan'208";a="21284881" Received: from nsziron-2.llnl.gov ([128.115.249.82]) by nspiron-2.llnl.gov with ESMTP; 10 Jul 2008 13:08:06 -0700 X-Attachments: None X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao4AAHkJdkjH1BOCmmdsb2JhbACSJgEBAQEBCAUIBxGfMgE X-IronPort-AV: E=McAfee;i="5200,2160,5336"; a="42743336" X-IronPort-AV: E=Sophos;i="4.30,340,1212390000"; d="scan'208";a="42743336" Received: from ecdow130.tor.ec.gc.ca (HELO OntExch1.ontario.int.ec.gc.ca) ([199.212.19.130]) by nsziron-2.llnl.gov with ESMTP; 10 Jul 2008 13:07:46 -0700 Received: from OntExch3.ontario.int.ec.gc.ca ([142.97.202.217]) by OntExch1.ontario.int.ec.gc.ca with Microsoft SMTPSVC(6.0.3790.3959); Thu, 10 Jul 2008 16:07:45 -0400 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-MimeOLE: Produced By Microsoft Exchange V6.5 Subject: RE: [Fwd: JOC-08-0098.R1 - Decision on Manuscript] Date: Thu, 10 Jul 2008 16:07:45 -0400 Message-ID: <33F9E32CDB0917428758DD583E747CC804095CEA@OntExch3.ontario.int.ec.gc.ca> In-Reply-To: <487663E3.1040309@llnl.gov> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Fwd: JOC-08-0098.R1 - Decision on Manuscript] Thread-Index: Acjiw9lJw91pKfupQQOFEbAg5s2/SgAAHtnA References: <48764B2C.5050004@llnl.gov> <33F9E32CDB0917428758DD583E747CC804095CB7@OntExch3.ontario.int.ec.gc.ca> <487663E3.1040309@llnl.gov> From: "Zwiers,Francis [Ontario]" To: X-OriginalArrivalTime: 10 Jul 2008 20:07:45.0611 (UTC) FILETIME=[9E3BB9B0:01C8E2C8] Hi Ben, sure, that would be fine. Cheers, Francis Francis Zwiers Director, Climate Research Division, Environment Canada 4905 Dufferin St., Toronto, Ont. M3H 5T4 Phone: 416 739 4767, Fax 416 739 5700 -----Original Message----- From: Ben Santer [mailto:santer1@llnl.gov] Sent: July 10, 2008 3:33 PM To: Zwiers,Francis [Ontario] Subject: Re: [Fwd: JOC-08-0098.R1 - Decision on Manuscript] Dear Francis, Thanks - this information will be extremely helpful in responding to Reviewer 2. I really do feel that the Reviewer is getting overly exercised about a relatively minor technical point. As you note, the key issue is that, in terms of the statistical significance testing, we are making it easier to get a "Douglass-like" result by using an AR-1 model for calculating the adjusted standard errors. I'm concerned that going down the road proposed by Reviewer 2 could leave us open to unjustified criticism. It would be a shame if Douglass et al. argued (erroneously) that our failure to find significant differences between modelled and observed trends was spurious, and arose primarily from use of higher-order autoregressive models for calculating the adjusted standard errors. Would it be o.k. to share your email with Glenn McGregor and with my other coauthors on the paper? Since you've looked at these issues in detail in your previous papers with Thiebaux and with Hans, your comments would be very useful background information for Glenn. With best regards, Ben Zwiers,Francis [Ontario] wrote: > Hi Ben, > > Sorry the 2nd reviewer is being a pain. As you say, there is already > quite a bit of literature on dealing with dependence in tests of the > mean (and this referree would have been critical if this paper had > gone over that ground again :)). > > Regardless, you might be interested in the attached papers. Both > contain relevant information and might help to formulate a response to > the editor. > > Thiebaux and Zwiers show that the equivalent sample size is hard to > estimate well, particularly from small samples. The approach proposed > by the reviewer is what we termed the "ARMA" method, and it produces > equivalent sample size estimates that have unacceptably large RMSE's > when the sample is small, even when the time series in question is not > very persistent (see Table 6). > > Zwiers and von Storch show the performance of an estimator of > equivalent sample size using the approach you use (i.e., assume the > data are AR(1)). They show that the equivalent sample size tends to be > over-estimated (Table 1) particularly when samples are small, and that > the corresponding t-test tends to operate at significance levels above > the nominal level (i.e., rejects too frequently - Table 2). So using > such a test in effect gives those who would like to reject the null > hypothesis a small leg up. > > Directly comparable results are not shown in the two papers, but you > can infer, from the comparison between equivalent sample size results > (Table > 6 in TZ, Table 2 in ZvS) that the "ARMA" approach for estimating > equivalent sample size would be much less reliable than the approach > that you are using (and thus, the sampled series would have to be very > far from being AR(1) for the ARMA approach to be beneficial). The > absolute key is to keep things as parsimonius as possible - there is > simply not enough data to entertain complex models of the > auto-covariance structure. > > Cheers, Francis > > > Francis Zwiers > Director, Climate Research Division, Environment Canada > 4905 Dufferin St., Toronto, Ont. M3H 5T4 > Phone: 416 739 4767, Fax 416 739 5700 > > -----Original Message----- > From: Ben Santer [mailto:santer1@llnl.gov] > Sent: July 10, 2008 1:47 PM > To: Thorne, Peter; Leopold Haimberger; Karl Taylor; Tom Wigley; John > Lanzante; ssolomon@frii.com; Melissa Free; peter gleckler; 'Philip D. > Jones'; Thomas R Karl; Steve Klein; carl mears; Doug Nychka; Gavin > Schmidt; Steven Sherwood; Frank Wentz > Subject: [Fwd: JOC-08-0098.R1 - Decision on Manuscript] > > Dear folks, > > I just returned from my trip to Australia - I had a great time there. > Now (sadly) it's back to the reality of Douglass et al. I'm forwarding > the second set of comments from the two Reviewers. As you'll see, > Reviewer 1 was very happy with the revisions we've made to the paper. > Reviewer 2 was somewhat crankier. The good news is that the editor > (Glenn McGregor) will not send the paper back to Reviewer 2, and is > requesting only minor changes in response to the Reviewer's comments. > > Once again, Reviewer 2 gets hung up on the issue of fitting > higher-order autoregressive models to the temperature time series used in our paper. > As noted in our response to the Reviewer, this is a relatively minor > technical point. The main point is that we include an estimate of the > standard error of the observed trend. DCPS07 do not, which is the main > error in their analysis. > > In calculating modeled and observed standard errors, we assume an AR-1 > model of the regression residuals. This assumption is not unreasonable > for many meteorological time series. We and others have made it in a > number of previous studies. > > Reviewer 2 would have liked us to fit higher-order autoregressive > models to the T2, T2LT, and TS-T2LT time series. This is a difficult > business, particularly given the relatively short length of the time > series available here. There is no easy way to reliably estimate the > parameters of higher-order AR models from 20 to 30 years of data. The > same applies to reliable estimation of the spectral density at > frequency zero (since we have only 2-3 independent samples for > estimating the spectral density at frequency zero). Reviewer 2's > comments are not particularly relevant to the specific problem we are dealing with here. > > It's also worth mentioning that use of higher-order AR models for > estimating trend standard errors would likely lead to SMALLER > effective sample sizes and LARGER standard errors, thus making it even > more difficult to find significant differences between modelled and > observed trends! Our use of an AR-1 model makes it easier for us to > obtain "DCPS07-like" results, and to find significant differences > between modelled and observed trends. DCPS cannot claim, therefore, > that our test somehow stacks the deck in favor of obtaining a > non-significance trend difference - which they might claim if we used > a > (poorly-constrained) higher-order AR model for estimating standard > errors. > > The Reviewer does not want to "see the method proposed in this paper > become established as the default method of estimating standard errors > in climatological time series". We do not claim universal > applicability of our approach. There may well be circumstances in > which it is more appropriate to use higher-order AR models in estimating standard errors. > > I'd be happy to make a statement to this effect in the revised paper. > > I have to confess that I was a little ticked off by Reviewer 2's > comments. The bit about "wilfully ignoring" time series literature was > uncalled for. Together with my former MPI colleague Wolfgang > Brueggemann, I've fooled around with a lot of different methods of > estimating standard errors, in both the time domain and frequency > domain. One could write a whole paper on this subject alone. Such a > paper would not help us to expose the statistical deficiencies in > DCPS07. Nor would in-depth exploration of this issue lead to the > shorter paper requested by the Reviewer. > > It should take me a few days to revise the paper and draft a response > to Reviewer 2's comments. I'll send you the revised paper and draft > response early next week. Slowly but surely, we are getting there! > > With best regards, > > Ben > ---------------------------------------------------------------------- > -- > ---- > Benjamin D. Santer > Program for Climate Model Diagnosis and Intercomparison Lawrence > Livermore National Laboratory P.O. Box 808, Mail Stop L-103 Livermore, > CA 94550, U.S.A. > Tel: (925) 422-3840 > FAX: (925) 422-7675 > email: santer1@llnl.gov > ---------------------------------------------------------------------- > -- > ---- > -- ------------------------------------------------------------------------ ---- Benjamin D. Santer Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory P.O. Box 808, Mail Stop L-103 Livermore, CA 94550, U.S.A. Tel: (925) 422-3840 FAX: (925) 422-7675 email: santer1@llnl.gov ------------------------------------------------------------------------ ----