cc: "Cawley Gavin Dr (CMP)" , "'Philip D. Jones'" , Gavin Schmidt , "Thorne, Peter" , Tom Wigley date: Fri, 31 Oct 2008 00:48:23 -0600 from: Tom Wigley subject: Re: Possible error in recent IJC paper to: santer1@llnl.gov SEE CAPS Ben Santer wrote: > Dear Gavin, > > Thanks very much for your email, and for your interest in our recent > paper in the International Journal of Climatology (IJoC). There is no > error in equation (12) in our IJoC paper. Let me try to answer the > questions that you posed. > > The first term under the square root in our equation (12) is a standard > estimate of the variance of a sample mean - see, e.g., "Statistical > Analysis in Climate Research", by Francis Zwiers and Hans von Storch, > Cambridge University Press, 1999 (their equation 5.24, page 86). The > second term under the square root sign is a very different beast - an > estimate of the variance of the observed trend. As we point out, our d1* > test is very similar to a standard Student's t-test of differences in > means (which involves, in its denominator, the square root of two pooled > sample variances). > > In testing the statistical significance of differences between the model > average trend and a single observed trend, Douglass et al. were wrong to > use sigma_SE as the sole measure of trend uncertainty in their > statistical test. Their test assumes that the model trend is uncertain, > but that the observed trend is perfectly-known. The observed trend is > not a "mean" quantity; it is NOT perfectly-known. Douglass et al. made a > demonstrably false assumption. > > Bottom line: sigma_SE is a standard estimate of the uncertainty in a > sample mean - which is why we use it to characterize uncertainty in the > estimate of the model average trend in equation (12). It is NOT > appropriate to use sigma_SE as the basis for a statistical test between > two uncertain quantities. The uncertainty in the estimates of both > modeled AND observed trend needs to be explicitly incorporated in the > design of any statistical test seeking to compare modeled and observed > trends. Douglass et al. incorrectly ignored uncertainties in observed > trends. > > I hope this answers your first question, and explains why there is no > inconsistency between the formulation of our d1* test in equation (12) > and the comments that we made in point #3 [immediately before equation > (12)]. As we note in point #3, "While sigma_SE is an appropriate measure > of how well the multi-model mean trend can be estimated from a finite > sample of model results, it is not an appropriate measure for deciding > whether this trend is consistent with a single observed trend." > > We could perhaps have made point #3 a little clearer by inserting > "imperfectly-known" before "observed trend". WE COULD ADD THIS, BUT BE CAREFUL. THE **SAMPLE** TREND **IS** PERFECTLY KNOWN. AFTER ALL, THIS IS A WELL-DEFINED NUMBER. WHAT IS UNCERTAIN IS THE POPULATION TREND THAT IT IS AN ESTIMATE OF. I thought, however, that > the uncertainty in the estimate of the observed trend was already made > very clear in our point #1 (on page 7, bottom of column 2). > > To answer your second question, d1* gives a reasonably flat line in > Figure 5B because the first term under the square root sign in equation > (12) (the variance of the model average trend, which has a dependence on > N, the number of models used in the test) is roughly a factor of 20 > smaller than the second term under the square root sign (the variance of > the observed trend, which has no dependence on N). The behaviour of d1* > with synthetic data is therefore dominated by the second term under the > square root sign - which is why the black lines in Figure 5B are flat. > > In answer to your third question, our Figure 6A provides only one of the > components from the denominator of our d1* test (sigma_SE). Figure 6A > does not show the standard errors in the observed trends at discrete > pressure levels. Had we attempted to show the observed standard errors > at individual pressure levels, we would have produced a very messy > Figure, since Figure 6A shows results from 7 different observational > datasets. > I HOPE THIS IS CLEAR IN THE TEXT OR CAPTION. > We could of course have performed our d1* test at each discrete pressure > level. This would have added another bulky Table to an already lengthy > paper. We judged that it was sufficient to perform our d1* test with the > synthetic MSU T2 and T2LT temperature trends calculated from the seven > radiosonde datasets and the climate model data. The results of such > tests are reported in the final paragraph of Section 7. As we point out, > the d1* test "indicates that the model-average signal trend (for T2LT) > is not significantly different (at the 5% level) from the observed > signal trends in three of the more recent radiosonde products (RICH, > IUK, and RAOBCORE v1.4)." So there is no inconsistency between the > formulation of our d1* test in equation (12) and the results displayed > in Figure 6. > > Thanks again for your interest in our paper, and my apologies for the > delay in replying to your email - I have been on travel (and out of > email contact) for the past 10 days. > > With best regards, > > Ben > > Cawley Gavin Dr (CMP) wrote: >> >> >> Dear Prof. Santer, >> >> I think there may be a minor problem with equation (12) in your >> paper "Consistency of modelled and observed temperature trends in the >> tropical trophosphere", namely that it includes the standard error of >> the models 1/n_m s{}^2 instead of the standard deviation >> s{}^2. Firstly the current formulation of (12) seems at odds >> with objection 3 raised at the start of the first column of page 8. >> Secondly, I can't see how the modified test d_1^* gives a flat line in >> Figure 5B as the test statistic is explicitly dependent on the size of >> the model ensemble n_m. Thirdly, the equation seems at odds with the >> results depicted graphically in Figure 6 which would suggest the >> models are clearly inconsistent at higher levels (400-850 hPa) using >> the confidence interval based on the standard error. Lastly, (12) >> seems at odds with the very lucid treatment at RealClimate written by >> Dr Schmidt. BEN -- DID YOU RESPOND TO THIS? BY THE WAY, I NOTE THAT GAVIN SCHMIDT IS NOT A STATISTICIAN. >> >> I congratulate all 17 authors for an excellent contribution that I >> have found most instructive! VERY PLEASING COMMENT !!!! >> >> I do hope I haven't missed something - sorry to have bothered you if >> this is the case. >> >> best regards >> >> Gavin >> > >