cc: Caspar Ammann , rbradley@geo.umass.edu, tcrowley@duke.edu, mhughes@ltrr.arizona.edu, omichael@princeton.edu, t.osborn@uea.ac.uk, jto@u.arizona.edu, Scott Rutherford , Tom Wigley , p.jones@uea.ac.uk, ckfolland@meto.gov.uk, tkarl@ncdc.noaa.gov date: Mon, 13 Oct 2003 14:21:14 -0400 from: "Michael E. Mann" subject: Re: draft to: Keith Briffa , Kevin Trenberth Dear Keith, Thanks a bunch for your comments, all of which are very helpful. I've attempted to incorporate these, along w/ those of Tim, Tom W, in the latest draft, as per my previous email. There is one point I wanted to comment on further, regarding the issue the how a potentially non-stationary time series (e.g. one with a significant trend near the end) should or should not be smoothed. The issue, happily, is not relevant to our Eos reply, because the proxy reconstruction (which ends in 1980) was smoothed based on a procedure that does not assume a continuation of the trend, the issue that seems controversial here. However, I do think that this is particularly important in smoothing of the instrumental surface temperature series. Those uninterested in this particular discussion need not read any further, but I would encourage those interested (particular as it might involve decisions about how to smooth time series in the next IPCC report) to read on... mike comment on "minimum roughness" constraint in smoothing time series with significant trends near the end of the series: I favor a "minimum roughness" constraint (which tends to retain trends near the end of a smooth) for smoothing of a series with a significant trend near (either) end for the following reason. Smoothing of a time series with a zero phase (i.e., centered) filter reflects a non-unique transformation of the data. It is non-unique because there is no information on one half of the filter center at either the beginning or end of the series. Because of that lack of information, an additional a priori constraint has to be placed on the filtering process. This contraint reflects an assumption about the data outside of the available interval. This can be cast (as often it is in the signal processing literature) as an inverse problem w/ non--unique constraints. The typical constraints that are typically employed [see Park, J., Envelope estimation for quasi-periodic geophysical signals in noise: A multitaper approach, in Statistics in the Environmental and Earth Sciences, edited by A. T. Walden and P. Guttorp, pp. 189-219, Edward Arnold, London, 1992.] involve a (i) "minimum norm", (ii) "minimum slope", and (iii) "minimum roughness" solution for the underlying statistical model. A possible way of insuring a reasonably objective smoothing of the time series is to minimize, among all possible linear combinations of these 3 models, that which minimizes the mean-square-misfit with respect to the raw data [see Ghil, M., Allen, M.R., Dettinger, M.D., Ide, K., Kondrashov, D., Mann, M.E., Robertson, A.W., Tian, Y., Varadi, F., Yiou, P., Advanced Spectral Methods for Climatic Time Series, Reviews of Geophysics, 40 (1), 1003, doi: 10.1029/2000RG000092, 2002.] Such an approach will favor contraints (i) or (ii) for a data series with stationary behavior in the mean near the end, and probably (iii) if there is non-stationary behavior (i.e. a long-term trend or, more specifically, a statistically signfiicant trend over the last 1/2 smoothing filter window width). (i) is called the "minimum norm" constraint because it chooses the smallest of all models for the smoothed data--it involves the minimization of the 0th derivative of the smooth. The implicit assumption is that the mean outside the available interval is equal to the mean of the available data. This is clearly wrong if there is a signfiicant trend. (ii) is called a "minimum slope" inversion. In this case, the solution involves the minimization of the mean-square first derivative of the model--the solution will favor a smooth that approaches the boundary with zero slope--the implicit assumption is that the mean outside the available interval may be different from the mean inside the available data, but that this reflects a step change in the mean value rather than any trend near the boundary. Ad hoc methods which pad the end of the series with e.g. the mean of the last 1/2 filter width, in essence, implement this constraint. (iii) is called the "minimum roughness" solution because it minimizes the mean-square 2nd derivative among all possible models. It favors a smooth with an inflection point at the boundary. and is consistent with the assumption that a trend exists as one approaches the boundary. Mathematically, it is most simply implemented in the time domain by padding the series with an extension of the trend over the past 1/2 filter width. However, the constraint can be implemented directly in the frequency-domain inversion [Park, 1992; Ghil et al, 2002]. This is the proper choice if there is a statistically significant trend within the final 1/2 smoothing window width of the edge of the data. Objectively, it is defensible in those situations where this choice minimizes the mean-square misfit with the raw data over all possible linear combinations of choices (i), (ii), and (iii). For the global or hemispheric mean instrumental series from 1856-present, that condition holds. So my concern is the opposite of Keith's. I believe that smoothing routines that explicitly invoke an assumption of stationarity are problematic when the series clearly is not stationary. This assertion has a rigorous foundation in the inverse theory literature [see Park, 1992 and other references therein] and is not a new or subjective approach. Though the point is actually irrelevant to the discussion at hand for reasons mentioned earlier, I would actually like to see this discussed, because I think that we (e.g. in IPCC '01) may have underplayed the significance of recent warming by employing improper boundary conditions on smooths of records like the global temperature or hemispheric temperature series. I've cc'd Chris Folland and Tom Karl in on this discussion, for their comments... Mike and all Hi , just back from a trip and only now catching up with important emails. Given the restricted time and space available to furnish a response to SB comments , I offer the following mix of comment and specific wording changes: I agree that the S+B response is designed to deflect criticism by confusing the issues rather than answering our points. In fact they fail to address any of the 3 specific issues we raised Namely , 1. the need for critical evaluation of proxy inputs , 2. the need for a consistent assimilation of widespread (dated and well resolved ) records, 3. the essential requirement for objective/quantitative calibration (scaling) of the input records to allow for assessment of the uncertainties when making comparisons of different reconstructions and when comparing early with recent temperatures. Their own , ill-conceived and largely subjective approach did not take account of the uncertainties and problems in the use of palaeodata that they chose to highlight in their opening remarks. I would be in favour of stating something to this effect at the outset of our response. Also , as regards the tree-ring bit , I fully concur with the sense of your text as regards Section 1, but suggest the following wording (to replace ",rarely for annual ring widths, and almost entirely at higher latitudes.") "but in certain high-latitude regions only. Where this is the case , these relatively recent (ie post 1950) data are not used in calibrating temperature reconstructions. In many other (even high-latitude) areas density or ring-width records display no bias." In the spirit of healthy debate - I agree with Tim's remarks , warning against presenting a too sanguine impression that the borehole debate is closed ( though I do think it is closing!). I also believe , as you already know, that the use of a recent padding algorithm to extend smoothed data to the present time, is inappropriate if it assumes the continuation of a recent trend. This is likely to confuse , rather than inform, the wider public about the current climate state . Finally , I repeat my earlier remarks (made before EOS piece published) that we are missing an opportunity to say that a warm Medieval period per se is not a refutation of anthropogenic warming , {as its absence is no proof}, if we do not understand the role of specific forcings (natural and anthropogenic) that influenced medieval and current climates. Cheers Keith At 12:48 PM 10/9/03 -0600, Kevin Trenberth wrote: Hi all Here are my suggested changes: toned down in several places. Tracking turned on Kevin Michael E. Mann wrote: Dear co-authors, Attached is a draft response, incorporating suggestions Kevin, Tom W, and Michael. I've aimed to be as brief as possible, but hard to go much lower than 750 words and still address all the key issues. 750 words, by the way, is our allotted limit. Looking forward to any comments. Feel free to send an edited version if you prefer, and I'll try to assimilate all of the suggested edits and suggestions into a single revised draft. If you can get comments to me within the next couple days, that would be very helpful as we're working on a late October deadline for the final version. Thanks for your continued help, mike ______________________________________________________________ Professor Michael E. Mann Department of Environmental Sciences, Clark Hall University of Virginia Charlottesville, VA 22903 _______________________________________________________________________ e-mail: <[1]mailto:mann@virginia.edu >mann@virginia.edu Phone: (434) 924-7770 FAX: (434) 982-2137 [2]http://www.evsc.virginia.edu/faculty/people/mann.shtml -- **************** Kevin E. Trenberth e-mail: <[3]mailto:trenbert@ucar.edu>trenbert@ucar.edu Climate Analysis Section, NCAR <[4]http://www.cgd.ucar.edu/cas/>[5]www.cgd.ucar.edu/cas/ P. O. Box 3000, (303) 497 1318 Boulder, CO 80307 (303) 497 1333 (fax) Street address: 1850 Table Mesa Drive, Boulder, CO 80303 -- Professor Keith Briffa, Climatic Research Unit University of East Anglia Norwich, NR4 7TJ, U.K. Phone: +44-1603-593909 Fax: +44-1603-507784 [6]http://www.cru.uea.ac.uk/cru/people/briffa/ ______________________________________________________________ Professor Michael E. Mann Department of Environmental Sciences, Clark Hall University of Virginia Charlottesville, VA 22903 _______________________________________________________________________ e-mail: mann@virginia.edu Phone: (434) 924-7770 FAX: (434) 982-2137 [7]http://www.evsc.virginia.edu/faculty/people/mann.shtml