cc: Edward Cook date: Fri, 13 Jun 2008 12:08:32 -0400 from: Edward Cook subject: Fwd: Some PR Challenge comments to: Keith Briffa Just what I sent Casper. FYI only. ================================== Dr. Edward R. Cook Doherty Senior Scholar and Director, Tree-Ring Laboratory Lamont-Doherty Earth Observatory Palisades, New York 10964 USA Email: [1]drdendro@ldeo.columbia.edu Phone: 845-365-8618 Fax: 845-365-8152 ================================== Begin forwarded message: From: Edward Cook <[2]drdendro@ldeo.columbia.edu> Date: June 13, 2008 10:46:43 AM EDT To: Caspar Ammann <[3]ammann@ucar.edu> Cc: Edward Cook <[4]drdendro@ldeo.columbia.edu>, Nicholas Graham <[5]ngraham@hrc-lab.org>, "Rosanne D'Arrigo" <[6]druidrd@ldeo.columbia.edu> Subject: Some PR Challenge comments Hi Casper, As you are aware, I have some serious concerns about how the PR Challenge has been formulated. So as of now, I have no intention of participating in it. I mentioned to you in Trieste that I thought the better way to go was to use real climate data and real proxies for a range of reconstruction problems (temperature, precipitation, drought, pressure on local, regional, hemispheric scales) such that TRUE uncertainties in the reconstructions and the methods used could be objectively and realistically evaluated. The work Keith, Phil, and I did for NATO is a very useful template for doing so and the results can be objectively evaluated by setting up the testing and comparison procedures in an a priori way. However, I also recognize the value of pseudo-proxies in this context and support the original PR Challenge idea of using model output to produce pseudo-climate and pseudo-proxy data for testing as well. Indeed, one could create the pseudo-data sets to match fairly closely the properties of the actual data sets chosen for testing. Including both real and pseudo data in the experiments would dramatically strengthen your PR Challenge in my opinion because it would reduce the dependence of the results and their interpretations on the considerable uncertainty that still remains in how to generate pseudo-proxy data using forward-modeling methods. Tree rings are most developed here for forward modeling I think (e.g. the Vaganov-Shashkin model), but to assume that the V-S model can magically model the tremendous complexity of growth responses to climate in tree rings at the genus and species levels is simply wrong. Maybe the V-S model will get a lot of it right, but it will likely be in a highly idealized way that will very likely under-estimate the true uncertainty in tree-ring reconstructions. I am also not sure that useful forward models for other proxies like corals even exist yet. Perhaps the biggest complaint I have with the PR Challenge is its "double-blind" design that gives the tester no information on the problem being tested even though no such situation ever occurs in practice. At that level alone, the "double-blind" design used in the PR Challenge is very odd. Regardless, I fully understand the value of "double-blind" statistical experiments as the only way to test for true causality in, for example, cancer drug tests. But this is simply not the kind of problem we are dealing with in the proxy reconstruction game (with all due respect to your statistician colleagues). We are much more involved in something more akin to epidemiological hypothesis testing on real pre-existing data, like the association between smoking and lung cancer in humans. Do medical epidemiologists perform "double-blind" experiments to determine if smoking causes lung cancer? Basically no. They use the data available to them to answer that hypothesis with a high degree of statistical certainty without the need to conduct "double-blind" experiments on humans (not possible for smoking and lung cancer in any case). So in my opinion, the "double-blind" approach is unnecessary and, I would argue, even inappropriate here. I also don't like it because that is not the way we should be conducting our science. When I worked with Keith and Phil for a month at CRU on our NATO test data sets and reconstruction methods, we worked in a completely objective and open way that enabled us to debate various options and interpretations of our programs and reconstructions. We were also able to thoroughly test our programs using identical data for the special case of full MLR. This enabled us to be absolutely certain that we could compare results in the various best-subset cases that are typically used for reconstructing climate from tree rings. There does not appear to be any such mechanism for doing so in your PR Challenge, so differences found may be as much related to odd programming matters rather than the methods being compared. In any case, comparison of reconstruction methods should be conducted openly and that is the way that I love to work. So if the "double-blind" requirement were removed from the PR Challenge, I would be much more inclined to participate. I am not adverse to being proven wrong (or "less right" perhaps) in an open environment that allows for direct "give-and-take" on the merits of the cases being tested and argued. A "double-blind" approach is explicitly designed to eliminate the scientific openness needed to evaluate and debate reconstruction methods in my opinion. At that level, and for reasons stated above, I reject it. Regards, Ed ================================== Dr. Edward R. Cook Doherty Senior Scholar and Director, Tree-Ring Laboratory Lamont-Doherty Earth Observatory Palisades, New York 10964 USA Email: [7]drdendro@ldeo.columbia.edu Phone: 845-365-8618 Fax: 845-365-8152 ==================================