date: Thu, 16 Sep 2004 09:46:34 +0100 from: Suraje Dessai subject: Fwd: Murphy et al. to: James Risbey , Jeroen van der SLuijs , Roger Pielke , Andrea Saltelli ,Mike Hulme , Milind Kandlikar Folks, Here's the first response to our brief communication from an ex-author of Murphy et al.! Perhaps someone would like to lead a reply to his points since some of his new points (and attached document) are just way over my head! Suraje >Subject: Murphy et al. >Date: Wed, 15 Sep 2004 14:26:12 +0100 >X-MS-Has-Attach: yes >X-MS-TNEF-Correlator: >Thread-Topic: Murphy et al. >Thread-Index: AcSbJ4zM2q56Fxh8QhCnvcniw9CQ3g== >From: "Myles Allen" >To: "Suraje Dessai" >X-UEA-MailScanner-Information: Please contact the ISP for more information >X-UEA-MailScanner: Found to be clean >X-UEA-MailScanner-SpamScore: s > >Dear Suraje, > > > >David Stainforth has passed me a copy of your Brief Communication arising >from Murphy et al, which I read with great interest. I certainly think it >s a good thing that these issues should be aired, and the points you raise >about the paper are very sensible ones. I should say immediately (as you >may well be aware) that I am an ex-author of that paper, having left the >author list voluntarily when it became clear that the (possibly infinite) >time required to reconcile our views on analysis methodology was not >compatible with very strong pressure from the Met Office hierarchy to get >the paper published quickly. As a result, I probably understand what was >actually done better than most and certainly better than you could be >expected to glean from the rather opaque wording of the paper. > > > >I suspect that Murphy et al would feel they have a satisfactory response >to some of the points raised by your Brief Communication as it stands, but >this does not mean their paper is immune from criticism. Here are some >specific thoughts: > > > >1) Your point that the expert s selections of parameter ranges are >closely related to the default values, so there is a risk of >double-counting observations: this is, of course, true, since none of >these parameters correspond to directly measurable quantities and only >make sense in the context of this specific model. Nevertheless, they will >simply deny it and argue that their extreme values depend on their >colleagues understanding of the physics and not on any observational >constraints. This argument has been ongoing for well over a year with no >sign of converging. I suspect you will get the same response that I did. > >2) You are on stronger ground focusing on the lack of a formal >expert elicitation procedure: a key issue here is their decision to treat >the high and low parameter values as extreme cases (i.e. 0th and 100th >percentiles!). I did manage to persuade them to include a sensitivity >study treating their experts values as 15th and 85th percentiles, and they >will no doubt point to the fact that results with the observational ( CPI >) constraint don t change much, but the CPI itself is problematic, for >reasons I ll go into below. Without the CPI constraint, the impact of >relaxing the interpretation of the expert ranges is to move the upper >bound on sensitivity up to 8K. A more fundamental problem is that these >parameter ranges were re-interpreted as the model responses came in: when >he started out on all this, David Stainforth was only asking for possible >high and low values (it wouldn t occur to me to ask a modeler to give me >the 0th percentile of anything, since I don t see how they could). I only >heard of the decision to consider these ranges as extreme when at least >half of the runs were complete which makes it very hard to separate prior >interpretation of parameters from prior expectations about sensitivity. If >one were very cynical, one might argue that with so many knobs to twiddle >on the methodology, it is hardly surprising they came up with a range so >astonishingly close to that of Senior and Mitchell (1993). > >3) Murphy et al don t neglect non-linearity altogether: they >performed 13 multiply-perturbed cases to assess the error in their linear >prediction model and added this error to their predicted lambda values >before computing distributions. The problem is that the 13 verifying cases >only span a relatively small range of sensitivities (3.1-4.9K), so there >is no way of checking whether non-linearity increases as we go to lower >lambda values (higher sensitivities). Crucially, also, they model the >impact of non-linearity as additive Gaussian noise on lambda, which makes >highly non-linear responses very unlikely. In fact, as you would expect, >we find when linearity fails (which is does by more than 2s.d. in about >25% of cases) it tends to fail completely, so it has much more impact on >the tails of the distribution than a simple additive Gaussian error. > > > >And now for some points you don t make: > > > >1) Their results depend critically on a single run, the low >entrainment rate case which gave them a sensitivity of about 7. If you >remove this run from the ensemble (and entrainment rate wasn t even >included in the original short-list of parameters to be perturbed, so it >was a close shave), you get a significantly narrower range. I guess one >could work this out from the table in the supp. info, but in my view >papers should be a bit more open than that. Early on they showed a >histogram of the actual sensitivities obtained, but that was dumped >despite my protests. > >2) Their assumption that parameter-perturbations are linear in >lambda combined with their decision to sample parameters uniformly between >the expert-determined limits is equivalent to assuming a uniform prior on >lambda, or the inverse of climate sensitivity, before any of the >perturbed-physics results come in (there is an additional complication >that they impose a further weighting in cases where they find the >parameters are not linear in lambda but in my view this is a red-herring >because none of these parameters corresponds to anything directly >measurable, so whether p or 1/p or p^n is used in the model is just a >historical accident). A uniform prior in lambda assumes a sensitivity of >0.5-0.55 is as likely as one of 3-6 before the study commences, which >explains why they get such a low upper bound on sensitivity. If we assume >a uniform prior on sensitivity but otherwise weight cases exactly as >Murphy et al do, we find no upper bound on sensitivity (see enclosed >note). Interestingly (and, of course, you would have no way of knowing >this), the first version of Murphy et al submitted in the summer of 2003 >used something much closer to a uniform prior on sensitivity, came up with >a much higher upper bound on sensitivity and was thrown out by the >reviewers because it didn t provide new information on sensitivity& a >fascinating example of the positive publication bias that makes >meta-analyses like the IPCC such a nightmare. > >3) Their CPI depends on an arbitrary scaling factor on the noise >which allows them to adjust how much impact they want the observations to >have pretty much at will (they scale up the noise by a factor of 20, and >one could argue for a further factor of 20). So you can make the red >curves as tight as you like at one extreme, or just like the blue curves >at the other. There is an additional problem that the functional form of >the CPI they use is appropriate only for a likelihood measure in one >dimension, and it s a very high-dimensional distance measure (they should >be using a Fisher not a Gaussian distribution), but this is starting to >get a bit technical (although it does make a substantial difference to the >relative weights given to different models). > > > >I enclose for your interest a 2-pager I wrote to accompany the >re-submission of Stainforth et al (the main climateprediction.net paper), >which should happen in the next week or so. Our problem is that the >reviewers of Stainforth et al are asking why we didn t do what Murphy et >al did, and demanding that we do it before publishing our paper. So we >have to explain at least to those reviewers why we aren t so keen. I wasn >t planning on circulating this note any further because I figured I was >probably the only one who cared. I m very pleased to see I was wrong. I d >be interested in your views and those of your co-authors, who have clearly >thought hard about Murphy et al, about the points raised and whether you >think there would be interest in taking them further. Certainly the point >about the prior will come back to haunt us, although I m not so sure it s >worth having a go at the CPI, since it s a bit like flogging a dead horse. >Feel free to share the enclosed with your co-authors, but please don t >circulate it further. > > > >Regards, > > > >Myles > >Climate Dynamics Group >Atmospheric, Oceanic and Planetary Physics >Department of Physics, University of Oxford >Tel: 44-1865-272085/095 >Fax: 44-1865-272923 >E-mail: myles.allen@physics.oxford.ac.uk > > ___________________________________________________________ Suraje Dessai PhD Researcher Tyndall Centre for Climate Change Research Zuckerman Institute for Connective Environmental Research School of Environmental Sciences University of East Anglia Norwich, NR4 7TJ United Kingdom Tel: + 44 (0)1603 593911 Fax: + 44 (0)1603 593901 E-mail: s.dessai@uea.ac.uk Web: http://www.tyndall.ac.uk ___________________________________________________________ Attachment Converted: "c:\eudora\attach\priorlhood.pdf"