cc: Edward Cook , Ned Guttman , potter4@bellsouth.net, Keith Briffa , "Klein Tank, Albert" , Richard Heim date: Mon, 4 Jun 2007 08:17:25 -0400 from: Edward Cook subject: Re: scPDSI program to: Gerard van der Schrier Hi Gerard, Thanks for the email. The choice of the calibration period on the PDSI estimates outside that period is something that probably affects all versions of the PDSI. The SPI ought to be similarly affected if the quantiles are based on a pdf fitted to a sub-period of data. That being the case, unless the calibration period is a truly unbiased expression of longer-term variability, I am not sure what can be done about it. I also agree that one should use the longest calibration period possible, but globally it is not really feasible to start before, say, 1950 over large areas of Asia. So this will make pre-1950 PDSI variability strongly conditional on climate statistics of the calibration period. Cheers, Ed ================================== Dr. Edward R. Cook Doherty Senior Scholar and Director, Tree-Ring Laboratory Lamont-Doherty Earth Observatory Palisades, New York 10964 USA Email: [1]drdendro@ldeo.columbia.edu Phone: 845-365-8618 Fax: 845-365-8152 ================================== On Jun 4, 2007, at 5:47 AM, Gerard van der Schrier wrote: Dear Ned, Ed, Thanks for your emails. I must admit that I haven't had the chance to work with the sc-pdsi program lately (=since 1 year or so......). The plans are to start working on this again in next autumn/winter. About the sensitivity to the length of the calibration period: The self-calibrating aspect makes that ca. 2% of the available months in the calibration period are in the "extemely dry" category (and 2% will be in the "extremely wet" category). The 2% threshold is more-or-less abritrary. The parameters in the PDSI-calculation are set to satisfy this constraint. When the calibration period does not cover the complete period over which the sc-pdsi is calculated, then you can expect PDSI values (hugely) outside the [-4, 4] range. Especially when a dry (or wet) period happens to fall outside the calibration period. I guess this was the reason why Wells et al. in the paper in which they introduce the sc-pdsi, advise to make the calibration period as long as possible. Note that this leaves another problem of the PDSI untouched: the median of a distribution of sc-pdsi values is not guaranteed to be zero. This remains unnoticed when a probability distribution is calculated over many different timeseries, but it can be seen in individual timeseries (e.g. fig. 4 of Wells et al. (2004), J. Clim. vol. 17, pp. 2335-2351). Obviously, I also included a spin-up period of 10 years for the waterbalance model, using climatological data. There are indeed problems with the sc-pdsi which need to be looked at before it is ready for an operational application. Nevertheless, we think it compares favourably to the "traditional" PDSI. Our plans are to calculate sc-pdsi for the updated global temperature and precipitation datasets of the Climatic Research Unit. There are some vague plans to use the index in a more operational way, and apply it to a network of European data which are continuously updated. Obviously, much needs to be done before this works satisfactory. Ned, could I get back to you when I actually start working on this? Many thanks for your thoughts. Best Regards, Gerard ---------------------------------------------------------- Gerard van der Schrier Royal Netherlands Meteorological Institute (KNMI) dept. KS/CK PO Box 201 3730 AE De Bilt The Netherlands [2]schrier@knmi.nl +31-30-2206597 www.knmi.nl/~schrier ----------------------------------------------------- Ed, Gerhard, After looking at the structure of Gerhard's code (module oriented) and Richard's code (hardwired for I/O), I decided that the easiest way to write something that we can use at NCDC would be to incorporate Gerhard's subroutines into Richard's program. The result is attached. I seem to remember that there may have been a sign problem in Gerhard's code, but I am not sure now. I did this a month or two ago, and then started getting ready for retirement (tomorrow is my last day). I did check out Gerhard's code with Palmer's and Nate Wells' papers to make sure the code truly reflects both Palmer's and Wells' logic. I did write a simple subroutine to calculate the regressions rather than use the sophisticated library subroutine that Gerhard used. The regression results from the two methods are close but not identical. Then I ran the attached program on US Climate Division data (1895-2006). The output was not satisfactory - I was getting outrageous PDSI values in some divisions for some periods. The calibration period for calculating the CAFEC values was 1950-2000, the period that is used for the North American Drought Monitor (Mexico does not have long term data). As a test, I reran the program using a calibration period of (I think) 1900-2000 and the results were much better. After thinking about it for a while, but not a long time, it is logical that the PDSI is sensitive to the calibration period, since the components of the water balance at the start of calibration period reflect existing and immediately preceding conditions such as droughts and wet spells. This is likely skewing the output of the sc-pdi in an unsatisfactory manner. I have not pursued this, and would hope to follow through with it somehow with contract work after tomorrow, but it is an issue that needs to be resolved. I also think the sc-pdi needs to be thoroughly tested before it is used in any kind of operational manner. There are too many things in the process that can go wrong when programmed. I also think that the characteristics of the sc-pdi need to be evaluated and affirmed. For example, is the method truly spatially invariant? Let me give you an email address where you can reach me after tomorrow (or maybe today since I don't know when all my computer accounts will be killed): [3]potter4@bellsouth.net Bottom line - I would not use the sc-pdi now. It needs much more review. Cheers, -- ---------------------------------------------------------- Gerard van der Schrier Royal Netherlands Meteorological Institute (KNMI) dept. KS/CK PO Box 201 3730 AE De Bilt The Netherlands [4]schrier@knmi.nl +31-30-2206597 www.knmi.nl/~schrier ----------------------------------------------------------