cc: Keith Briffa , Edward Cook date: Mon, 11 Apr 2005 07:33:06 -0400 from: Edward Cook subject: Re: North American SC-PDSI to: Gerard van der Schrier Hi Gerard, Thanks for the EOT source code. I will have a go at it and let you know if I encounter any problems with it. It looks like it won't be a problem to deal with. Interesting plot from the Dai data as well. Aside from what appears to be a calibration period around months 1000-1400, the remainder of the mins and maxs look like they have been mowed to keep them from getting too extreme as you suggest. I am afraid that the Dai data are looking more and more unreliable. At some point, this will need to be brought up to Dai. Of course, Trenberth will go nuts because his name is on the publications associated with the data. Below is an html version of an article written by Ned Guttman that discusses, among other things, the modified PDSI that does not require backtracking. Some of the equations and math terms did not translate from the paper to html, which is why there are occasional gaps in the rendering. A more accessible description of the modified PDSI can be found at http://nadss.unl.edu/PDSIReport/pdsi/, which is a fine review report by Nathan Wells, author of the SC-PDSI. Go to "The Weighted PDSI" to get a detailed description of the modified PDSI used by NOAA for operational purposes (see below). I hope this helps. Cheers, Ed CourierDETERMINATION OF DROUGHT FREQUENCIES FOR THE NATIONAL DROUGHT ATLASTimes    CourierNathaniel B. GuttmanTimes  CourierNational Climatic Data Center, Asheville, NCTimes    Courier 1.     INTRODUCTIONTimes  Courier        The meaning of the word "drought", as used in the National Drought Atlas, is the condition of widespread and negative economic, social, and environmental impacts resulting from less water than expected.  The water shortfall can come from a lack of precipitation, a deficiency in water storage and distribution systems, or inefficient use of water.  "Water management" refers to the planned intervention of man in the hydrologic cycle of rainfall, runoff and evapotranspiration in order to enhance water uses and reduce water hazards.  Pertaining to drought, it refers to reducing the adverse impacts of drought, to planning activities preceding a drought, and to operational activities during a drought.Times          Courier   One indicator of drought that is also used for water management is the Palmer Drought Severity Index.  This index, which was developed over 25 years ago (Palmer, 1965), relates the normal amount of precipitation that should have occurred in an area to that which actually did occur. ("Normal" is used in the sense that the moisture supply satisfied the average or climatically expected percentage of the absolute moisture requirements of the area.)  The index is relatively independent of time and space.  It is a representation of what Palmer called meteorological drought, i.e., an evaluation of meteorological anomalies characterized by prolonged and abnormal moisture deficiencies.    Times         Courier   This article describes the Palmer Drought Index and the data that were used to compute the Index for the National Drought Atlas.  It discusses the methodologies, assumptions and limitations used to prepare drought frequencies.  Times  Courier 2.     PALMER INDEX Times       Courier   The Palmer Drought Severity Index (PDSI) is based on a calculated water balance using methods devised by Thornthwaite (Thorthwaite and Mather, 1955).  For month i, the weighted moisture anomaly index, , is Times  Courierwhere k is a weighting factor that allows for spatial comparison; P, PE, PR, PRO, and PL are the actual precipitation, computed potential evapotranspiration, potential recharge (net gain in soil moisture), potential runoff, and potential loss of soil moisture, respectively, for the month;  and  are, respectively, the coefficients of evapotranspiration (mean evapotranspiration divided by the mean potential evapotranspiration), recharge (mean recharge divided by the mean potential recharge, runoff (mean runoff divided by mean potential runoff), and loss (mean loss divided by the potential runoff).Times         CourierA drought index, , for month i is computed on an incremental basis such that each successive month is evaluated in terms of its contribution to the severity of drought.  A recursive relationship linking one month to the next is defined asTimes           CourierComputation of the PDSI requires a tracking of the beginning, establishment and ending of wet and dry spells.  In practice, three values of  are computed simultaneously each month.  The first, a wetness index , is the PDSI for a wet spell that might become established.  The second, a dryness index , is the PDSI for a dry spell that might become established.  The third, , is the PDSI for any wet or dry spell that has definitely become established, i.e., .  It is also necessary to determine the percent of moisture required to end a spell that is actually received, where the end of spell is defined as .Times         CourierIf a spell has become established and the percent of moisture needed to end the spell is zero, then the PDSI is equal to . This condition also means that  and  are equal to zero.  If a wet spell is established but dry months occur so that the percent of moisture needed to end the spell is between 0 and 100, then  is computed along with .  If the percentage does not return to zero and reaches 100 in month m, then the PDSI for months i through m are the  values because the wet spell has definitely ended and the dry spell has begun.  If the percentage does return to zero in month m, then the PDSI for months i through m  are the  values because the established wet spell has not ended.  Similar computations are made for established dry spells using  and . Once an established spell has ended, the value of  equals zero.Times         CourierIf a spell is not established,  and  are computed simultaneously until the month i in which either or . At this time a new spell has become established and the procedure described in the preceding paragraph is followed.  The PDSI values for the months i backward to t are  if a wet spell has become established or  if a dry spell has become established.  At the point backwards in time when the appropriate index (wet or dry) is zero, the PDSI becomes the other index (dry or wet) value until it reaches zero.Times         CourierTable 1, taken from Palmer (1965), arbitrarily relates an index value to a qualitative measure of wetness or dryness.  Since  Palmer's original intent was to describe drought, he postulated that an index value of -4 would spell economic disaster in any region in which the established economy is significantly dependent upon the weather for its moisture supply.  Values between 0 and -4 represent conditions that are scaled between normal and economic disaster.  The meaning of positive index values is a mirror image of the meaning of the negative values.  It is based on Palmer's assumption that the abnormal moisture departures which lead to drought would lead to wet periods if they were positive instead of negative.Times         CourierThe PDSI is a retrospective index because current values depend on future conditions.  It is therefore useful as a climatological indicator but not as a "real-time" index for making operational decisions.  The necessity for an index that could be used operationally led to a variation of the PDSI known as the Palmer Hydrologic Drought Index or PHDI (Karl, 1986).  The PHDI avoids the backtracking problem, and can therefore be used in "real-time", by selecting  as the index value whenever the percent of moisture needed to end an established spell is between 0 and 100 and by selecting the nonzero  or  during the onset of a wet or dry spell.  The PDSI and the PHDI are identical during an established spell. They differ, however, during the onset and ending of a spell.  The PHDI during the months that a spell is incipient may change sign from month to month depending on the magnitude of the current month's departure from normal conditions (i.e., ); the PDSI values are those for the wet (dry) index for all the incipient months of a wet (dry) spell that does become established.  At the end of a spell, the sign of the PDSI values will reverse to signify the end sooner than with the PHDI values.Times         Courier   The tendency of the PHDI to switch signs during incipient conditions, as well as the slow response of the PHDI at the end of a spell, led to development of the modified PDSI as a better representation of existing conditions for real-time, operational use (Heddinghaus and Sabol, 1991).  During near normal or incipient conditions when  , the modified PDSI is equal to the  or  with the largest absolute value.  During an established spell, the modified PDSI is equal to the PDSI.  During the end of an established spell, the modified PDSI is equal to the weighted average of  and  if the spell is dry or  and  if the spell is wet.  The weighting factors are [1-(PCT/100)] for  and (PCT/100) for , where PCT is the percent of moisture needed to end the spell.Times         Courier   The National Drought Atlas is intended for use in real-time by water managers.   Because the modified PDSI values are readily available from the Weekly Weather and Crop Bulletin, the Weekly Climate Bulletin, or the Climate Analysis Center's Climate Dial-Up Service, it is the modified Palmer Index whose values are summarized in the Atlas.  Times  Courier 3.     LIMITATIONS OF THE PALMER INDEX Times         Courier   Since the index is widely used as a drought assessment tool for water management and planning activities (Wilhite et al., 1991; Wilhite, 1990; Grigg and Vlachos, 1990; South Carolina Water Resources Commission, 1987; American Meteorological Society, 1986), it is important to summarize its limitations.  Palmer developed his index from temperature and precipitation data observed in the Great Plains (Palmer, 1965).  He specifically designed the index to treat the drought problem in semiarid and dry subhumid climates, and cautioned that extrapolation beyond these conditions may lead to unrealistic results.  The index is, however, being computed for and used in all 48 contiguous States.  Palmer also clearly states that improvements in the computation of evapotranspiration, determination of the available water capacity in a layer of soil, and estimation of runoff would likely lead to better results.  Not considered in his development are conditions such as water supply from areas that are not in proximity to the location for which the index is computed, snowmelt, or frozen ground.Times         CourierFour previous studies examined the Palmer Index in terms of the various assumptions and parameterizations inherent in the computation of the index.  Karl (1983) looked at drought durations and found that increasing the available water capacity in the soil tends to increase both the length and severity of the more extreme droughts.  The effect is greatest in the Rocky Mountain states and least in the East.  For milder droughts, there is little effect in the East.  He also found that the magnitude of the PDSI, but not the duration of a drought, is sensitive to the weighting factor k that allows for spatial comparisons.   He concluded that his sensitivity experiments showed a negligible effect on drought duration.  In a second study, Karl (1986) examined the effect on both the PDSI and PHDI of the length of the period used to compute the normal climate of the area, and concluded that at least 50 years of data should be used.  Times         CourierAlley (1984), in an excellent critique of the PDSI, documented several limitations of the method for computing the PDSI.  He noted that the simplistic representation of runoff is very crude, and that it is difficult to account for the lag between moisture surplus and streamflow.  He advised that extreme caution should be exercised in using water balance variables such as soil moisture and evapotranspiration in developing indices of drought.  He also advised that because terms such as "severe" and "extreme" drought are loosely defined, care should be used when referring to drought severity classes.  Alley determined that the conditional distribution of the PDSI given the value for the previous month may be bimodal, therefore limiting the use of conventional time series models for describing the stochastic properties of the index.  He was also disturbed by the effect on the index of the arbitrary thresholds for determining the beginning and ending of wet and dry spells.  Times         Courier   Guttman (1992) examined the sensitivity of monthly time series of the PHDI to departures from average temperature and precipitation conditions.  He found that an initialization period of up to five years is necessary before the time series become reliable.  He also found that the effect of temperature departures from average conditions are, from a practical view, insignificant.  The PHDI is, however, sensitive to precipitation departures from normal.   The results also show that a warmer than normal temperature usually decreases the PHDI more than a colder than normal temperature of the same magnitude increases the PHDI, but drier than normal conditions tend to decrease the index less than wetter conditions increase the index.  Guttman (1992) suggests that  the distribution of PHDI values may be asymmetric and possibly bimodal, thereby possibly impacting threshold values used for making operational decisions. Times  Courier 4.     DATATimes         Courier   Monthly modified PDSI values were computed for 1,036 sites across the contiguous United States.  Time series go back from 1990 (except from 1988 or 1989 for a few sites) to at least 1930 after allowing for an initialization period of four years.  Computations required not only serially complete monthly temperature and precipitation data, but also soil water capacities, coefficients for computing potential evapotranspiration, and weighting factors.  The values of these latter constants and factors that were used for the at-site computations were the values on file at the National Climatic Data Center that are representative of the climatic division in which the site is located.  Although not examined, the use of climatic division values was not thought to significantly impact the at-site modified PDSI computations.  This belief is based on the results of Karl (1983) and Guttman (1992), on the relatively small area of a climatic division (there are 344 divisions in the 48 contiguous United States), and on the assumption that a division is climatologically homogeneous. Times       Courier   Monthly average temperature and total precipitation data were taken from the National Climatic Data Center's 1,219-station Historical Climatology Network (Karl et al., 1990).  Both unadjusted and adjusted data are contained in this data set.  The unadjusted data are original observations that have undergone quality assurance checks.  Missing data have not been estimated and remain missing.  The adjusted data are the original observations that have been quality checked and modified from the original data, when necessary, to account for non-climatic effects and biases such as those caused by changes in station location.  Missing data have been estimated so that the data for a station are essentially serially complete.Times         Courier   The estimation and modification techniques are based on the concept of making corrections from trends and patterns at neighboring stations.  The neighbors are the closest stations within the network, and could be large distances away.  Meteorologically, precipitation is often a localized phenomenon so that adjustments made on the basis of stations that are not within the localized area are suspect.  It was therefore decided to use the unadjusted precipitation data for the drought atlas.  The spatial representation of monthly average temperature, however, is  generally a smooth, continuous field so that more confidence can be placed on temperature adjustments than on precipitation adjustments.  This added confidence, coupled with the relative insensitivity of the Palmer Index to temperature, led to the decision to use the serially complete, adjusted temperature data for the Atlas. Times  Courier        Since the unadjusted precipitation data are not serially complete, the missing data had to be estimated.  In addition, some of the monthly precipitation totals are the sum of daily values for less than a full month, i.e., some of the monthly data are partial totals.  Because of the localized nature of precipitation, there is no generalized, accepted method of obtaining meteorologically reliable estimates of missing data.  Times          Courier   A simple precipitation estimation procedure was used that is based on the assumption that, even though precipitation is a localized phenomenon, an average monthly total over a small area should be reasonably representative of any site within the area.   A missing total for a given month and year was estimated by averaging the totals for the month and year for all stations in the National Climatic Data Center's Summary of the Month digital data base that are within 60 miles of the station with missing data.  The number of nearby stations whose totals were averaged varied from 1 to 87.  The number of stations depends on the density of the station network (most dense in the Northeast and least dense in the Rocky Mountains) and the year (fewer stations in the earlier years of the period of record).  If digital data were not available, Climatological Data publications were examined, and the published recorded or estimated values for the month and year in question at the Historical Climatology Network station were used.  The publications were also used to obtain a small number of values to extend the length of record of the temperature records (the length of record of the temperature data at a site is often shorter than that of precipitation data).Times          Courier   If a station had a partial precipitation total for a given month and year, the average of the totals for the nearby stations was computed and compared to the partial total.  The higher of the partial total or nearby station average was used as the estimate for the total for the month and year at the station.   Almost 13,000 comparisons were made between a nearby station average and a partial total.  The nearby station average was higher than the  partial total for 60 percent of the comparisons and lower for 40 percent.  These percentages subjectively confirm the difficulty in estimating monthly precipitation amounts over a small area and add credibility to the decision to use the unadjusted data from the Historical Climatology Network.          Times          Courier   Note that the number of stations for which the drought index has been computed is less than the number of stations for which precipitation probabilities have been computed.  It was necessary to eliminate some stations from the drought computations because of inadequate data.  The inadequacies resulted from the requirement that both temperature and precipitation data had to be serially complete from 1926 onward.  If either the temperature or precipitation data were not available and missing data could not be estimated because of lack of digital or published data from nearby stations, then the modified PDSI could not be computed. Times  Courier 5.     DROUGHT FREQUENCIES  Times          Courier   Drought is defined as beginning in the month and year when the modified PDSI equals or falls below -1 after having been above -1, and drought duration is defined as the interval of time for which the modified PDSI remains equal or below -1.  Using these definitions, frequencies of drought beginning in a specified calendar month were tabulated for fixed durations of 1, 2, 3, 6, 12, 24, 36 and 60 months.Times          Courier   The method of counting the occurrences of drought is best illustrated by example.  Consider a site with a modified PDSI above -1 from January through February, 1943; at or below -1 for each month from March 1943 through June 1946; and above -1 from July through December 1946.  For the time period from 1943 through 1946, there would be 3 occurrences of drought with a 1-month duration beginning (and ending) in January and February (the years 1944, 1945 and 1946), 4 occurrences in each month March through June (the years 1943, 1944, 1945 and 1946), and 3 occurrences in each month July through December (the years 1943, 1944 and 1945).  There would be 3 occurrences of 6-month droughts beginning in January (the years 1944, 1945 and 1946), 2 occurrences beginning in February (the years 1944 and 1945), and 3 occurrences beginning in each month March through December (the years 1943, 1944 and 1945).          Multiyear drought events with fixed durations of 24, 36 or 60 months beginning in a specified month are non-overlapping.  The drought index for a site was examined from the beginning of the series to determine whether or not a drought of specified duration had occurred.  Once an event was identified, the series was then examined beginning with the month after the end of the drought to determine whether or not another drought had occurred.  For example, consider a drought beginning in March 1943 and ending in June 1947 and a second drought beginning in January 1949 and ending in December 1950.  There would be two occurrences of a 24-month drought beginning in January (January 1944-December 1945 and January 1949-December 1950), one in February (February 1944-January 1946), two each in March through July (March 1943-February 1945 and March 1945-February 1947, etc.), and one each in August through December (August 1943-July 1945, etc.).  There would also be one occurrence of 36-month droughts beginning in each of the calendar months (January 1944-December 1946, February 1944-January 1947, March 1943-February 1946, etc.).  As defined for the Atlas, there were no occurrences of a drought with a duration of 60 months.Times          Courier   Drought frequencies are expressed in terms of percentage.  The number of occurrences of droughts of a specified duration was divided by the number of periods of the specified duration that are contained in the whole length of record.  This quotient was then multiplied by 100 to obtain a percentage.  The use of percentages allows comparisons among stations with varying lengths of data records.  Caution, however, should be exercised in making comparisons because cyclical, aperiodic or other climatic conditions may influence the number of droughts in one time period differently from that in another time period.     Times  Courier 6.     REFERENCESTimes  CourierAlley, W.M., 1984: The Palmer Drought Severity Index: Limitations  and assumptions, J. Clim. Appl. Meteor., 23, 1100-1109.Times  CourierAmerican Meteorological Society, 1986: Conference on Climate and  Water Management--A Critical Era, Boston, MA, 154pp.Times  CourierGuttman, N.B., 1992: A sensitivity analysis of the Palmer  Hydrologic Drought Index.  Water Resources Bull., in press.Times  CourierKarl, T.R., C.N. Williams, Jr., F.T. Quinlan and T.A. Boden, 1990:  United States Historical Climatology Network (HCN) serial  temperature and precipitation data.  ORNL/CDIAC-30, NDP- 019/R1, Carbon Dioxide Information Analysis Center, Oak Ridge  National Lab., Oak Ridge, TN, 374pp. Grigg, N.S. and E.C. Vlachos, 1990: Drought Water Management,  International School for Water Resources, Colorado St. Univ.,  Ft. Collins, CO, 252pp.Times  CourierHeddinghaus, T.R. and P. Sabol, 1991: A review of the Palmer  Drought Severity Index and where do we go from here? Proc. 7th  Conf. on Applied Climatology, September 10-13, 1991, American  Meteorological Society, Boston, MA, 242-246.Times  CourierKarl, T.R., 1983: Some spatial characteristics of drought duration  in the United States, J. Clim. Appl. Meteor., 22, 1356-1366.Times  CourierKarl, T.R., 1986: The sensitivity of the Palmer Drought Severity  Index and Palmer's Z-index to their calibration coefficients  including potential evapotranspiration, J. Clim. Appl.  Meteor., 25, 77-86. Palmer, W.C., 1965: Meteorological Drought. Res. Paper No. 45,  Weather Bureau, Washington, DC, 58pp.Times  CourierSouth Carolina Water Resources Commission, 1987: Southeastern  Drought Symposium Proceedings. South Carolina State  Climatology Office Publication G-30, Columbia, SC, 110pp.Times  CourierThornthwaite, C.W. and J.R. Mather, 1955: The Water Balance. Publ.  in Climatology, 8, 1, Drexel Inst. Technology, Lab.  Climatology, Centerton, NJ, 102pp.Times  CourierWilhite, D.A., 1990: Planning for Drought: A Process for State  Government. IDIC Tech. Rep. Series 90-1, International Drought  Information Center, Univ. Nebraska, Lincoln, NE, 52pp.Times  CourierWilhite, D.A., D.A. Wood and P.A. Kay, 1991: Drought Management and  Planning. IDIC Tech. Rep. Series 91-1, International Drought  Information Center, Univ. Nebraska, Lincoln, NE, 245pp.   Times ================================== Dr. Edward R. Cook Doherty Senior Scholar and Director, Tree-Ring Laboratory Lamont-Doherty Earth Observatory Palisades, New York 10964 USA Email: drdendro@ldeo.columbia.edu Phone: 845-365-8618 Fax: 845-365-8152 ==================================