cc: p.jones@uea.ac.uk, i.harris@uea.ac.uk, t.osborn@uea.ac.uk date: Tue, 22 Apr 2008 12:54:53 +0100 (BST) from: P.Jones@uea.ac.uk subject: Re: [Fwd: Re: {Spam?} Re: Status] to: "David Lister" David, Looks as though the time series data are wrong then for 61-96 assuming that you've not updated the earlier data with VAP data, where the earlier was something other than VAP. As I said earlier the time series data for the Gabon sites for VAP look like temperature data from somewhere like Ireland. Hope you can sort this out! Cheers Phil > Phil/Harry/Tim, > > I have looked in Mark New's normals database (I have not yet looked in the > "red book"). The Gabon stations are listed below: > > 6450000GAB8795 1719 301 308 302 309 304 274 248 250 268 288 297 > 306 > 6450300GAB6190100-1 287 289 296 308 294 242 223 232 254 277 287 > 277 > 6450400GAB6190100-1 298 301 305 301 301 279 255 261 286 287 294 > 289 > 6451000GAB8795 1419 272 256 255 260 263 256 247 243 249 260 261 > 258 > 6454500GAB6190100-1 239 241 247 250 249 242 226 226 240 244 243 > 257 > 6455000GAB8795 1119 301 299 294 301 305 274 249 250 267 283 290 > 301 > 6455200GAB6190100-1 261 264 266 253 268 254 237 236 249 254 258 > 249 > 6455600GAB6190100-1 251 249 253 255 250 239 227 226 239 244 243 > 240 > 6456500GAB6190100-1 227 263 268 268 260 230 219 226 248 261 263 > 258 > > > These values look reasonable to me. I will check further if required. > NB four of the above (code="-1") are derived variables. The others come > from CRU's GPD. > > Cheers > > David > On Tue, 22 Apr 2008 P.Jones@uea.ac.uk wrote: > >> ---------------------------- Original Message >> ---------------------------- >> Subject: Re: {Spam?} Re: Status >> From: f028@uea.ac.uk >> Date: Tue, April 22, 2008 7:45 am >> To: "Ian Harris" >> Cc: "Tim Osborn" >> "Phil Jones" >> -------------------------------------------------------------------------- >> >> >> Tim, Harry, >> First can't see why Harry's email has come as spam. >> >> A few thoughts on the data for the 2 Gabon stations. >> If these are supposed to VAP data then the data for the >> period 1961-1996 looks completely wrong. The later data looks >> better! >> >> Seems as though these two sites and the others for the country >> ought to be removed. I'm making this suggestion based on >> common sense. Gabon is near the equator and I think most is >> jungle. >> >> I could be wrong. The Sudan VAP for El Dueim looks OK. It should >> drop during teh dry season. >> >> Maybe ask David to have a look. Or have a look at a station in an >> adjacent country. >> >> No idea what the Gabon numbers are for the 61-96 period! They look >> like temperatures from somewhere like Ireland! >> >> Hope you can find the problem! >> >> Cheers >> Phil >> >> >>> Hi Tim, >>> >>> A little more detail now I've had a chance to look at the files. Try >>> and view this email in a monospaced font! Or it'll look awful. >>> >>> On 21 Apr 2008, at 16:23, Tim Osborn wrote: >>> >>>> Hi Harry, >>>> >>>> yes, please focus on WET first. >>>> >>>> then please take a look at the VAP station data in the countries of >>>> Gabon (1st priority) and Sudan (2nd priority), for 2003 onwards. >>>> >>>> I've checked the synthetic vap and that does not have extremely high >>>> recent values, so it seems to be coming from the stations. I masked >>>> out areas not near to stations and the high extremes are still >>>> there... so that's further evidence it is in the stations. Gabon >>>> shows particularly strong anomalies (approaching 20 hPa above normal >>>> at times), but there are some over 10 hPa above normal in the Sudan >>>> area too. >>>> >>>> The two ncview.*.pdf files attached show the anomaly maps for 2 >>>> selected months during 2003-2005 (see file names for specific >>>> months/years). I've masked in dark blue areas outside the CDD >>>> distance from a station (according to your ...vap.stn.nc file). The >>>> high values are red (surrounded a yellow stripe). Gabon in west >>>> equatorial Africa stands out in both, Sudan in one. Both have >>>> numerous other high anomalies like this from Jan 2003 onwards. They >>>> are the cause of spikes in country-mean VAP in the other attached file >>>> -- see page 15 for Gabon (and neighbouring countries) and page 13 for >>>> Sudan (and neighbouring countries). Sudan looks less severe, but >>>> that's partly because it is larger and the averaging takes in more >>>> synthetic vap values from outside the station's neighbourhood. >>>> >>>> The extreme positive anomalies occur close to (this is from visual >>>> reading off the screen, so not fully precise!): >>>> >>>> Gabon: 9.25 E, 0.25 S -- is there a station at Port-Gentil? >>> >>> There is - well, three, as this is a database I haven't cleansed: >>> >>> -645014 -187 1102 89 MOUILA GABON 1971 1996 >>> -999 -999.00 >>> -645009 50 940 15 LIBREVILLE GABON 1971 1996 >>> -999 0.85 >>> -645008 -70 875 4 PORT GENTIL GABON 1971 2002 >>> -999 -999 >>> -645007 -70 875 4 PORT GENTIL GABON 1994 2003 >>> -999 nocode >>> 6450000 45 942 15 LIBREVILLE/LEON MBA GABON 1979 2007 >>> -999 -999 >>> 6450100 -70 875 4 PORT GENTIL GABON 2003 2007 >>> -999 0 >>> 6451000 208 1148 599 BITAM GABON 1971 2007 >>> -999 -999 >>> 6450300 -342 1065 34 MAYUMBA GABON 1971 2005 >>> -999 -999 >>> 6450400 100 960 13 COCOBEACH GABON 1971 2004 >>> -999 -999 >>> 6450700 -285 1102 79 TCHIBANGA GABON 1971 2007 >>> -999 -999 >>> 6451000 208 1148 599 BITAM GABON 1971 1996 >>> -999 >>> 6455100 -72 1023 26 LAMBARENE GABON 1971 2007 >>> -999 -999 >>> 6455200 78 1153 583 MITZIC GABON 1971 2004 >>> -999 >>> 6455600 57 1287 515 MAKOKOU/EPASSENGUE GABON 1971 2006 >>> -999 >>> 6456000 -83 1271 485 LASTOURSVILLE GABON 1971 1996 >>> -999 -999 >>> 6456500 -153 1326 573 MOANDA GABON 1971 1996 >>> -999 >>> 6455000 -187 1102 89 MOUILA CITY GABON 2003 2007 >>> -999 0 >>> >>> Now, NONE of these stations should contribute to the product, since our >>> specified cutoff is 75%, or 23 values per month between 1961 and 1990. >>> But for a couple, a normals line is present: >>> 6450000 45 942 15 LIBREVILLE/LEON MBA GABON 1979 2007 >>> -999 -999 >>> 6451000 208 1148 599 BITAM GABON 1971 2007 >>> -999 -999 >>> >>> These two stations have several things in common. Firstly, they both >>> have a populated normals line, even though their data counts are too >>> low. Secondly, their values (and normals) are significantly below the >>> rest of GABON (ie, between 40 and 180 rather than between 200 and 350). >>> Thirdly, they both have a swathe of missing values 1997-2002, followed >>> by patchy data that looks like the rest of GABON (ie, too high for the >>> normals lines). Here they are, in full: >>> >>> 6450000 45 942 15 LIBREVILLE/LEON MBA GABON 1979 2007 >>> -999 -999 >>> 6190 65 65 75 87 111 135 149 147 135 110 83 72 >>> 1979-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 79 83 >>> 1980 59 80 77 85 99 136 149 160 149 100 76 71 >>> 1981 68 59 98 93 125 135 155 156 135 92 83 54 >>> 1982 51 60 73 78 107 147 161 150 149 114 94 70 >>> 1983 79 52 78 93 109 137 182 159 136 105 87 72 >>> 1984 69 62 66 78 109 121 139 154 131 111 95 74 >>> 1985 43 45 72 92 118 123 154 139 139 107 68 83 >>> 1986 64 37 71 82 115 140 146 130 106 113 91 77 >>> 1987 45 65 63 100 100 138 150 149 141 106 83 74 >>> 1988 80 67 76 84 120 136 150 148 134 114 86 93 >>> 1989 76 77 92 84 121 129 156 147 144 116 79 76 >>> 1990 79 85 89 86 112 137 136 151 121 115 84 69 >>> 1991 64 55 97 84 100 119 164 150 138 107 80 68 >>> 1992 66 76 83 93 130 146 153 160 134 95 93 72 >>> 1993 77 61 64 93 110 127 138 127 122 93 64 76 >>> 1994 72 64 91 89 112 133 173 150 137 104 115 84 >>> 1995 71 85 69 95 105 132 178 152 132 130 86 58 >>> 1996 59 60 69 86 101 138 147 153 114 106 89 58 >>> 1997-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 1998-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 1999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2001-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2002-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2003 305 320 311-9999-9999 275 277-9999-9999 300 303 313 >>> 2004 319 315-9999-9999 290 265-9999-9999 283-9999-9999 307 >>> 2005 316 318 311-9999 291 258-9999-9999 281-9999-9999 299 >>> 2006-9999-9999-9999-9999-9999 295 266-9999-9999-9999-9999-9999 >>> 2007 318 313-9999-9999-9999 277-9999-9999-9999-9999-9999-9999 >>> >>> 6451000 208 1148 599 BITAM GABON 1971 2007 >>> -999 -999 >>> 6190 67 65 74 84 110 135 149 147 133 111 84 72 >>> 1971 66-9999 65 85 122 128 157 149 117 104 79 81 >>> 1972 62 72 73 88 105 121 162 142 114 91 85 70 >>> 1973 69 69 70 78 113 140 150 154 137 100 79 68 >>> 1974 80 72 83 81 106 134 140 151 127 88 85 90 >>> 1975 85 64 76 90 101 129 158 161-9999 100 80-9999 >>> 1976 73 69 61 72 104 132 142 131 128 116 86 60 >>> 1977 70 78 84 77 100 136 149 151 120 120 85 80 >>> 1978 66 62 80 80 117 129 140 132 127 113 82 69 >>> 1979 44 53 74 82 108 143 144 140 127 116 85 83 >>> 1980 60 79 75 77 91 128 145 155 141 101 78 70 >>> 1981 69 59 97 86 120 130 154 159 134 102 90 59 >>> 1982 62 65 72 77 110 150 165 156 150 115 95 71 >>> 1983 82 55 81 91 105 138 181 160 133 107 86 73 >>> 1984 70 63 65 71 106 125 140 154 134 120 100 74 >>> 1985 46 47 70 88 118 124 147 137 137 106 68 83 >>> 1986 66 34 70 77 111 144 147 130 111 119 92 77 >>> 1987 46 66 64 95 98 139 152 153 150 112 87 77 >>> 1988 84 69 78 84 121 135 148 147 130 115 84 92 >>> 1989 77 71 82 76 114 119 145 138 141 116 76 75 >>> 1990 76 81 82 82 107 131 134 150 120 115 86 70 >>> 1991 66 53 88 75 94 117 162 150 137 107 81 68 >>> 1992 65 74 81 84 116 146 166 165 133 89 92 71 >>> 1993 77 63 70 100 123 140 147 135 131 101 68 80 >>> 1994 77 65 83 86 115 133 170 153 136 102 114 85 >>> 1995 71 84 70 96 109 134 178 157 131 134 88 60 >>> 1996 62 58 66 85 103 134 140 149 121 111 83 58 >>> 1997-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 1998-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 1999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2001-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2002-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 >>> 2003 255-9999 261-9999-9999 253 250-9999-9999 256 255 259 >>> 2004 265 250-9999-9999 259 245-9999-9999 245-9999-9999 266 >>> 2005 266-9999-9999-9999 257-9999-9999-9999-9999-9999-9999-9999 >>> 2006-9999-9999-9999-9999-9999 253 247-9999-9999-9999-9999-9999 >>> 2007 244 252-9999-9999-9999 253-9999-9999-9999-9999-9999-9999 >>> >>> I would say that somehow the presence of an illegal normals line is >>> allowing these 'bad' stations to be included. You'd almost call it >>> sabotage, wouldn't you? >>> >>> The obvious fix would be to replace the normals with missing values, >>> though it rankles to leave bad data in the database. And we'd end up >>> with no Gabon stations contributing at all! >>> >>>> Sudan: 32.75 E, 13.25 S -- perhaps Khartoum or somewhere just to the >>>> south of that? >>> >>> It's a similar story to Gabon, sorry. This is the culprit, showing >>> header, normals line, and 2003-2007: >>> >>> 6275000 1400 3233 378 ED DUEIM SUDAN 1971 2007 >>> -999 -999 >>> 6190 64 60 69 78 102 127 143 143 130 109 83 70 >>> 2003 89 69 63 85 213 210 247 266 247 198 119 102 >>> 2004 87 76 74 86 126 192 212 237 231 168 117 103 >>> 2005 77 107 69 97 125 186 231 247 233 176 108 122 >>> 2006 105 87 103 73 173 179 227 246 239 269 97 83 >>> 2007 80 70 82 95-9999 203-9999-9999-9999-9999-9999-9999 >>> >>> (again, there was a block of missing values 1997-2002). >>> >>> The only other Sudanese stations with enough data to contribute are OK: >>> 6260000 2182 3148 183 HALFA (WADI) SUDAN 1971 2007 >>> -999 -999 >>> 6265000 1917 3048 226 DONGOLA/DUNQULAH SUDAN 1971 2007 >>> -999 -999 >>> >>> In this case, I'd say the 2003-2007 additions to Ed Dueim should be >>> deleted. Now if we just had a data format with full historical >>> metadata.. >>> >>>> I presume you can set these to missing from Jan 2003 onwards? Or just >>>> set the outliers to missing? cc'd to Phil for his suggestion... we >>>> need something quick and easy, but defendable! They are so extreme >>>> relative to the standard deviation that these two particular ones >>>> cannot really be left as they are. >>>> >>>> There are also many other positive and negative vap anomalies >>>> (especially post 2003, but also some pre 2003) that seem dubious (5 to >>>> 10 hPa, compared with the vap standard deviation which is not far from >>>> 1 hPa in most places (well, 0.5 to 2 hPa). I haven't bothered to list >>>> the locations/years/months of these strange values because they >>>> typically last only 1 month and they are mostly in regions of the >>>> world with more dense observations and hence spatial averaging tends >>>> to reduce the anomalies rather quickly. The reason for picking out >>>> the Gabon and Sudan ones is that they seem to be affecting many months >>>> and also in regions with less dense networks and hence 1 station >>>> severely affects an entire national-average (and indeed neighbouring >>>> countries too). However I've cc'd this to Phil to get his opinion on >>>> how much time should be spent on these issues. >>> >>> I'd say that if these countries are anything to go by, the VAP database >>> needs a spring clean! >>> >>>> Note that it isn't all bad news :-) There were many dubious looking >>>> spikes in CRU TS 2.1 vap (even in "well-observed" countries like Spain >>>> and Portugal) that have improved in CRU TS 3.0 vap! (see the attached >>>> time series). >>> >>> Cool. >>> >>>> Cheers >>>> >>>> Tim >>>> >>>> P.S. I've just realised something... shouldn't these anomalies have >>>> been ignored if they are > 3 (or is it 4) S.D. from the normal? Or >>>> maybe that option isn't used for vap? >>> >>> As I stated previously - the 3 SD limit's applied by the anomaly >>> program and so will be in force for both 'real' and synthetic vap. >>> >>> Cheers >>> >>> Harry >>> >>> >>>> >>>> At 15:09 21/04/2008, you wrote: >>>>> Hi Tim, >>>>> >>>>> WET almost done; should be finished tonight (will complete it >>>>> remotely). Note that, if you're investigating it, WET uses a >>>>> percentage anomalies, I presume this is because the incoming >>>>> synthetics are PRE-derived. >>>>> >>>>> FRS is also within reach, but has a lower priority. >>>>> >>>>> Cheers >>>>> >>>>> Harry >>>>> Ian "Harry" Harris >>>>> Climatic Research Unit >>>>> School of Environmental Sciences >>>>> University of East Anglia >>>>> Norwich NR4 7TJ >>>>> United Kingdom >>>>> >>>> Dr >>>> Timothy J Osborn, Academic Fellow >>>> Climatic Research Unit >>>> School of Environmental Sciences >>>> University of East Anglia >>>> Norwich NR4 7TJ, UK >>>> >>>> e-mail: t.osborn@uea.ac.uk >>>> phone: +44 1603 592089 >>>> fax: +44 1603 507784 >>>> web: http://www.cru.uea.ac.uk/~timo/ >>>> sunclock: http://www.cru.uea.ac.uk/~timo/sunclock.htm >>>> >>> >>> >> >> >> >> >