Hindcast skill and predictability for precipitation and two-meter air temperature anomalies in global circulation models over the Southeast United States

Abstract

This paper presents an assessment of the seasonal prediction skill of current global circulation models, with a focus on the two-meter air temperature and precipitation over the Southeast United States. The model seasonal hindcasts are analyzed using measures of potential predictability, anomaly correlation, Brier skill score, and Gerrity skill score. The systematic differences in prediction skill of coupled ocean–atmosphere models versus models using prescribed (either observed or predicted) sea surface temperatures (SSTs) are documented. It is found that the predictability and the hindcast skill of the models vary seasonally and spatially. The largest potential predictability (signal-to-noise ratio) of precipitation anywhere in the United States is found in the Southeast in the spring and winter seasons. The maxima in the potential predictability of two-meter air temperature, however, reside outside the Southeast in all seasons. The largest deterministic hindcast skill over the Southeast is found in wintertime precipitation. At the same time, the boreal winter two-meter air temperature hindcasts have the smallest skill. The large wintertime precipitation skill, the lack of corresponding two-meter air temperature hindcast skill, and a lack of precipitation skill in any other season are features common to all three types of models (atmospheric models forced with observed SSTs, atmospheric models forced with predicted SSTs, and coupled ocean–atmosphere models). Atmospheric models with observed SST forcing demonstrate a moderate skill in hindcasting spring-and summertime two-meter air temperature anomalies, whereas coupled models and atmospheric models forced with predicted SSTs lack similar skill. Probabilistic and categorical hindcasts mirror the deterministic findings, i.e., there is very high skill for winter precipitation and none for summer precipitation. When skillful, the models are conservative, such that low-probability hindcasts tend to be overestimates, whereas high-probability hindcasts tend to be underestimates.

Appendices

Appendix 1: Potential predictability

Following Kumar and Hoerling (1995), consider an ensemble of forecasts for any forecast variable A. Let there be I = 1, N ensemble members with α = 1,M years of external forcing. The ensemble mean forecast, or the most likely outcome, for a given year α is then

$$ {\overline{{A_{\alpha } }} = \frac{1}{N}\sum\limits_{i = 1}^{N} {A_{i\alpha } } }. $$

The internal variance, or spread, of the ensemble members around this mean is

$$ {\sigma_{\alpha }^{2} = \frac{1}{N}\sum\limits_{i = 1}^{N} {\left( {A_{i\alpha } - \overline{{A_{\alpha } }} } \right)^{2} } }. $$

Since the spread can be dependent on the particular choice of year, the internal variance is then ${\sigma_{\alpha }^{2} }$averaged over all possible α, or

$$ {\sigma_{I}^{ 2} = \frac{1}{M}\sum\limits_{\alpha = 1}^{M} {\sigma_{\alpha }^{2} } = \frac{1}{M}\frac{1}{N}\sum\limits_{\alpha = 1}^{M} {\sum\limits_{i = 1}^{N} {\left( {A_{i\alpha } - \overline{{A_{\alpha } }} } \right)^{2} } .} } $$

The external variance is an estimate of the degree to which the difference between the ensemble mean forecast for different years is due to the boundary conditions rather than to “chance”; thus it is a measure of the forecast’s ability to distinguish between different regimes associated with different boundary conditions. The overall mean forecast, averaged over all realizations and boundary conditions, is given by

$$ {\bar{A} = \frac{1}{M}\frac{1}{N}\sum\limits_{\alpha = 1}^{M} {\sum\limits_{i = 1}^{N} {A_{i\alpha } } } }, $$

then the external variance is given by

$$ {\sigma_{E}^{2} = \frac{1}{M}\sum\limits_{\alpha = 1}^{M} {\left( {\overline{{A_{\alpha } }} - \overline{A} } \right)^{2} } }. $$

The total variance of the system is then

$$ {\sigma_{T}^{2} = \sigma_{E}^{2} + \sigma_{I}^{2} }. $$

By estimating the ratio of ${\sigma_{E}^{ 2} }$ to ${\sigma_{T}^{ 2} }$ or ${\sigma_{I}^{ 2} }$ we can then judge what part of the observed signal is due to boundary conditions and what part is due to the uncertainty of initial conditions, i.e., is effectively noise. The larger the ratio, the higher the predictability inherent to the system. The values of the ratio range between zero and one. In the case of zero, the ensemble does not see the boundary conditions, i.e., the outcome is entirely due to noise. In the case of one, the boundary conditions overwhelmingly mask out the effect of uncertainty of initial conditions.

The distance between the centroids of distributions for two different boundary condition regimes is a representation of the system’s external variability. The sharpness of the distribution associated with a particular regime is representative of the internal variance and represents the range of possible incomes associated with the particular boundary conditions. If the distributions are well separated, either because they are narrow or because their center points are far apart, the two states can be easily distinguished. If the reason for the distinguishability is that the curves are narrow, it can be said that the predictability is due to the internal variability being low. If what makes the distinction possible is that the two states are far apart, then the predictability is attributable to the large external variance.

Appendix 2: Brier skill score and reliability diagram

A probabilistic forecast is one that estimates the probability of occurrence of a chosen event ℰ, such as a precipitation rate anomaly relative to the mean state exceeding a preselected threshold level. For an ensemble of equally reliable models the probability P of the event ℰ is ${(m/M) \times 100},$ where m is the number of ensemble members forecasting ℰ, and M is the total number of ensemble forecasts. Since for a single realization a probability forecast is neither correct nor wrong, probability forecasts are verified by analyzing the joint (statistical) distribution of forecasts and observations.

The Brier score measures the magnitude of the probability forecast errors. It is defined as

$$ {b = \frac{1}{n}\sum\limits_{k = 1}^{n} {(f(k) - o(k))^{ 2} } }, $$

where the index k refers to the forecast/observation pairs, and n is the total number of such pairs within the data set, and both the forecast (f) and the observations (o) are in terms of probabilities. The lowest possible value of the Brier score is zero, and it can only be achieved with a perfect deterministic forecast.

Let the probabilistic forecast for ℰ be done within I discrete categories y _i . The frequency with which forecasts of y _i are issued is p(y _i ). The frequency within a category y _i forecast with which the event ℰ actually occurs is the conditional frequency ${\overline{{o_{i} }} = p\left( {o(k) = 1|y_{i} } \right)}$. A reliability diagram is a plot of ${\overline{{o_{i} }} }$ versus y _i , accompanied by the forecast frequency distribution p(y _i ) versus y _i . For a perfect forecast, the reliability diagram would be a line at 45°.

As suggested by Murphy (1973), it is useful to decompose the Brier score into three terms: reliability, resolution, and uncertainty:

$$ b = \underbrace {{\sum\limits_{i = 1}^{I} {p(y_{i} )\left( {y_{i} - \overline{{o_{i} }} } \right)^{2} } }}_{\text{reliability}} - \underbrace {{\sum\limits_{i = 1}^{I} {p(y_{i} )\left( {\overline{{o_{i} }} - \overline{o} } \right)^{2} } }}_{\text{resolution}} + \underbrace {{\overline{o} (1 - \overline{o} )}}_{\text{uncertainty}} = b_{\text{rel}} - b_{\text{res}} + b_{\text{unc}} , $$

where $\overline{o} = \frac{1}{n}\sum\nolimits_{k = 1}^{n} {o(k)}$ is the unconditional mean frequency of occurrence of the event ℰ.

The reliability term evaluates the statistical accuracy of the forecast–a perfectly reliable forecast is one for which the observed conditional frequency ${\overline{{o_{i} }} }$ is equal to the forecast probability (i.e., over all forecast for y percent chance of ℰ, ℰ will occur in y percent of the times). The resolution term addresses the distance between the forecast frequency and the unconditional climatological frequency. Forecasts that are always close to the climatological frequency exhibit good reliability (since the forecast frequency matches the observed frequency) but poor resolution (since they are not able to distinguish between different regimes). The uncertainty term is a measure of the variability of the system and is not influenced by the forecast. The Brier skill score is calculated with respect to a reference forecast as

$$ {BS = {\frac{{b - b_{\text{ref}} }}{{b_{\text{perf}} - b_{\text{ref}} }}} = 1 - {\frac{b}{{b_{\text{ref}} }}}}, \quad {\text{since}}\, {b_{\text{perf}} } \,{\text{is}}\, 0.$$

For a perfect forecast system, ${BS = BS_{\text{rel}} = BS_{\text{res}} = 1}$, while for a climatological forecast ${BS = BS_{\text{rel}} = BS_{\text{res}} = 0}$.