Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications (Vol. II)

The Kalman filter, invented initially for control systems, has been widely used in science and engineering including data assimilation. For the last several decades, the estimation theory for dynamical systems has been actively developed in control theory. In this paper, we survey several observers, including Kalman filters, for nonlinear systems. We also review some fundamental concepts on the observability of systems defined by either differential equations or a numerical model. The hope is that some of these ideas will inspire research that can benefit the area of data assimilation.

Markov chain Monte Carlo algorithms constitute flexible and powerful solutions to Bayesian inverse problems. They return a sample of the unapproximated posterior probability density, and make no assumptions as to linearity or the form of the prior or likelihood. MCMC algorithms are in principle easy to construct, however, they can prove difficult to implement in practice. This chapter describes the theory that underlies MCMC simulation, provides guidance for its practical implementation, and presents examples of applications of MCMC to satellite retrievals and model uncertainty characterization. Though the high dimensionality of Earth system datasets and the complexity of atmospheric, oceanic, and hydrologic models present significant challenges, continued advances in theory and practice are making application of MCMC algorithms increasingly feasible.

The influence matrix is used in ordinary least-squares applications for monitoring statistical multiple-regression analyses. Concepts related to the influence matrix provide diagnostics on the influence of individual data on the analysis, the analysis change that would occur by leaving one observation out, and the effective information content (degrees of freedom for signal) in any sub-set of the analysed data. In this paper, the corresponding concepts are derived in the context of linear statistical data assimilation in Numerical Weather Prediction. An approximate method to compute the diagonal elements of the influence matrix (the self-sensitivities) has been developed for a large-dimension variational data assimilation system (the 4D-Var system of the European Centre for Medium-Range Weather Forecasts). Results show that, in the ECMWF operational system, 18 % of the global influence is due to the assimilated observations, and the complementary 82 % is the influence of the prior (background) information, a short-range forecast containing information from earlier assimilated observations. About 20 % of the observational information is currently provided by surface-based observing systems, and 80 % by satellite systems.A toy-model is developed to illustrate how the observation influence depends on the data assimilation covariance matrices. In particular, the role of high-correlated observation error and high-correlated background error with respect to uncorrelated ones is presented. Low-influence data points usually occur in data-rich areas, while high-influence data points are in data-sparse areas or in dynamically active regions. Background error correlations also play an important role: high correlation diminishes the observation influence and amplifies the importance of the surrounding real and pseudo observations (prior information in observation space). To increase the observation influence in presence of high correlated background error is necessary to introduce the observation error correlation but also observation and background error variances must be of similar size. Incorrect specifications of background and observation error covariance matrices can be identified, interpreted and better understood by the use of influence matrix diagnostics for the variety of observation types and observed variables used in the data assimilation system.

The adequacy of observations to locate the minimum of the standard cost function for variational data assimilation under strong constraint has been investigated. A simplified yet meaningful Lagrangian air/sea interaction model that captures key aspects of air mass modification over the Gulf of Mexico in wintertime is the dynamical tool used to examine this question of adequacy. Two mathematically different yet equivalent variational schemes are used in numerical experiments with a fixed number of observations along a prior known trajectory over the Gulf. Research clearly indicates that sensitivity of model output to elements of control (initial condition, boundary condition, and physical parameter) is key to placement of observations in order to minimize the cost function and determine optimal corrections to control.

Adjoint models calculate the first order sensitivity of a scalar output parameter to an input vector. Adjoint numerical weather prediction models have been used for a variety of sensitivity and data assimilation studies to provide a gradient for a measure of error with respect to the model’s analysis variables. Recent work has shown that the adjoint of the data assimilation system can map the gradient information in analysis space onto individual observations to provide a quantitative estimate of an observation’s influence on short-term forecast error. This chapter will review the framework of an adjoint observation impact system and some reported applications. Aspects of the framework particular to limited area atmospheric models will be the main focus of this chapter and results from a specific system will be presented. Issues discussed include: the effect of horizontal grid spacing on observation impact, the influence of lateral boundaries on forecast error, the relative importance of observations for different physical locations, and appropriate error metrics for limited area forecast models.

Uncertainty in the position of a feature is a ubiquitous influence on data assimilation (DA) in geophysical applications. This chapter explores the properties of distributions arising from the uncertainty of the location of a flow feature. It is shown that distributions arising from phase uncertainty have surprisingly complex, non-Gaussian characteristics. These non-Gaussian characteristics are explored from an ensemble DA perspective in which the skewness (third-moment) is shown to be a significant contributor to the state-estimates obtained through Bayesian state estimation. Idealized examples, as well as an example in a real tropical cyclone using a state-of-the-art numerical weather prediction model, will be shown.

Many background error correlation (BEC) models in data assimilation are formulated in terms of a positive-definite smoothing operator B that is employed to simulate the action of correlation matrix on a vector in state space. In this chapter, a general procedure for constructing a BEC model as a rational function of the diffusion operator D is presented and analytic expressions for the respective correlation functions in the homogeneous case are obtained. It is shown that this class of BEC models can describe multi-scale stochastic fields whose characteristic scales can be expressed in terms of the polynomial coefficients of the model. In particular, the connection between the inverse binomial model and the well-known Gaussian model \(\mathbf{\mathsf{B}}_{g} =\exp \mathbf{\mathsf{D}}\) is established and the relationships between the respective decorrelation scales are derived.By its definition, the BEC operator has to have a unit diagonal and requires appropriate renormalization by rescaling. The exact computation of the rescaling factors (diagonal elements of B) is a computationally expensive procedure, therefore an efficient numerical approximation is needed. Under the assumption of local homogeneity of D, a heuristic method for computing the diagonal elements of B is proposed. It is shown that the method is sufficiently accurate for realistic applications, and requires 10² times less computational resources than other methods of diagonal estimation that do not take into account prior information on the structure of B.

Adjoint techniques are effective tools for the analysis and optimization of the observation performance on reducing the errors in the forecasts produced by atmospheric data assimilation systems (DASs). This chapter provides a detailed exposure of the equations that allow the extension of the adjoint-DAS applications from observation sensitivity and forecast impact assessment to diagnosis and tuning of parameters in the observation and background error covariance representation. The error covariance sensitivity analysis allows the identification of those parameters of potentially large impact on the forecast error reduction and provides a first-order diagnostic to parameter specification. A proof-of-concept is presented together with comparative results of observation impact assessment and sensitivity analysis obtained with the adjoint versions of the Naval Research Laboratory Atmospheric Variational Data Assimilation System – Accelerated Representer (NAVDAS-AR) and the Navy Operational Global Atmospheric Prediction System (NOGAPS).

A comprehensive set of linearized physical parameterizations has been developed for the global ECMWF Integrated Forecasting System. Implications of the linearity constraint for any parametrization scheme, such as the need for simplification and regularization, are discussed. The description of the methodology to develop linearized parameterizations highlights the complexity of obtaining a physics package that can be efficiently used in practical applications. The impact of the different physical processes on the tangent-linear approximation and adjoint sensitivities, as well as their performance in data assimilation are demonstrated.

Data assimilation with representer-based algorithms (also called “dual space” algorithms) are currently being used for weak-constraint four-dimensional variational data assimilation (W4D-Var) atmospheric prediction, distributed parameter estimation, and other hydrodynamic data assimilation problems. The iterative linear solvers at the core of these systems may display non-monotonic convergence in the norm defined by the primal objective function, and this behavior makes problematic the development of practical stopping criteria. One approach to this problem is described, namely an implementation of the inner solver using the generalized conjugate residual(GCR) algorithm. Additional elements of data assimilation systems are error model for the background, model forcings, and observations. An implementation of a posterior analysis method for diagnosing the error variances is described, and representative results from an atmospheric data assimilation systems are shown.

A fully three dimensional, multivariate, variational ocean data assimilation system has been developed that produces simultaneous analyses of temperature, salinity, geopotential and vector velocity. The analysis is run in real-time and is being evaluated as the data assimilation component of the Hybrid Coordinate Ocean Model (HYCOM) forecast system at the U.S. Naval Oceanographic Office. Global prediction of the ocean weather requires that the ocean model is run at very high resolution. Currently, global HYCOM is executed at 1/12 degree resolution ( ∼ 7 km mid-latitude grid mesh), with plans to move to a 1/25 degree resolution grid in the near future ( ∼ 3 km mid-latitude grid mesh). These high resolution global grids present challenges for the analysis given the huge model state vector and the ever increasing number of satellite and in situ ocean observations available for the assimilation. In this paper the development and evaluation of the new oceanographic three-dimensional variational (3DVAR) data assimilation is described. Special emphasis is placed on documenting the capabilities built into the 3DVAR to make the system efficient for use in global HYCOM.

In this chapter we will describe a comprehensive 4-dimensional variational ocean data assimilation system that is currently being used in the Regional Ocean Model System for the production of both near real-time and historical ocean analyses of the California Current circulation. The main focus of this article is on the practical aspects of the data assimilation system as applied to an energetic coastal mesoscale circulation environment.

A 4D-Variational system was recently developed for assimilating ocean observations with the Navy Coastal Ocean Model. It is described here, along with initial assimilation experiments in the Monterey Bay using a combination of real and synthetic ocean observations. For testing a new assimilation system it is advantageous to use this combination of real and synthetic data over simplified cases of climatology and twin data. Assimilation experiments are carried out in a weak constraint formulation, with the model’s external forcing assumed to be erroneous in addition to initial conditions. The system’s ability to fit assimilated and non assimilated observations is assessed, as well as the consistency and relevance of the retrieved model forcing. Experiment results show that the assimilation system fits the data with relatively high prior errors in the initial conditions and surface forcing fluxes. However, the retrieved model forcing errors are well within the range of acceptable corrections according to an independent study.

An ocean adaptive sampling algorithm, derived from the Ensemble Transform Kalman Filter (ETKF) technique, is illustrated in this Chapter using the glider observations collected during the Autonomous Ocean Sampling Network (AOSN) II field campaign. This algorithm can rapidly obtain the prediction error covariance matrix associated with a particular deployment of the observation and quickly assess the ability of a large number of future feasible sequences of observations to reduce the forecast error variance. The uncertainty in atmospheric forcing is represented by using a time-shift technique to generate a forcing ensemble from a single deterministic atmospheric forecast. The uncertainty in the ocean initial condition is provided by using the Ensemble Transform (ET) technique, which ensures that the ocean ensemble is consistent with estimates of the analysis error variance. The ocean ensemble forecast is set up for a 72 h forecast with a 24 h update cycle for the ocean data assimilation. Results from the atmospheric forcing perturbation and ET ocean ensemble mean are examined and discussed. Measurements of the ability of the ETKF to predict 24–48 h ocean forecast error variance reductions over the Monterey Bay due to the additional glider observations are displayed and discussed using the signal variance, signal variance summary map, and signal variance summary bar charts, respectively.

It has been noted that global warming is likely to increase both the frequency and severity of weather events such as heat waves and heavy rainfall. These could lead to large scale effects such as melting of large ice sheets with major impacts on low-lying regions throughout the world (Intergovernmental Panel on Climate Change, IPCC 2007a). Since these projected climate changes will impact water resources, agriculture, bio-diversity and health, one of the key challenges of climate research is the application of climate models to quantify both future climate change and its impacts on the physical and biological environment. One of the widely studied impacts is on hydrology, right from large scale river basins, river deltas through to small scale urban reservoirs. In this context, this chapter discusses some hydrological impact studies and presents results of a study done over the Sesan catchment in Lower Mekong Basin (in Southeast Asia). Sensitivity analysis and an optimization calibration scheme, SCE-UA algorithm, are applied to the SWAT model.

This article reports further theoretical development on the entropic balance theory applied to tornadogenesis (Sasaki 2009, 2010), and the first preliminary application of the theory to radar observations. The entropic balance is a newly found balance, different from the other balance conditions, such as hydrostatic, (quasi-)geostrophic, cyclostrophic, Boussinesq, and anelastic. The entropic balance condition is described as the sole diagnostic Euler-Lagrange equation derived from the Lagrangian of the variational formalism. The entropic balance is most general and involves no additional assumptions other than for the flow with high Reynolds and Rossby numbers estimated as appropriate for supercell storms and tornadoes. The entropic balance theory and the deduced wrap-around mechanism explain well the observations and simulations of tornado, RFD, hook-echo, upward tilting of horizontal vorticity, the vertical in-phase superimposition between upper and lower mesocyclones, and sudden transition from supercell, mesocyclones totornado. In the application, new variables DZ (temporal difference of radar reflectivity) and DZ_DR (temporal difference of differential reflectivity) are introduced to compute the entropy anomaly based on the entropic balance theory. The conditions necessary for the transition from supercell to tornado are clarified from the theory and verified from the DZ and DZ_DR analyses for a non-tornadic supercell case compared with VORTEX2 tornadic case.Since the entropic balance theory is found to fit well with all analyzed results of tornado and visual observations, it is suggested to use the entropic balance equation as a constraint for variational data assimilation in future development as a challenge.

Assimilation of satellite radiances is the backbone of today’s operational data assimilation. Satellites can cover all parts of globe and provide information in areas not accessible by any other observation type. Of special interest are high-impact weather areas, such as tropical cyclones and severe weather outbreaks, which are mostly covered by clouds. Unfortunately, in current operational practice only clear-sky satellite radiances are assimilated, with only few exceptions. This effectively filters out a potentially useful information from all-sky radiances related to clouds and microphysics, and consequently limits the utility of satellite data. In this paper we will address numerous challenges related to the use of all-sky satellite radiances.All-sky satellite radiances present a formidable challenge for data assimilation as they relate to numerous technical aspects of data assimilation such as: (1) forecast error covariance, (2) correlated observation errors, (3) nonlinearity and non-differentiability, and (4) non-Gaussian errors. Assimilation of all-sky radiances is also challenging from a dynamical/physical point of view, since observing clouds implies a need for better understanding and ultimately simulation of cloud microphysical processes. Given that a reliable prediction of clouds requires a high-resolution cloud-resolving model, assimilation of all-sky radiancesis also a high-dimensional problem that requires addressing computational challenges.

A two-way nested Local Ensemble Transform Kalman Filter (LETKF) system has been developed to improve the accuracy of numerical forecasts on local heavy rainfalls. In this system, mesoscale convergence which drives local heavy rainfalls, is first reproduced by the LETKF with a grid interval of 15 km (Outer LETKF) which assimilates conventional data. The convection cells associated with the local heavy rainfall are then reproduced by the higher resolution LETKF with a grid interval of 1.875 km (Inner LETKF) which assimilates local data. The boundary conditions of the Inner LETKF are given by the forecast of the Outer LETKF. To consider the upward cascade effect from storm scale to mesoscale, the forecast results of the Inner LETKF are reflected into the Outer LETKF every 6 h.This system was applied to a thunderstorm that caused a local heavy rainfall event on the Osaka Plain on 5th September 2008. The rainfall distributions similar to the observed ones were reproduced in a few ensemble members of the Inner LETKF, although the observed scattered convection cells were expressed as weak rainfall regions in the Outer LETKF. When the precipitable water vapor or slant-path water vapor data obtained by GPS and horizontal wind or radial wind data observed by Doppler radars were assimilated in the Inner LETKF, the number of ensemble forecasts, which reproduced the local heavy rainfall, increased. The experiments on the small-scale disturbances in the initial seeds of the Inner LETKF and on the initial conditions produced by the no-cost smoother showed that these improvements might enhance the accuracy of local heavy rainfall forecasts.

Recent activities on ensemble data assimilation and its application to observing-system research at the Japan Agency for Marine-Earth Science and Technology are reviewed. A revised version of an ensemble-based data assimilation system for global atmospheric data has been developed on the second-generation Earth Simulator. This system assimilates conventional atmospheric observations and satellite-based wind data into an atmospheric general circulation model using the local ensemble transform Kalman filter (LETKF), a deterministic ensemble Kalman filter algorithm that is extremely efficient with parallel computer architecture. The updated system incorporates improvements to the previous system in the forecast model, data assimilation algorithm and input data. Using the LETKF system, observations taken during field campaigns are evaluated by data assimilation experiments involving adding or removing observations. The results of these observing-system experiments successfully demonstrate the value of the observations and are highly useful for exploring the predictability of atmospheric disturbances.

In this paper, variational data assimilation techniques to retrieve 3-dimensional wind fields from weather radars and LIDAR are discussed. The retrieved wind field from the 3-dimensional variational (3DVAR) technique applied to the weather radar data are found useful to delineate the mesoscale features leading to the convective development in a rainstorm event that brought significant lightning and thunderstorms near the Hong Kong airport and heavy precipitation over the territory. Impacts in improving analysis and forecast of a non-hydrostatic NWP model are also obtained through the data assimilation of wind retrieval data as additional observations in the model analysis. To capture the low-level windshear due to complex wind flow around the Hong Kong airport, 3DVAR and 4DVAR techniques are applied to LIDAR data. The performance of the wind retrieval algorithms and results of case studies will be illustrated. It is found that the wind fields obtained are useful to depict salient features of terrain-induced airflow disturbances at HKIA, such as mountain waves and vortices in a gustnado event.

In this chapter, we review the recent progresses in targeted observations for tropical cyclone prediction based on Conditional Nonlinear Optimal Perturbation (CNOP) method. The CNOP is a natural extension of the singular vector (SV) into the nonlinear regime and it has been used to identify the sensitive areas for tropical cyclone predictions.The properties of the sensitive areas identified by CNOP have been first studied, including the sensitivity to the horizontal resolution, the verification area design, and the optimization period. It has been found that the CNOP sensitive areas have similarities at different horizontal resolutions, and a small variation of the verification area has minimal influence on the CNOP sensitive areas. The CNOP sensitive areas identified for special forecast times when the initial time is fixed resemble those identified for other forecast times in the linear case, while the similarities among the sensitive areas identified for different forecast times are more limited in the nonlinear case. When the forecast time is fixed, the CNOP sensitive areas are much different when they are identified at different time period ahead.Then the influence of the initial conditions in the sensitive areas on the targeted forecasts have been examined, and the observing system simulation experiments (OSSEs) have been performed to assess whether or not the sensitive areas can be considered as dropping sites in real time targeting. Also, the observation system experiments (OSEs) have been carried out to demonstrate the utility of the CNOP method. It is found that the impact of initial errors introduced into the CNOP sensitive areas on the forecasts is greater than that of errors fixed in the SV sensitive areas or other randomly selected areas. The OSSEs have shown that assimilating the ideal observations in the CNOP sensitive areas results in the improvements of 13–46 % in typhoon track forecasts, while the improvements of 14–25 % are obtained by assimilating the ideal observations in the SV sensitive areas. Besides, the improvements have been achieved for longer forecast times. The OSEs have shown that the DOTSTAR data in the CNOP sensitive areas has a more positive impact on the typhoon track forecast than that in the SV sensitive areas.All the above results have demonstrated that the CNOP is a useful tool in the adaptive observations to identify the sensitive areas.

In this study, the impact of directly assimilating Advanced TIROS Operational Vertical Sounder (ATOVS) radiances using the Community Radiative Transfer Model (CRTM) was evaluated to determine the impact on forecasts over Southwest Asia. The CRTM was developed by the Center for Satellite Applications and Research (STAR) and its application was promoted by the Joint Center for Satellite Data Assimilation (JCSDA). The ATOVS radiance data from the National Environmental Satellite Data and Information Service (NESDIS), the Gridpoint Statistical Interpolation (GSI) three-dimensional variational analysis (3DVAR) system from the National Centers for Environmental Prediction (NCEP), and the Advanced Research WRF (WRF-ARW) model from the National Center for Atmospheric Research (NCAR) were employed in this study.First, this paper will describe the forecasting errors encountered from running the WRF-ARW model in the complex terrain of Southwest Asia from 1–31 May 2006. The subsequent statistical evaluation is designed to assess the model’s surface and upper-air forecast accuracy. The results show that the model biases caused by inadequate parameterizations of physical processes are relatively small, except for the 2-m temperature, as compared to the nonsystematic errors resulting in part from the uncertainty in initial conditions. The total model forecast errors at the surface show a substantial spatial heterogeneity and the errors are relatively larger in higher elevation mountain areas. The performance of 2-m temperature forecasts is different from the other surface variables’ forecasts; the model forecast errors in 2-m temperature forecasts are closely related to the terrain configuration. The simulated diurnal variation of near-surface temperature is much smaller than the observed diurnal variation.Second, to understand the impact of initial conditions on the accuracy of the model forecasts, the satellite radiances are assimilated into the numerical model through GSI data assimilation system. The results indicate that on average over a 30-day experiment for the 24- and 48-h (second 24-h) forecasts, the satellite data provides beneficial information for improving the initial conditions and the model errors are reduced to some degree over some of the study locations. The diurnal cycle of some forecast variables can be improved by using adequate initial conditions with satellite radiance data assimilation.

Variational data assimilation provides a convenient means of optimally combining the “first-guess” or “background” meteorological fields with the observations. The background fields are typically obtained from the numerical weather prediction output of a model while the observations can be either the meteorological model variables or even non-model variables. In the three-dimensional variational (3D-VAR) method the analysis state is obtained by optimally combining the “first-guess” and the “observations” at the same analysis time.The present article begins with a brief overview of the characteristics of the monsoon disturbances that form over the Indian region during the summer monsoon season. Subsequently, the 3D-VAR method is briefly introduced together with details of the mesoscale model employed in this study. The next section outlines the results of the impact of the 3D-VAR assimilation of satellite observations in the simulation of a few monsoon disturbances over India using the Weather Research and Forecast (WRF) model. The satellite observations utilized in the 3D-VAR assimilation study presented in this article include (1) temperature and humidity profiles from Moderate Resolution Imaging Spectroradiometer (MODIS), (2) temperature and humidity profiles from Advanced TIROS Vertical Sounder (ATOVS), and (3) total precipitable water from Special Sensor microwave imager (SSMI), respectively. In order to discern the impact of 3D-VAR assimilation of satellite observations a (base or control) numerical experiment called “control run” is performed, which is identical to the assimilated run (called “3D-VAR run”) except that no observations are assimilated in the control run. The results of the simulation between the assimilated run and the control run are compared with one another as well as with global analysis and Tropical Rainfall Measurement Mission (TRMM) and Quick Scatterometer (QuikSCAT) observations.The results of the study indicate that the assimilation of satellite observations, in general, does improve the simulation of the various monsoon disturbances over India, although the improvements are not uniformly very marked for all the monsoon disturbances and for all the satellite observations. Assimilating MODIS temperature and humidity profiles have yielded better results for two of the depressions as compared to the ATOVS and SSM/I assimilations. Also the results of the study indicate that assimilating total precipitable water from SSM/I has lower impact as compared to assimilating temperature and humidity profiles from ATOVS and MODIS.

In this study the quantitative precipitation forecast (QPF) related to a tropical cyclone is performed using a high-resolution mesoscale model and an evolutionary algorithm. For this purpose two parameters of the Kain-Fritsch convective parameterization scheme, in the Weather Research and Forecasting (WRF) model, are optimized using the micro-genetic algorithm (GA). The auto-conversion rate (c) and the convective time scale (T _c ) are target parameters. The fitness function is based on a QPF skill score. Typhoon Rusa (2002) is simulated in a grid spacing of 25 km. The default value of c is 0. 03 s^{− 1} while that of T _c is limited to a range between 1800 s and 3600 s as a function of grid resolution. To produce the best QPF skill, at least for this tropical cyclone case, c is optimized to 0. 0004 s⁻ and T _c to 1922s. Our results indicate that parameters of subgrid-scale physical processes need to be adjusted to produce better QPF in a tropical cyclone, sometimes to values far different from the default values in a numerical model. Such adjustment may be dependent on the characteristics of weather systems as well as geographical locations.