This paper investigates the problem of global sensitivity analysis (GSA) of Dynamical Earth System Models and proposes a basis for how such analyses should be performed. We argue that (a) performance ...metric‐based approaches to parameter GSA are actually identifiability analyses, (b) the use of a performance metric to assess sensitivity unavoidably distorts the information provided by the model about relative parameter importance, and (c) it is a serious conceptual flaw to interpret the results of such an analysis as being consistent and accurate indications of the sensitivity of the model response to parameter perturbations. Further, because such approaches depend on availability of system state/output observational data, the analysis they provide is necessarily incomplete. Here we frame the GSA problem from first principles, using trajectories of the partial derivatives of model outputs with respect to controlling factors as the theoretical basis for sensitivity, and construct a global sensitivity matrix from which statistical indices of total period time‐aggregate parameter importance, and time series of time‐varying parameter importance, can be inferred. We demonstrate this framework using the HBV‐SASK conceptual hydrologic model applied to the Oldman basin in Canada and show that it disagrees with performance metric‐based methods regarding which parameters exert the strongest controls on model behavior. Further, it is highly efficient, requiring less than 1,000 base samples to obtain stable and robust parameter importance assessments for our 10‐parameter example.
Plain Language Summary
When developing and using computer‐based models to (a) understand Earth and environmental systems, (b) make predictions, and/or (c) make management or policy decisions, it is very important to know which factors most strongly control the behaviors of the model. Tools to determine this are called sensitivity analysis (SA) methods. This paper shows that the use of model performance metrics to assess sensitivity is based in faulty reasoning. By framing the problem from first principles, a logical approach is developed that provides accurate and cost‐effective assessments of both time‐aggregate and time‐varying parameter importance. Because the approach does not require availability of system output data, it enables a comprehensive assessment and can be applied to historical and predictive conditions, as well as to future scenarios.
Key Points
Questions the use of the performance‐metric‐based sensitivity analysis of Dynamical Earth Systems Models and shows that the analysis it provides is both inaccurate and incomplete
Theoretically frames the global sensitivity analysis problem from first principles and develops a performance metric‐free approach to assessing parameter importance
Demonstrates that the new approach is efficient, stable, and robust and disagrees with metric‐based methods regarding which parameters exert the strongest controls on model behavior
Sensitivity analysis is an essential paradigm in Earth and Environmental Systems modeling. However, the term “sensitivity” has a clear definition, based in partial derivatives, only when specified ...locally around a particular point (e.g., optimal solution) in the problem space. Accordingly, no unique definition exists for “global sensitivity” across the problem space, when considering one or more model responses to different factors such as model parameters or forcings. A variety of approaches have been proposed for global sensitivity analysis, based on different philosophies and theories, and each of these formally characterizes a different “intuitive” understanding of sensitivity. These approaches focus on different properties of the model response at a fundamental level and may therefore lead to different (even conflicting) conclusions about the underlying sensitivities. Here we revisit the theoretical basis for sensitivity analysis, summarize and critically evaluate existing approaches in the literature, and demonstrate their flaws and shortcomings through conceptual examples. We also demonstrate the difficulty involved in interpreting “global” interaction effects, which may undermine the value of existing interpretive approaches. With this background, we identify several important properties of response surfaces that are associated with the understanding and interpretation of sensitivities in the context of Earth and Environmental System models. Finally, we highlight the need for a new, comprehensive framework for sensitivity analysis that effectively characterizes all of the important sensitivity‐related properties of model response surfaces.
Key Points:
No unique definition exists for “global sensitivity” across the problem space
Existing approaches to sensitivity analysis are limited in consistency and utility
We identify important properties of response surfaces that relate to sensitivity
Computer simulation models are continually growing in complexity with increasingly more factors to be identified. Sensitivity Analysis (SA) provides an essential means for understanding the role and ...importance of these factors in producing model responses. However, conventional approaches to SA suffer from (1) an ambiguous characterization of sensitivity, and (2) poor computational efficiency, particularly as the problem dimension grows. Here, we present a new and general sensitivity analysis framework (called VARS), based on an analogy to “variogram analysis,” that provides an intuitive and comprehensive characterization of sensitivity across the full spectrum of scales in the factor space. We prove, theoretically, that Morris (derivative‐based) and Sobol (variance‐based) methods and their extensions are special cases of VARS, and that their SA indices can be computed as by‐products of the VARS framework. Synthetic functions that resemble actual model response surfaces are used to illustrate the concepts, and show VARS to be as much as two orders of magnitude more computationally efficient than the state‐of‐the‐art Sobol approach. In a companion paper, we propose a practical implementation strategy, and demonstrate the effectiveness, efficiency, and reliability (robustness) of the VARS framework on real‐data case studies.
Key Points:
The VARS framework enables sensitivity analysis across a full range of scales
Sobol and Morris are special cases of the VARS framework
VARS is highly efficient because it utilizes information from pairs of points
This paper is derived from a keynote talk given at the Google's 2020 Flood Forecasting Meets Machine Learning Workshop. Recent experiments applying deep learning to rainfall‐runoff simulation ...indicate that there is significantly more information in large‐scale hydrological data sets than hydrologists have been able to translate into theory or models. While there is a growing interest in machine learning in the hydrological sciences community, in many ways, our community still holds deeply subjective and nonevidence‐based preferences for models based on a certain type of “process understanding” that has historically not translated into accurate theory, models, or predictions. This commentary is a call to action for the hydrology community to focus on developing a quantitative understanding of where and when hydrological process understanding is valuable in a modeling discipline increasingly dominated by machine learning. We offer some potential perspectives and preliminary examples about how this might be accomplished.
Key Points
Hydrology lacks scale‐relevant theories, but deep learning experiments suggest that these theories should exist
The success of machine learning for hydrological forecasting has potential to decouple science from modeling
It is up to hydrologists to clearly show where and when hydrological theory adds value to simulation and forecasting
Precipitation-extremes-driven floods, which compose an important proportion of streamflow but cause severe adverse impacts in the Loess Plateau of China, urged the progressive implementation of ...ecological restoration (ER) strategies in the Loess Plateau (LP) of China. Knowledge of the linkage between climate variables (especially precipitation extremes) and streamflow generation become more essential for advanced catchment management as ER and climate variability have resulted in reduced streamflow and freshwater stress. Here, a partial least squares regression (PLSR) approach was used to investigate this issue at 16 main catchments of the LP over a reference period (1961–1979). Then, we quantified streamflow decline during the “Integrated Soil and Water Conservation” (1980–1999) and the “Grain for Grain” (2000–2015) strategies by PLSR modeling. We found that the dominant climatic variables controlling annual streamflow include heavy precipitation amount and heavy precipitation days, maximum precipitation event amount, number of consecutive wet days, annual total precipitation (daily precipitation ≥1 mm), and effective precipitation amount (daily precipitation ≥5 mm). Further, the effect of precipitation extremes on streamflow generation is stronger in drier catchments. The impacts of precipitation extremes on streamflow generation can be strengthened by agricultural cultivation and weakened by revegetation (especially reforestation). Overall, we found that climate-driven annual streamflow decreased by 7.5 mm during 1980–1999 and by 5.6 mm during 2000–2015, in comparison to 1961–1979. The dominant cause of streamflow reduction was ER, with the contribution increasing from 59% in 1980–1999 to 82% in 2000–2015. The PLSR approach enables the identification of linkages between climate variables and streamflow generation, and the prediction of climate-driven streamflow. This study yields a greater understanding of the influences of climate variability and ER on streamflow change, and is helpful to identify hydroclimatological trends and projections.
Identified predominant climatic factors on streamflow generation (a) and predicted streamflow by the PLSR model in Period-I (b) for streamflow change attribution analysis during Period-II and Period-III in the CSHC region of LP (c). Display omitted
•Precipitation extremes dominate streamflow generation in the Loess Plateau.•Roles of precipitation extremes in streamflow generation can be enhanced by cultivation but weakened by revegetation.•Effects of precipitation extremes on streamflow generation are stronger in drier catchments.•Ecological restoration (ER) contributed 59% and 82% to streamflow reduction during the two ER periods.
Based on the theoretical framework for sensitivity analysis called “Variogram Analysis of Response Surfaces” (VARS), developed in the companion paper, we develop and implement a practical ...“star‐based” sampling strategy (called STAR‐VARS), for the application of VARS to real‐world problems. We also develop a bootstrap approach to provide confidence level estimates for the VARS sensitivity metrics and to evaluate the reliability of inferred factor rankings. The effectiveness, efficiency, and robustness of STAR‐VARS are demonstrated via two real‐data hydrological case studies (a 5‐parameter conceptual rainfall‐runoff model and a 45‐parameter land surface scheme hydrology model), and a comparison with the “derivative‐based” Morris and “variance‐based” Sobol approaches are provided. Our results show that STAR‐VARS provides reliable and stable assessments of “global” sensitivity across the full range of scales in the factor space, while being 1–2 orders of magnitude more efficient than the Morris or Sobol approaches.
Key Points:
Star‐sampling (STAR) enables VARS to fully characterize global sensitivity
Case studies show VARS is highly efficient, even for high‐dimensional problems
STAR‐VARS is more robust, stable, and efficient than either Sobol or Morris
Pumping tests are widely used to estimate parameters such as transmissivity and storativity, using aquifer response equations that assume a time‐constant pumping rate. However, in actual practice the ...discharge rate will often vary erratically and follow a generally decreasing trend as the test proceeds. In such cases, if the discharge history is recorded with sufficient temporal fidelity, accurate solutions can be obtained via time domain piecewise‐linear numerical integration. However, if the discharge data do not adequately characterize the variability in the dynamics of the pumping rate, the result can be information loss leading to bias in the inferred parameter estimates. Here, we investigate the severity of this problem for six selected aquifer types, including those that are confined, leaky, and unconfined. Our results indicate that the effects of information loss due to inadequate temporal resolution of the discharge data and systematic observational error are significantly more severe than due to random observational error. The implication is that operators should make a concerted effort to record the pumping rate at the highest practical temporal resolution throughout the duration of the test.
Key Points
Pumping test discharge rates often vary erratically while following a generally decreasing trend as the test proceeds
If the discharge history is not recorded with sufficient temporal resolution to characterize the system dynamics, information loss can occur
The result can be severe bias in the inferred estimates of aquifer properties
We study the sensitivity of aquifer‐scale estimates of transmissivity (T) and storativity (S) to the variance and correlation length scale of aquifer heterogeneity, when such estimates are obtained ...by the traditional approach of analyzing pumping test data. We consider both constant‐rate and variable‐rate pumping tests, and a variety of Theis‐based solution methodologies (single‐ and multiple‐observation well methods, and interpreted single value or transient values for T and S parameters) applied in pumping test data analysis. Our results indicate that achieving reliable inference of effective T and S requires that pumping be continued until the radius of the test‐induced cone of depression exceeds a “representative length” that corresponds to ∼15 times than the correlation length scale of the aquifer heterogeneity. Independent of solution type, pumping history, correlation length scale, magnitude of the sill, and location of observation well, the estimates for T will converge toward the geometric mean. For S, the estimation uncertainty is relatively large for observation wells that are close to the pumping well, but diminishes for observation wells located beyond the representative distance, resulting in convergence to the arithmetic mean. Given that these results have practical implications for how pumping tests should be carried out, we present a simple “rule‐of‐thumb” for estimating the correlation length scale of the heterogeneity of an aquifer.
Key Points
The T estimates for constant and heterogeneous S aquifers differ for every observation well; instead, the difference for S strongly depends on observation well distance
Pumping rate and history does not affect the inferred aquifer parameters; however, Theis‐based solution type results in different interpretations
As a rule‐of‐thumb, temporal variability of T and S diminishes when the depression cone radius exceeds 15 times the correlation length scale plus the observation well distance
1 Despite significant recent developments in computational power and distributed hydrologic modeling, the issue of how to adequately address the uncertainty associated with hydrological predictions ...remains a critical and challenging one. This issue needs to be properly addressed for hydrological modeling to realize its maximum practical potential in environmental decision-making processes. Arguably, the key to properly addressing hydrologic uncertainty is to understand, quantify, and reduce uncertainty involved in hydrologic modeling in a cohesive, systematic manner. Although general principles and techniques on addressing hydrologic uncertainty are emerging in the literature, there exist no well-accepted guidelines about how to actually implement these principles and techniques in various hydrologic settings in an integrated manner. This paper reviews, in relevant detail, the common data assimilation methods that have been used in hydrologic modeling to address problems of state estimation, parameter estimation, and system identification. In particular, the paper discusses concepts, methods, and issues involved in hydrologic data assimilation from a systems perspective. An integrated hierarchical framework is proposed for pursuing hydrologic data assimilation in several progressive steps to maximally reduce uncertainty in hydrologic predictions.
PDF
