Mean squared error of prediction as a criterion for evaluating and comparing system models

doi:10.1016/0304-3800(89)90035-5

Ecological Modelling

Volume 44, Issues 3–4, January 1989, Pages 299-306

https://doi.org/10.1016/0304-3800(89)90035-5 Get rights and content

Abstract

Model evaluation is an essential aspect of the process of development of system models. When the main purpose of the model is prediction, a reasonable criterion of model quality is the mean squared error of prediction. This criterion is defined here, and it is shown how it can be estimated from available data in a number of situations, including the situation where the parameters of the model are adjusted to the data. An example of the use of this criterion for choosing between alternative models is presented.

References (10)

M. Cabelguenne et al.
Conduite des systèmes de grandes cultures et prévision des rendements: Tentatives de modélisation. II. Etalonage du modèle: resultats et perspectives
C.R. Acad. Agric. Fr.
(1986)
N. Carter
Simulation modelling
J.B. Dent et al.
Systems Simulation in Agriculture
Applied Science
(1979)
B. Efron
Estimating the error rate of a prediction rule: improvement on cross-validation
J. Am. Stat. Assoc.
(1983)
R.M. Feldman et al.
Statistical procedure for validating a simple population model
Environ. Entomol.
(1984)

There are more references available in the full text version of this article.

Cited by (239)

Mathematical models for predicting protodioscin in tropical forage grasses
2024, Toxicon
Protodioscin is a saponin present in grasses that can lead to losses in animal production. Our hypothesis was that mathematical models can accurately and precisely predict the protodioscin concentration in tropical grasses. We evaluated the ability of four mathematical models to describe the protodioscin concentration in Brachiaria and Panicum cultivars with different regrowth periods. Six cultivars of Panicum: Aruana, Massai, Mombaça, Tanzânia, Tamani, and Zuri; and five of Brachiaria-grass: Marandu, Paiaguás, Piatã, Xaraés and Basilisk. Protodioscin concentration evaluations were carried out at 51, 84, 110, and 111 days of age. Linear, Quadratic, Exponential, and Logarithmic models were evaluated, and the adequacy of the models was verified. The models were compared for accuracy and precision by pairwise mean squared error analysis and the delta Akaike information criterion. The models did not differ from each other in terms of accuracy and precision. The exponential model showed a high ability to explain the observed variability between protodioscin concentration and plant age for Brachiaria grasses. Panicum grasses have constant protodioscin concentration. Mathematical models are capable of predicting the protodioscin concentration in grasses of the genus Brachiaria based on plant age. We recommend Exponential model to predict the concentration of protodioscin in Brachiaria grasses.
Generalized building energy and carbon emissions benchmarking with post-prediction analysis
2024, Developments in the Built Environment
This study proposes a generalized building energy and carbon emissions benchmarking approach. Leveraging eleven years of real-world data from twelve U.S. cities, an advanced ensemble learning model is employed to predict building site, source energy, and carbon emissions. The generalizability is validated across seven climate zones, yielding R² values ranging from 0.3 to 0.9. Additionally, a post-prediction analysis method is proposed that categorizes actual and predicted values into well-estimated, underestimated, and overestimated groups by an [−15%, 15%] acceptance interval (AI). This approach allows for assessing predictions related to energy, building types, and climate zones. The findings indicate the feasibility of a reliable generalized benchmarking tool. It’s been observed that energy in most underestimated groups tends to be higher than that of their overestimated counterparts. Over 63% of buildings' predictions perform within the well-estimated group by [−15%, 15%] AI.
Overview of emerging semiconductor device model methodologies: From device physics to machine learning engines
2024, Fundamental Research
Advancements in the semiconductor industry introduce novel channel materials, device structures, and integration methods, leading to intricate physics challenges when characterizing devices at circuit level. Nevertheless, accurate models for emerging devices are crucial for physics-driven TCAD-to-SPICE flows to enable the increasingly vital design technology co-optimization (DTCO). Particularly for ultra-scaled devices where quantum effects become significant, this led to the introduction of empirical model parameters and a disconnection to manufacturing processes. To catch up with these developments, an alternative to the traditional white-box modeling methods has attracted much attention: machine learning-assisted compact modeling (MLCM). These black-box methods target towards general-purpose modeling of complex mathematics and physics through training of neural networks on experimental and simulated data, generating an accurate closed-form mapping between output characteristics and input parameters for fabrication process and device operation. To address this new trend, this work provides a comprehensive overview of emerging device model methodologies, spanning from device physics to machine learning engines. By analyzing, structuring, and extending distributed efforts on this topic, it is shown how MLCM can overcome limitations of traditional compact modeling and contribute to effective DTCO to further advance semiconductor technologies.
Optimizing agronomic practices to harness climate change impacts on potato production in tropical highland regions
2024, European Journal of Agronomy
A simulation experiment was conducted to evaluate the effects of planting time, nitrogen rate, and crop variety choice on potato productivity and to assess the adaptation role of these practices in a changing climate in the northwestern tropical highlands of Ethiopia. The study used agroecosystem (AES) as the lens for spatial analysis. Thirty years of simulations were performed for the baseline (1981–2010), near-term (2011–2040), and mid-century (2041–2070) climate periods using a calibrated and validated SUBSTOR-Potato (DSSAT) model. The results showed that shifting planting time forward and changing the nitrogen application rate had greater productivity benefits than switching varieties in all climate periods and AESs. Late planting and higher nitrogen rates increased tuber yield during the near-term climate period. However, in the mid-century climate period, early planting of both medium and long maturity varieties with higher nitrogen rates showed potential adaptation benefits despite its negative impact on yield in the current climate. This result highlights the distinction between optimal management under current conditions and practices that are truly adaptive to climate change.
Development of land-use-specific pedotransfer functions for predicting bulk density of acidic topsoil in eastern Himalayas (India)
2023, Geoderma Regional
Soil bulk density (BD), although a key input parameter in many soil functions is often overlooked in soil studies, including surveys. Direct measurements over a large area are slow and expensive, especially in resource-poor mountain ecosystems. For soils of eastern Himalayan ecosystems, cost-effective indirect methods such as pedotransfer functions (PTFs) are lacking. We developed seven landuse-specific PTFs for BD estimation using a soil particle size distribution (sand, and silt) and soil organic carbon database of 1206 sampling sites. Land uses were 3 uncultivated (dense forests-DF, open forests-OF, and grasslands-GL), 3 cultivated (shifting cultivation-SC, upland agriculture -UA, lowland paddy-LP), and a perennial plantation (PL). PTFs were developed using five machine learning algorithms (MLAs) and a multi-linear regression (MLR). The best PTFs were chosen based on cross-validation performance with the values of the greatest coefficient of determination (R²) and the smallest root mean square error (RMSE). In addition, the relative performance of thirteen (13) documented PTFs was evaluated on soils in the region. Among the MLAs, the artificial neural network (ANN) showed the best BD prediction for the whole dataset (R²: 0.58, RMSE: 0.09 Mg m⁻³) and four (e.g. GL, UA, SC, and LP: R² of 0.65 to 0.69, RMSE of 0.06 to 0. 09 Mg m⁻³) of the seven land use classes. For the remaining three land uses (e.g., DF, OF and PL), based on the greatest R² values (0.55 to 0.75), PTFs based on random forests (RF) made the best predictions. However, considering the smallest RMSE values (0.06–0.08 Mg m⁻³), ANN-PTFs were another option for these three land uses. Of the land uses, the best prediction for BD was obtained for PL (R²: 0.75), while the DF with variable soil properties had the lowest predictive performance (R²: 0.55). The performance of MLA-based PTFs such as the support vector machine (SVM), ridge regression (RR), and extreme gradient boosting (XGB) and MLR-PTFs were lower than the ANN-PTFs and RF-PTFs. The precision of BD prediction for 13 published PTFs was inconsistent and inferior (R²: 0.13 to 0.39) to the ANN-PTFs and RF-PTFs (R²: 0.55 to 0.75) developed in this study. Since PTFs are very region-specific, we suggest ANN-PTFs to estimate the BD of grasslands and cultivated soils while the RF-PTFs (based on the greatest R²) for dense and open forests and plantation soils in the eastern Himalayas and similar mountain ecosystems.
Speckle noise reduction for medical ultrasound images based on cycle-consistent generative adversarial network
2023, Biomedical Signal Processing and Control
Medical ultrasound (US) images are corrupted by speckle noise, which can adversely affect the disease diagnosis and treatment. Recently, the cycle-consistent adversarial network (CycleGAN) has been used in the style transfer for both natural and medical images. In this study, we aim to develop a US despeckling method based on the CycleGAN by the style transfer between the speckled noisy data domain and noise-free data domain by forming a bi-directional universal mapping. The inputs of noisy and noise-free images are designed in the CycleGAN model. For simulation work, we use both the multiplicative model and the spatial impulse response model to obtain noisy images from noise-free images. However, noise-free US images are not clinically available. Hence, for the real US despeckling scenario, the clinical images of hearts, lymph nodes, and breast tumors are used as noisy images; and the high-quality images that are derived from the clinical images by despeckling with the Gabor-based anisotropic diffusion (GAD) and selected with a new metric named the edge-to-noise ratio, are used as the noise-free images. We compare our CycleGAN based denoising method with nine existing denoising methods, i.e., the speckle reduction anisotropic diffusion, GAD, non-local means, wavelet transform, unbiased non-local means, statistical nearest-neighbors, TVHTVM, improved non-local self-similarity measures, and generative adversarial network. Our method outperforms other methods by visual assessment and quantitative comparison, which demonstrates its superiority for noise reduction and detail preservation.

View all citing articles on Scopus

View full text

Mean squared error of prediction as a criterion for evaluating and comparing system models

Abstract

Conduite des systèmes de grandes cultures et prévision des rendements: Tentatives de modélisation. II. Etalonage du modèle: resultats et perspectives

C.R. Acad. Agric. Fr.

Simulation modelling

Systems Simulation in Agriculture

Applied Science

Estimating the error rate of a prediction rule: improvement on cross-validation

J. Am. Stat. Assoc.

Statistical procedure for validating a simple population model

Environ. Entomol.