Replied in thread
Accuracy! To counter regression dilution, a method is to add a constraint on the statistical modeling.
Regression Redress restrains bias by segregating the residual values.
My article: http://data.yt/kit/regression-redress.html
Recent searches
Search options
#ModelEvaluation
0 posts0 participants0 posts today
Replied in thread
How to assess a statistical model?
How to choose between variables?
Pearson's #correlation is irrelevant if you suspect that the relationship is not a straight line.
If monotonic relationship:
"#Spearman’s rho is particularly useful for small samples where weak correlations are expected, as it can detect subtle monotonic trends." It is "widespread across disciplines where the measurement precision is not guaranteed".
"#Kendall’s Tau-b is less affected [than Spearman’s rho] by outliers in the data, making it a robust option for datasets with extreme values."
Ref: https://statisticseasily.com/kendall-tau-b-vs-spearman/
LEARN STATISTICS EASILY · Kendall Tau-b vs Spearman: Which Correlation Coefficient Wins?Discover why Kendall Tau-b vs Spearman Correlation is crucial for your data analysis and which coefficient offers the most reliable results.
Continued thread
4/
Below, key points:
- "lack of #ModelEvaluation"
- statistical #uncertainty & "#robustness of event attribution results"
[4] Seneviratne, et al., 2021. Chapter 11: weather and climate extreme events in a changing climate. In: Climate Change 2021: The Physical Science Basis - Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. IPCC, Geneva, Switzerland, pp. 1513–1766. https://purl.org/INRMM-MiD/z-ED8RQFV5
https://www.ipcc.ch/report/ar6/wg1/downloads/report/IPCC_AR6_WGI_Chapter11.pdf#page=28
![Excerpt from [4], p. 1540 (p. 28 of the PDF). Cited literature: pp. 1706-1758
Section "11.2.3 Attribution of Extremes"
"Apart from the detection and attribution of trends in extremes, new approaches have been developed to answer the question of whether, and to what extent, external drivers have altered the probability and intensity of an individual extreme event (NASEM, 2016). In AR5, there was an emerging consensus that the role of external drivers of climate change in specific extreme weather events could be estimated and quantified in principle, but related assessments were still confined to particular case studies, often using a single model, and typically focusing on high-impact events with a clear attributable signal.
However, since AR5, the attribution of extreme weather events has emerged as a growing field of climate research with an increasing body of literature (see series of supplements to the annual State of the Climate report (Peterson et al., 2012, 2013a; Herring et al., 2014, 2015, 2016, 2018), including the number of approaches to examining extreme events (described in Easterling et al., 2016; Otto, 2017; Stott et al., 2016))."](https://hostux.social/system/media_attachments/files/110/436/695/826/336/639/original/02df0d0d0f116aeb.png "Excerpt from [4], p. 1540 (p. 28 of the PDF). Cited literature: pp. 1706-1758
Section \"11.2.3 Attribution of Extremes\"
\"Apart from the detection and attribution of trends in extremes, new approaches have been developed to answer the question of whether, and to what extent, external drivers have altered the probability and intensity of an individual extreme event (NASEM, 2016). In AR5, there was an emerging consensus that the role of external drivers of climate change in specific extreme weather events could be estimated and quantified in principle, but related assessments were still confined to particular case studies, often using a single model, and typically focusing on high-impact events with a clear attributable signal.
However, since AR5, the attribution of extreme weather events has emerged as a growing field of climate research with an increasing body of literature (see series of supplements to the annual State of the Climate report (Peterson et al., 2012, 2013a; Herring et al., 2014, 2015, 2016, 2018), including the number of approaches to examining extreme events (described in Easterling et al., 2016; Otto, 2017; Stott et al., 2016)).\"")
![Excerpt from [4], p. 1540 (p. 28 of the PDF).
Section "11.2.3 Attribution of Extremes"
"The outcome of event attribution is dependent on the definition of the event [...], as well as the framing [...] and uncertainties in observations and modelling. Observational uncertainties arise in estimating the magnitude of an event as well as its rarity [...]. Results of attribution studies can also be very sensitive to the choice of climate variables [...]. Attribution statements are also dependent on the spatial [...] and temporal [...] extent of event definitions, as events of different scales involve different processes [...] and large-scale averages generally yield higher attributable changes in magnitude or probability due to the smoothing out of noise. In general, confidence in attribution statements for large-scale heat and lengthy extreme precipitation events have higher confidence than shorter and more localized events, such as extreme storms, an aspect also relevant for determining the emergence of signals in extremes or the confidence in projections [...].
The reliability of the representation of the event in question in the climate models used in a study is essential [...]. Extreme events characterized by atmospheric dynamics that stretch the capabilities of current-generation models [...] limit the applicability of the probability-based approach of event attribution."](https://hostux.social/system/media_attachments/files/110/436/699/097/893/935/original/217b89d894620f5b.png "Excerpt from [4], p. 1540 (p. 28 of the PDF).
Section \"11.2.3 Attribution of Extremes\"
\"The outcome of event attribution is dependent on the definition of the event [...], as well as the framing [...] and uncertainties in observations and modelling. Observational uncertainties arise in estimating the magnitude of an event as well as its rarity [...]. Results of attribution studies can also be very sensitive to the choice of climate variables [...]. Attribution statements are also dependent on the spatial [...] and temporal [...] extent of event definitions, as events of different scales involve different processes [...] and large-scale averages generally yield higher attributable changes in magnitude or probability due to the smoothing out of noise. In general, confidence in attribution statements for large-scale heat and lengthy extreme precipitation events have higher confidence than shorter and more localized events, such as extreme storms, an aspect also relevant for determining the emergence of signals in extremes or the confidence in projections [...].
The reliability of the representation of the event in question in the climate models used in a study is essential [...]. Extreme events characterized by atmospheric dynamics that stretch the capabilities of current-generation models [...] limit the applicability of the probability-based approach of event attribution.\"")
![Excerpt from [4], p. 1541 (p. 29 of the PDF). Cited literature: pp. 1706-1758
Section "11.2.3 Attribution of Extremes"
"The lack of model evaluation, in particular in early event attribution studies, has led to criticism of the emerging field of attribution science as a whole (Trenberth et al., 2015) and of individual studies (Angélil et al., 2017). In this regard, the storyline approach (Shepherd, 2016) provides an alternative option that does not depend on the model’s ability to represent the circulation reliably. In addition, several ways of quantifying statistical uncertainty (Paciorek et al., 2018) and model evaluation (Lott and Stott, 2016; Philip et al., 2018b, 2020) have been employed to evaluate the robustness of event attribution results. For the unconditional probability-based approach, multi-model and multi-approach (e.g., combining observational analyses and model experiments) methods have been used to improve the robustness of event attribution (Hauser et al., 2017; Otto et al., 2018a; Philip et al., 2018b, 2019, 2020; van Oldenborgh et al., 2018; Kew et al., 2019)."](https://hostux.social/system/media_attachments/files/110/436/701/534/018/455/original/a8ca7a02f72b9522.png "Excerpt from [4], p. 1541 (p. 29 of the PDF). Cited literature: pp. 1706-1758
Section \"11.2.3 Attribution of Extremes\"
\"The lack of model evaluation, in particular in early event attribution studies, has led to criticism of the emerging field of attribution science as a whole (Trenberth et al., 2015) and of individual studies (Angélil et al., 2017). In this regard, the storyline approach (Shepherd, 2016) provides an alternative option that does not depend on the model’s ability to represent the circulation reliably. In addition, several ways of quantifying statistical uncertainty (Paciorek et al., 2018) and model evaluation (Lott and Stott, 2016; Philip et al., 2018b, 2020) have been employed to evaluate the robustness of event attribution results. For the unconditional probability-based approach, multi-model and multi-approach (e.g., combining observational analyses and model experiments) methods have been used to improve the robustness of event attribution (Hauser et al., 2017; Otto et al., 2018a; Philip et al., 2018b, 2019, 2020; van Oldenborgh et al., 2018; Kew et al., 2019).\"")
![Excerpt from [4], p. 1553 (p. 41 of the PDF).
Section "11.3.4 Detection and Attribution, Event Attribution"
"Local forcing may mask or enhance the warming effect of greenhouse gases [...]
Irrigation and crop intensification [...] lead to a cooling in some regions [...]
Deforestation has contributed about one third of the total warming of hot extremes in some mid-latitude regions [...]. Despite [...] larger uncertainties at the regional scale, nearly all studies demonstrated that human influence has contributed to an increase in the frequency or intensity of hot extremes and to a decrease in the frequency or intensity of cold extremes.
In summary, long-term changes in various aspects of long- and short-duration extreme temperatures, including intensity, frequency, and duration have been detected in observations and attributed to human influence at global and continental scales. It is extremely likely that human influence is the main contributor to the observed increase in the intensity and frequency of hot extremes and the observed decrease in the intensity and frequency of cold extremes on the global scale. It is very likely that this applies on continental scales as well. Some specific recent hot extreme events would have been extremely unlikely to occur without human influence on the climate system. Changes in aerosol concentrations have affected trends in hot extremes in some regions, with [...] aerosols leading to attenuated warming, in particular from 1950 to 1980."](https://hostux.social/system/media_attachments/files/110/436/705/589/734/582/original/97c57ff676291bd7.png "Excerpt from [4], p. 1553 (p. 41 of the PDF).
Section \"11.3.4 Detection and Attribution, Event Attribution\"
\"Local forcing may mask or enhance the warming effect of greenhouse gases [...]
Irrigation and crop intensification [...] lead to a cooling in some regions [...]
Deforestation has contributed about one third of the total warming of hot extremes in some mid-latitude regions [...]. Despite [...] larger uncertainties at the regional scale, nearly all studies demonstrated that human influence has contributed to an increase in the frequency or intensity of hot extremes and to a decrease in the frequency or intensity of cold extremes.
In summary, long-term changes in various aspects of long- and short-duration extreme temperatures, including intensity, frequency, and duration have been detected in observations and attributed to human influence at global and continental scales. It is extremely likely that human influence is the main contributor to the observed increase in the intensity and frequency of hot extremes and the observed decrease in the intensity and frequency of cold extremes on the global scale. It is very likely that this applies on continental scales as well. Some specific recent hot extreme events would have been extremely unlikely to occur without human influence on the climate system. Changes in aerosol concentrations have affected trends in hot extremes in some regions, with [...] aerosols leading to attenuated warming, in particular from 1950 to 1980.\"")
Replied in thread
Anyway, I keep meaning to write up a blog post on “falsehoods I have believed about measuring model performance” touching on #AppliedML issues related to #modelEvaluation, #metrics, #monitoring, #observability, and #experiments (#RCTs). The cool kids would call this #AIAlignment in their VC pitch decks, but even us #NormCore ML engineers have to wrestle with how to measure and optimize the real-world impact of our models.