Skip to main content
Top

2021 | OriginalPaper | Chapter

5. Identification and Processing of Outliers

Authors : Xavier Romão, Emilia Vasanelli

Published in: Non-Destructive In Situ Strength Assessment of Concrete

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

When analyzing real data sets, observations different from the majority of the data are sometimes found. These observations are usually called outliers and can be defined as individual data values that are numerically distant from the rest of the sample, thus masking its probability distribution. Outliers require special attention because they can have a significant impact in the concrete strength estimation process and because they may signal the presence of a different concrete population that deserves a separate assessment. The two-step process involved in an outlier analysis (outlier identification and outlier handling) is presented, discussing several statistical methodologies that are available for its implementation. To illustrate the application of an outlier analysis, examples involving univariate and multivariate datasets are presented. Several statistical methodologies are implemented for outlier identification, while outlier handling is illustrated by using robust statistics, i.e. outlier accommodation approaches that reduce the effect of existing outliers on the outcomes of statistical analyses of the data.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Iglewicz, B., Hoaglin, D.: How to Detect and Handle Outliers. ASQC Quality Press, Milwaukee (1993) Iglewicz, B., Hoaglin, D.: How to Detect and Handle Outliers. ASQC Quality Press, Milwaukee (1993)
2.
go back to reference Barnett, V., Lewis, T.: Outliers in Statistical Data. Wiley, New York (1994) Barnett, V., Lewis, T.: Outliers in Statistical Data. Wiley, New York (1994)
3.
4.
5.
go back to reference ISO 16269-4:2010: Statistical interpretation of data—Part 4: Detection and treatment of outliers (2010) ISO 16269-4:2010: Statistical interpretation of data—Part 4: Detection and treatment of outliers (2010)
6.
go back to reference ASTM D7915.14: Standard Practice for Application of Generalized Extreme Studentized Deviate (GESD) Technique to Simultaneously Identify Multiple Outliers in a Data Set. ASTM International, West Conshohocken (2014) ASTM D7915.14: Standard Practice for Application of Generalized Extreme Studentized Deviate (GESD) Technique to Simultaneously Identify Multiple Outliers in a Data Set. ASTM International, West Conshohocken (2014)
7.
go back to reference ASTM E178-16: Standard practice for dealing with outlying observations, ASTM International, West Conshohocken (2016) ASTM E178-16: Standard practice for dealing with outlying observations, ASTM International, West Conshohocken (2016)
8.
go back to reference Aggarwal, C.C.: Outlier Analysis. Springer Publishing Company, Inc (2017) Aggarwal, C.C.: Outlier Analysis. Springer Publishing Company, Inc (2017)
9.
go back to reference Rousseeuw, P.J., Van Driessen, K.: A fast algorithm for the minimum covariance determinant estimator. Technometrics 41(3), 212–223 (1999)CrossRef Rousseeuw, P.J., Van Driessen, K.: A fast algorithm for the minimum covariance determinant estimator. Technometrics 41(3), 212–223 (1999)CrossRef
10.
go back to reference Tukey, J.W.: Mathematics and the picturing of data. In: Proceeding of the International Congress of Mathematicians. Vancouver, Canada, 21.29 Aug 1974, vol. 2, pp. 523–531 Tukey, J.W.: Mathematics and the picturing of data. In: Proceeding of the International Congress of Mathematicians. Vancouver, Canada, 21.29 Aug 1974, vol. 2, pp. 523–531
11.
go back to reference Donoho, D.L., Gasko, M.: Breakdown properties of location estimates based on halfspace depth and projected outlyingness. Ann Stat 20(4), 1803–1827 (1992)MathSciNetCrossRef Donoho, D.L., Gasko, M.: Breakdown properties of location estimates based on halfspace depth and projected outlyingness. Ann Stat 20(4), 1803–1827 (1992)MathSciNetCrossRef
12.
go back to reference Hubert, M., Van der Veeken, S.: Outlier detection for skewed data. J. Chemom. 22(3–4), 235.246 (2008) Hubert, M., Van der Veeken, S.: Outlier detection for skewed data. J. Chemom. 22(3–4), 235.246 (2008)
13.
go back to reference Hartigan, J.A.: Clustering Algorithms. Wiley, 1975 Hartigan, J.A.: Clustering Algorithms. Wiley, 1975
14.
go back to reference Kaufman, L., Rousseeuw, P.J.: Partitioning around medoids (program PAM), In: Finding Groups in Data: An Introduction to Cluster Analysis. Wiley, New York, pp. 68–125 (1990) Kaufman, L., Rousseeuw, P.J.: Partitioning around medoids (program PAM), In: Finding Groups in Data: An Introduction to Cluster Analysis. Wiley, New York, pp. 68–125 (1990)
15.
16.
go back to reference Hubert, M., Vandervieren, E.: An adjusted boxplot for skewed distributions. Comput. Stat. Data. Anal. 52(12), 5186–5201 (2008)MathSciNetCrossRef Hubert, M., Vandervieren, E.: An adjusted boxplot for skewed distributions. Comput. Stat. Data. Anal. 52(12), 5186–5201 (2008)MathSciNetCrossRef
17.
go back to reference Rousseeuw, P.J., Hubert, M.: Robust statistics for outlier detection. WIREs Data Min. Knowl. Discov. 1, 73–79 (2011)CrossRef Rousseeuw, P.J., Hubert, M.: Robust statistics for outlier detection. WIREs Data Min. Knowl. Discov. 1, 73–79 (2011)CrossRef
18.
go back to reference Rousseeuw, P.J., Ruts, I., Tukey, J.W.: The bagplot: a bivariate boxplot. Am. Stat. 53(4), 382–387 (1999) Rousseeuw, P.J., Ruts, I., Tukey, J.W.: The bagplot: a bivariate boxplot. Am. Stat. 53(4), 382–387 (1999)
19.
go back to reference Pison, G., Van Aelst, S., Willems, G.: Small sample corrections for LTS and MCD. Metrika 55(1.2), 111.123 (2002) Pison, G., Van Aelst, S., Willems, G.: Small sample corrections for LTS and MCD. Metrika 55(1.2), 111.123 (2002)
20.
go back to reference Olive, D.J.: A resistant estimator of multivariate location and dispersion. Comput. Stat. Data Anal. 46(1), 93–102 (2004)MathSciNetCrossRef Olive, D.J.: A resistant estimator of multivariate location and dispersion. Comput. Stat. Data Anal. 46(1), 93–102 (2004)MathSciNetCrossRef
21.
go back to reference Hoaglin, D.C., Mosteller, F., Tukey, J.W. (eds.) Understanding Robust and Exploratory Data Analysis. Wiley, New York (1983) Hoaglin, D.C., Mosteller, F., Tukey, J.W. (eds.) Understanding Robust and Exploratory Data Analysis. Wiley, New York (1983)
22.
go back to reference Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics: the Approach Based on Influence Functions. Wiley (1986) Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics: the Approach Based on Influence Functions. Wiley (1986)
23.
go back to reference Staudte, R.G., Sheather, S.J.: Robust estimation and testing. Wiley, New York (1990) Staudte, R.G., Sheather, S.J.: Robust estimation and testing. Wiley, New York (1990)
24.
go back to reference Wilcox, R.: Introduction to Robust Estimation and Hypothesis Testing, 2nd edn. Academic press, Cambridge (2005)MATH Wilcox, R.: Introduction to Robust Estimation and Hypothesis Testing, 2nd edn. Academic press, Cambridge (2005)MATH
25.
go back to reference Maronna, R.A., Martin, D.R., Yohai, V.J.: Robust Statistics—Theory and Methods. Wiley, New York (2006) Maronna, R.A., Martin, D.R., Yohai, V.J.: Robust Statistics—Theory and Methods. Wiley, New York (2006)
26.
go back to reference Rousseeuw, P.J., Verboven, S.: Robust estimation in very small samples. Comput. Stat. Data Anal. 40(4), 741.758 (2002) Rousseeuw, P.J., Verboven, S.: Robust estimation in very small samples. Comput. Stat. Data Anal. 40(4), 741.758 (2002)
27.
go back to reference Romão, X., Delgado, R., Costa, A.: Statistical characterization of structural demand under earthquake loading. Part 1: Robust estimation of the central value of the data. J. Earthq. Eng. 16(5), 686–718 (2012) Romão, X., Delgado, R., Costa, A.: Statistical characterization of structural demand under earthquake loading. Part 1: Robust estimation of the central value of the data. J. Earthq. Eng. 16(5), 686–718 (2012)
28.
go back to reference Romão, X., Delgado, R., Costa, A.: Statistical characterization of structural demand under earthquake loading. Part 2: Robust estimation of the dispersion of the data. J. Earthq. Eng. 16(6) (2012) Romão, X., Delgado, R., Costa, A.: Statistical characterization of structural demand under earthquake loading. Part 2: Robust estimation of the dispersion of the data. J. Earthq. Eng. 16(6) (2012)
29.
go back to reference Randal, J.A.: A reinvestigation of robust scale estimation in finite samples. Comput. Stat. Data Anal. 52(11), 5014–5021 (2008)MathSciNetCrossRef Randal, J.A.: A reinvestigation of robust scale estimation in finite samples. Comput. Stat. Data Anal. 52(11), 5014–5021 (2008)MathSciNetCrossRef
30.
go back to reference Yohai, V.J., Zamar, R.H.: High breakdown-point estimates of regression by means of the minimization of an efficient scale. J. Am. Stat. Assoc. 83(402), 406–413 (1988)MathSciNetCrossRef Yohai, V.J., Zamar, R.H.: High breakdown-point estimates of regression by means of the minimization of an efficient scale. J. Am. Stat. Assoc. 83(402), 406–413 (1988)MathSciNetCrossRef
Metadata
Title
Identification and Processing of Outliers
Authors
Xavier Romão
Emilia Vasanelli
Copyright Year
2021
DOI
https://doi.org/10.1007/978-3-030-64900-5_5