Skip to main content
Erschienen in: International Journal of Speech Technology 4/2016

19.09.2016

Improved automatic English proficiency rating of unconstrained speech with multiple corpora

verfasst von: David O. Johnson, Okim Kang, Romy Ghanem

Erschienen in: International Journal of Speech Technology | Ausgabe 4/2016

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

The performance of machine learning classifiers in automatically scoring the English proficiency of unconstrained speech has been explored. Suprasegmental measures were computed by software, which identifies the basic elements of Brazil’s model in human discourse. This paper explores machine learning training with multiple corpora to improve two of those algorithms: prominent syllable detection and tone choice classification. The results show that machine learning training with the Boston University Radio News Corpus can improve automatic English proficiency scoring of unconstrained speech from a Pearson’s correlation of 0.677–0.718. This correlation is higher than any other existing computer programs for automatically scoring the proficiency of unconstrained speech and is approaching that of human raters in terms of inter-rater reliability.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Anhänge
Nur mit Berechtigung zugänglich
Literatur
Zurück zum Zitat Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3), 3–30. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3), 3–30.
Zurück zum Zitat Bernstein, J. (1999). PhonePass testing: Structure and construct. Menlo Park: Ordinate Corporation. Bernstein, J. (1999). PhonePass testing: Structure and construct. Menlo Park: Ordinate Corporation.
Zurück zum Zitat Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27(3), 355–377.CrossRef Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27(3), 355–377.CrossRef
Zurück zum Zitat Boersma, P., & Weenink, D. (2014). Praat: doing phonetics by computer (Version 5.3.83), [Computer program]. Retrieved August 19, 2014. Boersma, P., & Weenink, D. (2014). Praat: doing phonetics by computer (Version 5.3.83), [Computer program]. Retrieved August 19, 2014.
Zurück zum Zitat Brazil, D. (1997). The communicative value of intonation in English. Cambridge: Cambridge University Press. Brazil, D. (1997). The communicative value of intonation in English. Cambridge: Cambridge University Press.
Zurück zum Zitat Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., et al. (1998). Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment essays. ETS Research Report Series, 1998(1), i-67.CrossRef Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., et al. (1998). Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment essays. ETS Research Report Series, 1998(1), i-67.CrossRef
Zurück zum Zitat Černý, V. (1985). Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications, 45(1), 41–51.MathSciNetCrossRefMATH Černý, V. (1985). Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications, 45(1), 41–51.MathSciNetCrossRefMATH
Zurück zum Zitat Chodorow, M., & Burstein, J. (2004). Beyond essay length: evaluating e‐rater®’s performance on toefl® essays. ETS Research Report Series, 2004(1), i-38.CrossRef Chodorow, M., & Burstein, J. (2004). Beyond essay length: evaluating e‐rater®’s performance on toefl® essays. ETS Research Report Series, 2004(1), i-38.CrossRef
Zurück zum Zitat Chun, D. M. (2002). Discourse intonation in L2: From theory and research to practice (Vol. 1). Philadelphia: John Benjamins Publishing.CrossRef Chun, D. M. (2002). Discourse intonation in L2: From theory and research to practice (Vol. 1). Philadelphia: John Benjamins Publishing.CrossRef
Zurück zum Zitat Evanini, K., & Wang, X. (2013). Automated speech scoring for non-native middle school students with multiple task types. In INTERSPEECH (pp. 2435–2439). Evanini, K., & Wang, X. (2013). Automated speech scoring for non-native middle school students with multiple task types. In INTERSPEECH (pp. 2435–2439).
Zurück zum Zitat Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., & Pallett, D. S. (1993). DARPA TIMIT acoustic-phonetic continous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon Technical Report N, 93, 27403. Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., & Pallett, D. S. (1993). DARPA TIMIT acoustic-phonetic continous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon Technical Report N, 93, 27403.
Zurück zum Zitat Johnson, D. O., & Kang, O. (2015). Automatic prominent syllable detection with machine learning classifiers. International Journal of Speech Technology, 18(4), 583–592.CrossRef Johnson, D. O., & Kang, O. (2015). Automatic prominent syllable detection with machine learning classifiers. International Journal of Speech Technology, 18(4), 583–592.CrossRef
Zurück zum Zitat Johnson, D. O., & Kang, O. (2016). Automatic prosodic tone choice classification with Brazil’s intonation model. International Journal of Speech Technology, 19(1), 95–109.CrossRef Johnson, D. O., & Kang, O. (2016). Automatic prosodic tone choice classification with Brazil’s intonation model. International Journal of Speech Technology, 19(1), 95–109.CrossRef
Zurück zum Zitat Kahn, D. (1976). Syllable-based generalizations in English phonology (Vol. 156). Bloomington: Indiana University Linguistics Club. Kahn, D. (1976). Syllable-based generalizations in English phonology (Vol. 156). Bloomington: Indiana University Linguistics Club.
Zurück zum Zitat Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38(2), 301–315.CrossRef Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38(2), 301–315.CrossRef
Zurück zum Zitat Kang, O., & Johnson, D. O. (2015). Comparison of inter-rater reliability of human and computer prosodic annotation using brazil’s prosody model. English Linguistics Research, 4(4), p58.CrossRef Kang, O., & Johnson, D. O. (2015). Comparison of inter-rater reliability of human and computer prosodic annotation using brazil’s prosody model. English Linguistics Research, 4(4), p58.CrossRef
Zurück zum Zitat Kang, O., & Johnson, D. O. (2016). Systems and Methods for Automated Evaluation of Human Speech. U.S. Patent Application No. 15/054,128. Washington, DC: U.S. Patent and Trademark Office. Kang, O., & Johnson, D. O. (2016). Systems and Methods for Automated Evaluation of Human Speech. U.S. Patent Application No. 15/054,128. Washington, DC: U.S. Patent and Trademark Office.
Zurück zum Zitat Kang, O., Rubin, D., & Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. The Modern Language Journal, 94(4), 554–566.CrossRef Kang, O., Rubin, D., & Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. The Modern Language Journal, 94(4), 554–566.CrossRef
Zurück zum Zitat Kang, O., & Wang, L. (2014). Impact of different task types on candidates’ speaking performances and interactive features that distinguish between CEFR levels. ISSN 1756-509X, 40. Kang, O., & Wang, L. (2014). Impact of different task types on candidates’ speaking performances and interactive features that distinguish between CEFR levels. ISSN 1756-509X, 40.
Zurück zum Zitat KayPENTAX. (2008). Multi-speech and CSL software. Lincoln Park: KayPENTAX. KayPENTAX. (2008). Multi-speech and CSL software. Lincoln Park: KayPENTAX.
Zurück zum Zitat Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211.CrossRef Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211.CrossRef
Zurück zum Zitat Leacock, C. (2004). Scoring free-responses automatically: A case study of a large-scale assessment. Examens, 1(3). Leacock, C. (2004). Scoring free-responses automatically: A case study of a large-scale assessment. Examens, 1(3).
Zurück zum Zitat Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405.CrossRef Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405.CrossRef
Zurück zum Zitat Longman, P. (2013). Official guide to Pearson test of English academic. Longman, P. (2013). Official guide to Pearson test of English academic.
Zurück zum Zitat MathWorks, Inc. (2013). MATLAB Release 2013a. [Computer program]. Retrieved February 15, 2013. MathWorks, Inc. (2013). MATLAB Release 2013a. [Computer program]. Retrieved February 15, 2013.
Zurück zum Zitat Ostendorf, M., Price, P. J., & Shattuck-Hufnagel, S. (1995). The Boston University radio news corpus. Linguistic Data Consortium, pp. 1–19. Ostendorf, M., Price, P. J., & Shattuck-Hufnagel, S. (1995). The Boston University radio news corpus. Linguistic Data Consortium, pp. 1–19.
Zurück zum Zitat Pickering, L. (1999). An analysis of prosodic systems in the classroom discourse of native speaker and nonnative speaker teaching assistants (Doctoral dissertation, University of Florida). Pickering, L. (1999). An analysis of prosodic systems in the classroom discourse of native speaker and nonnative speaker teaching assistants (Doctoral dissertation, University of Florida).
Zurück zum Zitat Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlíček, P., Qian, Y., Schwarz, P., Silovsky, J., Stemmer, G., & Vesel, K. (2011). The Kaldi speech recognition toolkit. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlíček, P., Qian, Y., Schwarz, P., Silovsky, J., Stemmer, G., & Vesel, K. (2011). The Kaldi speech recognition toolkit.
Zurück zum Zitat Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetric™ essay scoring system. The Journal of Technology, Learning and Assessment, 4(4), 1–22. Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetric™ essay scoring system. The Journal of Technology, Learning and Assessment, 4(4), 1–22.
Zurück zum Zitat Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883–895.CrossRef Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883–895.CrossRef
Metadaten
Titel
Improved automatic English proficiency rating of unconstrained speech with multiple corpora
verfasst von
David O. Johnson
Okim Kang
Romy Ghanem
Publikationsdatum
19.09.2016
Verlag
Springer US
Erschienen in
International Journal of Speech Technology / Ausgabe 4/2016
Print ISSN: 1381-2416
Elektronische ISSN: 1572-8110
DOI
https://doi.org/10.1007/s10772-016-9366-0

Weitere Artikel der Ausgabe 4/2016

International Journal of Speech Technology 4/2016 Zur Ausgabe

Neuer Inhalt