Top

Published in:

2020 | OriginalPaper | Chapter

Evolving Gaussian Process Kernels for Translation Editing Effort Estimation

Authors : Ibai Roman, Roberto Santana, Alexander Mendiburu, Jose A. Lozano

Published in: Learning and Intelligent Optimization

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In many Natural Language Processing problems the combination of machine learning and optimization techniques is essential. One of these problems is estimating the effort required to improve, under direct human supervision, a text that has been translated using a machine translation method. Recent developments in this area have shown that Gaussian Processes can be accurate for post-editing effort prediction. However, the Gaussian Process kernel has to be chosen in advance, and this choice influences the quality of the prediction. In this paper, we propose a Genetic Programming algorithm to evolve kernels for Gaussian Processes. We show that the combination of evolutionary optimization and Gaussian Processes removes the need for a-priori specification of the kernel choice, and achieves predictions that, in many cases, outperform those obtained with fixed kernels.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Assessing Simulated Annealing with Variable Neighborhoods

next chapter Predicting the Execution Time of the Interior Point Method for Solving Linear Programming Problems Using Artificial Neural Networks

http://vectors.nlpl.eu/repository/.

https://fasttext.cc/docs/en/crawl-vectors.html.

https://www.spinningbytes.com/resources/wordembeddings/.

Beck, D.: Modelling representation noise in emotion analysis using Gaussian processes. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), vol. 2, pp. 140–145 (2017)

Beck, D., Cohn, T., Hardmeier, C., Specia, L.: Learning structural kernels for natural language processing. CoRR abs/1508.02131 (2015). http://arxiv.org/abs/1508.02131

Beck, D.E.: Gaussian processes for text regression. Ph.D. thesis, University of Sheffield (2017)

Bing, W., Wen-qiong, Z., Ling, C., Jia-hong, L.: A GP-based kernel construction and optimization method for RVM. In: 2010 the 2nd International Conference on Computer and Automation Engineering (ICCAE), vol. 4, pp. 419–423, February 2010. https://doi.org/10.1109/ICCAE.2010.5451646

Callison-Burch, C., Koehn, P., Monz, C., Zaidan, O.F.: Findings of the 2011 workshop on statistical machine translation. In: Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 22–64. Association for Computational Linguistics (2011)

Cohn, T., Preotiuc-Pietro, D., Lawrence, N.: Gaussian processes for natural language processing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Tutorials, pp. 1–3 (2014)

Cohn, T., Specia, L.: Modelling annotator bias with multi-task Gaussian processes: an application to machine translation quality estimation. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 32–42 (2013)

Deriu, J., et al.: Leveraging large amounts of weakly supervised data for multi-language sentiment classification. In: Proceedings of the 26th International Conference on World Wide Web, pp. 1045–1052. International World Wide Web Conferences Steering Committee (2017)

Diosan, L., Rogozan, A., Pecuchet, J.P.: Evolving kernel functions for SVMs by genetic programming. In: Sixth International Conference on Machine Learning and Applications (ICMLA 2007), pp. 19–24 (2007). https://doi.org/10.1109/ICMLA.2007.70

10.

Dioşan, L., Rogozan, A., Pecuchet, J.P.: Improving classification performance of support vector machine by genetically optimising kernel shape and hyper-parameters. Appl. Intell. 36(2), 280–294 (2012). https://doi.org/10.1007/s10489-010-0260-1. https://link.springer.com/article/10.1007/s10489-010-0260-1CrossRef

11.

Duvenaud, D.: Automatic model construction with Gaussian processes. Thesis, University of Cambridge (2014). http://www.repository.cam.ac.uk/handle/1810/247281

12.

Duvenaud, D., Lloyd, J., Grosse, R., Tenenbaum, J., Zoubin, G.: Structure discovery in nonparametric regression through compositional kernel search. In: Proceedings of the 30th International Conference on Machine Learning, pp. 1166–1174 (2013). http://jmlr.org/proceedings/papers/v28/duvenaud13.html

13.

Friedman, M.: The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. Am. Stat. Assoc. 32(200), 675–701 (1937)CrossRef

14.

Gagné, C., Schoenauer, M., Sebag, M., Tomassini, M.: Genetic programming for kernel-based learning with co-evolving subsets selection. In: Runarsson, T.P., et al. (eds.) PPSN 2006. LNCS, vol. 4193, pp. 1008–1017. Springer, Heidelberg (2006). https://doi.org/10.1007/11844297_102CrossRef

15.

Genton, M.G.: Classes of kernels for machine learning: a statistics perspective. J. Mach. Learn. Res. 2, 299–312 (2002). http://dl.acm.org/citation.cfm?id=944790.944815MathSciNetMATH

16.

Grave, E., Bojanowski, P., Gupta, P., Joulin, A., Mikolov, T.: Learning word vectors for 157 languages. CoRR abs/1802.06893 (2018). http://arxiv.org/abs/1802.06893

17.

Hintze, J.L., Nelson, R.D.: Violin plots: a box plot-density trace synergism. Am. Stat. 52(2), 181–184 (1998)

18.

Howley, T., Madden, M.G.: An evolutionary approach to automatic kernel construction. In: Kollias, S., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4132, pp. 417–426. Springer, Heidelberg (2006). https://doi.org/10.1007/11840930_43CrossRef

19.

Koch, P., Bischl, B., Flasch, O., Bartz-Beielstein, T., Weihs, C., Konen, W.: Tuning and evolution of support vector kernels. Evol. Intell. 5(3), 153–170 (2012). https://doi.org/10.1007/s12065-012-0073-8. https://link.springer.com/article/10.1007/s12065-012-0073-8CrossRef

20.

Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992). google-Books-ID: Bhtxo60BV0ECMATH

21.

Kronberger, G., Kommenda, M.: Evolution of covariance functions for gaussian process regression using genetic programming. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) EUROCAST 2013. LNCS, vol. 8111, pp. 308–315. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-53856-8_39CrossRef

22.

Lampos, V., Zou, B., Cox, I.J.: Enhancing feature selection using word embeddings: the case of flu surveillance. In: Proceedings of the 26th International Conference on World Wide Web, pp. 695–704 (2017)

23.

Lloyd, J.R., Duvenaud, D., Grosse, R., Tenenbaum, J., Ghahramani, Z.: Automatic construction and natural-language description of nonparametric regression models. In: Twenty-Eighth AAAI Conference on Artificial Intelligence, June 2014. https://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8240

24.

Mercer, J.: XVI. Functions of positive and negative type, and their connection the theory of integral equations. Phil. Trans. R. Soc. Lond. A 209(441–458), 415–446 (1909). https://doi.org/10.1098/rsta.1909.0016. http://rsta.royalsocietypublishing.org/content/209/441-458/415 CrossRefMATH

25.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

26.

Montana, D.J.: Strongly typed genetic programming. Evol. Comput. 3(2), 199–230 (1995). https://doi.org/10.1162/evco.1995.3.2.199CrossRef

27.

Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), vol. 14, pp. 1532–1543 (2014)

28.

Polajnar, T., Rogers, S., Girolami, M.: Protein interaction detection in sentences via Gaussian processes: a preliminary evaluation. Int. J. Data Min. Bioinf. 5(1), 52–72 (2011)CrossRef

29.

Powell, M.J.D.: An efficient method for finding the minimum of a function of several variables without calculating derivatives. Comput. J. 7(2), 155–162 (1964). https://doi.org/10.1093/comjnl/7.2.155. http://comjnl.oxfordjournals.org/content/7/2/155MathSciNetCrossRefMATH

30.

Preoţiuc-Pietro, D., Cohn, T.: A temporal model of text periodicities using Gaussian processes. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 977–988 (2013)

31.

Rasmussen, C.E., Williams, C.K.: Gaussian Processes for Machine Learning. MIT Press, Cambridge (2006)MATH

32.

Santana, R.: Reproducing and learning new algebraic operations on word embeddings using genetic programming. CoRR abs/1702.05624 (2017). https://arxiv.org/abs/1702.05624v

33.

Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978). https://doi.org/10.1214/aos/1176344136. https://projecteuclid.org/euclid.aos/1176344136MathSciNetCrossRefMATH

34.

Shaffer, J.P.: Modified sequentially rejective multiple test procedures. J. Am. Stat. Assoc. (2012). https://amstat.tandfonline.com/doi/abs/10.1080/01621459.1986.10478341

35.

Shah, K., Cohn, T., Specia, L.: An investigation on the effectiveness of features for translation quality estimation. In: Proceedings of the Machine Translation Summit, vol. 14, pp. 167–174. Citeseer (2013)

36.

Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: Proceedings of Association for Machine Translation in the Americas, vol. 200 (2006)

37.

Specia, L.: Exploiting objective annotations for measuring translation post-editing effort. In: Proceedings of the 15th Conference of the European Association for Machine Translation, pp. 73–80 (2011)

38.

Specia, L., Shah, K., Souza, J.G., Cohn, T.: QuEst-a translation quality estimation framework. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 79–84 (2013)

39.

Sullivan, K.M., Luke, S.: Evolving kernels for support vector machine classification. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO 2007, pp. 1702–1707. ACM, New York (2007). https://doi.org/10.1145/1276958.1277292. http://doi.acm.org/10.1145/1276958.1277292

40.

Vapnik, V.: The Nature of Statistical Learning Theory. Springer-Verlag, New York (2000). https://doi.org/10.1007/978-1-4757-3264-1CrossRefMATH

Title: Evolving Gaussian Process Kernels for Translation Editing Effort Estimation
Authors: Ibai Roman
Roberto Santana
Alexander Mendiburu
Jose A. Lozano
Publisher: Springer International Publishing
Book: Learning and Intelligent Optimization
Print ISBN: 978-3-030-38628-3

Electronic ISBN: 978-3-030-38629-0

Copyright Year: 2020
DOI: https://doi.org/10.1007/978-3-030-38629-0_25

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner