Top

Arabian Journal for Science and Engineering

Published in:

22-05-2021 | Research Article-Computer Engineering and Computer Science

An Enhanced Gated Recurrent Unit with Auto-Encoder for Solving Text Classification Problems

Authors: Muhammad Zulqarnain, Rozaida Ghazali, Yana Mazwin Mohmad Hassim, Muhammad Aamir

Published in: Arabian Journal for Science and Engineering | Issue 9/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Classification has become an important task for automatically categorizing documents based on their respective group. The purpose of classification is to assign the pre-specified group or class to an instance based on the observed features related to that instance. For accurate text classification, feature selection techniques are normally used to identify important features and to remove irrelevant, undesired and noisy features for minimizing the dimensionality of feature space. Therefore, in this research, a new model namely Encoder Simplified GRU (ES-GRU) is proposed to reduce dimension of data using an auto-encoder (AE). Gated Recurrent Unit (GRU) is a deep learning algorithm that contains update gate and reset gate, which is considered as one of the most efficient text classification technique, specifically on sequential datasets. Accordingly, the reset gate is replaced with an update gate in order to reduce the redundancy and complexity in the standard GRU. The proposed model has been evaluated on five benchmark text datasets and compared with six baseline well-known text classification approaches, which includes standard GRU, AE, Long Short-Term Memory, Convolutional Neural Network, Support Vector Machine, and Naïve Bayes. Based on various types of performance evaluation parameters, a considerable amount of improvement has been observed in the performance of the proposed model as compared to state-of-the-art approaches.

previous article JayaL: A Novel Jaya Algorithm Based on Elite Local Search for Optimization Problems

next article Hybrid and Double Modular Redundancy (DMR)-Based Fault-Tolerant Carry Look-Ahead Adder Design

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Wang Z.; and Qu, Z. “Research on web text classification algorithm based on improved CNN and SVM,” IEEE, pp. 1958–1961, 2017.

Sharif, W.; Samsudin, N.A; M. M. Deris, M.M and M. Aamir, “Improved relative discriminative criterion feature ranking technique for text classification.” Int. J. Artif. Intell., 15(2), pp. 61–78, 2017.

Ahmed, R.; Al-Khatib, W.G. and Mahmoud, S.: “A Survey on handwritten documents word spotting.” Int. J. Multimed. Inf. Retr., 6(1), pp. 31–47, 2016 https://doi.org/10.1007/s13735-016-0110-y.

Zulqarnain, M.; Ishak, S.A.; Ghazali, R.; Nawi, N.M.: An improved deep learning approach based on variant two-state gated recurrent unit and word embeddings for sentiment classification. Int. J. Adv. Comput. Sci. Appl. 11(1), 594–603 (2020)

Yi, J.; Zhang, Y.; Zhao, X. and Wan, J.: “A novel text clustering approach using deep-learning vocabulary network.” Math. Probl. Eng., 2017, 2017 https://doi.org/10.1155/2017/8310934.

Barushka, A.; Hajek, P.: Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks. Appl. Intell. 48(10), 3538–3556 (2018). https://doi.org/10.1007/s10489-018-1161-yCrossRef

Li, L.; Goh, T.T.; Jin, D.: How textual quality of online reviews affect classification performance: a case of deep learning sentiment analysis. Neural Comput. Appl. 32(9), 4387–4415 (2020). https://doi.org/10.1007/s00521-018-3865-7CrossRef

Jadhav, S.; Kasar, R.; Lade, N.; Patil, M.; Kolte, S.: Disease prediction by machine learning from healthcare communities. Int. J. Sci. Res. Sci. Technol. 5, 8869–8869 (2019). https://doi.org/10.32628/ijsrst19633CrossRef

Sharif, W.; Tun, U.; Onn, H.; Tri, I.; Yanto, R.: An optimised support vector machine with ringed seal search algorithm for efficient text classification. J. Eng. Sci. Technol. 14(3), 1601–1613 (2019)

10.

Kowsari, K.; Brown, D.E.; Heidarysafa, M.; Meimandi, K.J.; Gerber, M.S. and Barnes, L.E.: “HDLTex: Hierarchical deep learning for text classification.” 2017 16th IEEE Int. Conf. Mach. Learn. Appl., pp. 364–371, 2017 https://doi.org/10.1109/ICMLA.2017.0-134.

11.

Dawar, M.: “Fast fuzzy feature clustering for text classification.” Acad. Ind. Res. Collab. Centre, Comput. Sci. Inf. Technol., 2, pp. 167–172, 2012 https://doi.org/10.5121/csit.2012.2317.

12.

Onan, A.; Korukoǧlu, S.; Bulut, H.: Ensemble of keyword extraction methods and classifiers in text classification. Expert Syst. Appl. 57, 232–247 (2016). https://doi.org/10.1016/j.eswa.2016.03.045CrossRef

13.

Berger, M.J.: “Large scale multi-label text classification with semantic word vectors.” Tech. Rep., pp. 1–8, 2014.

14.

Yeh, C.K.; Wu, W.C.; Ko, W.J.; Wang, Y.C.F.: “Learning deep latent spaces for multi-label classification.” 31st AAAI Conf Artif. Intell. AAAI 2017, 2838–2844 (2017)

15.

Xu, J.; Xu, C.; Zou, B.; Tang, Y.Y.; Peng, J. and You, X.: “New incremental learning algorithm with support vector machines.” IEEE Trans. Syst. Man, Cybern. Syst., vol. 49, no. 11, pp. 2230–2241, 2019 https://doi.org/10.1109/TSMC.2018.2791511.

16.

Xu, S.: Bayesian Naïve Bayes classifiers to text classification. J. Inf. Sci. 44(1), 48–59 (2018). https://doi.org/10.1177/0165551516677946CrossRef

17.

Ghiassi, M.; Olschimke, M.; Moon, B.; Arnaudo, P.: Automated text classification using a dynamic artificial neural network model. Expert Syst. Appl. 39(12), 10967–10976 (2012). https://doi.org/10.1016/j.eswa.2012.03.027CrossRef

18.

Liu, L.: Hierarchical learning for large multi-class network classification. Proc. Int. Conf. Pattern Recognit. (2016). https://doi.org/10.1109/ICPR.2016.7899980CrossRef

19.

Hochreiter, S.: Long short term memory. Neural Comput. 9(8), 1–32 (1997). https://doi.org/10.1144/GSL.MEM.1999.018.01.02MathSciNetCrossRef

20.

Cho, K. et al. “Learning phrase representations using rnn encoder-decoder for statistical machine translation.” arXiv, pp. 1–15, 2014 https://doi.org/10.3115/v1/D14-1179.

21.

Zhou, G.: Minimal gated unit for recurrent neural networks. ICML 7, 153–163 (2016)

22.

Kim, D.; Seo, D.; Cho, S.; Kang, P.: Multi-co-training for document classification using various document representations: TF–IDF, LDA, and Doc2Vec. Inf. Sci. (Ny) 477, 15–29 (2019). https://doi.org/10.1016/j.ins.2018.10.006CrossRef

23.

Tang, D.; Qin, B. and Liu, T.: “Document modeling with gated recurrent neural network for sentiment classification.” Proc. 2015 Conf. Empir. Methods Nat. Lang. Process., pp. 1422–1432, 2015 https://doi.org/10.18653/v1/D15-1167.

24.

Conneau, A.; Schwenk, H.; Barrault, L. and Lecun, Y.: “Very deep convolutional networks for text classification,” arXiv, pp. 1–10, 2017.

25.

Kowsari, K.; Meimandi, K.J.; Heidarysafa, M.; Mendu, S.; Barnes, L.; Brown, D.: Text classification algorithms: A survey. Inf. 10(4), 1–68 (2019). https://doi.org/10.3390/info10040150CrossRef

26.

Vincent, P.: “A Neural Probabilistic Language Model.” Neural Probabilistic Lang Model. J. Mach. Learn. Res. 3, 1137–1155 (2003)MATH

27.

Socher, R.; Huval, B.; Manning, C.D. and Ng, A.Y.: “Semantic compositionality through recursive matrix-vector spaces.” Proc. 2012 Jt. Conf. Empir. methods Nat. Lang. Process. Comput. Nat. Lang. Learn., pp. 1201–1211, 2012.

28.

Joulin, A.; Grave, E.; Bojanowski, P. and Mikolov, T.: “Bag of tricks for efficient text classification.” 2016 1511.09249v1.

29.

Prusa, J.D. and Khoshgoftaar, T.M.: “Designing a better data representation for deep neural networks and text classification,” Proc.-2016 IEEE 17th Int. Conf. Inf. Reuse Integr. IRI 2016, pp. 411–416, 2016 https://doi.org/10.1109/IRI.2016.61.

30.

Zhang, X.; Zhao, J. and Lecun, Y.: “Character-level convolutional networks for text classification.” Adv. Neural Inf. Process. Syst., vol. 2015 pp. 649–657, 2015.

31.

Chung, J.; Gulcehre, C.; Cho, K. and Bengio, Y.: “Empirical evaluation of gated recurrent neural networks on sequence modeling.” pp. 1–9, 2014 https://doi.org/10.1109/IJCNN.2015.7280624.

32.

Zhou, P.; Qi, Z.; Zheng, S.; Xu, J.; Bao, H. and Xu, B.: “Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling.” COLING 2016-26th Int. Conf. Comput. Linguist. Proc. COLING 2016 Tech. Pap., 2(1), pp. 3485–3495, 2016.

33.

Wei, J. and Zou, K.: “EDA: Easy data augmentation techniques for boosting performance on text classification tasks.” EMNLP-IJCNLP 2019-2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. Process. Proc. Conf., pp. 6382–6388, 2020 https://doi.org/10.18653/v1/d19-1670.

34.

Wang, Z. and Wu, Q.: “An Integrated Deep Generative Model for Text Classification and Generation,” Math. Probl. Eng., 2018, 2018 https://doi.org/10.1155/2018/7529286.

35.

Liao, R. et al.: “Reviving and improving recurrent back-propagation,” arXiv, pp. 3082–3091, 2018.

36.

Noaman, H.M.; Sarhan, S.S.; Rashwan, M.A.A.: Enhancing recurrent neural network-based language models by word tokenization. Human-centric Comput. Inf. Sci. 8(1), 1–13 (2018). https://doi.org/10.1186/s13673-018-0133-xCrossRef

37.

Ghazali, R.; Husaini, N.A.; Ismail, L.H.; Herawan, T. and Hassim, Y.M.M.: “The performance of a Recurrent HONN for temperature time series prediction.” Proc. Int. Jt. Conf. Neural Networks, pp. 518–524, 2014 https://doi.org/10.1109/IJCNN.2014.6889789.

38.

Wang, Y.; Wang, H.; Zhang, X.; Chaspari, T.; Choe, Y. and Lu, M.: “An Attention-aware Bidirectional Multi-residual Recurrent Neural Network (Abmrnn): A Study about Better Short-term Text Classification,” ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process. Proc., 2019, pp. 3582–3586, 2019 https://doi.org/10.1109/ICASSP.2019.8682565.

39.

Samarawickrama, A.J.P. and Fernando, T.G.I.: “A recurrent neural network approach in predicting daily stock prices an application to the Sri Lankan stock market.” 2017 IEEE Int. Conf. Ind. Inf. Syst. ICIIS 2017-Proc., 2018, pp. 1–6, 2018 https://doi.org/10.1109/ICIINFS.2017.8300345.

40.

Hochreiter, S.; Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735CrossRef

41.

Sundermeyer, M.; Ney, H. and Schlüter, R.: “From feedforward to recurrent LSTM neural networks for language modeling.” IEEE/ACM Trans. Audio, Speech, Lang. Process, 23(3), pp. 517–529, 2015.

42.

Lee, H.: “For modeling sentences and documents.” Proc. 15th Annu. Conf. North Am. Chapter Assoc. Comput., pp. 1512–1521, 2015.

43.

Pascanu, R.; Tour, D.; Mikolov, T.; Tour, D.: On the difficulty of training recurrent neural networks. Conf. ICLR 2, 1310–1318 (2013)

44.

Hao, Y.; Sheng, Y.; Wang, J.: Variant gated recurrent units with encoders to preprocess packets for payload-aware intrusion detection. IEEE Access 7, 49985–49998 (2019). https://doi.org/10.1109/ACCESS.2019.2910860CrossRef

45.

Aamir, M.; Nawi, N.M.; Bin Mahdin, H.; Naseem, R. and Zulqarnain, M.: “Auto-encoder variants for solving handwritten digits classification problem.” Int. J. Fuzzy Log. Intell. Syst., 20(1), pp. 8–16, 2020 https://doi.org/10.5391/IJFIS.2020.20.1.8.

46.

Metwally, A.A.; Yu, P.S.; Reiman, D.; Dai, Y.; Finn, P.W.; Perkins, D.L.: Utilizing longitudinal microbiome taxonomic profiles to predict food allergy via long short-term memory networks. PLoS Comput. Biol. 15(2), 1–16 (2019).CrossRef

47.

Pennington, J.; Socher, R. and Manning, C.D.: “GloVe : Global vectors for word representation.” Proc. Conf. Empir. Methods Nat. Lang. Process., pp. 1532–1543, 2014.

48.

Hinton, G.: Dropout: a simple way to prevent neural networks from overfitting. J. Machine Learn. Res. 15(1), 1929–1958 (2014)MathSciNetMATH

49.

Chu, A.; Stratos, K. and Gimpel, K.: “Unsupervised label refinement improves dataless text classification,” arXiv, 2020.

50.

Shen, D. et al.: “Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms,” ACL 2018-56th Annu. Meet. Assoc. Comput. Linguist. Proc. Conf. (Long Pap., 1, pp. 440–450, 2018 https://doi.org/10.18653/v1/p18-1041.

51.

Yogatama, D.; Dyer, C.; Ling, W.; Blunsom, P.: “Generative and discriminative text classification with recurrent neural networks.” arXiv, no. May, pp. 1–9, 2017.

52.

Kong, L.; Jiang, H.; Zhuang, Y.; Lyu, J.; Zhao, T.; and Zhang, C.: “ibrated language model fine-tuning for in- and out-of-distribution dataCal.” axXiv, pp. 1326–1340, 2020 https://doi.org/10.18653/v1/2020.emnlp-main.102.

53.

J. Xu and Q. Du, “A deep investigation into fasttext.” Proc. 21st IEEE Int. Conf. High Perform. Comput. Commun. 17th IEEE Int. Conf. Smart City 5th IEEE Int. Conf. Data Sci. Syst. HPCC/SmartCity/DSS 2019, pp. 1714–1719, 2019 https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00234.

54.

Wang, T.; Liu, L.; Zhang, H.; Zhang, L.; Chen, X.: Joint character-level convolutional and generative adversarial networks for text classification. Complexity 2020, 1–11 (2020). https://doi.org/10.1155/2020/8516216CrossRef

55.

Ma, Y.; Fan, H.; Zhao, C.: Feature-based fusion adversarial recurrent neural networks for text sentiment classification. IEEE Access 7, 132542–132551 (2019). https://doi.org/10.1109/ACCESS.2019.2940506CrossRef

56.

Fu, X.; Yang, J.; Li, J.; Fang, M.; Wang, H.: Lexicon-enhanced LSTM with attention for general sentiment analysis. IEEE Access 6, 71884–71891 (2018). https://doi.org/10.1109/ACCESS.2018.2878425CrossRef

57.

Camacho-Collados, J. and Pilehvar, M. T.: “on the role of text preprocessing in neural network architectures: An evaluation study on text categorization and sentiment analysis,” arXiv Prepr. arXiv1707.01780, pp. 40–46, 2018, https://doi.org/10.18653/v1/w18-5406.

58.

Liu, B.: Text sentiment analysis based on CBOW model and deep learning in big data environment. J. Ambient Intell. Humaniz. Comput. 11(2), 451–458 (2020). https://doi.org/10.1007/s12652-018-1095-6CrossRef

Title: An Enhanced Gated Recurrent Unit with Auto-Encoder for Solving Text Classification Problems
Authors: Muhammad Zulqarnain
Rozaida Ghazali
Yana Mazwin Mohmad Hassim
Muhammad Aamir
Publication date: 22-05-2021
Publisher: Springer Berlin Heidelberg
Published in: Arabian Journal for Science and Engineering / Issue 9/2021
Print ISSN: 2193-567X
Electronic ISSN: 2191-4281
DOI: https://doi.org/10.1007/s13369-021-05691-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 9/2021

A Simultaneous Moth Flame Optimizer Feature Selection Approach Based on Levy Flight and Selection Operators for Medical Diagnosis

Classification of Marine Plankton Based on Few-shot Learning

Arabic Sentiment Analysis Using Deep Learning and Ensemble Methods

Accurate Classification of COVID-19 Based on Incomplete Heterogeneous Data using a KNN Variant Algorithm

Addressing Limited Vocabulary and Long Sentences Constraints in English–Arabic Neural Machine Translation

Chaos-Based Mutual Synchronization of Three-Layer Tree Parity Machine: A Session Key Exchange Protocol Over Public Channel

Premium Partners