Skip to main content
SpringerLink
Log in

Neural Network Architecture Selection: Can Function Complexity Help?

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

This work analyzes the problem of selecting an adequate neural network architecture for a given function, comparing existing approaches and introducing a new one based on the use of the complexity of the function under analysis. Numerical simulations using a large set of Boolean functions are carried out and a comparative analysis of the results is done according to the architectures that the different techniques suggest and based on the generalization ability obtained in each case. The results show that a procedure that utilizes the complexity of the function can help to achieve almost optimal results despite the fact that some variability exists for the generalization ability of similar complexity classes of functions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Arai M (1993) Bounds on the number of hidden units in binary-valued three-layer neural networks. Neural Netw 6(6): 855–860

    Article  Google Scholar 

  2. Barron AR (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inform Theory 39(3): 930–945

    Article  MATH  MathSciNet  Google Scholar 

  3. Barron AR (1994) Approximation and estimation bounds for artificial neural networks. Mach Learn 14(1): 115–133

    MATH  Google Scholar 

  4. Baum EB, Haussler D (1990) What size net gives valid generalization?. Neural Comput 1(1): 151–160

    Article  Google Scholar 

  5. Blumer A, Ehrenfeucht A, Haussler D, Warmuth MK (1989) Learnability and the Vapnik-Chervonenkis dimension. J ACM 36(4): 929–965

    Article  MATH  MathSciNet  Google Scholar 

  6. Camargo LS, Yoneyama T (2001) Specification of training sets and the number of hidden neurons for multilayer perceptrons. Neural Comput 13(12): 2673–2680

    Article  MATH  Google Scholar 

  7. Demuth H , Beale M (1994) MATLAB neural networks toolbox—user’s guide version 4. The Math Works, USA

    Google Scholar 

  8. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7: 1–30

    MathSciNet  Google Scholar 

  9. Franco L (2006) Generalization ability of Boolean functions implemented in feedforward neural networks. Neurocomputing 70: 351–361

    Article  Google Scholar 

  10. Franco L, Anthony M (2004) On a generalization complexity measure for Boolean functions. In: Proceedings of the 2004 IEEE international joint conference on neural networks. pp 973–978

  11. Franco L, Anthony M (2006) The influence of oppositely classified examples on the generalization complexity of Boolean functions. IEEE Trans Neural Netw 17(3): 578–590

    Article  Google Scholar 

  12. Franco L, Cannas SA (2000) Generalization and selection of examples in feedforward neural networks. Neural Comput 12(10): 2405–2426

    Article  Google Scholar 

  13. Franco L, Cannas SA (2001) Generalization properties of modular networks, implementing the parity function. IEEE Trans Neural Netw 12: 1306–1313

    Article  Google Scholar 

  14. Frean M (1990) The upstart algorithm, a method for constructing and training feedforward neural networks. Neural Comput 2(2): 198–209

    Article  Google Scholar 

  15. Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32: 675–701

    Article  Google Scholar 

  16. Hájek J, Šidák Z, Sen PK (1999) Theory of rank tests, 2nd ed. Academic Press, Orlando

    MATH  Google Scholar 

  17. Haykin S (1994) Neural networks (a comprehensive foundation). Prentice Hall, USA

    MATH  Google Scholar 

  18. Lawrence S, Giles CL, and Tsoi A (1996) What size neural network gives optimal generalization? Convergence properties of backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617, University of Maryland

  19. LeCun Y, Denker J, Solla S, Howard RE, Jackel LD (1990) Optimal brain damage. In: Touretzky DS (eds) Advances in neural information processing systems II. Morgan Kaufman, San Mateo

    Google Scholar 

  20. Liang X (2006) Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation. Neural Comput Appl 16(1): 57–68

    Article  Google Scholar 

  21. Linial N, Mansour Y, Nisan N (1993) Constant depth circuits, Fourier transform and learnability. J ACM 40: 607–620

    Article  MATH  MathSciNet  Google Scholar 

  22. Liu Y, Starzyk JA, Zhu Z (2007) Optimizing number of hidden neurons in neural networks. In: AIAP’07 proceedings of the 25th conference on proceedings of the 25th IASTED international multi-conference. Anaheim, CA, USA, pp 121–126, ACTA Press

  23. Masters T (1993) Practical neural network recipes in C++. Academic Press Professional, Inc., San Diego

    Google Scholar 

  24. Mayoraz E (1991) On the Power of Networks of Majority Functions’. In: IWANN ’91 Proceedings of the international workshop on artificial neural networks. Springer-Verlag, London, UK, pp 78–85

  25. Mezard M, Nadal J-P (1989) Learning in feedforward layered networks (the tiling algorithm. J Phys A Math Gen 22(12): 2191–2203

    Article  MathSciNet  Google Scholar 

  26. Mirchandani G, Cao W (1989) On hidden nodes for neural nets. IEEE Trans Circuit Systems 36(5): 661–664

    Article  MathSciNet  Google Scholar 

  27. Moller MF (1993) Scaled conjugate gradient algorithm for fast supervised learning. Neural Netw 6(4): 525–533

    Article  Google Scholar 

  28. Neuralware I (2001) The reference guide. http://www.neuralware.com

  29. Piramuthu S, Shaw M, Gentry J (1994) A classification approach using multi-layered neural networks. Decis Support Syst 11: 509–525

    Article  Google Scholar 

  30. Scarselli F, Tsoi AC (1998) Universal approximation using feedforward neural networks a survey of some existing methods, and some new results. Neural Netw 11(1): 15–37

    Article  Google Scholar 

  31. Siu K-Y, Roychowdhury VP (1994) On optimal depth threshold circuits for multiplication and related problems. SIAM J Discret Math 7(2): 284–292

    Article  MATH  MathSciNet  Google Scholar 

  32. Vázquez EG, Yanez A, Galindo P, Pizarro J (2001) Repeated measures multiple comparison procedures applied to model selection in neural networks. In: IWANN ’01 proceedings of the 6th international work-conference on artificial and natural neural networks. Springer–Verlag, London, UK, pp 88–95

  33. Šmieja FJ (1993) Neural network constructive algorithms (trading generalization for learning efficiency?. Circuits Syst Signal Process 12(2): 331–374

    Article  MATH  Google Scholar 

  34. Wang J, Yi Z, Zurada JM, Lu B-L, Yin H (eds) (2006) Advances in neural networks—ISNN 2006, Third international symposium on neural networks, Chengdu, China, May 28–June 1, 2006, Proceedings, Part I, Vol. 3971 of Lecture notes in computer science. Springer

  35. Weigend AS, Rumelhart DE, Huberman BA (1990) Generalization by weight-elimination with application to forecasting. In: NIPS. pp 875–882

  36. Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics 1: 80–83

    Article  Google Scholar 

  37. Witten IH, Frank E (2005) Data mining, practical machine learning tools and techniques. Morgan Kaufmann, San Mateo

    MATH  Google Scholar 

  38. Yuan HC, Xiong FL, Huai XY (2003) A method for estimating the number of hidden neurons in feed-forward neural networks based on information entropy. Comput Electr Agric 40: 57–64

    Article  Google Scholar 

  39. Zhang Z, Ma X, Yang Y (2003) Bounds on the number of hidden neurons in three-layer binary neural networks. Neural Netw 16(7): 995–1002

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Lenguajes y Ciencias de la Computación, Universidad de Málaga, Campus de Teatinos S/N, 29071, Málaga, Spain

    Iván Gómez, Leonardo Franco & José M. Jerez

Authors
  1. Iván Gómez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leonardo Franco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José M. Jerez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonardo Franco.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gómez, I., Franco, L. & Jerez, J.M. Neural Network Architecture Selection: Can Function Complexity Help?. Neural Process Lett 30, 71–87 (2009). https://doi.org/10.1007/s11063-009-9108-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-009-9108-2

Keywords

Search

Navigation