Two-Stage Orthogonal Least Squares Methods for Neural Network Construction

Long Zhang, Kang Li, Er Wei Bai, George W. Irwin

Research output: Contribution to journalArticlepeer-review

30 Citations (Scopus)

Abstract

A number of neural networks can be formulated as the linear-in-the-parameters models. Training such networks can be transformed to a model selection problem where a compact model is selected from all the candidates using subset selection algorithms. Forward selection methods are popular fast subset selection approaches. However, they may only produce suboptimal models and can be trapped into a local minimum. More recently, a two-stage fast recursive algorithm (TSFRA) combining forward selection and backward model refinement has been proposed to improve the compactness and generalization performance of the model. This paper proposes unified two-stage orthogonal least squares methods instead of the fast recursive-based methods. In contrast to the TSFRA, this paper derives a new simplified relationship between the forward and the backward stages to avoid repetitive computations using the inherent orthogonal properties of the least squares methods. Furthermore, a new term exchanging scheme for backward model refinement is introduced to reduce computational demand. Finally, given the error reduction ratio criterion, effective and efficient forward and backward subset selection procedures are proposed. Extensive examples are presented to demonstrate the improved model compactness constructed by the proposed technique in comparison with some popular methods.

Original languageEnglish
Article number6895303
Pages (from-to)1608-1621
Number of pages14
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume26
Issue number8
Early online date15 Jul 2015
DOIs
Publication statusPublished - Aug 2015

Bibliographical note

Neural networks have been widely used in many different areas due to their powerful approximation capability, yet to effectively and efficiently construct a compact network with excellent generalization capability has been the most challenging topic in the literature. Forward selection methods are popular, fast but greedy approaches for training many neural networks that can be formulated as the linear-in-the-parameters models. This paper proposes unified two-stage orthogonal least squares methods, combining forward selection and backward model refinement, to significantly reduce the computational demand and improve the network compactness. Extensive simulation studies confirm the effectiveness and efficiency of the proposed approach.

Keywords

  • Backward model refinement
  • computational complexity
  • forward selection
  • linear-in-the-parameters model
  • orthogonal least square (OLS)

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Software

Fingerprint

Dive into the research topics of 'Two-Stage Orthogonal Least Squares Methods for Neural Network Construction'. Together they form a unique fingerprint.

Cite this