A number of neural networks can be formulated as the linear-in-the-parameters models. Training such networks can be transformed to a model selection problem where a compact model is selected from all the candidates using subset selection algorithms. Forward selection methods are popular fast subset selection approaches. However, they may only produce suboptimal models and can be trapped into a local minimum. More recently, a two-stage fast recursive algorithm (TSFRA) combining forward selection and backward model refinement has been proposed to improve the compactness and generalization performance of the model. This paper proposes unified two-stage orthogonal least squares methods instead of the fast recursive-based methods. In contrast to the TSFRA, this paper derives a new simplified relationship between the forward and the backward stages to avoid repetitive computations using the inherent orthogonal properties of the least squares methods. Furthermore, a new term exchanging scheme for backward model refinement is introduced to reduce computational demand. Finally, given the error reduction ratio criterion, effective and efficient forward and backward subset selection procedures are proposed. Extensive examples are presented to demonstrate the improved model compactness constructed by the proposed technique in comparison with some popular methods.
|Number of pages||14|
|Journal||IEEE Transactions on Neural Networks and Learning Systems|
|Early online date||15 Jul 2015|
|Publication status||Published - Aug 2015|
Bibliographical noteNeural networks have been widely used in many different areas due to their powerful approximation capability, yet to effectively and efficiently construct a compact network with excellent generalization capability has been the most challenging topic in the literature. Forward selection methods are popular, fast but greedy approaches for training many neural networks that can be formulated as the linear-in-the-parameters models. This paper proposes unified two-stage orthogonal least squares methods, combining forward selection and backward model refinement, to significantly reduce the computational demand and improve the network compactness. Extensive simulation studies confirm the effectiveness and efficiency of the proposed approach.
- Backward model refinement
- computational complexity
- forward selection
- linear-in-the-parameters model
- orthogonal least square (OLS)
ASJC Scopus subject areas
- Artificial Intelligence
- Computer Networks and Communications
- Computer Science Applications