Efficient neural network architecture search

Abstract

In recent years an increasing number of researchers and practitioners focused on algorithms for large-scale neural network architecture search. Consequently, numerous publications started to push the boundaries of accuracy, however at immense training costs. For tens of days, hundreds of GPUs evaluate thousands of candidates in order to discover a configuration that surpasses the accuracy of hand designed networks or of the results of other automated searches. After designing our own type of search and analysing its drawbacks, we observed repetitive evaluations of similar candidates. Unless the intermediate results are wisely reused, architecture searches started from scratch have high chances of wasting compute time on similar or identical candidates. We propose to speed up the evaluation phase by replacing the slow training process with a fast and reliable prediction of the candidate’s performance.

In contrast to previously proposed approaches, our prediction is not only calibrated on the topological network information, but also on the characterization of the dataset-difficulty which allows us to re-tune the prediction for unseen datasets without any training.

We employ our predictor in image and text classification tasks, and emphasize oil the delivered speed-up in comparison with related work.

Date of Award	Dec 2019
Original language	English
Awarding Institution	Queen's University Belfast
Supervisor	Dimitrios Nikolopoulos (Supervisor), Cristiano Malossi (Supervisor) & Costas Bekas (Supervisor)