Structural Test Coverage Criteria for Deep Neural Networks

Youcheng Sun, Xiaowei Huang, Daniel Kroening, James Sharp, Matthew Hill, Rob Ashmore

Research output: Contribution to journalArticlepeer-review

69 Citations (Scopus)
1183 Downloads (Pure)

Abstract

Deep neural networks (DNNs) have a wide range of applications, and software employing them must bethoroughly tested, especially in safety-critical domains. However, traditional software test coverage metricscannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we proposea family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that test inputs that are generated with guidance by ourproposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods,we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test input generation. Ourexperiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, includingMNIST, CIFAR-10 and ImageNet.
Original languageEnglish
Article number94
Number of pages23
JournalACM Transactions on Embedded Computing Systems
Volume18
Issue number5s
Early online date08 Oct 2019
DOIs
Publication statusEarly online date - 08 Oct 2019

Fingerprint

Dive into the research topics of 'Structural Test Coverage Criteria for Deep Neural Networks'. Together they form a unique fingerprint.

Cite this