NEUKONFIG: Reducing Edge Service Downtime When Repartitioning DNNs

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Deep Neural Networks (DNNs) may be partitioned across the edge and the cloud to improve the performance efficiency of inference. DNN partitions are determined based on operational conditions such as network speed. When operational conditions change DNNs will need to be repartitioned to maintain the overall performance. However, repartitioning using existing approaches, such as Pause and Resume, will incur a service downtime on the edge. This paper presents the NEUKONFIG framework that identifies the service downtime incurred when repartitioning DNNs and proposes approaches for reducing edge service downtime. The proposed approaches are based on ‘Dynamic Switching’ in which, when the network speed changes and given an existing edge-cloud pipeline, a new edge-cloud pipeline is initialised with new DNN partitions. Incoming inference requests are switched to the new pipeline for processing data. Two dynamic switching scenarios are considered: when a second edge-cloud pipeline is always running and when a second pipeline is only initialised when the network speed changes. Experimental studies are carried out on a lab-based testbed to demonstrate that Dynamic Switching reduces the downtime by at least an order of magnitude when compared to a baseline using Pause and Resume that has a downtime of 6 seconds. A trade-off in the edge service downtime and memory required is noted. The Dynamic Switching approach that requires the same amount of memory as the baseline reduces the edge service downtime to 0.6 seconds and to less than 1 millisecond in the best case when twice the amount of memory as the baseline is available.
Original languageEnglish
Title of host publication12th International Conference on Cloud Computing (CLOUD)
Publication statusAccepted - 01 Apr 2021
EventIEEE International Conference on Cloud Engineering -
Duration: 04 Oct 202108 Oct 2021
https://conferences.computer.org/IC2E/2021/

Publication series

NameInternational Conference on Cloud Computing (CLOUD): Proceedings
PublisherIEEE
ISSN (Print)2159-6182
ISSN (Electronic)2159-6190

Conference

ConferenceIEEE International Conference on Cloud Engineering
Abbreviated titleIC2E
Period04/10/202108/10/2021
Internet address

Fingerprint

Dive into the research topics of 'NEUKONFIG: Reducing Edge Service Downtime When Repartitioning DNNs'. Together they form a unique fingerprint.

Cite this