Spatio-temporal prompting network for robust video feature extraction

Guanxiong Sun, Chi Wang, Zhaoyu Zhang, Jiankang Deng, Stefanos Zafeiriou, Yang Hua

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Frame quality deterioration is one of the main challenges in the field of video understanding. To compensate for the information loss caused by deteriorated frames, recent approaches exploit transformer-based integration modules to obtain spatio-temporal information. However, these integration modules are heavy and complex. Furthermore, each integration module is specifically tailored for its target task, making it difficult to generalise to multiple tasks. In this paper, we present a neat and unified framework, called Spatio-Temporal Prompting Network (STPN). It can efficiently extract robust and accurate video features by dynamically adjusting the input features in the backbone network. Specifically, STPN predicts several video prompts containing spatio-temporal information of neighbour frames. Then, these video prompts are prepended to the patch embeddings of the current frame as the updated input for video feature extraction. Moreover, STPN is easy to generalise to various video tasks because it does not contain task-specific modules. Without bells and whistles, STPN achieves state-of-the-art performance on three widely-used datasets for different video understanding tasks, i.e., ImageNetVID for video object detection, YouTubeVIS for video instance segmentation, and GOT-10k for visual object tracking. Codes are available at https://github.com/guanxiongsun/STPN
Original languageEnglish
Title of host publicationProceedings of the IEEE/CVF International Conference on Computer Vision
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages13587-13597
Number of pages11
ISBN (Electronic)9798350307184
ISBN (Print)9798350307191
DOIs
Publication statusPublished - 15 Jan 2024
Event2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) - Paris, France
Duration: 02 Oct 202306 Oct 2023

Publication series

NameProceedings of the IEEE/CVF International Conference on Computer Vision
ISSN (Print)1550-5499
ISSN (Electronic)2380-7504

Conference

Conference2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
Country/TerritoryFrance
CityParis
Period02/10/202306/10/2023

Fingerprint

Dive into the research topics of 'Spatio-temporal prompting network for robust video feature extraction'. Together they form a unique fingerprint.

Cite this