Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion

Rui Zhang; Yang Hua; Tao Song; Zhengui Xue; Ruhui Ma; Haibing Guan

doi:10.1109/ICASSP49357.2023.10097015

Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion

Rui Zhang, Yang Hua, Tao Song, Zhengui Xue, Ruhui Ma^*, Haibing Guan

^*Corresponding author for this work

School of Electronics, Electrical Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Key frame sampling is a common component in video tasks. Putting more effort into key frames, rather than processing all frames equally, can significantly reduce computational costs and improve processing efficiency. This paper presents ORSampler, an adaptive Online Residual-based key frame Sampler. ORSampler relies on feature residuals to sample key frames and decouples from subsequent video tasks. To facilitate ORSampler, a self-coached mechanism is designed to speed up learning, and an adaptive multi-level feature fusion is proposed to fit the diversity of subsequent video tasks. OR-Sampler has a fast inference speed and can work online. Extensive experiments on two typical video tasks verify the effectiveness and generality of our proposed ORSampler.

Original language	English
Title of host publication	ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728163277
ISBN (Print)	9781728163284
DOIs	https://doi.org/10.1109/ICASSP49357.2023.10097015
Publication status	Published - 05 May 2023
Event	48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023 - Rhodes Island, Greece Duration: 04 Jun 2023 → 10 Jun 2023

Publication series

Name	IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISSN (Print)	1520-6149
ISSN (Electronic)	2379-190X

Conference

Conference	48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023
Country/Territory	Greece
City	Rhodes Island
Period	04/06/2023 → 10/06/2023

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Keywords

key frame sampling
non-uniform sampling
reinforcement learning
Video signal processing

ASJC Scopus subject areas

Software
Signal Processing
Electrical and Electronic Engineering

Access to Document

10.1109/ICASSP49357.2023.10097015Licence: Unspecified

Cite this

Zhang, R., Hua, Y., Song, T., Xue, Z., Ma, R., & Guan, H. (2023). Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings (IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP49357.2023.10097015

Zhang, Rui ; Hua, Yang ; Song, Tao et al. / Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings. Institute of Electrical and Electronics Engineers Inc., 2023. (IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings).

@inproceedings{68f711e53d23459eb31929322528b259,

title = "Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion",

abstract = "Key frame sampling is a common component in video tasks. Putting more effort into key frames, rather than processing all frames equally, can significantly reduce computational costs and improve processing efficiency. This paper presents ORSampler, an adaptive Online Residual-based key frame Sampler. ORSampler relies on feature residuals to sample key frames and decouples from subsequent video tasks. To facilitate ORSampler, a self-coached mechanism is designed to speed up learning, and an adaptive multi-level feature fusion is proposed to fit the diversity of subsequent video tasks. OR-Sampler has a fast inference speed and can work online. Extensive experiments on two typical video tasks verify the effectiveness and generality of our proposed ORSampler.",

keywords = "key frame sampling, non-uniform sampling, reinforcement learning, Video signal processing",

author = "Rui Zhang and Yang Hua and Tao Song and Zhengui Xue and Ruhui Ma and Haibing Guan",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023 ; Conference date: 04-06-2023 Through 10-06-2023",

year = "2023",

month = may,

day = "5",

doi = "10.1109/ICASSP49357.2023.10097015",

language = "English",

isbn = "9781728163284",

series = "IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings",

address = "United States",

}

Zhang, R, Hua, Y, Song, T, Xue, Z, Ma, R & Guan, H 2023, Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion. in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings. IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings, Institute of Electrical and Electronics Engineers Inc., 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023, Rhodes Island, Greece, 04/06/2023. https://doi.org/10.1109/ICASSP49357.2023.10097015

Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion. / Zhang, Rui; Hua, Yang; Song, Tao et al.
ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings. Institute of Electrical and Electronics Engineers Inc., 2023. (IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion

AU - Zhang, Rui

AU - Hua, Yang

AU - Song, Tao

AU - Xue, Zhengui

AU - Ma, Ruhui

AU - Guan, Haibing

PY - 2023/5/5

Y1 - 2023/5/5

N2 - Key frame sampling is a common component in video tasks. Putting more effort into key frames, rather than processing all frames equally, can significantly reduce computational costs and improve processing efficiency. This paper presents ORSampler, an adaptive Online Residual-based key frame Sampler. ORSampler relies on feature residuals to sample key frames and decouples from subsequent video tasks. To facilitate ORSampler, a self-coached mechanism is designed to speed up learning, and an adaptive multi-level feature fusion is proposed to fit the diversity of subsequent video tasks. OR-Sampler has a fast inference speed and can work online. Extensive experiments on two typical video tasks verify the effectiveness and generality of our proposed ORSampler.

AB - Key frame sampling is a common component in video tasks. Putting more effort into key frames, rather than processing all frames equally, can significantly reduce computational costs and improve processing efficiency. This paper presents ORSampler, an adaptive Online Residual-based key frame Sampler. ORSampler relies on feature residuals to sample key frames and decouples from subsequent video tasks. To facilitate ORSampler, a self-coached mechanism is designed to speed up learning, and an adaptive multi-level feature fusion is proposed to fit the diversity of subsequent video tasks. OR-Sampler has a fast inference speed and can work online. Extensive experiments on two typical video tasks verify the effectiveness and generality of our proposed ORSampler.

KW - key frame sampling

KW - non-uniform sampling

KW - reinforcement learning

KW - Video signal processing

U2 - 10.1109/ICASSP49357.2023.10097015

DO - 10.1109/ICASSP49357.2023.10097015

M3 - Conference contribution

AN - SCOPUS:85180553169

SN - 9781728163284

T3 - IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings

BT - ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023

Y2 - 4 June 2023 through 10 June 2023

ER -

Zhang R, Hua Y, Song T, Xue Z, Ma R, Guan H. Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): Proceedings. Institute of Electrical and Electronics Engineers Inc. 2023. (IEEE International Conference on Acoustics, Speech and Signal Processing: Proceedings). doi: 10.1109/ICASSP49357.2023.10097015

Online residual-based key frame sampling with self-coach mechanism and adaptive multi-level feature fusion

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this