Spatio-temporal attention model for foreground detection in cross-scene surveillance videos

Dong Liang; Jiaxing Pan; Han Sun; Huiyu Zhou

doi:10.3390/s19235142

Spatio-temporal attention model for foreground detection in cross-scene surveillance videos

Dong Liang^*, Jiaxing Pan, Han Sun, Huiyu Zhou

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

6 Citations (Scopus)

13 Downloads (Pure)

Abstract

Foreground detection is an important theme in video surveillance. Conventional background modeling approaches build sophisticated temporal statistical model to detect foreground based on low-level features, while modern semantic/instance segmentation approaches generate high-level foreground annotation, but ignore the temporal relevance among consecutive frames. In this paper, we propose a Spatio-Temporal Attention Model (STAM) for cross-scene foreground detection. To fill the semantic gap between low and high level features, appearance and optical flow features are synthesized by attention modules via the feature learning procedure. Experimental results on CDnet 2014 benchmarks validate it and outperformed many state-of-the-art methods in seven evaluation metrics. With the attention modules and optical flow, its F-measure increased 9% and 6% respectively. The model without any tuning showed its cross-scene generalization on Wallflower and PETS datasets. The processing speed was 10.8 fps with the frame size 256 by 256.

Original language	English
Article number	5142
Journal	Sensors (Switzerland)
Volume	19
Issue number	23
DOIs	https://doi.org/10.3390/s19235142
Publication status	Published - 24 Nov 2019
Externally published	Yes

Bibliographical note

Funding Information:
This work is supported by the National Key R&D Program of China under Grant 2017YFB0802300, National Natural Science Foundation of China 61601223. H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union?s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325.

Funding Information:
Funding: This work is supported by the National Key R&D Program of China under Grant 2017YFB0802300, National Natural Science Foundation of China 61601223. H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union’s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325.

Publisher Copyright:
© 2019 by the authors. Licensee MDPI, Basel, Switzerland.

Copyright:
Copyright 2019 Elsevier B.V., All rights reserved.

Keywords

Attention model
Background modeling
Foreground detection
Optical flow

ASJC Scopus subject areas

Analytical Chemistry
Biochemistry
Atomic and Molecular Physics, and Optics
Instrumentation
Electrical and Electronic Engineering

Access to Document

10.3390/s19235142Licence: CC BY

Spatio-temporal attention model for foreground detection in cross-scene surveillance videos
Copyright 2019 the authors. This is an open access article published under a Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the author and source are cited.
Final published version, 3.39 MBLicence: CC BY

Cite this

@article{f195786cf5614cb0bcd07301cc558863,

title = "Spatio-temporal attention model for foreground detection in cross-scene surveillance videos",

abstract = "Foreground detection is an important theme in video surveillance. Conventional background modeling approaches build sophisticated temporal statistical model to detect foreground based on low-level features, while modern semantic/instance segmentation approaches generate high-level foreground annotation, but ignore the temporal relevance among consecutive frames. In this paper, we propose a Spatio-Temporal Attention Model (STAM) for cross-scene foreground detection. To fill the semantic gap between low and high level features, appearance and optical flow features are synthesized by attention modules via the feature learning procedure. Experimental results on CDnet 2014 benchmarks validate it and outperformed many state-of-the-art methods in seven evaluation metrics. With the attention modules and optical flow, its F-measure increased 9% and 6% respectively. The model without any tuning showed its cross-scene generalization on Wallflower and PETS datasets. The processing speed was 10.8 fps with the frame size 256 by 256.",

keywords = "Attention model, Background modeling, Foreground detection, Optical flow",

author = "Dong Liang and Jiaxing Pan and Han Sun and Huiyu Zhou",

note = "Funding Information: This work is supported by the National Key R&D Program of China under Grant 2017YFB0802300, National Natural Science Foundation of China 61601223. H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union?s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325. Funding Information: Funding: This work is supported by the National Key R&D Program of China under Grant 2017YFB0802300, National Natural Science Foundation of China 61601223. H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union{\textquoteright}s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325. Publisher Copyright: {\textcopyright} 2019 by the authors. Licensee MDPI, Basel, Switzerland. Copyright: Copyright 2019 Elsevier B.V., All rights reserved.",

year = "2019",

month = nov,

day = "24",

doi = "10.3390/s19235142",

language = "English",

volume = "19",

journal = "Sensors (Basel, Switzerland)",

issn = "1424-8220",

publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",

number = "23",

}

TY - JOUR

T1 - Spatio-temporal attention model for foreground detection in cross-scene surveillance videos

AU - Liang, Dong

AU - Pan, Jiaxing

AU - Sun, Han

AU - Zhou, Huiyu

N1 - Funding Information: This work is supported by the National Key R&D Program of China under Grant 2017YFB0802300, National Natural Science Foundation of China 61601223. H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union?s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325. Funding Information: Funding: This work is supported by the National Key R&D Program of China under Grant 2017YFB0802300, National Natural Science Foundation of China 61601223. H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union’s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 720325. Publisher Copyright: © 2019 by the authors. Licensee MDPI, Basel, Switzerland. Copyright: Copyright 2019 Elsevier B.V., All rights reserved.

PY - 2019/11/24

Y1 - 2019/11/24

N2 - Foreground detection is an important theme in video surveillance. Conventional background modeling approaches build sophisticated temporal statistical model to detect foreground based on low-level features, while modern semantic/instance segmentation approaches generate high-level foreground annotation, but ignore the temporal relevance among consecutive frames. In this paper, we propose a Spatio-Temporal Attention Model (STAM) for cross-scene foreground detection. To fill the semantic gap between low and high level features, appearance and optical flow features are synthesized by attention modules via the feature learning procedure. Experimental results on CDnet 2014 benchmarks validate it and outperformed many state-of-the-art methods in seven evaluation metrics. With the attention modules and optical flow, its F-measure increased 9% and 6% respectively. The model without any tuning showed its cross-scene generalization on Wallflower and PETS datasets. The processing speed was 10.8 fps with the frame size 256 by 256.

AB - Foreground detection is an important theme in video surveillance. Conventional background modeling approaches build sophisticated temporal statistical model to detect foreground based on low-level features, while modern semantic/instance segmentation approaches generate high-level foreground annotation, but ignore the temporal relevance among consecutive frames. In this paper, we propose a Spatio-Temporal Attention Model (STAM) for cross-scene foreground detection. To fill the semantic gap between low and high level features, appearance and optical flow features are synthesized by attention modules via the feature learning procedure. Experimental results on CDnet 2014 benchmarks validate it and outperformed many state-of-the-art methods in seven evaluation metrics. With the attention modules and optical flow, its F-measure increased 9% and 6% respectively. The model without any tuning showed its cross-scene generalization on Wallflower and PETS datasets. The processing speed was 10.8 fps with the frame size 256 by 256.

KW - Attention model

KW - Background modeling

KW - Foreground detection

KW - Optical flow

U2 - 10.3390/s19235142

DO - 10.3390/s19235142

M3 - Article

C2 - 31771250

AN - SCOPUS:85075476482

VL - 19

JO - Sensors (Basel, Switzerland)

JF - Sensors (Basel, Switzerland)

SN - 1424-8220

IS - 23

M1 - 5142

ER -

Spatio-temporal attention model for foreground detection in cross-scene surveillance videos

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this