Context-aware Mouse Behaviour Recognition using Hidden Markov Models

Zheheng Jiang, Danny Crookes, Brian D Green, Yunfeng Zhao, Haiping Ma, Ling Li, Shengping Zhang, Dacheng Tao, Huiyu Zhou

Research output: Contribution to journalArticlepeer-review

30 Citations (Scopus)
558 Downloads (Pure)


Automated recognition of mouse behaviours is crucial in studying psychiatric and neurologic diseases. To achieve this objective, it is very important to analyse temporal dynamics of mouse behaviours. In particular, the change between mouse neighbouring actions is swift in a short period. In this paper, we develop and implement a novel Hidden Markov Model (HMM) algorithm to describe the temporal characteristics of mouse behaviours. In particular, we here propose a hybrid deep learning architecture, where the first unsupervised layer relies on an advanced spatial-temporal segment Fisher Vector (SFV) encoding both visual and contextual features. Subsequent supervised layers based on our segment aggregate network (SAN) are trained to estimate the state dependent observation probabilities of the HMM. The proposed architecture shows the ability to discriminate between visually similar behaviours and results in high recognition rates with the strength of processing imbalanced mouse behaviour datasets. Finally, we evaluate our approach using JHuang's and our own datasets, and the results show that our method outperforms other state-of-the-art approaches.

Original languageEnglish
Pages (from-to)1133 - 1148
Number of pages16
JournalIEEE Trans. on Image Processing
Issue number3
Early online date10 Oct 2018
Publication statusPublished - Mar 2019


Dive into the research topics of 'Context-aware Mouse Behaviour Recognition using Hidden Markov Models'. Together they form a unique fingerprint.

Cite this