Gesture recognition for musician computer interaction

Abstract

This thesis investigates how machine learning can be applied to the automatic recognition of a musical gesture by a computer. The recognition of gestures in a musical domain provides a number of interesting research challenges over and above the recognition of gestures in the more general field of human-computer interaction as, due to its real-time musical application, a low-latency, highly robust, user-configurable recognition system is required. These research challenges raise a number of fundamental questions related to the application of gesture recognition for musician-computer interaction; such as what differences, if any, are there between the application of machine learning for the classification of musical gestures from that of the classification of other gestures used throughout various fields within human-computer interaction? This thesis addresses such questions and in doing so tests the applicability of the leading machine learning algorithms for the recognition of musical gestures along with developing a number of new algorithms specifically for the recognition of musical gestures.

The work in this thesis focuses primarily on the discrete classification of a musical gesture, as opposed to the continuous mapping of a movement to a sound or control parameter. The scope of the research presented in this thesis is therefore constrained to the design and evaluation of machine learning algorithms that can be used to classify both static musical postures and musical gestures that consist of a cohesive sequence of movements that occur over a variable time period (i.e. temporal gestures).

The principal contributions of this thesis include a number of novel machine learning algorithms that have been specifically developed for the recognition of both static musical postures and temporal musical gestures. Another major contribution of this thesis is the development of a real-time gesture recognition software tool that has been designed to operate independently from any one specific piece of sensor hardware or audio software. Rather than pre-training the software tool to recognise a specific set of musical gestures, such as the communicative gestures of a conductor or the expressive gestures of a pianist for example; the software has instead been designed to facilitate a musician to train the recognition algorithms with the specific gestures that musician wants to use. The software has therefore been designed to enable a performer, regardless of their programming abilities, to quickly train a computer to recognise their musical gestures using a number of powerful machine learning algorithms, including the algorithms that have been specifically developed as the result of this thesis. The work in this thesis therefore not only contributes to the domains of musician-computer interaction and the more general field of human-computer interaction; it also facilitates performers to directly apply these contributions to their compositions, performances and/or research.

Date of Award	Jul 2011
Original language	English
Awarding Institution	Queen's University Belfast
Supervisor	Sile O'Modhrain (Supervisor) & Benjamin R. Knapp (Supervisor)