Statistical Tests for Joint Analysis of Performance Measures

Alessio Benavoli, Cassio P. de Campos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)
175 Downloads (Pure)

Abstract

Recently there has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and architectural complexity). Once one has learned a model based on their devised method, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Unfortunately, the standard tests used for this purpose are not able to jointly consider performance measures. The aim of this paper is to resolve this issue by developing statistical procedures that are able to account for multiple competing measures at the same time. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameter of such models, as usually the number of studied cases is very reduced in such comparisons. Real data from a comparison among general purpose classifiers is used to show a practical application of our tests.
Original languageEnglish
Title of host publicationAdvanced Methodologies for Bayesian Networks
EditorsJoe Suzuki, Maomi Ueno
PublisherSpringer
Pages76-92
Number of pages17
ISBN (Electronic)978-3-319-28379-1
ISBN (Print)978-3-319-28378-4
DOIs
Publication statusPublished - 08 Jan 2016
EventSecond International Workshop AMBN 2015 - Yokohama, Japan
Duration: 16 Nov 201518 Nov 2015

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume9505
ISSN (Print)0302-9743

Conference

ConferenceSecond International Workshop AMBN 2015
CountryJapan
CityYokohama
Period16/11/201518/11/2015

Fingerprint Dive into the research topics of 'Statistical Tests for Joint Analysis of Performance Measures'. Together they form a unique fingerprint.

  • Cite this

    Benavoli, A., & de Campos, C. P. (2016). Statistical Tests for Joint Analysis of Performance Measures. In J. Suzuki, & M. Ueno (Eds.), Advanced Methodologies for Bayesian Networks (pp. 76-92). (Lecture Notes in Computer Science; Vol. 9505). Springer. https://doi.org/10.1007/978-3-319-28379-1_6