TY - GEN
T1 - Efficient benchmarking of content-based image retrieval via resampling
AU - Shen, Jialie
AU - Shepherd, John
PY - 2006/12/1
Y1 - 2006/12/1
N2 - While content-based image retrieval (CBIR) is an expanding field, and new approaches to ever more effective retrieval are frequently proposed, relatively little attention has so far been paid to the process of evaluating the effectiveness of CBIR methods. Most of the reported evaluations use standard IR evaluation methodologies, with little consideration of their statistical significance or appropriateness for CBIR, which makes it difficult to assess the precise impact of individual methods. In this paper, we present a new approach for evaluating CBIR systems which provides both efficient and statistically-sound performance evaluation. The approach is based on stratified sampling, and provides a significant improvement over existing evaluation approaches. Comprehensive experiments using our approach to evaluate a range of CBIR methods have shown that the approach reduces not only the estimation error, but also reduces the size of the test data set required to achieve specific estimation error levels.
AB - While content-based image retrieval (CBIR) is an expanding field, and new approaches to ever more effective retrieval are frequently proposed, relatively little attention has so far been paid to the process of evaluating the effectiveness of CBIR methods. Most of the reported evaluations use standard IR evaluation methodologies, with little consideration of their statistical significance or appropriateness for CBIR, which makes it difficult to assess the precise impact of individual methods. In this paper, we present a new approach for evaluating CBIR systems which provides both efficient and statistically-sound performance evaluation. The approach is based on stratified sampling, and provides a significant improvement over existing evaluation approaches. Comprehensive experiments using our approach to evaluate a range of CBIR methods have shown that the approach reduces not only the estimation error, but also reduces the size of the test data set required to achieve specific estimation error levels.
KW - Evaluation
KW - Image retrieval
KW - Sampling
UR - http://www.scopus.com/inward/record.url?scp=34547166901&partnerID=8YFLogxK
U2 - 10.1145/1180639.1180758
DO - 10.1145/1180639.1180758
M3 - Conference contribution
AN - SCOPUS:34547166901
SN - 1595934472
SN - 9781595934475
T3 - Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006
SP - 569
EP - 578
BT - Proceedings of the 14th Annual ACM International Conference on Multimedia, MM 2006
T2 - 14th Annual ACM International Conference on Multimedia, MM 2006
Y2 - 23 October 2006 through 27 October 2006
ER -