Not all knowledge is created equal: mutual distillation of confident knowledge

Ziyun Li, Xinshao Wang*, Christoph Meinel, Neil M. Robertson, David A. Clifton, Haojin Yang

*Corresponding author for this work

Research output: Contribution to conferencePaperpeer-review

Abstract

Mutual knowledge distillation (MKD) improves a model by distilling knowledge from another model. However, not all knowledge is certain and correct, especially under adverse conditions. For example, label noise usually leads to less reliable models due to undesired memorization. Wrong knowledge harms the learning rather than helps it. This problem can be handled by two aspects: (i) knowledge source, improving the reliability of each model (knowledge producer) improving the knowledge source’s reliability; (ii) selecting reliable knowledge for distillation. Making a model more reliable is widely studied while selective MKD receives little attention. Therefore, we focus on studying selective MKD and highlight its importance in this work. Concretely, a generic MKD framework, Confident knowledge selection followed by Mutual Distillation (CMD), is designed. The key component of CMD is a generic knowledge selection formulation, making the selection threshold either static (CMD-S) or progressive (CMD-P). Additionally, CMD covers two special cases: zero knowledge and all knowledge, leading to a unified MKD framework. Extensive experiments are present to demonstrate the effectiveness of CMD and thoroughly justify the design of CMD.

Original languageEnglish
Publication statusAccepted - 27 Oct 2022
EventNeurIPS 2022 Workshop on Trustworthy and Socially Responsible Machine Learning - virtual, online
Duration: 09 Dec 202209 Dec 2022
https://tsrml2022.github.io/

Workshop

WorkshopNeurIPS 2022 Workshop on Trustworthy and Socially Responsible Machine Learning
Abbreviated titleNeurIPS 2022 - TSRML
Cityvirtual, online
Period09/12/202209/12/2022
Internet address

Fingerprint

Dive into the research topics of 'Not all knowledge is created equal: mutual distillation of confident knowledge'. Together they form a unique fingerprint.

Cite this