Knowledge transfer in neural language models

Peter John Hampton; Hui Wang; Zhiwei Lin

doi:10.1007/978-3-319-71078-5_12

Knowledge transfer in neural language models

Peter John Hampton^*, Hui Wang, Zhiwei Lin

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

The complexity and depth of Information Extraction becomes increasingly apparent as time goes on. Heuristics, shocastic and more recently, neural models have proved challenging to scale into and out of various domains. In this paper we discuss the limitations of current approaches and explore if transferring human knowledge into a neural language model could improve performance in an deep learning setting. We approach this by constructing gazetteers from existing public resources. We demonstrate that leveraging existing knowledge we can increase performance and train such networks faster. We argue a case for further research into leveraging pre-existing domain knowledge and engineering resources to train neural models.

Original language	English
Title of host publication	Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings
Editors	Miltos Petridis, Max Bramer
Place of Publication	Switzerland
Publisher	Springer Verlag
Pages	143-148
Number of pages	6
ISBN (Print)	9783319710778
DOIs	https://doi.org/10.1007/978-3-319-71078-5_12
Publication status	Published - Dec 2017
Externally published	Yes
Event	37th SGAI International Conference on Artificial Intelligence, AI 2017 - Cambridge, United Kingdom Duration: 12 Dec 2017 → 14 Dec 2017

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10630 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	37th SGAI International Conference on Artificial Intelligence, AI 2017
Country/Territory	United Kingdom
City	Cambridge
Period	12/12/2017 → 14/12/2017

Bibliographical note

Funding Information:
1 Implementation: https://github.com/zhiweiuu/SGAITagger. 2 This work is partially supported by the EPSRC (Grant REF: EP/P031668/1).

Publisher Copyright:
© Springer International Publishing AG 2017.

Copyright:
Copyright 2017 Elsevier B.V., All rights reserved.

Keywords

Information extraction
Named entity recognition

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-319-71078-5_12Licence: Unspecified

Cite this

Hampton, P. J., Wang, H., & Lin, Z. (2017). Knowledge transfer in neural language models. In M. Petridis, & M. Bramer (Eds.), Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings (pp. 143-148). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10630 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-319-71078-5_12

Hampton, Peter John ; Wang, Hui ; Lin, Zhiwei. / Knowledge transfer in neural language models. Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings. editor / Miltos Petridis ; Max Bramer. Switzerland : Springer Verlag, 2017. pp. 143-148 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{7f473aee5a4c4e41aab60249fe810820,

title = "Knowledge transfer in neural language models",

abstract = "The complexity and depth of Information Extraction becomes increasingly apparent as time goes on. Heuristics, shocastic and more recently, neural models have proved challenging to scale into and out of various domains. In this paper we discuss the limitations of current approaches and explore if transferring human knowledge into a neural language model could improve performance in an deep learning setting. We approach this by constructing gazetteers from existing public resources. We demonstrate that leveraging existing knowledge we can increase performance and train such networks faster. We argue a case for further research into leveraging pre-existing domain knowledge and engineering resources to train neural models.",

keywords = "Information extraction, Named entity recognition",

author = "Hampton, {Peter John} and Hui Wang and Zhiwei Lin",

note = "Funding Information: 1 Implementation: https://github.com/zhiweiuu/SGAITagger. 2 This work is partially supported by the EPSRC (Grant REF: EP/P031668/1). Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017. Copyright: Copyright 2017 Elsevier B.V., All rights reserved.; 37th SGAI International Conference on Artificial Intelligence, AI 2017 ; Conference date: 12-12-2017 Through 14-12-2017",

year = "2017",

month = dec,

doi = "10.1007/978-3-319-71078-5_12",

language = "English",

isbn = "9783319710778",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "143--148",

editor = "Miltos Petridis and Max Bramer",

booktitle = "Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings",

address = "Germany",

}

Hampton, PJ, Wang, H & Lin, Z 2017, Knowledge transfer in neural language models. in M Petridis & M Bramer (eds), Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10630 LNAI, Springer Verlag, Switzerland, pp. 143-148, 37th SGAI International Conference on Artificial Intelligence, AI 2017, Cambridge, United Kingdom, 12/12/2017. https://doi.org/10.1007/978-3-319-71078-5_12

Knowledge transfer in neural language models. / Hampton, Peter John; Wang, Hui; Lin, Zhiwei.
Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings. ed. / Miltos Petridis; Max Bramer. Switzerland: Springer Verlag, 2017. p. 143-148 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10630 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Knowledge transfer in neural language models

AU - Hampton, Peter John

AU - Wang, Hui

AU - Lin, Zhiwei

N1 - Funding Information: 1 Implementation: https://github.com/zhiweiuu/SGAITagger. 2 This work is partially supported by the EPSRC (Grant REF: EP/P031668/1). Publisher Copyright: © Springer International Publishing AG 2017. Copyright: Copyright 2017 Elsevier B.V., All rights reserved.

PY - 2017/12

Y1 - 2017/12

N2 - The complexity and depth of Information Extraction becomes increasingly apparent as time goes on. Heuristics, shocastic and more recently, neural models have proved challenging to scale into and out of various domains. In this paper we discuss the limitations of current approaches and explore if transferring human knowledge into a neural language model could improve performance in an deep learning setting. We approach this by constructing gazetteers from existing public resources. We demonstrate that leveraging existing knowledge we can increase performance and train such networks faster. We argue a case for further research into leveraging pre-existing domain knowledge and engineering resources to train neural models.

AB - The complexity and depth of Information Extraction becomes increasingly apparent as time goes on. Heuristics, shocastic and more recently, neural models have proved challenging to scale into and out of various domains. In this paper we discuss the limitations of current approaches and explore if transferring human knowledge into a neural language model could improve performance in an deep learning setting. We approach this by constructing gazetteers from existing public resources. We demonstrate that leveraging existing knowledge we can increase performance and train such networks faster. We argue a case for further research into leveraging pre-existing domain knowledge and engineering resources to train neural models.

KW - Information extraction

KW - Named entity recognition

U2 - 10.1007/978-3-319-71078-5_12

DO - 10.1007/978-3-319-71078-5_12

M3 - Conference contribution

AN - SCOPUS:85037722549

SN - 9783319710778

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 143

EP - 148

BT - Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings

A2 - Petridis, Miltos

A2 - Bramer, Max

PB - Springer Verlag

CY - Switzerland

T2 - 37th SGAI International Conference on Artificial Intelligence, AI 2017

Y2 - 12 December 2017 through 14 December 2017

ER -

Hampton PJ, Wang H, Lin Z. Knowledge transfer in neural language models. In Petridis M, Bramer M, editors, Artificial Intelligence XXXIV - 37th SGAI International Conference on Artificial Intelligence, AI 2017, Proceedings. Switzerland: Springer Verlag. 2017. p. 143-148. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-71078-5_12

Knowledge transfer in neural language models

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this