Statistically-constrained shallow text marking: Techniques, evaluation paradigm and results

B. Murphy, C. Vogel

Research output: Chapter in Book/Report/Conference proceedingChapter

10 Citations (Scopus)

Abstract

We present three natural language marking strategies based on fast and reliable shallow parsing techniques, and on widely available lexical resources: lexical substitution, adjective conjunction swaps, and relativiser switching. We test these techniques on a random sample of the British National Corpus. Individual candidate marks are checked for goodness of structural and semantic fit, using both lexical resources, and the web as a corpus. A representative sample of marks is given to 25 human judges to evaluate for acceptability and preservation of meaning. This establishes a correlation between corpus based felicity measures and perceived quality, and makes qualified predictions. Grammatical acceptability correlates with our automatic measure strongly (Pearson's r = 0.795, p = 0.001), allowing us to account for about two thirds of variability in human judgements. A moderate but statistically insignificant (Pearson's r = 0.422, p = 0.356) correlation is found with judgements of meaning preservation, indicating that the contextual window of five content words used for our automatic measure may need to be extended.
Original languageEnglish
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
Volume6505
Publication statusPublished - 01 Jan 2007

Fingerprint

Semantics
Language

Cite this

Murphy, B., & Vogel, C. (2007). Statistically-constrained shallow text marking: Techniques, evaluation paradigm and results. In Proceedings of SPIE - The International Society for Optical Engineering (Vol. 6505)
Murphy, B. ; Vogel, C. / Statistically-constrained shallow text marking : Techniques, evaluation paradigm and results. Proceedings of SPIE - The International Society for Optical Engineering. Vol. 6505 2007.
@inbook{b437344690634714a2353d4bbe663d91,
title = "Statistically-constrained shallow text marking: Techniques, evaluation paradigm and results",
abstract = "We present three natural language marking strategies based on fast and reliable shallow parsing techniques, and on widely available lexical resources: lexical substitution, adjective conjunction swaps, and relativiser switching. We test these techniques on a random sample of the British National Corpus. Individual candidate marks are checked for goodness of structural and semantic fit, using both lexical resources, and the web as a corpus. A representative sample of marks is given to 25 human judges to evaluate for acceptability and preservation of meaning. This establishes a correlation between corpus based felicity measures and perceived quality, and makes qualified predictions. Grammatical acceptability correlates with our automatic measure strongly (Pearson's r = 0.795, p = 0.001), allowing us to account for about two thirds of variability in human judgements. A moderate but statistically insignificant (Pearson's r = 0.422, p = 0.356) correlation is found with judgements of meaning preservation, indicating that the contextual window of five content words used for our automatic measure may need to be extended.",
author = "B. Murphy and C. Vogel",
year = "2007",
month = "1",
day = "1",
language = "English",
isbn = "9780819466181",
volume = "6505",
booktitle = "Proceedings of SPIE - The International Society for Optical Engineering",

}

Murphy, B & Vogel, C 2007, Statistically-constrained shallow text marking: Techniques, evaluation paradigm and results. in Proceedings of SPIE - The International Society for Optical Engineering. vol. 6505.

Statistically-constrained shallow text marking : Techniques, evaluation paradigm and results. / Murphy, B.; Vogel, C.

Proceedings of SPIE - The International Society for Optical Engineering. Vol. 6505 2007.

Research output: Chapter in Book/Report/Conference proceedingChapter

TY - CHAP

T1 - Statistically-constrained shallow text marking

T2 - Techniques, evaluation paradigm and results

AU - Murphy, B.

AU - Vogel, C.

PY - 2007/1/1

Y1 - 2007/1/1

N2 - We present three natural language marking strategies based on fast and reliable shallow parsing techniques, and on widely available lexical resources: lexical substitution, adjective conjunction swaps, and relativiser switching. We test these techniques on a random sample of the British National Corpus. Individual candidate marks are checked for goodness of structural and semantic fit, using both lexical resources, and the web as a corpus. A representative sample of marks is given to 25 human judges to evaluate for acceptability and preservation of meaning. This establishes a correlation between corpus based felicity measures and perceived quality, and makes qualified predictions. Grammatical acceptability correlates with our automatic measure strongly (Pearson's r = 0.795, p = 0.001), allowing us to account for about two thirds of variability in human judgements. A moderate but statistically insignificant (Pearson's r = 0.422, p = 0.356) correlation is found with judgements of meaning preservation, indicating that the contextual window of five content words used for our automatic measure may need to be extended.

AB - We present three natural language marking strategies based on fast and reliable shallow parsing techniques, and on widely available lexical resources: lexical substitution, adjective conjunction swaps, and relativiser switching. We test these techniques on a random sample of the British National Corpus. Individual candidate marks are checked for goodness of structural and semantic fit, using both lexical resources, and the web as a corpus. A representative sample of marks is given to 25 human judges to evaluate for acceptability and preservation of meaning. This establishes a correlation between corpus based felicity measures and perceived quality, and makes qualified predictions. Grammatical acceptability correlates with our automatic measure strongly (Pearson's r = 0.795, p = 0.001), allowing us to account for about two thirds of variability in human judgements. A moderate but statistically insignificant (Pearson's r = 0.422, p = 0.356) correlation is found with judgements of meaning preservation, indicating that the contextual window of five content words used for our automatic measure may need to be extended.

UR - http://www.scopus.com/inward/record.url?eid=2-s2.0-34548217225&partnerID=8YFLogxK

M3 - Chapter

AN - SCOPUS:34548217225

SN - 9780819466181

VL - 6505

BT - Proceedings of SPIE - The International Society for Optical Engineering

ER -

Murphy B, Vogel C. Statistically-constrained shallow text marking: Techniques, evaluation paradigm and results. In Proceedings of SPIE - The International Society for Optical Engineering. Vol. 6505. 2007