Text corpus

Article
Talk
Read
Edit
View history

Tools
From Wikipedia, the free encyclopedia
(Redirected from Corpus of text)
In linguistics and natural language processing, a corpus (pl.: corpora) or text
corpus is a dataset, consisting of natively digital and older, digitalized,
language resources, either annotated or unannotated.

Annotated, they have been used in corpus linguistics for statistical hypothesis
testing, checking occurrences or validating linguistic rules within a specific
language territory.

In search technology, a corpus is the collection of documents which is being

searched.

Overview
A corpus may contain texts in a single language (monolingual corpus) or text data
in multiple languages (multilingual corpus).

In order to make the corpora more useful for doing linguistic research, they are
often subjected to a process known as annotation. An example of annotating a corpus
is part-of-speech tagging, or POS-tagging, in which information about each word's
part of speech (verb, noun, adjective, etc.) is added to the corpus in the form of
tags. Another example is indicating the lemma (base) form of each word. When the
language of the corpus is not a working language of the researchers who use it,
interlinear glossing is used to make the annotation bilingual.

Some corpora have further structured levels of analysis applied. In particular,

smaller corpora may be fully parsed. Such corpora are usually called Treebanks or
Parsed Corpora. The difficulty of ensuring that the entire corpus is completely and
consistently annotated means that these corpora are usually smaller, containing
around one to three million words. Other levels of linguistic structured analysis
are possible, including annotations for morphology, semantics and pragmatics.

Applications
Corpora are the main knowledge base in corpus linguistics. Other notable areas of
application include:

Language technology, natural language processing, computational linguistics

The analysis and processing of various types of corpora are also the subject of
much work in computational linguistics, speech recognition and machine translation,
where they are often used to create hidden Markov models for part of speech tagging
and other purposes. Corpora and frequency lists derived from them are useful for
language teaching. Corpora can be considered as a type of foreign language writing
aid as the contextualised grammatical knowledge acquired by non-native language
users through exposure to authentic texts in corpora allows learners to grasp the
manner of sentence formation in the target language, enabling effective writing.[1]
Machine translation
Multilingual corpora that have been specially formatted for side-by-side comparison
are called aligned parallel corpora. There are two main types of parallel corpora
which contain texts in two languages. In a translation corpus, the texts in one
language are translations of texts in the other language. In a comparable corpus,
the texts are of the same kind and cover the same content, but they are not
translations of each other.[2] To exploit a parallel text, some kind of text
alignment identifying equivalent text segments (phrases or sentences) is a
prerequisite for analysis. Machine translation algorithms for translating between
two languages are often trained using parallel fragments comprising a first-
language corpus and a second-language corpus, which is an element-for-element
translation of the first-language corpus.[3]
Philologies
Text corpora are also used in the study of historical documents, for example in
attempts to decipher ancient scripts, or in Biblical scholarship. Some
archaeological corpora can be of such short duration that they provide a snapshot
in time. One of the shortest corpora in time may be the 15–30 year Amarna letters
texts (1350 BC). The corpus of an ancient city, (for example the "Kültepe Texts" of
Turkey), may go through a series of corpora, determined by their find site dates.
Some notable text corpora
Main article: List of text corpora
See also
Concordance
Corpus linguistics
Distributional–relational database
Linguistic Data Consortium
Natural language processing
Natural Language Toolkit
Parallel text alignment
Search engines: they access the "web corpus".
Speech corpus
Translation memory
Treebank
Zipf's Law
References
Yoon, H., & Hirvela, A. (2004). ESL Student Attitudes toward Corpus Use in L2
Writing. Journal of Second Language Writing, 13(4), 257–283. Retrieved 21 March
2012.
Wołk, K.; Marasek, K. (7 April 2014). "Real-Time Statistical Speech Translation".
New Perspectives in Information Systems and Technologies, Volume 1. Advances in
Intelligent Systems and Computing. Vol. 275. Springer. pp. 107–114.
arXiv:1509.09090. doi:10.1007/978-3-319-05951-8_11. ISBN 978-3-319-05950-1. ISSN
2194-5357. S2CID 15361632.
Wolk, Krzysztof; Marasek, Krzysztof (2015). "Tuned and GPU-accelerated parallel
data mining from comparable corpora". In Král, Pavel; Matousek, Václav (eds.).
Text, Speech, and Dialogue – 18th International Conference, TSD 2015, Pilsen, Czech
Republic, September 14–17, 2015, Proceedings. Lecture Notes in Computer Science.
Vol. 9302. Springer. pp. 32–40. arXiv:1509.08639. doi:10.1007/978-3-319-24033-6_4.
External links
ACL SIGLEX Resource Links: Text Corpora Archived 2013-08-13 at the Wayback Machine
Developing Linguistic Corpora: a Guide to Good Practice
Free samples (not free), web-based corpora (45-425 million words each): American
(COCA, COHA, TIME), British (BNC), Spanish, Portuguese
Intercorp Building synchronous parallel corpora of the languages taught at the
Faculty of Arts of Charles University.
Sketch Engine: Open corpora with free access
TS Corpus – A Turkish Corpus freely available for academic research.
Turkish National Corpus – A general-purpose corpus for contemporary Turkish
Corpus of Political Speeches, Free access to political speeches by American and
Chinese politicians, developed by Hong Kong Baptist University Library
Russian National Corpus

vte
Natural language processing
General terms
AI-completeBag-of-wordsn-gram BigramTrigramComputational linguisticsNatural-
language understandingStop wordsText processing
Text analysis
Argument miningCollocation extractionConcept miningCoreference resolutionDeep
linguistic processingDistant readingInformation extractionNamed-entity
recognitionOntology learningParsingPart-of-speech taggingSemantic analysisSemantic
role labelingSemantic decompositionSemantic similaritySentiment analysisSyntactic
parsingTerminology extractionText miningTextual entailmentTruecasingWord-sense
disambiguationWord-sense induction
Text segmentation
Compound-term processingLemmatisationLexical analysisText chunkingStemmingSentence
segmentationWord segmentation
Automatic summarization
Multi-document summarizationSentence extractionText simplification
Machine translation
Computer-assistedExample-basedRule-basedStatisticalTransfer-basedNeural
Distributional semantics models
BERTDocument-term matrixExplicit semantic analysisfastTextGloVeLanguage model
(large)Latent semantic analysisSeq2seqWord embeddingWord2vec
Language resources,
datasets and corpora
Types and
standards
Corpus linguisticsLexical resourceLinguistic Linked Open DataMachine-readable
dictionaryParallel textPropBankSemantic networkSimple Knowledge Organization
SystemSpeech corpusText corpusThesaurus (information retrieval)TreebankUniversal
Dependencies
Data
BabelNetBank of EnglishDBpediaFrameNetGoogle Ngram ViewerUBYWordNet
Automatic identification
and data capture
Speech recognitionSpeech segmentationSpeech synthesisNatural language
generationOptical character recognition
Topic model
Document classificationLatent Dirichlet allocationPachinko allocation
Computer-assisted
reviewing
Automated essay scoringConcordancerGrammar checkerPredictive textPronunciation
assessmentSpell checkerSyntax guessing
Natural language
user interface
ChatbotInteractive fictionQuestion answeringVirtual assistantVoice user interface
Related
HallucinationNatural Language ToolkitspaCy
Categories: Discourse analysisCorpus linguisticsComputational linguisticsWorks
based on multiple worksTest items