Statistical language modeling based on automatic classification of words
Sepesy Maučec, Mirjam
In statistical language modeling the model's parameters are extracted from large amounts of text. This kind of models can be built for any language without requireing any linguistic knowledge. Bigram ...and trigram language models will be discussed. In statistical modeling there is always a problem of sparse data. We will compare two proposed solutions: smoothing method proposed by Katz and automatic word clustering proposed by Ney. In the first case, some probability mass is redistributed over bigrams (trigrams) which never occured in the text. In the second case, the words are mapped into classes in such a way that the perplexity of the model is minimized. By comparing word based models and class based models we see that the use of clustered words leads to a significant improvement, as measured by the perplexity.
It was not necessary to add the material to the shelf.
Bibliographic material was successfully added on shelf.
Adding bibliographical material on shelf was not successful.
Material is already on the shelf.
Permalink
URL:
Impact factor
Access to the JCR database is permitted only to users from Slovenia. Your current IP address is not on the list of IP addresses with access permission, and authentication with the relevant AAI accout is required.
Year
Impact factor
Edition
Category
Classification
JCR
SNIP
JCR
SNIP
JCR
SNIP
JCR
SNIP
Select the library membership card:
DRS,
in which the journal is indexed
Database name
Field
Year
Links to authors' personal bibliographies
Links to information on researchers in the SICRIS system
Subject headings in COBISS General List of Subject Headings
Select pickup location
The material from the parent unit is free. If the material is delivered to the pickup location from another unit, the library may charge you for this service.
