ArXiv Preprint
Language modeling, a central task in natural language processing, involves
estimating a probability distribution over strings. In most cases, the
estimated distribution sums to 1 over all finite strings. However, in some
pathological cases, probability mass can ``leak'' onto the set of infinite
sequences. In order to characterize the notion of leakage more precisely, this
paper offers a measure-theoretic treatment of language modeling. We prove that
many popular language model families are in fact tight, meaning that they will
not leak in this sense. We also generalize characterizations of tightness
proposed in previous works.
Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, Ryan Cotterell
2022-12-20
