Assignment # 1

Back-off and Interpolation in Language Modeling

1. Back-off in Language Modeling:

Back-off is a technique used in language modeling to estimate the probability of a sequence of

words by recursively estimating probabilities of shorter sequences when the full sequence's
probability is not available or reliable. This is commonly employed in N-gram models.

In N-gram models, the probability of a word sequence is often estimated using the chain rule of
probability:

P(w1,w2,...,wn)=P(w1) × P(w2∣w1) × ... × P(wn∣wn−1,...,w1)

If the probability of a longer sequence is not available, the back-off technique uses a shorter
sequence's probability as an estimate. For example, if P(wn∣wn−1,...,w1) is not reliable, the
model might back off to P(wn∣wn−1) and so on.

2. Interpolation in Language Modeling:

Interpolation is another technique used in language modeling to combine probabilities from

different sources or models. It involves assigning a weighted sum of probabilities based on
multiple language models. The weights indicate the importance of each model in contributing to
the overall probability.

In the context of N-gram models, interpolation might be represented as follows:

P(wn∣wn−1,...,w1)=λ1×P1(wn∣wn−1,...,w1)+λ2×P2(wn∣wn−1,...,w1)+...+λk×Pk(wn
∣wn−1,...,w1)
Here, Pi(wn∣wn−1,...,w1) represents the probability from the i-th language model, and λi is
the weight assigned to the i-th model. The weights are typically determined through training on
a held-out dataset.