1
Review of N-grams approach for Language
Detection, Correction and Distance Calculation
16010420127 Aditya Phatak, 16010420075 Guneet Sura,
16010420092 Malay Thakkar, 16010420146 Tarash Budhrani
Abstract—This paper is a review research paper comprising of
an analysis of four different applications of N-gram distance and
N-gram model for detecting, correcting and finding relationships
between words of different language models. One of the papers
aims at finding similarities in 41 European languages using N-
gram distance using Common N-Grams model. Its main use case
involves authorship attribution tasks which works fairly well with
the proposed model. Rest of the papers shift their approach to
provide new concepts based on N-gram distance. This helps us
understand the real-world use cases of this concept and how
it was further analyzed to provide a forthcoming between the
outputs of synthesized language correction models. The extended
applications not only gave us a new approach, but provided a
gateway for new technologies to build their foundations and
expand itself in the future. Since, n-gram distance does not
provided complete transposition operations, the concept of E-
N-DIST was introduced for providing a clearer search result
in one of the papers. In another paper, it was suggested that
an another method a two-phase index-less search procedure for
application in misuse detection-based IDS that makes use of
q-gram distance since the complexity is significantly high as
compared to constraint edit distance. In the final paper, the study
suggests a technique for speech recognition that uses a syllable
lattice and n-gram array indices to identify words that are not
commonly used. Compared to other methods, the methodology
has high accuracy and efficiency.
I. I NTRODUCTION
T HE first research paper suggests the approach for cal-
culating the distance between two languages using n-
grams, which are contiguous sequences of n letters or words.
The authors contend that conventional methods for comparing
languages, such as lexical and phonetic distance, are not
necessarily reliable or useful. The common n-grams approach,
which counts the amount of shared n-grams between two
languages and uses this number as a gauge of how similar
they are, is what they recommend using as an alternative. The
tests in this study demonstrate that, in terms of accuracy and
computational economy, the common n-grams approach per-
forms better than the more conventional metrics of language
distance.
The second paper begins with noting down the common
use cases of each type of name matching in different fields. It
addresses name-errors that can occur when we spell things and
how they can be fixed using different techniques of distance
and similarity. Each word can be classified based on its
distance and similarity to the error word that was initially used.
The distance between both the words can be used for either
replacing, transposing or adding a letter for correction. The
authors state Edit Distance, Damerau-Levenshtein Distance
(DLD), Modified DLD (MDLD) and finally N-Dist that will be
used in the paper. The N-Dist method is simply an improved
method over all the other methods, that is, it provides an
overall increase in accuracy.
The third paper uses the concept of IDS to keep searching
for attack signatures i.e. arrangement of information that can
be used to identify an attacker’s attempt to exploit a known
operating system or application vulnerability. For executing
such a test we need distance measure to determine the dif-
ference between attack signature and traffic packet. In this
paper we use q gram distance to detect the difference between
attack signature and packet. The results are then arranged in
increasing order of q gram distance and finer inspection is
done on that packet. This paper tells that efficient algorithms
for exact string search were found several years ago but the
problem with this algorithm were they are not fault tolerant.
But the q gram distance continues operating uninterrupted
despite the failure of one or more of its components. For faster
response to query it uses indexing but the drawback is it is
memory and time consuming.
The fourth paper says that textual search engines can be
used to find information on the web if the subject data has
textual information such as transcribed broadcasts and news.
Spoken Term Detection (STD) methods use Large Vocabu-
lary Continuous Speech Recognizer (LVCSR) transcripts to
perform textual search, but Out of Vocabulary (OOV) terms
are not recognized as the LVCSR doesn’t have the given
word as an output, for various languages, the Levenshtein
distance between two syllables, syllable unit as basic units
of recognition and elastic matching between two syllable
sequences have been tried to reduce recognition errors for
OOV terms and ignore grammatical constraints, all of which
are better suited to specific languages based on their syllable
properties. Phenome based n-grams have also been suggested,
Dynamic Time Warping (DTW) has also been used but is more
time consuming than index-based processing. An n-gram array
has been used in the paper with distance measures which has a
syllable recognition lattice to reduce recognition errors, along
with a pruning method for n-gram array indices based on the
probability of recognition.
II. L ITERATURE S URVEY
The first paper begins with the concept of Automatic
Language Identification (LID) that refers to the process of
identifying the language of a spoken utterance or text au-
tomatically. It is an active area of research. It involves the

process of applying a probability distribution over a sequence
of tokens. Although the language can be identified, we need
to understand how different the language is from a particular
language. That is, the degree of difference between two or
more languages. We can tell if the two languages are similar
or different discreetly, either yes or no. But the degree of
difference gives us the relation between those languages.
This paper focuses on extending the work done on language
similarity analysis using the Common N-Grams(CNG) by
taking a different approach.
In the earlier versions of language similarity models, Mul-
tiple Discriminant Analysis was used to test the results where
some character level features like the presence of diphthongs
or phrases which were exclusive to some language were
used. The first usage of unigram and bigram frequencies was
implemented while exploring the LID. It was an ensemble of
seven using majority voting. The classifier was constructed
on Kolmogor-Smirnov and Yule K test. The accuracy was as
high as 89 percent but it only classified English and Spanish
and did not give degree of difference between the two. Later,
a character trigram Bayesian Model was used for language
identification. The best known work by far on LID is by
Cavnar et al where they used an out of order similarity metric
on th n-gram ranks. Their model approached an accuracy of
99.8 percent. This implementation later served as a base for
TextCat, which supports 69 languages.
Language Analysis by previous researchers have analysed
some basic rules for evaluating languages. In any large corpus
of data, frequency of the word is inversely proportional to
its ranking in the frequency table. Some researchers had built
15 linguistic complex networks based on their corresponding
syntactic treebanks. The results were astonishing.English came
out as more flexible and powerful in expression, Spanish and
French turned out to be constrained by rules and Chinese
showed that although it has less characters, it has got more
meaning than any other language.
Asgari et al. build word co-occurrence networks for fifty
languages. The edges between nodes are weighted using cosine
similarity between word embeddings. In addition to that, they
perform word alignment between two graphs, which means
that the words from different language networks are aligned
by semantic similarity.
Generating language distance based on perplexity measure
was much computationally less expensive compared to the
previous approaches.Perplexity is an evaluation metric for
language models used to measure fitness of test data built with
n-grams which was, in this case, adapted to measure distance
between languages.
The study in second paper suggests an enhanced name
matching technique based on N-gram distance. In several
areas, including record linking, data cleaning, and information
retrieval, name matching is a crucial task. An effective way
to compare two strings based on their shared N-gram-length
subsequences is to utilise the N-gram distance. Nonetheless,
multiple research have pointed out the shortcomings of the
conventional N-gram distance algorithm and suggested various
methods to enhance its functionality.
The classic N-gram distance algorithm’s sensitivity to string
2
length is one of its drawbacks. This problem has been solved in
a number of experiments by standardising the input strings. For
instance, a string normalisation algorithm based on character
frequency and positioning information was proposed by Wang
et al. in 2007. The method gives less frequently occurring
characters and those that are in the middle of the string more
weights.
The classic N-gram distance algorithm’s inability to handle
synonyms and homonyms is another drawback. The use of
outside resources like dictionaries and thesauri has been sug-
gested in several research as a way to address this problem.
For instance, Han et al. (2017) introduced a name matching
technique that maps ambiguous terms to the appropriate rep-
resentation using a predetermined synonym/homonym dictio-
nary.
Another crucial difficulty with N-gram distance algorithms
is weighting. For the informative and uncommon N-gram sub-
sequences, greater weights have been suggested by a number
of studies. For instance, Luan et al. (2017) developed a
weighting scheme that gives N-grams that appear at the start
or end of the string larger weights and those that appear in
the middle of the string lower weights.
In order to provide precise and reliable matching for many
sorts of names and variations, the work by Al-Hagree proposes
an enhanced N-gram distance method that includes normali-
sation, synonym/homonym management, and weighting com-
ponents. The authors test their algorithm against numerous
cutting-edge techniques, including Jaro-Winkler distance, Lev-
enshtein distance, and Smith-Waterman algorithm, using two
benchmark datasets of Arabic and English names. The findings
demonstrate the proposed algorithm’s superior performance in
both datasets, where it achieves an F-measure of up to 0.98.
In order to increase the effectiveness of misuse detection
using the q-gram distance measure, a novel strategy is put
forward in the study of third paper.
Intrusion detection systems (IDSs) and their difficulties are
introduced in the beginning of the paper. IDSs have a high
false-positive rate, which the authors note is one of their
key problems. This problem can be solved by increasing the
effectiveness of the detection process. The q-gram distance
metric, a gauge of the similarity between two strings that takes
into account the frequency of substrings of length q in each
string, is then introduced.
The authors next go over their suggested strategy for boost-
ing misuse detection’s effectiveness when employing the q-
gram distance metric. The method entails employing a q-gram
index to pre-process the input data in order to effectively locate
substrings that match recognised attack patterns. The authors
also suggest a variety of modifications to boost the strategy’s
effectiveness even more.
The proposed approach is assessed in the research using
a network traffic dataset that is available to the public. The
outcomes demonstrate that the method maintains a low false-
positive rate while achieving a high detection rate. Addi-
tionally, the authors compare their strategy to other cutting-
edge strategies and demonstrate that it beats them in terms of
efficiency and detection rate.

The problem of out-of-vocabulary term detection in the
fourth paper, which refers to the challenge of identifying terms
that are not present in a language model or dictionary, is
described in the paper’s introduction. According to the authors,
this issue is particularly severe in languages with intricate
morphology, such as Japanese, which has several potential
morpheme combinations.
The authors suggest a technique that uses a syllable lattice
with n-gram array indices to solve this issue. An effective
data structure for storing and retrieving n-gram frequencies is
the n-gram array indices. The conceivable syllabic sequences
in a language are represented by the syllable lattice, a graph
structure.
III. C OMPARISON AND M ETHODOLOGY U SED
The first paper compares two approaches, using the Perplex-
ity Method and the CNG method. CNG stands for Common
N-Grams Language Distance Measure. It is a text classification
algorithm which compares the frequencies of character n-
grams, that is the strings of characters of length n that are
most common in the considered documents. To get a higher
level of understanding, it can be interpreted as a k nearest
neighbor with the k value as 1, where instead of the standard
Euclidean distance, the CNG distance is used. For each defined
class and a new unlabeled document, the algorithm builds
a class profile, which consists of frequencies of the most
common character n-grams with the length of n. The n-gram
frequencies are normalized: n-gram counts divided by the
total number of n-grams. They used this CNG algorithm on
the train subsets, that they prepared language profiles for, by
using character n-gram range 3–7 and word uni-grams. For the
profile length hyperparameter, for all experiments they used
the maximum length of the smallest profile in the dataset.
They evaluated the test subsets by preparing test profiles and
applied pairwise CNG between each train-test language pair.
The result of this is asymmetrical distance matrix among train
and test language profiles. Then, using the evaluation method
described by Gamallo et al. They obtained accuracy in the
reference to gold standard. To evaluate the significance and
stability of the results, they employ McNemar’s statistic and
Spearman correlation coefficient. These parameters are also
calculated for the Perplexity Method. On comparing the two,
CNG tends to perform way better than the old, time consuming
and complex Perplexity Method.
In the second paper the author clearly states that the methods
used are varied as orthographic distance and orthographic
similarity for the name matching approach. By comparing
various methods and summing their equations, the author
has provided a clear contrast of how the N-DIST approach
has been the most efficient one. This approach has been
used to provide name matching in English as well as the
Arabic language (A-N-DIST). The purpose of this paper is
to improve the N-DIST algorithm using Latin based language
by considering multiple transpositions. This will be called the
E-N-DIST Algorithm (Enhanced N-DIST).
This algorithm proposed by the author promises improve-
ment over the original N-DIST algorithm. The first one being
3
that the E-N-DIST algorithm can provide computations for a
greater number of states as compared to the original algorithm.
The second being that the estimated cost of performing op-
erations (transposition, substitution, insertion and deletion) on
the word for name matching will depend on the number of
states of the word, rather than diving towards a fixed cost as
provided by the N-DIST algorithm. This minimizes the error
in correction and provides highly reliable results.
The author puts the equations used for the E-N-DIST algo-
rithm into algorithms to compute the cost of each operation
that takes place for name matching. This is done by comparing
weights for each N that depends on the word’s error. The
results vary from the N-DIST algorithm and it provides a
wider range of solutions. The pseudo code for insertion and
deletion is much simpler as compared to the pseudo code of
transposition and substitution due to the complexity of the
titles.
Moreover, there were experiments conducted on the thesis.
It begins by the preparation of datasets which explains the
names of multi languages that were put into test the algorithms
highlighted in the paper. In total, 11 datasets of different
languages were added and the performance of the E-N-
DIST algorithm was examined, each with spelling errors and
corrections. Overall, the E-N-DIST algorithm provided the
best results (as highlighted by the authors in a given table).
Using two benchmark datasets of Arabic and English names,
the authors assess the suggested algorithm and compare it
to many cutting-edge techniques, including Jaro-Winkler dis-
tance, Levenshtein distance, and Smith-Waterman algorithm.
They quantify the algorithms’ effectiveness using F-measure,
recall, and precision. The findings demonstrate the proposed
algorithm’s superior performance in both datasets, where it
achieves an F-measure of up to 0.98. The authors also do a
sensitivity analysis to examine the impact of several factors
on the algorithm’s performance, including N-gram size, a
synonym/homonym dictionary, and a weighting method. For
further analysis, the author provided the formula for calculat-
ing the correctness using the F-measure, precision and recall
techniques, and with no surprise, the E-N-DIST outperformed
the other algorithms.
The third paper discusses that there are two phases in the
algorithm. First phase is detecting differences between packet
and attack signatures.Then q gram distance are arranged in
increasing order.Second phase to deep dive search into packets.
In this algorithm we focused on first phase. Steps of algorithm
used in this paper are:
• Pre-processing: The raw network traffic data is prepro-
cessed to extract features, which are subsequences of
length q. These features are represented as a set of q-
grams, where a q-gram is defined as a sub-string of length
q.
• Computing the q-gram distance: The q-gram distance is
computed between each pair of network traffic samples
in the dataset. The q-gram distance between two samples
is defined as the number of unique q-grams that are
present in both samples. This distance measure captures
the similarity between the two samples based on the q-
grams they share.

• Feature selection: The q-gram distance values are used
to select a subset of the most discriminative q-grams for
use as features in the classification model. The authors
propose using a threshold value to determine which q-
grams are most discriminative. The threshold value is
set based on the desired number of features and the
distribution of the q-gram distances in the dataset.
• Classification: The selected q-grams are used as features
in a classification model to distinguish between normal
and anomalous network traffic. The authors evaluate
the performance of several classification algorithms, in-
cluding Naive Bayes, decision trees, and support vector
machines (SVM).
Then an experiment was conducted.For every string con-
strained edit distance and the q-gram distance were found in
advance. Threshold distance was known prior,for each string
if the constrained edit distance/q-gram distance between the
record and the search string was calculated. If this condition
was true then record/string will move in second phase.The
approach is based on a well-established metric (q-gram dis-
tance) and is therefore easy to implement.The experimental
evaluation is extensive and uses both synthetic and real-world
datasets. The results show that the approach outperforms other
state-of-the-art approaches in terms of efficiency and accuracy.
In the fourth paper, to find candidate positions for a known
sub-word sequence, we used syllable as a sub-word unit, the
recurrence equation of a DTW between an input sub-word
sequence (a1 a2 a3 a4. . . ) and a query sub-word sequence
(b1 b2 b3 b4. . . ) is required. The position where the query
appears is the result and the neighborhood of the spotting
position is taken as an input configuration. Distance with
the query is measured by DTW. Distance between the query
and candidate is measured by edit distance or Bhattacharya
distance. A Spoken Document Retrieval (SDR) for OOV words
with a syllable lattice with mis-recognized syllables:- The
spoken document is recognized by an LVCSR with a syllable
recognition system for OOV words, then indexing is applied
to the lattice which contains plural candidates at every best
candidate. OOV terms are searched using the top m-best in
the syllable lattice. Syllables are first assigned their appearance
position in a spoken document. The syllable is then built as an
n-gram in each appearance place. The n-gram is then sorted
in lexical order in order to swiftly search it with a binary
search method. Three steps make up the search procedure on
an n-gram array. A query is first transformed into a series of
syllables. Then, the query is turned into an n-gram. The final
step is to retrieve the query from the n-gram array. A mixture
of n-grams is used to obtain a query that has more than n+1
syllables. A query is divided into two n-grams in the first half
and the second half if it has less than 2n syllables and more
than n+1 syllables. As a result, the query is twice taken from
the n-gram array. The positions where the detection results
occurred in the first half and the latter half are taken into
account when combining the findings to determine whether
they appeared within one position or not. A query with less
than 3n syllables and more than 2n+1 syllables can be found
by splitting it into three parts. For substitution errors: n-gram
4
array is made using the combination of syllables in the m-best
lattice, for one position in the lattice, there are mn kinds of n-
gram. For insertion error: make an n-gram array which permits
one distant n-gram, considering the gap between appearance
positions. For deletion error: search the query as above while
allowing for the case where one syllable in the query is deleted.
Pruning Method for array indices : to attack the problem of a
large index, tri-gram indices are pruned based on probability of
recognition results, difference in 1-best and m-best likelihood
is calculated for every position in the lattice, The indexer
compares the pre-set threshold with the difference and adds
the tri-gram to the index if it is smaller than the threshold.
Decision Maker : a new distance measure for the number of
errors allowed is defined, to allow fast pruning of unreliable
candidates, the syllable in the 1 best result is always zero
distance Next, the syllable distance for an insertion error is
represented by 1. Finally, the syllable distance for deletion
errors is equal to the number of syllables that were eliminated
in a query.
IV. C ONTRIBUTION
V. P ROPOSED W ORK
The authors in first paper claim that a number of appli-
cations, including language categorization and machine trans-
lation, may benefit from their method. This model has the
capability of finding and asserting the relevance of a conversa-
tion between two individuals speaking two different languages.
This is especially useful in training and giving guidance to
(human) translators, who can perform the translation, evaluate
their progress and improve.
In the second paper, the scope widens to practical usage of
day-to-day spell checking in devices like “Auto-correct” that
also uses this concept to bring out the most of user experience.
Overall, this paper has provided a better future outlook by
adding phonetic and other conditions for name-matching.
In the third paper, Q-gram distance has the potential to
increase the effectiveness of misuse detection. To further
improve the precision and effectiveness of intrusion detection
systems, future research can concentrate on investigating the
usage of additional distance measures and machine learning
approaches. Another interesting area of research is examining
the efficiency of incorporating domain-specific knowledge into
the detection process.
An important topic for future work in fourth paper is to
improve the retrieval precision. Using just low confidence
portions as OOV candidates from the LVCSR results is one
technique to increase the retrieval accuracy. Combining the
output of various decoders is another technique to raise the
pace of syllable recognition. Finally, to increase retrieval
accuracy, we can substitute the syllable probability received
from the decoder for the syllable distance.
VI. C ONCLUSION
The CNG algorithm is addressed in the first paper as
a suitable approach for author identification tasks, offering
an automated and effective technique to represent linguistic
similarity.
 5

The second study offers a novel notion of n-gram distance,

which has applications in Arabic and other languages, includ-
ing useful ones like name-matching and spell-checking.
In the third study, an unique way for increasing the effec-
tiveness and precision of misuse detection in IDS is presented.
This method makes use of the q-gram distance measure, which
has been shown to be more effective and precise than existing
approaches for datasets with a lot of features.
Last but not least, the fourth research suggests a pruning
method employing a trigram array index that outperformed
earlier approaches in terms of search time in retrieval tasks.
In conclusion, these four publications introduce cutting-edge
ideas and strategies to tackle a range of problems. These
are significant contributions to their respective domains in
information retrieval, network security, and other disciplines
of natural language processing. They serve as an example of
the ongoing work being done to create new algorithms and
methods that will enhance our capacity to understand natural
language, secure networks, and rapidly retrieve information.

R EFERENCES
[1] Dijana Kosmajac, Vlado Keselj, ”Language Distance using Common
N-Grams”, 19th International Symposium INFOTEH-JAHORINA, 18-20
March 2020
[2] Salah AL-Hagree,Maher Al-Sanabani,Mohammed Hadwan,Mohammed
A.Al-Hagery, ”An Improved N-gram Distance for Names Matching”, 2019
First International Conference of Intelligent Computing and Engineering
(ICOICE)
[3] Maher Al-Sanabani, Slobodan Petrovic, Sverre Bakke, ”Improving the
Efficiency of Misuse Detection by Means of the q-gram Distance”, The
Fourth International Conference on Information Assurance and Security
[4] Keisuke wami, Yasuhisa Fujii , Kazumasa Yamamoto, Seiichi Nakagawa,
”Efficient Out-of-Vocabulary Term Detection by N-Gram array Indices
With Distance from a syllable Lattice” .Department of Computer Science
and Engineering, Toyohashi University of Technology, Japan