IETE Technical Review

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/titr20

Studying the Effect of Syntactic Simplification on

Text Summarization

Niladri Chatterjee & Raksha Agarwal

To cite this article: Niladri Chatterjee & Raksha Agarwal (2023) Studying the Effect of
Syntactic Simplification on Text Summarization, IETE Technical Review, 40:2, 155-166, DOI:
10.1080/02564602.2022.2055670

To link to this article: https://doi.org/10.1080/02564602.2022.2055670

Published online: 31 Mar 2022.

Submit your article to this journal

Article views: 227

View related articles

View Crossmark data

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=titr20
IETE TECHNICAL REVIEW
2023, VOL. 40, NO. 2, 155–166
https://doi.org/10.1080/02564602.2022.2055670

Studying the Effect of Syntactic Simplification on Text Summarization

Niladri Chatterjee and Raksha Agarwal
Department of Mathematics, Indian Institute of Technology Delhi, New Delhi 110 016, India

ABSTRACT KEYWORDS
The need for automatic text summarization (ATS) is increased manifold in recent times due to the Abstractive summary;
overwhelming growth of textual data available in electronic form. However, existing ATS systems Relative clause; ROUGE
suffer from two major shortcomings. Summarizers of extractive type, that is, the ones which select metric; Syntactic
important sentences of the documents in their original form as the output, tend to copy some irrel- simplification; Text
processing; Text
evant or unimportant parts of the input text in the output summary. On the other hand, abstractive simplification; Text
summarizers, that is, the ones that produce a gist of the limited size of the original document, often summarization
fail to include important contents in the generated summary. Simplification of the input texts before
submitting them to the ATS system(s) may obliterate the above difficulties. The present work exam-
ines the effectiveness of simplification of input for five different known ATS systems. In this work,
DEPSYM++ simplifier has been used for the above purpose, which carries out four different kinds of
simplification on sentences of the input text corresponding to the presence of appositive clause, rela-
tive clause, conjoint clause, and passive voice. The results obtained are found to be very encouraging
when experiments were carried out on three different gold data sets and under different evaluation
metrics commonly used for performance evaluation for summarizers.

1. INTRODUCTION
Automatic text summarization (ATS) aims at providing
a concise and crisp overview containing the important
information of a given text. ATS systems can be classified
as extractive and abstractive [1]. Extractive summarizers
rank the input sentences on the basis of their impor-
tance using different techniques, such as occurrence of
frequent words [2], graphical representation [3,4], matrix
factorization [5,6], fuzzy sentence similarity [7]. The top-
ranking sentences are then selected to constitute the sum-
mary. The major limitation of the above approach is that
it may result in summaries of low quality, particularly
when the input text contains long sentences having many
different concepts. Presenting them in toto in summary
leads to the inclusion of unimportant contents in sum-
mary. On the other hand, in abstractive summarization,
the input text is typically modified or rewritten to convey
the information more meaningfully and succinctly.
The aim of the current paper is to improve the quality of
summarization, both extractive and abstractive, by first
simplifying the input text at the preprocessing stage and
then applying the summarization technique to the sim-
plified text. More, specifically, here we try to investigate
the impact of prior simplification of the input text on
the quality of the summary generated through different
summarizers.
The above motivation emerges from the observation of
the following research gap with respect to abstractive
summarization. Abstractive summarization through sen-
tence compression is being practiced for more than two
decades now [8,9], their performance is often poor. In
compression-based abstractive summarization, parts of
an input sentence are deleted on the basis of its syntactic
structure and their statistical properties [10]. However,
the decision regarding which parts of the sentence are
to be deleted often ignores the information contained in
them [11]. As a consequence, the sentence compression
process may end up pruning parts of the sentence with
higher information content. As a remedy to this problem,
in the present work, syntactic simplification is performed
on the sentences of the given text before performing
summarization.
In syntactic simplification, the structure of a sentence is
modified by rearranging the words and/or splitting a sen-
tence into two or more shorter sentences [12]. The inher-
ent purpose is to reduce the structural complexity of a
given sentence while preserving its meaning. The present
work uses the syntactic simplification system DEPSYM
[13] for the above purpose. However, in order to use it
more effectively for summarization purposes, we made
some modifications to the original DEPSYM algorithm,
which is termed DEPSYM++. DEPSYM++ splits the

© 2023 IETE
156 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
Figure 1: Overview of the proposed approach
input sentence into multiple parts without deleting any
information. The simplified version is then served as
input to different extractive and abstractive summariza-
tion systems. Figure 1 presents an overview of the pro-
posed approach. The purpose is to compare the perfor-
mance of different summarization systems on simplified
and original texts. Different metrics have been used to
evaluate their performance.
The novelty of the proposed scheme lies not only in sug-
gesting syntactic simplification as a preprocessing step for
generating summaries of better quality but also in fine-
tuning the DEPSYM algorithm for modifying the raw
text suitably. Text simplification is a complicated NLP
task with various ramifications. Different types of simpli-
fication techniques have been used in literature for sum-
marization, including end-to-end dual objective train-
ing, as discussed in Section 2. However, despite being a
lightweight simplification system based only on depen-
dency parsing, experiments conducted for this research
show that the application of DEPSYM++ in the prepro-
cessing step provides a significant boost to the perfor-
mance of five different summarization systems as shown
in section 5.
The paper is organized as follows. Section 2 describes
related works on application of text simplification for text
summarization. Syntactic simplification algorithm used
in the present work is described in Section 3. Section 4
provides a description of different summarization algo-
rithms. Experimental details and data statistics are pre-
sented in Section 5. Results corresponding to different
evaluation metrics are presented in Section 6. The paper
is concluded in Section 7.
2. RELATED WORK
The utility of text simplification has been demonstrated
in various NLP tasks, such as machine translation [14,15],
question answering [16], semantic role labeling [17,18],
and text summarization [11,19]. The effect of simplifi-
cation has also been studied for the cognitive tasks of
reading comprehension [20] and listening comprehen-
sion [21]. In this section, a review of existing works
on application of text simplification for the task of text
summarization has been presented.
The use of simplification techniques as a preprocess-
ing step for text summarization is being practiced for
nearly two decades. Siddharthan et al. [11] used syn-
tactic simplification rules for improving content selec-
tion in multi-document summarization. Here, relative
clauses and appositive clauses of the input sentences
are deleted to improve sentence clustering. Summary is
generated by selecting at most one sentence from each
cluster.
Vanderwende et al. [22] used heuristic templates for the
elimination of syntactic units from the input sentences
based on parser output. Here, syntactic patterns, such
as noun appositives, gerundive clauses, nonrestrictive
relative clause, and lead adverbials, were deleted from
the input sentences. It was observed that although such
deletions perform well on non-redundancy and content
responsiveness, the grammatical quality and referential
clarity of the summary were compromised.
Silveira and Branco [19] used sentence simplification at
the summary generation phase to produce highly infor-
mative summaries. A double clustering approach based
on sentence similarity and keyword similarity is used for
content selection. In order to reduce redundancy, only
the highest scoring sentence from each cluster is retained
at each step. After content selection, the sentences are
simplified by removing dispensable phrases. These were
identified by isolating subtrees corresponding to five dif-
ferent syntactic structures from the constituency parse
tree of the sentences. However, as pointed out by the
authors themselves that such a simplification system, if
applied before content selection, may affect the efficiency
of double clustering by removing significant informative
parts of the sentence.
Finegan-Dollak et al. [23] performed sentence simplifica-
tion, compression, and disaggregation for summarization
 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
of texts from biomedical and legal domains. The sim-
plification system performs syntactic as well as lexical
simplifications. The disaggregation system splits an input
sentence into multiple sentences while maintaining the
relevant dependencies. The summaries for input docu-
ments with different levels of preprocessing were gener-
ated using LexRank [4]. It was observed that summaries
generated after applying simplification and disaggrega-
tion performed better than compression with respect to
comprehension score and extrinsic evaluation.
Zaman et al. [24] combined text sim plification and text
summarization by extending the end-to-end pointer gen-
erator model. The authors used a combined loss function
consisting of likelihood loss, coverage loss, and simplifi-
cation loss. The work suggests that inclusion of simpli-
fication objectives in the summarization model yielded
better ROUGE scores for generation of summaries for
scholarly articles.
Vale et al. [25] studied the effect of different simplifi-
cation systems on extractive summarization techniques.
Four simplification systems corresponding to rule-based
method, optimization method, supervised deep learn-
ing model, and unsupervised deep learning model were
used to preprocess the input documents. For ranking, the
sentences word-based, sentence-based and graph-based
scoring techniques were used. It was observed that with
respect to different scoring techniques, the simplified
inputs obtained higher ROUGE-1 scores in comparison
with raw inputs. The summaries generated after applying
rule-based and supervised deep learning model-based
simplification demonstrated significant gains in terms of
the evaluation metrics.
3. TEXT SIMPLIFICATION
Transformations based on the syntactic structure of the
sentence have been utilized for performing text sim-
plification by many different works in literature. These
include transformation rules using typed dependency
representations [26] and synchronous dependency gram-
mars [27]. Ferres et al. [28] and Scranton et al. [29] used
dependency parse structure, while Garin et al. [30] used
constituency parse trees to perform simplification trans-
formations. Evans and Orason [31] used a sign tagger to
identify compound clauses and rule-based transforma-
tions for rewriting.
DEPSYM++ used in this work utilizes rules based on
the dependency tree structure of the sentence for sim-
plification of appositive clause, relative clause, and con-
joint clause, along with passive-to-active conversion. The
157
modifications performed on DEPSYM for simplification
of relative and conjoint clause is described in Sections 3.1
and 3.2, respectively.
3.1 Relative Clause
For sentences containing relative clauses, the output pro-
vided by DEPSYM does not insert the noun phrase
within the relative clause, and the relative pronoun is also
retained in the output.
For illustration, the sentence
John teaches art in Sun City, where he goes to play golf
is simplified as a pair of sentences:
John teaches art in Sun City.
Sun City is where he goes to play golf.
The following improvements have been incorporated
in the DEPSYM++ algorithm1 for different relative
pronouns:
• For the relative pronoun where, the noun phrase along
with its syntactical head is inserted after the relative
phrase as described in Figure 2. In the figure, the noun
phrase Sun City and its head in is inserted after the
relative phrase.
• For relative pronoun whose, the noun phrase is con-
verted to its possessive form and the relative phrase is
inserted after the noun phrase as described in Figure 3.
In the figure, John is converted to John’s and the rela-
tive phrase house is in Sun City is attached with it.
• For other relative pronouns, if the relative token
(token with relcl/rcmod tag) succeeds the relative pro-
noun in the sentence, then the noun phrase is inserted
just before the relative phrase. If the relative token
does not succeed the relative pronoun in the sentence,
then the noun phrase is inserted in the relative phrase
before the phrase corresponding to the right subtree
of the relative token.
3.2 Conjoint Clause
DEPSYM algorithm splits sentences containing conjoint
clauses. The relevant conjunctions are appended as the
first token of the sentence.
For illustration, consider the sentence
John went to play golf but did not take his car.
This sentence is simplified as:
John went to play golf. But John did not take his car.
158 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
Figure 2: Simplification of relative clause with relative pronoun
where
Figure 3: Simplification of relative clause with relative pronoun
whose
After sentence splitting, the selection of each indi-
vidual sentence in the summary is independent of
the presence of its context in the summary. Thus, in
DEPSYM++ the conjunctions “and”, “but” are removed
from the simplified sentences if they appear as the first
word.
4. TEXT SUMMARIZATION
For summarization of input documents, six different
techniques covering both extractive and abstractive sum-
marization have been considered. For extraction graph-
based summarization [3,4], term-frequency-based sum-
marization [2] and matrix factorization-based summa-
rization [5] have been used while Transformer-based
neural summarization [32] has been used for gen-
eration of abstractive summaries. The details are as
follows:
4.1 TextRank
TextRank [3] is a graph-based sentence ranking tech-
nique. Here, a graph is constructed where each node
represents a sentence and weighted edges represent the
similarity between the respective sentences. For illustra-
tion, suppose Si and Sj are two sentences with N and M
words, respectively. Si ∩ Sj contains the common words
of Si and Sj . Then, Sim(Si , Sj ) is defined as shown in
Equation (1). Nodes of the graph are ranked using the
PageRank [33] Algorithm. Rank of a sentence Si is calcu-
lated using Equation (2), where N(Si ) denotes the neigh-
bors of the node representing Si , and d is the damping
factor which is set to 0.85 [33]
Count(Si ∩ Sj )
Sim(Si , Sj ) = (1)
log(N) + log(M)
 Sim(Si , Sj )
Rank(Si ) = (1 − d) + d · 
Sj ∈N(Si ) Sk ∈N(Sj ) Sim(Sj , Sk )
× Rank(Sj ) (2)
4.2 LexRank
LexRank [4] is a ranking algorithm based on eigenvector
centrality in a graph representation of sentences. Here,
the nodes of the graph are connected with weighted
edges representing the cosine similarity of the term
frequency-inverse document frequency (TF-IDF) vec-
tors of the respective sentences. For illustration, suppose
Si and Sj are two sentences with TF-IDF vector Vi and
Vj , respectively. Then, Sim(Si , Sj ) is defined as shown in
Equation (3). The adjacency matrix of this sentence graph
is a connectivity matrix based on intra-sentence cosine
similarity. A similarity matrix is constructed by divid-
ing each element of the adjacency matrix by the sum
of the corresponding column entry. The left eigenvec-
tor corresponding to the eigenvalue 1 of this similarity
matrix corresponds to its stationary distribution. The
sentence/node ranking in LexRank is similar to that of
TextRank, as discussed in Section 4.1.
Sim(Si , Sj ) = CosineSimilarity(Vi , Vj )
Vi · Vj
= (3)
||Vi ||·||Vj ||
4.3 LSA
Latent Semantic Analysis (LSA) [5] summarizers use
Singular Value Decomposition for the extraction of
important sentences of a given document. Here, the
input documents are represented using a sparse sen-
tence matrix, where each column represents the weighted
term-frequency vector of an individual sentence. Sup-
pose a given input document D contains n sentences and
m unique words. The sentence matrix M is an m × n
matrix where Mij is the number of times ith word appears
 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION 159

in sentence Sj . Sentences of the document are ranked on Table 1: Data statistics

the basis of the singular values of M. DUC-2001 DUC-2002 CNN/DM
Number of documents 100 100 100
Range of number of sentences per (8,123) (8,82) (14,46)
4.4 SumBasic document
Range of number of sentences after (18,247) (19, 148) (24,165)
SumBasic [2] is a frequency-based summarizer. Here, the simplification per document
Average number of sentence per 37.49 27.71 39.20
sentences are ranked on the basis of the average prob- document
ability of the words in the sentence. The probability of Average number of sentence per 86.9 60.27 94.93
document after simplification
each word is calculated as the ratio of the number of Range of number of words (210,2193) (182, 1376) (284,1678)
times it appears in the input document to the total num- Range of number of words after (228,2497) (200, 1447) (311,1846)
simplification
ber of words in the document. Suppose the input docu- Average number of words per 899.09 624.39 935.12
ment consists of N words. The score of a sentence S = document
w1 w2 . . . wm of m words is computed using Equation (4). Average number of words per 996.06 691.44 1032.41
document after simplification
Here, freq(wi ) denotes the frequency of wi in the Number of reference summaries 3 2 1
document. per document
Average number of words per 113.463 104.695 98.62
 m 
1  freq(wi )
reference summary
Score(S) = (4)
m N
i=1

three data sets, namely DUC-2001, DUC-2002, and

4.5 BART
BART [32] is a denoising autoencoder used for pretrain-
ing sequence-to-sequence Transformer models for vari-
ous NLP tasks, such as Question Answering, Token Clas-
sification, and Abstractive Summarization. It pretrains a
model by combining Bidirectional and Auto-Regressive
Transformers. The pretraining procedure is a two-step
process where first, the input text is corrupted with
an arbitrary noising function. A sequence-to-sequence
model is learned to reconstruct the original text.
5. EXPERIMENTAL DETAILS
For summarization using sentence ranking techniques,
namely, TextRank, LexRank, LSA, and SumBasic, sen-
tences of the input documents are preprocessed. In the
preprocessing step, tokenization and stemming are per-
formed using NLTK [34]. Stopwords are also removed
using NLTK list of English Stopwords. Every input sen-
tence is ranked (scored) using the TextRank, LexRank,
LSA, and SumBasic techniques, as discussed in Section 4.
The implementation of TextRank, LexRank, LSA, and
SumBasic available in the Sumy2 package has been used
in the present work. For each of the above techniques,
the top-ranking sentences are included in the summary
till word limit of 100 is attained. For abstractive sum-
marization using BART,3 the pre-trained model from
summarization pipeline of Transformers [35] has been
used to generate 100 word summaries.
For evaluating the efficacy of the application of syn-
tactic simplification on text summarization outputs,
experiments have been conducted on documents from
CNN/Daily Mail [36] Test Dataset. In DUC-2001 and
DUC-2002, each document is paired with 2–3 man-
ual summaries. Experiments have been conducted on
100 documents from each DUC-2001 and DUC-2002
datasets. In CNN/Daily Mail test data set, each docu-
ment is paired with a set of 1–5 highlight sentences.
The number of words in the highlights (reference sum-
mary) ranged from 9 to 676, with an average of about
58 words. A set of 100 documents are selected such that
the total number of words in the reference highlight sen-
tences of the selected documents lies in the range of
90–110. This helps in ensuring that the generated 100-
word summary is compared with a reference summary
having similar number of words during the evaluation
process. A huge mismatch in the length of the generated
and reference summary could lead to bias in the calcu-
lation of precision and recall terms. Another way could
be to compare the generated summary with only the
first 100 words of the reference summary, but this could
lead to deletion of important facts from the reference,
which may be otherwise present in the generated sum-
mary resulting in a lower evaluation metric score. The
data4 statistics for the three datasets is reported in Table 1.
It can be observed that the selected data set consists of
documents having diverse characteristics. Upon appli-
cation of DEPSYM++, it was observed that syntactic
simplifications were performed for 80.91%, 81.91%, and
84.60% input sentences of DUC-2001, DUC-2002, and
CNN/Daily Mail datasets, respectively. This is because
the proposed simplification algorithm simplifies only the
sentences containing complex syntactical constructions,
namely passive voice, appositive, relative, and/or conjoint
clauses.
160 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION

6. EVALUATION METRICS AND RESULTS using n-gram co-occurrence statistics. The

In order to study the effect of syntactic simplifica-
tion on text summarization, summaries were gener-
ated using different summarization techniques from
raw inputs as well as after applying the preprocess-
ing step of Syntactic simplification. The performance
has been measured using different evaluation met-
rics, such as ROUGE, BERTScore, and different shal-
low metrics. Tables 2–4 show the results with respect
to ROUGE, BERTScore, and different shallow metrics.
Higher score with respect to simplified and raw (i.e.
non-simplified) inputs for each summarization strategy
has been bold-faced in the corresponding tables, and
the highest value for each column has been italicized.
The column Simplified indicates whether DEPSYM-
based simplification on the input text has been applied
or not.
6.1 Performance with Respect to ROUGE
Different ROUGE metrics [37] have been used to eval-
uate the summaries. ROUGE is a measure of overlap
between system summaries and reference summaries
ROUGE-n score of a system summary is computed using
Equation (5).

 R∈{References}
gramn ∈R Countmatch (gramn )
ROUGE − n =  (5)
 R∈{References}
gramn ∈R Count(gramn )
where Countmatch (gramn ) is the maximum number of n-
grams co-occurring in a system summary and a reference
summary (R), and Count(gramn ) is the total number of
n-grams in the reference summary.
ROUGE-n scores for n = 1, 2 for summaries gener-
ated using different summarization methods have been
reported in Table 2. Additionally, ROUGE-L, which is
based on Longest Common Subsequence matching, and
ROUGE-SU4, which matches bigrams with a maximum
skip distance of 4, have also been reported.
Figure 4 indicates the percentage change in ROUGE
scores after simplification for different summarization
schemes on different datasets. Let RSraw and RSsimple

Table 2: ROUGE (recall) scores for different summarization methods

DUC-2001 DUC-2002 CNN/DM
Input R-1 R-2 R-L R-SU4 R-1 R-2 R-L R-SU4 R-1 R-2 R-L R-SU4
TextRank Raw 32.86 13.79 30.38 14.73 37.41 15.93 29.58 16.94 34.91 12.77 32.30 14.19
Simplified 34.13 14.26 31.96 15.14 40.30 17.66 32.96 18.48 40.70 16.16 38.03 17.64
LexRank Raw 30.44 11.43 28.26 12.66 35.18 14.81 28.45 15.56 35.05 12.08 32.43 13.45
Simplified 34.78 14.24 32.76 15.02 37.96 15.06 30.87 16.43 40.62 14.81 38.29 16.66
LSA Raw 27.41 11.21 25.29 11.66 32.96 13.56 25.92 14.33 28.52 9.63 26.37 10.61
Simplified 31.83 13.88 29.82 13.96 37.41 16.53 30.92 16.87 36.90 14.76 34.57 15.17
SumBasic Raw 33.64 16.83 31.43 16.97 36.53 17.69 29.74 18.04 42.11 19.03 40.00 18.91
Simplified 33.69 16.16 32.01 16.21 38.32 18.12 32.00 18.40 41.16 17.99 39.42 18.20
BART Raw 32.06 14.31 30.39 14.90 34.42 15.26 28.63 15.46 42.13 20.22 40.59 20.24
Simplified 33.00 14.96 31.48 15.15 35.60 16.16 29.92 16.31 42.59 21.26 40.49 20.47

Table 3: Precision, Recall and F1-Score of BERTScore for different summarization methods
DUC-2001 DUC-2002 CNN/DM
Input PBERT RBERT F1BERT PBERT RBERT F1BERT PBERT RBERT F1BERT
TextRank Raw 58.79 59.73 59.23 60.66 62.66 61.61 58.03 60.51 59.21
Simplified 58.24 60.44 59.29 61.31 62.16 61.70 60.91 61.71 61.28
LexRank Raw 57.70 59.98 58.79 60.44 61.64 60.99 57.67 60.43 58.99
Simplified 58.84 59.99 59.39 60.63 62.48 61.50 60.42 61.84 61.10
LSA Raw 56.27 58.78 57.47 58.71 60.68 59.65 55.47 58.19 56.76
Simplified 57.80 59.26 58.49 60.11 61.54 60.78 58.80 60.28 59.50
SumBasic Raw 59.39 60.90 60.11 61.50 61.62 61.53 61.03 63.31 62.12
Simplified 59.47 59.63 59.52 61.50 62.38 61.91 61.45 61.54 61.46
BART Raw 61.28 59.80 60.51 62.53 60.99 61.72 64.71 63.51 64.09
Simplified 61.87 59.88 60.83 63.39 60.79 62.03 65.57 65.08 65.30
 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION 161

Table 4: Properties of the generated summaries

DUC-2001 DUC-2002 CNN/DM
Input ASL SCR FKGL ASL SCR FKGL ASL SCR FKGL
Reference 15.3 3.1 10.3 14.9 0.2 9.8 13.4 1.4 7.7
Average of all Raw 18.8 69.7 11.1 17.9 69.7 10.2 21.3 70.7 11.7
methods Simplified 14.1 14.0 8.8 13.4 14.0 8.2 14.7 10.4 8.2
TextRank Raw 22.8 100.0 14.3 20.5 100.0 12.0 24.7 100.0 14.5
Simplified 16.4 16.2 10.4 15.1 18.3 9.4 16.8 11.9 9.9
LexRank Raw 19.5 100.0 11.5 18.1 100.0 10.5 21.5 100.0 12.2
Simplified 11.3 15.1 7.7 11.9 16.6 7.7 12.3 9.9 7.3
LSA Raw 22.1 100.0 13.8 20.9 100.0 12.8 25.0 100.0 15.0
Simplified 13.7 12.8 8.8 12.7 13.8 7.9 14.6 10.2 8.6
SumBasic Raw 19.1 100.0 11.4 18.0 100.0 10.8 21.0 100.0 12.2
Simplified 17.5 24.8 11.5 16.4 23.7 10.7 19.1 24.9 11.7
BART Raw 15.0 16.2 8.7 14.6 16.4 8.1 18.2 20.6 8.9
Simplified 12.9 12.1 8.2 11.7 9.6 7.5 12.8 4.6 7.0

Figure 4: Percentage change in ROUGE Scores after application of DEPSYM Simplification for different summarization schemes

be the ROUGE scores obtained when summarization is
performed on raw text and simplified text, respectively.
The percentage change is calculated using the following
equation:
RSsimple − RSraw
× 100
RSraw
It can be observed from Table 5 that TextRank, LexRank,
LSA, and BART gave better performance when simplifi-
cation is applied for all the three datasets under all the
four ROUGE metrics mentioned above. Application of
simplification with LSA summarization yields the highest
increase in ROUGE. In fact, it can be observed that with
respect to ROUGE-2, there is an increase of more than
50% for LSA summarizer on the CNN/DailyMail dataset.
SumBasic summarizer shows better performance with-
out simplification for DUC 2001 and CNN/DailyMail
datasets. However, the magnitude of decrease in scores
after the application of simplification is only marginal.
This may be attributed to the naïve word frequency-based
sentence ranking approach of SumBasic.
It can also be observed that for DUC datasets the
outputs of sentence ranking or extractive summarizers
have significantly higher ROUGE score than the out-
(6) put generated using the abstractive neural summarizer,
namely BART. The outputs from all other summarizers
except LSA have comparable performance for documents
of CNN/Daily Mail.
6.2 Performance with Respect to BERTScore
Although ROUGE metrics are widely accepted for sum-
mary evaluation, it is biased towards lexical overlap. It
overlooks the fact that semantic overlap between texts
can occur irrespective of lexical dissimilarity. In the
present work, BERTScore [38] addresses the above draw-
back in the following way. Similarity using BERTScore
is calculated as a weighted sum of cosine similari-
ties between the tokens. It correlates well with human
162 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
Table 5: Summary for article la082889-0067 of DUC2002
Reference Summary
Dave Johnson of the US clinched the decathlon gold medal in the final event
of the World University Games. Another American, Sheldon Blockburger,
was in third place in the decathlon until that last event but was overtaken
by Szabo of Hungary. Medved of the USSR won the silver. Americans won
gold and bronze in the men’s 400-meter hurdles. Ana Quirot of Cuba beat
Jearl Miles of the US in the women’s 400 meters. American Llewellyn
Starks was third in the long jump. Romanian Paula Ivan set a meet record
in winning the women’s 3,000-meter race
TextRank
Johnson trailed Mikhail Medved of the Soviet Union by 11 points before the
1,500-meter race, the final event of the two-day competition. Olympic
steeplechase champion Julius Kariuki of Kenya, running the distance
for the first time this year, won the men’s 10,000-meter race, while the
Romanian team earned six gold medals. Kariuki, who won the gold medal
at the 1988 Seoul Games in the 3,000-meter steeplechase, beat out Zeki
Oeztuerk of Turkey to win the 10,000 in the slow time of 28:35. Four
of Romania’s golds came in rowing and two in track and field, one of them
from Olympic champion Paula Ivan, who set a meet record in winning
the women’s 3,000-meter race.
Syntactic Simplification + TextRank
Dave Johnson of the United States clinched the Decathlon gold medal in
the final event at the world University games Sunday. Johnson trailed
Mikhail Medved of the Soviet Union by 11 points before the 1,500-Meter
race, the final event of the two-day competition. Blockburger earned only
569 points in the final event. Kariuki beat out Zeki Oeztuerk of Turkey
to win the 10 000 in the slow time of 28:35. Kevin Henderson took
the bronze for the U. S. in the men’s 400 - meter intermediate hurdles.
Olympic champion Paula Ivan set a meet record in winning the
women’s 3 000-Meter race.
judgment for evaluating different language generation
tasks [39].
Let the system-generated sentence be S = s1 s2 ...sn , and
the reference be R = r1 r2 ...rm then precision (PBERT ) and
recall (RBERT ) for BERTScore are calculated as shown in
Equation (7). Here, −→
si and − →rj , denote the contextualized
5
BERT embedding of the words si and rj , respectively.
F1-Score (F1BERT ) is calculated as the harmonic mean of
PBERT and RBERT .
1 1 
n m
PBERT = max − →
rj T −
→
si ; RBERT = max − →
rj T −
→
si
n i=1 rj ∈R m j=1 si ∈S
(7)
Table 3 provides the evaluation results for the generated
summaries with respect to BERTScore for simplified and
non-simplified inputs. Figure 5 indicates the percentage
change in BERTScores after simplification for different
summarization schemes on different datasets. The per-
centage change is calculated similar to the way described
in Equation (6).
It can be seen that the summaries generated using simpli-
fied inputs have higher BERTScores for most of the meth-
ods. Similar to ROUGE, the application of simplification
with LSA summarizer leads to high gains in terms of
precision, recall, and F1-Score across the three datasets.
Here, the overall best values are obtained for summaries
generated by BART with simplified inputs. From Table 3,
it can be observed that, with respect to BERTScore, out-
puts generated using BART have higher6 values for all
the datasets in comparison with other extractive systems.
This can be ascribed to the similarity in the training
objectives of BART and BERT. It can also be observed that
for CNN/DailyMail dataset, the magnitude of increase in
scores after application of simplification is very high for
extractive summarizers TextRank, LexRank, and LSA.
6.3 Shallow Properties of the Generated Summary
In order to analyze the properties of the generated sum-
maries, Average Sentence Length (ASL), Sentence Copy
Rate (SCR), and Flesch-Kincaid Grade Level (FKGL)
have been calculated. ASL is the mean number of words
in the summary sentences. SCR of a summary is mea-
sured as the proportion of summary sentences that are
exact copies of the source text sentences. FKGL [40] score
is calculated using the formula given in Equation (9).
It is a readability metric designed to indicate the diffi-
culty level of a given text. A lower FKGL score implies
a lower level of difficulty. It can be noted that although
the FKGL score does not measure the overall informa-
tion content of the summary, it is a good indicator of how
simple the summary is in terms of its ease of readabil-
ity. Moreover, studying the change in FKGL scores of the
summary after the application of DEPSYM+ also high-
lights the efficacy of simplification as a preprocessing step
to summarization.
number of words
FKGL = 0.39
number of sentences
number of syllables
+ 11.8 − 15.59 (9)
number of words
ASL, SCR, and FKGL of the reference and generated
summaries are reported in Table 4. Here, the rows cor-
responding to Average of all methods represent the mean
of the metric values of different summarization methods.
Figure 6 gives a visual representation of the percent-
age decrease in ASL, SCR, and FKGL after simplifica-
tion for different summarization schemes on different
datasets. Let Valraw and Valsimple be the feature value
obtained when summarization is performed on raw text
and simplified text, respectively. The percentage decrease
is calculated using the following equation
Valraw − Valsimple
× 100 (10)
Valraw
 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION 163

Figure 5: Percentage change in BERT Scores after application of DEPSYM Simplification for different summarization schemes

Figure 6: Percentage decrease in Shallow feature values after application of DEPSYM Simplification for different summarization schemes

It can be observed that summaries generated using raw
inputs contain sentences longer than those generated
using simplified inputs. Moreover, the ASL of summaries
generated with simplified inputs is closer to the ASL of
manually written reference summaries.
There is a high decrease in SCR values for all the
summarizers across the three datasets. This indicates
that the summaries generated after applying simplifi-
cations are abstractive summaries. That is, even for
extractive methods, such as TextRank, LSA, simplified
sentences are included in the summary. Even for the
generative BART summarizer, the decrease in SCR is
more than 20% for CNN/DailyMail and more than 40%
for DUC.
FKGL values for the summaries with simplified inputs
are lower than those generated with raw inputs. For
DUC documents, the FKGL of simplified summaries are
even lower than reference summaries. This indicates that
syntactic simplification enhances the readability of the
generated summary.
An example of the generated summary with and without
simplification along with a reference summary, is pre-
sented in Table 5. Here, it can be observed that syntactic
simplification helps to break long sentences such that
only the informative parts of the sentence are included
in the summary. The FKGL score of the summary gener-
ated using TextRank is 10.9, while it reduces to 6.5 for the
summary generated using TextRank after simplification.
7. CONCLUSION
In the era of internet and smartphones, when an abun-
dance of textual information can be retrieved with a
single click, the need for efficient ATS systems is of prime
importance. ATS systems help a user to go through the
important information of a text in lesser time and also
to decide its relevance for any specific task. Summary
generation by the selection of top-ranked sentences often
includes irrelevant phrases and concepts in the summary.
This is especially true when the input text consists of long
sentences. Deletion of phrases corresponding to a specific
164 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
syntactic structure without any consideration of its infor-
mation content and relevance can affect the quality of
the summary. In the present work, a syntactic simplifica-
tion system, DEPSYM++, has been applied on the input
text as a preprocessing step. In this research, the effect
of syntactic simplification has been studied for five dif-
ferent summarization techniques of both extractive and
abstractive types. Experiments have been conducted on
documents from three datasets containing news articles,
namely DUC-2001, DUC-2002, and CNN/Daily Mail.
Results indicate that the application of syntactic simpli-
fication helps to generate better summaries in terms of
the different evaluation metrics.
In the present work, complexity of the input text is
identified by the presence of four complex syntacti-
cal structures. Various machine learning-based meth-
ods have been developed in the literature for automatic
assessment of text complexity using sequence process-
ing [41], linguistic features [42], shallow semantic fea-
tures [43], and knowledge graphs [44]. In the future, we
would also like to incorporate this in the text simplifica-
tion module. The present work can also be extended to
study the effects of simplification on texts pertaining to
specific domains, such as medical, legal among others.
The summary evaluation module may be expanded to
include semantically enhanced Jaccard similarity, which
have been used in literature for evaluating automatic
summaries of clinical texts [45].
DISCLOSURE STATEMENT
No potential conflict of interest was reported by the author(s).
FUNDING
Raksha Agarwal acknowledges Council of Scientific and Indus-
trial Research (CSIR), India, for supporting the research [grant
no: SPM-06/086(0267)/2018-EMR-I].
NOTES
1. Code available at https://github.com/RakshaAg/DEPSYM
Sum.
2. https://pypi.org/project/sumy/.
3. https://huggingface.co/facebook/bart-large-cnn.
4. Dataset is available at https://github.com/RakshaAg/DEPS
YMSum.
5. bert-base-uncased.
6. Except recall for DUC.
ORCID
Niladri Chatterjee http://orcid.org/0000-0003-3832-4003
Raksha Agarwal http://orcid.org/0000-0002-9356-4387
REFERENCES
1. I. Mani, and M. T. Maybury. Advances in automatic text
summarization. Cambridge, MA: MIT Press, 1999.
2. A. Nenkova, and L. Vanderwende, “The impact of fre-
quency on summarization,” Microsoft Res Redmond
Washington Tech. Rep. MSR-TR-2005, Vol. 101, 2005.
3. R. Mihalcea, and P. Tarau. “TextRank: Bringing order into
text,” in Proceedings of the 2004 conference on empirical
methods in natural language processing, 2004, pp. 404–11.
4. G. Erkan, and D. R. Radev, “Lexrank: graph-based lexical
centrality as salience in text summarization,” J. Artif. Intell.
Res., Vol. 22, pp. 457–79, 2004.
5. Y. Gong, and X. Liu. “Generic text summarization using
relevance measure and latent semantic analysis,” in Pro-
ceedings of the 24th annual international ACM SIGIR
conference on Research and development in information
retrieval, 2001, pp. 19–25.
6. S. Park, B. Cha, and D. U. An, “Automatic multi-
document summarization based on clustering and non-
negative matrix factorization,” IETE Tech. Rev., Vol. 27,
no. 2, pp. 167–78, 2010.
7. N. Chatterjee, and N. Yadav, “Fuzzy rough set-based sen-
tence similarity measure and its application to text sum-
marization,” IETE Tech. Rev., Vol. 36, no. 5, pp. 517–25,
2019.
8. H. Jing. “Sentence reduction for automatic text summa-
rization,” in Sixth Applied Natural Language Processing
Conference, (Seattle, Washington, USA), Association for
Computational Linguistics, Apr. 2000, pp. 310–5.
9. K. Knight, and D. Marcu, “Summarization beyond sen-
tence extraction: A probabilistic approach to sentence
compression,” Artif. Intell., Vol. 139, no. 1, pp. 91–107,
2002.
10. C. Zong, R. Xia, and J. Zhang. Text data mining. Singapore:
Springer, 2021.
11. A. Siddharthan, A. Nenkova, and K. McKeown. “Syn-
tactic simplification for improving content selection in
multi-document summarization,” in COLING 2004: Pro-
ceedings of the 20th International Conference on Computa-
tional Linguistics, (Geneva, Switzerland), COLING, 2004,
pp. 896–902.
12. M. Shardlow, “A survey of automated text simplification,”
Int. J. Adv. Comput. Sci. Appl., Vol. 4, no. 1, pp. 58–70, 2014.
 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
13. N. Chatterjee, and R. Agarwal. “DEPSYM: A lightweight
syntactic text simplification approach using dependency
trees,” in Proceedings of the First Workshop on Current
Trends in Text Simplification (CTTS 2021), co-located with
SEPLN, 2021, pp. 42–56.
14. S. Štajner, and M. Popovic. “Can text simplification help
machine translation?” in Proceedings of the 19th Annual
Conference of the European Association for Machine Trans-
lation, 2016, pp. 230–42.
15. E. Hasler, A. de Gispert, F. Stahlberg, A. Waite, and
B. Byrne, “Source sentence simplification for statistical
ma-chine translation,” Comput. Speech. Lang., Vol. 45,
pp. 221–35, 2017.
16. T. Dadu, K. Pant, S. Nagar, F. A. Barbhuiya, and
K. Dey. “Text simplification for comprehension-based
question-answering,” arXiv preprint arXiv:2109.13984,
2021.
17. D. Vickrey, and D. Koller. “Sentence simplification for
semantic role labeling,” in Proceedings of ACL-08: HLT,
(Columbus, Ohio), Association for Computational Linguis-
tics, June 2008, pp. 344–52.
18. R. Evans, and C. Orasan. “Sentence simplification for
semantic role labelling and information extraction,” in Pro-
ceedings of the International Conference on Recent Advances
in Natural Language Processing (RANLP), (Varna, Bul-
garia), 2019, pp. 285–94.
19. S. B. Silveira, and A. Branco. “Combining a double clus-
tering approach with sentence simplification to produce
highly informative multi-document summaries,” in2012
IEEE 13thInternational Conference on Information Reuse
Integration(IRI), 2012, pp. 482–89.
20. B. M. Rebello, G. L. d. Santos, C. R. B. d. Ávila, and A. d.
S. B. Kida, “Effects of syntactic simplification on reading
comprehension of elementary school students,” Audiol-
Commun. Res., Vol. 24, pp. 1–8, 2019.
21. R. Cervantes, and G. Gainer, “The effects of syntactic sim-
plification and repetition on listening comprehension,”
Tesol Q., Vol. 26, no. 4, pp. 767–70, 1992.
22. L. Vanderwende, H. Suzuki, C. Brockett, and A. Nenkova,
“Beyond sumbasic: task-focused summarization with sen-
tence simplification and lexical expansion,” Inf. Pro-
cess. Manage., Vol. 43, no. 6, pp. 1606–18, 2007. Text
Summarization.
23. C. Finegan-Dollak, and D. R. Radev, “Sentence simplifica-
tion, compression, and disaggregation for summarization
of sophisticated documents,” J. Assoc. Inf. Sci. Technol.,
Vol. 67, no. 10, pp. 2437–53, 2016.
24. F. Zaman, M. Shardlow, S.-U. Hassan, N. R. Aljohani, and
R. Nawaz, “HTSS: A novel hybrid text summarisation and
simplification architecture,” Inf. Process. Manage., Vol. 57,
no. 6, pp. 102351, 2020.
165
25. R. Vale, R. D. Lins, and R. Ferreira. “An assessment
of sentence simplification methods in extractive text
summarization,” in Proceedings of the ACM Sympo-
sium on Document Engineering 2020, DocEng ‘20, (New
York, NY, USA), Association for Computing Machinery,
2020.
26. A. Siddharthan. “Text simplification using typed depen-
dencies: A comparison of the robustness of different gen-
eration strategies,” in Proceedings of the 13th European
Workshop on Natural Language Generation, 2011, pp. 2–11.
27. A. Siddharthan, and A. Mandya. “Hybrid text simplifica-
tion using synchronous dependency grammars with hand-
written and automatically harvested rules,” in Proceedings
of the14th Conference of the European Chapter of the Asso-
ciation for Computational Linguistics, 2014, pp. 722–31.
28. D. Ferres, M. Marimon, H. Saggion, and A. AbuRa’ed.
“Yats: yet another text simplifier,” in International Confer-
ence on Applications of Natural Language to Information
Systems, Springer, 2016, pp. 335–42.
29. C. Scarton, A. P. Aprosio, S. Tonelli, T. M. Wanton, and
L. Specia. “Musst: a multilingual syntactic simplification
tool,” in Proceedings of the IJCNLP 2017, System Demon-
strations, 2017, pp. 25–8.
30. A. Garain, A. Basu, R. Dawn, and S. K. Naskar. “Sentence
simplification using syntactic parse trees,” in 4th Inter-
national Conference on Information Systems and Com-
puter Networks (ISCON), 2019, pp. 672–6.
31. R. Evans, and C. Orasan, “Identifying signs of syntactic
complexity for rule-based sentence simplification,” Nat.
Lang. Eng, Vol. 25, no. 1, pp. 69–119, 2019.
32. M. Lewis, et al. “BART: Denoising sequence-to-sequence
pre-training for natural language generation, translation,
and comprehension,” in Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics,
(Online), Association for Computational Linguistics, July
2020, pp. 7871–80.
33. S. Brin, and L. Page, “The anatomy of a large-scale hyper-
textual web search engine,” Comput. Netw. ISDN Syst.,
Vol. 30, no. 1–7, pp. 107–117, 1998.
34. S. Bird, E. Klein, and E. Loper. Natural language process-
ing with Python: analyzing text with the natural language
toolkit. Sebastopol, CA: O’Reilly Media, Inc. 335, 2009.
35. T. Wolf, et al. Transformers: State-of-the-art natural lan-
guage processing, in: Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing: System
Demonstrations, Association for Computational Linguistics,
Online, 2020, pp. 38–45.
36. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt,
W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines
to read and comprehend,” Adv. Neural. Inf. Process. Syst.,
Vol. 28, pp. 1693–701, 2015.
166 N. CHATTERJEE AND R. AGARWAL: STUDYING THE EFFECT OF SYNTACTIC SIMPLIFICATION ON TEXT SUMMARIZATION
37. C.-Y. Lin. “Looking for a few good metrics: Auto-
matic summarization evaluation-how many samples are
enough?” in NTCIR, 2004.
38. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y.
Artzi. “Bertscore: Evaluating text generation with BERT,”
in 8th International Conference on Learning Represen-
tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
2020, OpenReview.net, 2020.
39. S. Li, D. Lei, P. Qin, and W. Y. Wang. “Deep reinforcement
learning with distributional semantic rewards for abstrac-
tive summarization,” in Proceedings of the 2019 Confer-
ence on Empirical Methods in Natural Language Process-
ing and the 9th International Joint Conference on Natu-
ral Language Processing (EMNLP-IJCNLP), (Hong Kong,
China), Association for Computational Linguistics, Nov.
2019, pp. 6038–44.
40. J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S.
Chissom. “Derivation of new readability formulas (auto-
mated readability index, fog count and flesch reading ease
formula) for navy enlisted personnel,” tech. rep., Naval
Technical Training Command Millington TN Research
Branch, 1975.
41. D. Schicchi, G. L. Bosco, and G. Pilato, “Machine learn-
ing models for measuring syntax complexity of English
AUTHORS
Niladri Chatterjee is the chair professor
of artificial intelligence in IIT Delhi. He is
a professor of statistics and computer sci-
ence in the Department of Mathematics,
IIT Delhi, School of IT and School of AI
of IIT Delhi. His primary research areas
are artificial intelligence, natural language
processing, big data analytics, statistical
modelling. His association with IIT Delhi has been more than
22 years. Prior to that, he had worked as a lecturer in University
College London, and a computer engineer at Indian Statistical
Institute, Calcutta. He has been a visiting professor in Depart-
ment of Informatics, University of Pisa, Italy. He has supervised
ten PhDs and over a hundred master’s thesis in mathematics
and computing.
Corresponding author. Email: niladri@maths.iitd.ac.in
text,” in Biologically inspired cognitive architectures meeting,
Advances in Intelligent Systems and Computing, vol. 948, A.
Samsonovich Ed. Cham: Springer, 2019, pp. 449–54.
42. R. Agarwal, and N. Chatterjee. “Gradient boosted trees
for identification of complex words in context,” in Pro-
ceedings of the First Workshop on Current Trends in Text
Simplification (CTTS 2021) co-located with SEPLN, 2021,
pp. 12–28.
43. S. Stajner, and I. Hulpus. “When shallow is good enough:
Automatic assessment of conceptual text complexity using
shallow semantic features,” in Proceedings of the 12th Lan-
guage Resources and Evaluation Conference, (Marseille,
France), European Language Resources Association, May
2020, pp. 1414–22.
44. S. Stajner, and I. Hulpus. Automatic assessment of concep-
tual text complexity using knowledge graphs,” in Proceed-
ings of the 27th International Conference on Computational
Linguistics, (Santa Fe, New Mexico, USA), Association for
Computational Linguistics, Aug. 2018, pp. 318–30.
45. M. Afzal, F. Alam, K. M. Malik, and G. M. Malik, “Clin-
ical context–aware biomedical text summarization using
deep neural network: model development and validation,”
J. Med. Internet Res., Vol. 22, no. 10, pp. e19810, 2020.
Raksha Agarwal is a PhD scholar in the
Department of Mathematics, IIT Delhi.
Her primary research areas are natural
language processing and machine learn-
ing, with focus on abstractive text summa-
rization. She has done Master of Science
in mathematics from IIT Delhi and Bach-
elor of Science in mathematics from the
University of Delhi. She is the recipient of Shyama Prasad
Mukherjee Fellowship awarded by the Council of Scientific and
Industrial Research, India.
Email: raksha.agarwal@maths.iitd.ac.in