Praveen Soni (2018CH70296)

Supervisor – Hari Prasad Kodamana

Chemical Engineering Department, IIT Delhi

Generation of
Knowledge Graph
for PET
 Introduction
• PET is a thermoplastic polymer made from
polyester
• Excellent mechanical, chemical, and thermal
stability
• Packaging, electronics etc. commercial usage of PET
• Extensive data on the research of PET production
and recycling  Need for automated techniques
• Data isolation, diffusion, and heterogeneity
• Knowledge Graph (KG) for PET poses a solution for
that
• Huge data, therefore, abstractive text
summarization and other automated techniques
required

P. Verma, "An Overview of Polyethylene Terephthalate (PET) Uses and Features“.

X. Zhao et al., "Knowledge Graph-Empowered Materials Discovery," 2021 IEEE International Conference on Big
Data (Big Data), Orlando, FL, USA, 2021, pp. 4628-4632.
Related Work
• Early methods for pretraining were based on language models
• BERT (Devlin et al., 2019) introduced masked language modelling, allowing pre-
training to learn interactions between left and right context words
• Recent work has shown that very strong performance can be achieved by training for
longer (Liu et al., 2019), by tying parameters across layers
• Raffel et al. (2019) pre-trained a denoising sequence-to-sequence model named T5,
experimenting with a similar range of noising tasks
• XL-Net (Yang et al., 2019) extends BERT by predicting masked tokens auto-regressively
in a permuted order
• BART uses a slightly different objective, in which spans are masked from the input but
the complete output is predicted, to improve the decoder’s language modelling ability
Objective

• Collecting information on PET synthesis & recycling

Source
• Manually summarizing them
• Cleaning the text
• Fine tuning an abstractive text summarization model on our dataset
• Breaking the processed text into 3 categories: Manufacturing
Methodology
 Source
 Edge
 Target
• Construct a knowledge graph on the summarized data tailored Target
specifically for PET.
 Seq2Seq Model
• Sequence to Sequence (seq2seq) architecture is an
important segment in the models described nex
• Subset of RNNs used to solve linguistic issues including
machine translation, text summarization, etc.
• An encoder and a decoder make up a basic seq2seq
architecture
• Source input: x = (x1, x2, ..., xJ ) (read by the encoder)
• Input transformed into hidden states: he = (he1, he2, ...,
heJ) • xi, yj = one-hot token encodings
• The hidden states are then used to output the of abstract and summary
summary, denoted as y = (y1, y2, ..., yT ) • J = no. of tokens (document
length)
• T = no. of tokens of the summary

Shi, T., Keneshloo, Y., Ramakrishnan, N., & Reddy, C. K. (2018). Neural Abstractive Text Summarization with Sequence-to-Sequence Models. ArXiv.
/abs/1812.02303
 Seq2Seq Model
• Feed-forward networks, CNNs, or RNNs can be used as encoders and decoders
• The most popular RNN designs for seq2seq models are GRU and LSTM.
• In the figure, a straightforward RNN seq2seq model that makes use of a bidirectional LSTM encoder and
decoder is shown
• Since a bi-directional LSTM typically provides better document representations than a forward LSTM, it
is taken into consideration.
• The model configuration used in training the model is:

CONFIGURATION VALUE
Optimizer RMS Prop
LSTM layers 4
Epochs 50
Loss function Sparse Categorical Cross entropy
Learning rate 5x10-4
 BART Model
• BART stands for bidirectional autoregressive transformer.
• A Seq2Seq model with a left-to-right autoregressive decoder and a
bidirectional encoder over tainted text.
• BART is trained by first corrupting documents and then the loss (cross
entropy) is optimized between the original document and the
decoder’s output.
• BART is pretrained on large-scale corpora using self-supervised
learning techniques.
• Well-suited for tasks like machine translation, text summarization, and
dialogue generation.
• Our architecture employs an encoder and decoder with 12 layers
each.

Lewis, Mike, et al. "BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension." ArXiv,
2019, /abs/1910.13461. Accessed 21 Jun. 2023.
 BART Model
The model configuration used in training the model:

CONFIGURATION VALUE

max_length 300

min_length 30

temperature 1

epochs 20

loss function Cross entropy flat

α 10-4
 GPT Model

• GPT stands for generative pre-trained transformer

• It’s also based on the transformer architecture, incorporating self-attention mechanisms
• The GPT version of the transformer model lacks an encoder in their network.
• Unlike RNNs, these models examine tokens simultaneously by making predictions for all time steps at once
Montesinos, D. M. (2020). Modern Methods for Text Generation. ArXiv. /abs/2009.04968.
Radford, Alec et al. “Language Models are Unsupervised Multitask Learners.” (2019).
 GPT Model
• To prevent repetitive outputs, the following 2
generation methods are used:
1. top_k_top_p_filtering:
• Only considers the top k most probable tokens at CONFIGURATION VALUE
each decoding stage (top-k filtering) Optimizer Adam
ε 10-8
• top-p uses the cumulative probability distribution
Temperature 0.8
rather than the top-k tokens. Epochs 20

2. Beam Search: Loss function Cross entropy

α 5x10-4
• A greedy algorithm, like Breadth-First Search
GPT model configuration
(BFS)
• β (Beam width) indicates the number of nodes
from which to grow
• Our heuristic cost decides which nodes are the
optimal nodes to go to a certain node
 Understanding the dataset
• The dataset consists of 379 rows of Abstracts and their
corresponding summaries.
• The abstracts has detailed information about PET, its
manufacturing and recycling processes.
• The following information can be drawn from the figure:
a) Most abstract lengths are below 300
b) Most summary lengths are below 40
• Through experimentation with BART, GPT, and Seq2Seq
models, we aim to evaluate their performance in
summarizing the PET abstracts.
Data Pre-Processing
Text cleaning on literature abstracts:
• Removing extra space
• Removing numbers
• Removing contractions
Text cleaning on scientific abstracts (unconventional approach):
• Removing extra spaces & numbers can cause data loss
o Ex: Polyethylene terephthalate (PET) or C6H12O6
• Removing contractions may change the meaning of sentence
o poly-l-lysine (PLL)-modified conical nanochannels
• A list of keywords was defined to make those particular
sentences uninterrupted.
What is Knowledge Graph?
• A directed labeled graph having 4-tuple G = (N, E, L, f),
where N is a set of nodes, E ⊆ N × N is a set of edges, L
is a set of labels, and f: E→L, is an assignment function PET
from edges to labels.
• An assignment of a label B to an edge E=(A,C) can be
viewed as a triplet (A, B, C) . Glycolysis
• Example texts: glycolysis
&
• “BHET was produced by glycolysis of PET.” esterification
• “Polyester polyols are made via glycolysis and
esterification of recycled polyethylene terephthalate
(rPET) scrap.”
• Python library used for constructing KGs: networkx . Polyester BHET
polyols
Results on BART
Results on GPT
Results on Seq2Seq
 BART Summaries
S No. Actual Summary Predicted Summary

1. CO2 is employed as a pyrolysis CO2 is employed as a pyrolysis medium for thermal treatment of polyethylene terephthalate
medium for thermal treatment of (PET) waste to recovery of energy from the PET waste and reduce acidic byproducts such as
polyethylene terephthalate (PET) benzoic acid. The generation of carbon monoxide (CO) for pyrolysis of the PET helps to
waste to recovery of energy. energy recovery.

2. the production of gaseous
products containing acetylene,
ethylene and carbon monoxide
from PET particles using a rotating
direct current are plasma reactor.
the production of gaseous products containing acetylene, ethylene and carbon monoxide
from PET particles using a rotating direct current arc plasma reactor with a product gas
containing 42% acetylene, 53% carbon monox and 4% ethylene.
 GPT Summaries
S No. Actual Summary
1. depolymerization of poly(ethylene terephthalate)
using Sulfonic acid ionic liquids as catalyst in hot
compressed water to yield terephthalic acid.
2. In the presenece of xylene and an emulsifier,
glycolysis and neutral hydrolysis of waste PET was
carried out. After separating the product from EG,
water and xylene by filtration, water insoluble
fraction was obtained.
Predicted Summary
Depolymerization of poly(ethylene terephthalate) in hot compressed water to form
terephthalic acidic acidene terephthalate compressed waterSulfonic acid ionic liquids
as catalysts in hot compressed water liquids witholy acidic ionic acid acid assts in hot
compressed waterDepolymerization of poly(ethylene terephthalate) in hot
compressed wateric acid
Simultaneous glycolysis and neutral hydrolysis of waste PET wastes PET flakes
obtained from grinding postconsumer bottles was carried out in the presence of
xylene and an emulsifier at 190A°C Simultaneous glycolysis and neutral hydrolysis of
waste PET wastes PET wastes PET from PET PET grinding postconsumer bottles was
carried out in the presence of xylene and an emulsifier at 180A°C.
Knowledge Graph
Conclusion
• BART fine tuning hasn’t performed well enough on the dataset
• The predicted summaries by BART contain some unnecessary information
• For Seq2Seq model, the test loss is converging, suggesting complete training with the
average loss coming out to be 1.6
• Seq2Seq predicted summaries contain a lot of repetitive words making to ineligible to
use for practical purposes
• The performance of GPT model is not much different from that obtained by BART
• Summaries predicted by GPT are better than BART because it does contain valuable
information, although the output is repetitive
• Finally ,the summaries predicted by GPT are used for KG creation, by simply taking
only the first non-repetitive sentence
Future Work
• The size of the dataset can be increased to a few 1000
samples.
• A more complex transformers architecture can be tested on
the current dataset to increase the model performance.
• To extend this project for other polymers too besides PET
THANKYOU!
-Praveen Soni