2024 International Conference on Computing, Power, and Communication Technologies (IC2PCT)

Evaluating Top-k RAG-based approach for Game

2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT) | 979-8-3503-8354-6/24/$31.00 ©2024 IEEE | DOI: 10.1109/IC2PCT60090.2024.10486273

Review Generation
Pratyush Chauhan
dept of computer scienceBennett
University Greater Noida India
pratyushchauhan62@gmail.com
Rahul Kumar Sahani Soham Datta
dept of computer science Bennett dept of computer scienceBennett
University Greater Noida India University Greater Noida India
rahul.sahan810@gmail.com sohamdatta34@gmail.com

Ali Qadir Manish Raj Mohd Mohsin Ali

dept of computer scienceBennett dept of computer science Bennett dept of computer scienceBennett
University Greater Noida India University Greater Noida India University Greater Noida India
ali.qadir.007@outlook.com manish.raj@bennett.edu.in mohdmohsin066@gmail.com

Abstract—Having access to public opinion for a particular A. Conventional approach

product can be a cumbersome task. There are multiple reviews
for the same product. Some may be good or bad depending on the
bias of the reviewer. Using LLMs for the interpretation of this
data would make it easier to understand the overall perception of
a product.This is a study regarding how well RAG+LLMs can be
used as Game Review Generators, built using state-of-the-art
open source LLM LLaMA 2 13b and the Retrieval Augmented
Generation framework llamaindex. The goal here is to generate
and evaluate game reviews that take elements from a set of game
reviews regarding a particular game without using any form of
fine-tuning. This is achieved by using a rudimentary ‘query
engine’ over a subset of publicly available game reviews. Game
reviews are converted to vector stores which allow us to use top- k
semantic retrieval for inference. This technique of providing data
to an LLM from a document is called Retrieval Augmented
Generation or RAG .Upon experimenting, game-specific reviews
were generated (without using any form of fine-tuning) with the
help of RAG combined with a top-k semantic retrieval ranking
system.The application of this technique goes beyond simple game
review generation, it can be used to generate and query context-
specific information on any product given enough base
information.
Keywords—video game review generation, large language
models (LLM), LLAMaindex, na¨ıve retrieval augmented gener-
ation (RAG), top-k semantic retrieval.
I. INTRODUCTION
Game reviews serve as an important resource of
information for prospective players, it aids in their decision-
making process regarding whether to buy a particular game or
not. These critic reviews offer valuable insights into the
strengths and weaknesses of a game. LLMs[1]offer us a chance
to make inferences from these vast amounts of unstructured
data as they are a form of AI that undergo extensive training on
vast datasets of text, enabling them to grasp human language
patterns and produce content that closely resembles human-
generated content. In the context of generating game reviews,
LLMs prove beneficial in crafting reviews that consider the
opinions of underlying human-written game reviews.
Refinement through fine-tuning represents a conventional
method for adapting a model for a task that is specific to a
domain. This process usually entails modifying the model
weights of a pre-trained model to enhance its performance of a
desirable domain addressing a target task. For example, one
might consider fine-tuning an LLM for generating game
reviews.
The fine-tuning process is characterized by the intensity of
time and resources. It necessitates using a substantial volume
of training data, this requirement poses a challenge in domains
where finding quality data is difficult. In the case of game
reviews, a limited number of game reviews can impede the
acquisition of a sufficiently large dataset for training.
II. PROPOSED APPROACH
In this study, an innovative approach to game review
generation is presented that eliminates the need for fine- tuning
for this specific task. This approach involves naive or top-k
Retrieval Augmented Generation (RAG)[3] combining
retrieval with context-specific generation. The retrieval com-
ponent extracts top-k information chunks most similar to the
query provided and the generation component produces text
based on the retrieved information.
This approach offers an efficiency advantage compared to
fine-tuning. Since there is no training involved, resources are
conserved in that aspect. It demonstrates an efficient
adaptability to novel domains.
A. Key contributions
• Introduction of a novel method for generating game
reviews without reliance on fine-tuning.
• Demonstration of the capability of our method to
producehigh-quality, context-specific game reviews.
• Illustration of the efficiency and cost-effectiveness of
our method in contrast to fine-tuning. Validation of the
generalizability of our method to diversedomains.

Copyright © IEEE–2024 ISBN: 979-8-3503-8354-6 258

Authorized licensed use limited to: Universidade Federal Rural do Semiárido - Campus Mossoró. Downloaded on June 21,2024 at 16:13:36 UTC from IEEE Xplore. Restrictions apply.
 2024 International Conference on Computing, Power, and Communication Technologies (IC2PCT)
III. RELATED WORKS
Evaluating Large Language Models for Document-
grounded Response Generation in Information-Seeking
Dialogues explore the application of large language models
[4] (LLMs). This work examines how well LLMs such as
ChatGPT work in producing responses based on documents
in information-seeking conversations. The authors perform
a human evaluation in which annotators evaluate the shared
task-winning system’s output after realizing that automatic
evaluation measures are unable to adequately capture the
subtleties of these verbose responses.
Benchmarking Large Language Models for News Sum-
marization[5]. This paper talks about the effectiveness of
LLMs in summarization. The main point is that instruction
tuning, rather than model size, is the key to LLMs’ success
in zero- shot summarization. The state-of-the-art LLM per-
forms on par with summaries written by freelance writers,
with instruction tuning being the key factor for success.
The evaulation of Large Language Models. Assessment are
conducted on the established CNN/DM dataset and XSUM
dataset. However, challenges arose due to the presence of low-
quality reference summaries in these datasets. The reference
summaries, as judged by human annotators, were inferior
to the outputs generated by most automated systems. This
inferior quality of references hampers the correlation between
metric results and human judgment when computing automatic
metrics. This not only complicates the evaluation process
but also diminishes the performance of systems that receive
supervision through methods such as finetuning or few-shot
prompting, making meaningful comparisons challenging.
To tackle the issues associated with the quality of reference
summaries and gain a deeper understanding of how LLMs
compare to human summary writers, this study has enlisted the
services of freelance writers from Upwork. These freelancers
were tasked with re-annotating 100 articles from the test sets
of CNN/DM and XSUM[25].
Upon comparing the performance of the most proficient
LLM, OpenAI’s Instruct-Davinci, with that of the freelance
writers, a notable distinction emerged. The Instruct-Davinci
summaries exhibited a significantly more extractive nature.
Through manual annotation of summarization operations em-
ployed in these summaries, it is observed that Instruct-Davinci
engages in less frequent paraphrasing, although it demonstrates
the ability to coherently combine copied segments.
Google Bard [6] uses retrieval augmented generation to
give context-specific answers with sources. Results show
promise for accelerating knowledge expression using AIin
literature reviews, with a brief comparison to OpenAI
ChatGPT. Despite higher plagiarism rates in paraphrased
texts, the study suggests increasing use of AI tools in
academic literature.
On Faithfulness and Factuality in Abstractive Sum-
marization [12]. Conventional likelihood training and ap-
Copyright © IEEE–2024 ISBN: 979-8-3503-8354-6
Authorized licensed use limited to: Universidade Federal Rural do Semiárido - Campus Mossoró. Downloaded on June 21,2024 at 16:13:36 UTC from IEEE Xplore. Restrictions apply.
proximate decoding objectives implemented in neural text
generation models result in responses that deviate from human-
like qualities, particularly in open-ended tasks like language
modeling and story generation. This investigation delves into
the limitations of these models when applied to abstractive
document summarization. The findings reveal a high suscep-
tibility of these models to generate content that lacks fidelity
to the input document, a phenomenon commonly referred to
as hallucination.
To comprehensively examine this issue, an extensive
human evaluation involving multiple neural abstractive
summarization systems is conducted. The objective was to
gain insights into the nature of hallucinations produced by
these systems. The outcomes which were executed by human
annotators indicate the presence of significant amounts of
hallucinated content across all summaries generated by the
models under consideration.
This analysis presents evidence supporting the notion
that pretrained models exhibit superior summarization
capabilities, not only in terms of conventional metrics like
ROUGE[26] but also in their proficiency to generate faithful
and factually accurate summaries, as discerned through
human evaluation. Additionally, it is observed that textual
entailment measures exhibit a stronger correlation with
faithfulness compared to standard metrics. This discovery
holds promise for the development of automatic evaluation
metrics, as well as refined training and decoding criteria in
the domain of abstractive document summarization. NLU
[20] driven pretraining generates factual information but
it’s not sufficient. Semantic inference-based automatic
measures are better representations of true summarization
quality
BERTScore: Evaluating Text Generation with BERT
[10]. The paper in question talks about Bertscore, an automatic
evaluation metric for generated text. Unlike traditional metrics,
it calculates token similarity using contextual embeddings
rather than exact matches between candidate and reference
sentences.BERTSCORE exhibits significantly higher
performance compared to the othermetrics.
Understanding the Extent to which Content Quality
Metrics Measure the Information Quality of Summaries
[13]By comparing a summary to a reference, reference-
based metrics like ROUGE or BERTScore are frequently used
to assess the content quality of a summary. The fundamental
idea is to use the degree of commonality to gauge the
information quality of summaries. Nonetheless, in the context
of sum- mary comparisons, this study critically investigates the
token alignments used by ROUGE and BERTScore. The
analysis casts doubt on the idea that the scores produced by
thesemetrics invariably represent information overlap; rather, it
is more appropriate to interpret them as measures of how
closelyrelated topics are covered in the summaries.
259
 2024 International Conference on Computing, Power, and Communication Technologies (IC2PCT)
Fig. 1. Document parsing
Furthermore, there is strong evidence to suggest that this
observation holds true for a wide range of other evaluation
metrics that are employed in summarization. This finding is
important because it casts doubt on how well-aligned popular
summarization evaluation metrics are with the main objective
of the research, which is to produce summaries that contain
high-quality information. However, the study highlights a
promising direction for future work: the recently proposed
QAEval assessment metric, which involves using question-
answering to score summaries, seems to better capture the
quality of the information than the evaluation methodologies
that are currently in use.This realization points to a possible
direction for improving the way the research community
assesses summarization methods. QAEval generates all
possible wh-questions in the reference summary, looks for
matching answers in the candidate summary, and calculates the
final score based on the total weighted alignment to achieve
alignment between two summaries.
Understanding and Improving Zero-shot Multi-hop Rea-
soning in Generative Question Answering [14]. Question
Answering (QA) models demonstrate the capacity to produce
answers to questions in two scenarios: in an open-book sce-
nario where pertinent data is also retrieved, or in a closed-
book scenario where they rely only on their model parameters.
We still don’t fully understand the mechanisms behind the
success of generative QA models, despite their ability to
handle complex questions. Through a number of experiments,
this study seeks to explore the multi-hop reasoning capabilities
of generative QA models.
First, the research breaks down multi-hop questions into
separate single-hop questions and finds significant discrep-
ancies in the responses given by QA models for questions that
appear to be the same. This observation calls for a more
thorough investigation of these models’ underlying reasoning
abilities, particularly in the context of multi-hop structures.
Moreover, the analysis reveals a weakness in these models’
zero-shot multi-hop reasoning capacity. Suboptimal general-
ization to multi-hop questions is the outcome of exclusive
training on single-hop questions. This emphasizes how crucial
it is to enhance the models’ ability to reason in multiple hops
without explicit training on these kinds of situations.
Copyright © IEEE–2024 ISBN: 979-8-3503-8354-6
Authorized licensed use limited to: Universidade Federal Rural do Semiárido - Campus Mossoró. Downloaded on June 21,2024 at 16:13:36 UTC from IEEE Xplore. Restrictions apply.
The study offers evidence indicating that generative QA
models can be made more capable of zero-hop multi-hop
reasoning in two different ways. In the first, training is done on
single-hop question concatenation, and in the second, training
is done on logical forms (SPARQL) that approximate real
multi-hop natural language (NL) questions. Together, these
results show that multi-hop reasoning in generative QA models
can be encouraged by improving model architectures or
training strategies rather than being innate.
Learning to summarize from human feedback [8]. The data
and metrics utilized for certain tasks limit the effective- ness of
language models’ training and evaluation techniques as they
get more complex. For example, summarization models are
usually assessed using criteria such as ROUGE and trained to
anticipate human-generated reference summaries. These
measures, however, are not perfect gauges and might not fully
represent the intended conclusion of summary quality. This
study shows that training a machine to optimize for human
preferences can lead to notable gains in summary quality.
In order to do this, a sizable, carefully selected dataset of
human comparisons between summaries is used. Next, the
algorithm is taught to forecast the summary that is preferred by
humans. This trained model then serves as a reward function,
enabling reinforcement learning to be used to fine-tune a
summarization policy. Positive outcomes are obtained when
this method is applied to an altered version of Reddit’s TL;DR
dataset [23]. Comparisons show that the models created with
this technique perform better than larger models that are
exclusively refined by supervised learning, as well as human
reference summaries. Additionally, these models work well on
CNN/DM [22] news items, producing summaries that are on
par with human references without the need for extra fine-
tuning for news-specific content.
The fine-tuned models and the human feedback dataset are
understood through extensive analyses. According to human
evaluations, the study demonstrates that the reward model is
generalizable to new datasets and that optimizing this reward
model produces better summaries than optimizing based on
ROUGE. This emphasizes how crucial it is for machine
learning researchers to carefully analyze the metrics they use
to evaluate their models’ performance in addition to how they
are trained and assessed.
260
 2024 International Conference on Computing, Power, and Communication Technologies (IC2PCT)
Large Language Models are Zero-Shot Reasoners [15]. The
paper in question talks about the surprising zero-shot reasoning
capabilities of pretrained large language models (LLMs).
While LLMs are known for their few-shot learning abilities,
the authors show that by adding a specific prompt, ”Let’s think
step by step,” LLMs significantly outperform existing zero-
shot LLM performance.“Let’s think step by step” followed by
the remaining prompt acts as a single zero-shot prompt ,
expanding the ability to extract cognitive answers.This study
introduces Zero-shot-CoT, a novel approach employing zero-
shot templates for chain of thought reasoning prompts. It
distinguishes itself from the initial chain of thought prompting
method proposed by Wei et al. (2022)[24] by eliminating the
necessity for step-by-step few-shot examples. It also diverges
from the majority of template prompting methodologies used
before, such as those outlined earlier, by being task-agnostic
and facilitating multi-hop reasoning across a diverse array of
tasks through the utilization of a single template. The
fundamental concept underlying this method is characterized
by its simplicity.
A Transformer-based Approach for Source Code
Summarization [16]. The paper in question talks about source
code summarization, which is the task of generating human-
readable summaries for computer programs. The study focuses
on learning code representation using Transformer models,
leveraging self-attention to capture long-range dependencies
between code tokensBase model outperforms the baselines
(except for ROGUE-L in java) while full model improves the
performance further.
Evaluating large language models on medical evidence
summarization [17]. The paper in question talks about the
assessment of large language models, specifically GPT-3.5 and
ChatGPT[21], in zero-shot medical evidence summarization
across various clinical domains. Findings show that having
longer text actually negatively impacts ChatGPT’s capability
to identify and extract the most pertinent information.
Large Language Models are Diverse Role-Players for
Summarization Evaluation [18]. The paper in question talks
about the challenge of evaluating text summarization quality
and proposes a new evaluation framework based on Large
Language Models (LLMs). The framework assesses both
objective aspects like correctness and subjective dimensions
like comprehensiveness and interestingness. It uses roleplayer
prompting and context-based mechanisms to provide a
comprehensive evaluation. Currently, there exists a disparity
between automated metrics for text generation and human
evaluation. Automated metrics typically assess only surface
similarities (at the lexicon or semantic level). This limitation
can result in a biased perception and evaluation of the text
generation capabilities of Large Language Models. In
this section, we expand upon our suggested measurement
framework for text generation, which emphasises on the
generation of diversified roleplayers and an evaluation based
on these roleplayers. A context-based prompting mechanism
that is able to generate dynamic roleplayer profiles based on
input context.
Copyright © IEEE–2024 ISBN: 979-8-3503-8354-6
Authorized licensed use limited to: Universidade Federal Rural do Semiárido - Campus Mossoró. Downloaded on June 21,2024 at 16:13:36 UTC from IEEE Xplore. Restrictions apply.
Fig. 2. Flowchart for Sub-Approach 1
IV. METHODOLOGY
To develop a methodology to summarize game reviews
with a constantly updated knowledge base. Here the traditional
fine-tuning approach runs into a massive hurdle: constantly
updating data, which takes a lot of time and resources. Due
to constantly updating data, we can not feasibly fine-tune an
LLM every time a detail updates.
A. Key components:
LLM: Open source large language model - LLaMA V2 13b
[9] to serve as generation component as it is relatively small in
size and fast for our computational processing setup due to its
relatively small memory footprint.
Vector DB: Weaviate for storing vector stores generated
from our documents [2]. Using a vector DB eliminates the
need to generate indexes of the knowledge base again and
again and saves time and resources.
Web Scraper: Python web scraper to collect updated game
reviews from various public websites.
Data Framework: Python along with LlamaIndex to
seamlessly feed document text-based data into the LLM (Fig
2).
B. Document parsing:
In order for the documents to be useable for RAG they
must be parsed and converted to vector stores. This is because
mathematical operations are applied are applied on them.
Documents are first converted into chunks of 512 tokens via
chunking. These chunks are further converted to mathematical
representations or embeddings. These embeddings are repre-
sented as vector stores. Embeddings are necessary to enable
top-k semantic search where the top-k most relevant chunks
are filtered and fed into the LLM context for generation.
Upon receiving a query, a “query engine”[2] is created
using the vector store of a particular game. This vector store
contains game-review embeddings for a specific game. The
query engine then retrieves the top-k most relevant chunks
under the process of “Retrieval” (fig 2). These chunks are fed
into the LLM context for game review generation. With this
new context-specific information the LLM is prompted to
generate a game review (Fig 1)
261
 2024 International Conference on Computing, Power, and Communication Technologies (IC2PCT)

V. EXPERIMENTS AND RESULTS B. Unbounded Newline Outburst

Rouge and BERTScores were calculated on generated After the llm is finished gen- erating game reviews we
game reviews for 10 different games based on a knowledge observed that sometimes it continues to return empty lines
base of 120+ human-written game reviews. indefinitely. We call this an Unbounded Newline Outburst,
where the llm does not stop returning new lines. Further
The following results were obtained after generating game research is required to reach the root of this issue.
reviews using LLaMA 2 13b with top-k RAG.
A more complex RAG technique other than top-k similarity
TABLE I. RESULT ANALYSIS METRICS AND SCORES RAG may be used to improve the coverage of game reviews.
Metric Score
ROUGE-1 Precision 0.019032
ROUGE-1 Recall 0.877474
ROUGE-2 Precision 0.005211
ROUGE-2 Recall 0.519979
ROUGE-L Precision 0.018611
ROUGE-L Recall 0.857376
BERTScore 0.813583

TABLE II. F1-SCORE COMPARISON TO MODLES WITHOUT

RAG.[27]
Model R-1 R-2 R-L B-S
GPT-3.5 0.3071 0.0620 0.1861 0.6102
GPT-4 0.3211 0.0611 0.1841 0.6152
PaLM-2 0.2201 0.0420 0.1321 0.5123
LLaMA-2-13b 0.2864 0.0639 0.1829 0.5532
LLaMA-2-7b 0.2965 0.0637 0.1741 0.5766
LLaMA-2-13b (top-3 RAG) 0.0186 0.0051 0.01821 0.8135

A. Inference from results Fig. 3. Result analysis

Low Rouge Precision - In this case a low rouge precision
indicates that the generated review is significantly smaller than
the reviews provided for validation. It also means only certain
parts of the validation review were taken into account while
generating a game review.
High Rouge Recall - A high rouge recall in this case means
on average, a large portion of the information present in the
reference reviews is captured in the generated review.
High BERTScore - BERTScore[10] is a new metric that
considers contextual embeddings while evaluating text. In this
case, it means that the reference review and generated review
are highly similar in terms of context.
VI. DISCUSSION
In observation, the game review generator suffers from a
small context window. Due to this size, it is only able to fit in
up 2 or 3 of the most relevant game-review chunks (top-k
where k=3) (where each chunk is 512 tokens long). The quality
of game reviews and the number of game review chunks can
be further increased if this context limit is somehow bypassed.
[11]
A. Hardware limitations
This experiment was performed on limited hardware (RTX
3050 4GB laptop GPU). The above- specified context limit
could also have been mitigated by using a GPU with higher
VRAM which would allow us to use a language model with a
larger context window.
VII. CONCLUSION
In summary, context-specific and comprehensible game re-
views were successfully generated using human-written game
reviews stored in a vector database. The game reviews
generated are readable and well formulated, the information
within a game review lies well within what was provided in
human- written documents.
It was observed that LLMs with RAG can be effectively
used for game review generation and summarization tasks
without involving any additional fine-tuning costs. However,
RAG may not take all information into consideration and is
limited by the context length of the underlying LLM.
REFERENCES
[1] Zhang, L., Li, Y., Liu, Z., Liu, J., & Yang, M. (2023). Marathon: A
Race Through the Realm of Long Context with Large Language Models.
arXiv preprint arXiv:2312.09542.V.
[2] Schumacher, D. (2023). V3CTRON— Data Retrieval & Access
System For Flexible Semantic Search & Retrieval Of Proprietary
Document Collections Using Natural Language Queries. Available at
SSRN.
[3] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ...
& Kiela, D. (2020). Retrieval- augmented generation for knowledge-
intensive nlp tasks. Advances in Neural Information Processing Systems,
33,9459-9474.
[4] Braunschweiler, N., Doddipatla, R., Keizer, S., & Stoyanchev, S. (2023).
Evaluating Large Language Models for Document-grounded Response
Genera- tion in Information-Seeking Dialogues. arXiv preprint
arXiv:2309.11838.

Copyright © IEEE–2024 ISBN: 979-8-3503-8354-6 262

Authorized licensed use limited to: Universidade Federal Rural do Semiárido - Campus Mossoró. Downloaded on June 21,2024 at 16:13:36 UTC from IEEE Xplore. Restrictions apply.
 2024 International Conference on Computing, Power, and Communication Technologies (IC2PCT)

[5] Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & [26] Lin, C. Y. (2004, July). Rouge: A package for auto- matic evaluation of
Hashimoto, T. B. (2023). Benchmarking large language models for news summaries. In Text summarization branches out (pp. 74-81).
summarization. arXiv preprintarXiv:2301.13848. [27] Laskar, M. T. R., Fu, X. Y., Chen, C., & Tn, S. B. (2023). Building Real-
[6] AYDIN, Ö . (2023). Google Bard generated literature review: World Meeting Summarization Systems using Large Language Models:
metaverse. Journal of AI, 7(1), 1-14. A Practical Perspective. arXiv preprint arXiv:2310.19233.
[7] Ranjit, M., Ganapathy, G., Manuel, R., & Ganu, T. (2023).
Retrieval augmented chest x-ray report generation using openai gpt
models. arXiv preprint arXiv:2305.03660.
[8] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., ...
& Christiano, P. F. (2020). Learning to summarize with human feedback.
Advances in Neural Information Processing Systems, 33, 3008-3021.
[9] Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas,A., Jones, M., ...
& Gadepally, V. (2023, September). From Words to Watts:
Benchmarking the Energy Costs of Large Language Model Inference. In
2023 IEEE High Performance Extreme Computing Conference (HPEC)
(pp. 1-9). IEEE.
[10] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019).
Bertscore: Evaluating text generation with bert. arXiv preprint
arXiv:1904.09675.
[11] Packer, C., Fang, V., Patil, S. G., Lin, K., Wooders, S., &
Gonzalez, J. E. (2023). Memgpt: Towards llmsas operating systems.
arXiv preprint arXiv:2310.08560.
[12] Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). On
faithfulness and factuality in abstractive summarization. arXiv preprint
arXiv:2005.00661.
[13] Deutsch, D., & Roth, D. (2021, November). Understand- ing the extent to
which content quality metrics measure the information quality of
summaries. In Proceedings of the 25th Conference on Computational
Natural Lan- guage Learning (pp. 300-309).
[14] Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2022). Un- derstanding and
Improving Zero-shot Multi-hop Reason- ing in Generative Question
Answering. arXiv preprint arXiv:2210.04234.
[15] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022).
Large language models are zero-shot reason- ers. Advances in neural
information processing systems,35, 22199-22213.
[16] Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K. W. (2020). A
transformer-based approach for source code summarization. arXiv
preprint arXiv:2005.00653.
[17] Tang, L., Sun, Z., Idnay, B., Nestor, J. G., Soroush, A., Elias, P. A.,
... & Peng, Y. (2023). Evaluating large language models on medical
evidence summarization. npj Digital Medicine, 6(1), 158.
[18] Wu, N., Gong, M., Shou, L., Liang, S., & Jiang, D. (2023).
Large language models are diverse role- players for summarization
evaluation. arXiv preprint arXiv:2303.15078.
[19] Pérez, J., Arenas, M., & Gutierrez, C. (2009). Semantics and complexity
of SPARQL. ACM Transactions on Database Systems (TODS), 34(3),
1-45.
[20] McShane, M. (2017). Natural language understanding (NLU, not NLP)
in cognitive systems. AI Magazine, 38(4), 43-56.
[21] Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., ... & Ge, B. (2023).
Summary of chatgpt-related research and perspective towards the future
of large language models.Meta-Radiology, 100017.
[22] Saito, I., Nishida, K., Nishida, K., & Tomita, J. (2020). Abstractive
summarization with combination of pre-trained sequence-to-sequence
and saliency models. arXiv preprint arXiv:2003.13028.
[23] Völske, M., Potthast, M., Syed, S., & Stein, B. (2017, September). Tl;
dr: Mining reddit to learn automatic summarization. In Proceedings of the
Workshop on NewFrontiers in Summarization (pp. 59-63).
[24] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts,
A., ... & Fiedel, N. (2023). Palm: Scaling language modeling with
pathways. Journal of Machine Learning Research, 24(240), 1-113.
[25] Qi, W., Gong, Y., Jiao, J., Yan, Y., Chen, W., Liu, D., & Duan, N.
(2021, July). Bang: Bridging autoregres- sive and non-autoregressive
generation with large scale pretraining. In International Conference on
Machine Learning (pp. 8630-8639). PMLR.

Copyright © IEEE–2024 ISBN: 979-8-3503-8354-6 263

Authorized licensed use limited to: Universidade Federal Rural do Semiárido - Campus Mossoró. Downloaded on June 21,2024 at 16:13:36 UTC from IEEE Xplore. Restrictions apply.