Technovation 133 (2024) 103002

Contents lists available at ScienceDirect

Technovation
journal homepage: www.elsevier.com/locate/technovation

Future applications of generative large language models: A data-driven case

study on ChatGPT
Filippo Chiarello a, d, *, Vito Giordano a, d, Irene Spada b, d, Simone Barandoni c, d,
Gualtiero Fantoni b, d
a
Department of Energy, Systems, Territory and Construction Engineering, University of Pisa, Italy
b
Department of Civil and Industrial Engineering, University of Pisa, Italy
c
Department of Computer Science, University of Pisa, Italy
d
Business Engineering for Data Science (B4DS) Research Lab, University of Pisa, Italy

A R T I C L E I N F O A B S T R A C T

Handling Editor: Dr Stelvia Matos This study delves into the evolving role of generative Large Language Models (LLMs). We develop a data-driven
approach to collect and analyse tasks that users are asking to generative LLMs. Thanks to the focus on tasks this
Keywords: paper contributes to give a quantitative and granular understanding of the potential influence of LLMs in
Generative artificial intelligence different business areas. Utilizing a dataset comprising over 3.8 million tweets, we identify and cluster 31,747
Generative large language models
unique tasks, with a specific case study on ChatGPT. To reach this goal, the proposed method combines two
ChatGPT
Natural Language Processing (NLP) Techniques, Named Entity Recognition (NER) and BERTopic. The combi­
Social media analysis
Technology adoption nation makes it possible to collect granular tasks of LLMs (NER) and clusters them in business areas (BERTopic).
Emerging technologies Our findings reveal a wide spectrum of applications, from programming assistance to creative content genera­
tion, highlighting LLM’s versatility. The analysis highlighted six emerging areas of application for ChatGPT:
human resources, programming, social media, office automation, search engines, education. The study also
examines the implications of these findings for innovation management, proposing a research agenda to explore
the intersection of the identified areas, with four stages of the innovation process: idea generation, screening/
idea selection, development, and diffusion/sales/marketing.

1. Introduction engendering an unprecedented impact in terms of user numbers, esti­

In the rapidly evolving landscape of artificial intelligence (AI),
generative large language models (LLMs) have emerged as a pivotal
innovation. Language models are probabilistic models that enable the
processing of natural language through algorithms, and they are the core
of the natural language processing (NLP) techniques. The term ‘large’
refers to the huge number of parameters needed to train these models,
whereas the term ‘generative’ indicates a subset of LLMs specifically
designed to generate text. Generative LLMs are at the forefront of
numerous applications, ranging from summarisation to translation,
typically delivered through dialogue-like interactions with the user
(Zhao et al., 2023).
The dynamic of the dialogue eases the interaction of users on the
system (Dang et al., 2022). According to the Technology Acceptance
Model, perceived ease of use and usefulness are the factors that influence
user acceptance of technology (Davis, 1989). Indeed, these systems are
mated to be over 100 million in January 2023 (Huang et al., 2023). The
rise of generative LLMs has been led by ChatGPT, the openAI conver­
sational system built on the generative pre-trained transformer (GPT)
model launched in late 2022. Other relevant generative LLMs are
Microsoft’s Bing, which focuses on interacting with the web to provide
answers based on present information, and Google’s PaLM, known for its
improved multilingual, reasoning, and coding capabilities. Meta’s
LLaMA is another contender in the space, with an approach to
multi-modal learning that integrates both text and image data.
Despite already reaching millions of users, generative LLMs are still
in their infancy, aligning with what innovation literature defines as
emerging technologies. These are characterized by radical novelty, rapid
growth, coherence, prominent impact, and a degree of uncertainty and
ambiguity (Rotolo et al., 2015). Moreover, these new generative models
have found applications across a wide spectrum of fields, making them
general-purpose technologies (Bresnahan and Trajtenberg, 1995).

* Corresponding author. Department of Energy, Systems, Construction and Territory (DESTEC), Largo Lucio Lazzarino, 56122, Pisa, PI, Italy.
E-mail address: filippo.chiarello@unipi.it (F. Chiarello).

https://doi.org/10.1016/j.technovation.2024.103002
Received 27 February 2023; Received in revised form 7 February 2024; Accepted 26 March 2024
Available online 29 March 2024
0166-4972/© 2024 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
F. Chiarello et al.
Earlier NLP models were often highly task-specific, whereas newer NLP
technologies exhibit remarkable versatility and can seamlessly handle a
variety of tasks (Bouschery et al., 2023).
The success of these models is not solely determined by the quality of
their output and their general-purpose nature (refer to Section 2 for
more details), but primarily by the way users can interact with them.
Fig. 1 elucidates this interaction. A user starts by defining a task. The user
then interacts with the model by describing a task (e.g. writing a research
paper, generating novel ideas for a market advertising, or debugging a
Python code) via a prompt (i.e., a request made to the generative sys­
tem), which is then sent to the model. The model analyses the prompt
and responds with a textual output. Since the user expects a certain result,
the result is compared with the actual output. The user is then able to
iterate a new prompt to refine the output, thus engaging in a form of
conversation with the system until the output does not match the ex­
pected result.
At present, generative LLMs have characteristics that limit the pos­
sibility to study their behaviour and how users are interacting with
them. First, the systems are black boxes in the sense that their huge size
(i.e., 175 billion parameters for ChatGPT) limits their explainability, a
problem largely studied in the field of explainable artificial intelligence
(XAI) (Barredo Arrieta et al., 2020). Second, the prompts and responses
of the models are proprietary to the owner companies. Some websites
that are collecting this information are emerging, but the information
stored is still limited and unable to give a full understanding of LLM
behaviour.
LLMs are poised to change people’s relationship with technology and
with AI, a phenomenon that is challenging to study. Considering the
growing importance of the topic of AI for the innovation management
community (Mariani et al., 2023), innovation researchers are expected
to better elucidate the evolution of generative LLMs. Given the chal­
lenging characteristics of the technology, we identified a research gap
that lies in the point where the user interacts with the systems: the tasks
(i.e. what generative LLMs are asked to do). This problem has been
identified for AI in general, but no clear research path has been identi­
fied to solve it (Cho et al., 2023).
We aimed to address this gap by analysing descriptions of task that
users are asking generative LLMs. Specifically, we sought to answer the
following research question: Which are the tasks that users are asking
generative LLMs? Answering such a question has implications both for
theory and practice. A clear map of LLM tasks, can enable researchers to
study how users are interacting with LLMs while this technology is
emerging. A quantitative view in this direction is essential to comple­
ment the scientific discourse on the topic, that given the impossibility to
have access to first hand data, risks to focus only on opinion papers and
qualitative research. Also, companies can explore the map to understand
how LLMs can be used in their context, and to spot priorities of users that
are early adopters of the technology. Given the paste at which LLMs has
pervaded the market, companies are missing maps to explore this un­
known territory. From the theoretical side, developing the correct
metrics on the collection of tasks can enable to very if and how much
emerging and general-purpose technologies as LLMs fit to consolidated
innovation theories.
We conducted a case study based on textual analysis of Twitter data
(renamed X from July 2023 but called Twitter in the rest of the paper for
the sake of simplicity) on ChatGPT. Textual analysis of Twitter has been
Fig. 1. A model for the interaction between a user and a Generative LLM. Verbs on the arrows.
2
Technovation 133 (2024) 103002
adopted in literature using various methods for analysing emerging
technologies. Some methods are based on bibliometric data, such as
citation analysis (Small et al., 2014), whereas others are based on textual
data (Porter et al., 2019; Ranaei et al., 2020), such as patents, scientific
paper, and social media. The analysis of textual data with NLP out­
performs bibliometric analysis (Arts et al., 2021), and Twitter has been
confirmed to be a reliable source to explore trends in product develop­
ment (Ozcan et al., 2021). The choice of Twitter was also informed by
our aim to concentrate on the tasks users assign to ChatGPT. Many re­
searchers have utilized Twitter to capture this user feedback, even to
analyse user opinions on emerging technologies (Kohl et al., 2018; Li
et al., 2019, 2022). Finally, we selected ChatGPT considering its wide­
spread adoption.
We collected 3,821,843 tweets talking about ChatGPT published
between November 2022 and May 2023. Using named entity recogni­
tion (NER) techniques, a branch of NLP, we filtered the tweets
describing tasks assigned to ChatGPT and isolated the textual descrip­
tion of these tasks. Next, we clustered the tasks using a topic modelling
technique based on bidirectional encoder representations from trans­
formers (BERT), namely, BERTopic (Grootendorst, 2022), to identify
groups of similar tasks. We thus obtained a map of the uses of ChatGPT
and discussed the emerging areas of its application. With this map, we
identified the business areas where ChatGPT is having an impact in
terms of users’ requests, as well as the opportunities for research and
innovation in the realm of generative LLMs.
Our findings show that NLP can help having a clear view of the dy­
namics that stands at the intersection of an emerging technology and
users. NLP has proven to be a proper method to identify relevant part of
texts from large social media databases (thanks to NER) and to synthe­
sise this information to give a map of LLMs tasks (thanks to topic
modelling). Empirically, the proposed map will help orient towards
LLMs applications in business. Our findings uncover a broad spectrum of
ChatGPT applications, ranging from programming assistance to creative
content generation, underscoring the versatility and potential of LLMs in
revolutionizing business processes and services. We identified six
emerging areas of significant application: human resources, program­
ming, social media, office automation, search engines, and education.
The integration of the results with the phases of innovation, will help
researchers focus on promising areas that are having an impact on users
practice with LLMs. In fact, we synthesized the research opportunities in
a preliminary research agenda and highlighted future research chal­
lenges that innovation management scholars will need to face to answer
the proposed questions.
Finally, our study also contributes to open research on the topic, to
incentivise quantitative research, by opening the collected dataset and
analysis (https://github.com/NutrientAxis/tasks_asked_to_ChatGPT).
2. Technological advancement of large language models
LLMs represent a class of NLP technologies that leverage extensive
textual data for both understanding and generating natural language. To
elucidate how these models operate and their key features, we con­
ducted a review of the major technological advancements in NLP from
its inception.
One of the first and simplest algorithms applied to extract informa­
tion from texts is the bag-of-words (BoW). It operates by representing a
F. Chiarello et al.
document as a collection (‘bag’) of its constituent words, with no
consideration for the order or structure of the words in the document
(Hofmann et al., 2019). To address this limitation, researchers have used
two-dimensionality reduction techniques, namely, latent semantic
analysis (LSA) and singular value decomposition (SVD). Another
commonly used dimensionality reduction technique is latent Dirichlet
allocation (LDA). LDA is employed to extract topics from a collection of
documents and then classify them based on the distribution of these
topics (Hannigan et al., 2022). Recently, NLP research has shifted its
focus towards word-embedding algorithms, which are trained on large
amounts of textual data to capture the semantic meaning of words.
These algorithms can be categorised as LLMs, representing a significant
leap in NLP technology (Ameye et al., 2023; Just, 2024; Song et al.,
2023). Word-embedding techniques leverage artificial neural networks
to represent words as dense semantic vectors of real numbers (i.e. vec­
tors that embed the meaning of words) (Mikolov et al., 2013). Tradi­
tional word embeddings provide a single vector for a given word,
regardless of context and possible polysemy. In the past few years, a
novel class of LLM algorithms has been created, transformers (or
contextual word embeddings), of which BERT (Devlin et al., 2018) is one
of the first and more popular architectures. Transformers address pre­
vious system limitations by considering the textual context of words or
phrases.
One of the first generative LLMs was GPT. Introduced by OpenAI in
2018, GPT employs the power of transformer architecture, such as
BERT, to generate natural language. Over the years, the LLM field has
witnessed significant advancements, leading to the emergence of a new
category of GPT technologies, notably OpenAI’s GPT-3 (Ouyang et al.,
2022). GPT-3 stands out as a large, task-agnostic model capable of
performing a wide range of NLP tasks based on user prompts presented
in natural language (see Section 1 and Fig. 1 for more details). Models
like GPT-3 are trained on extensive document corpora, enabling them to
learn the distribution of words in text and generate new text sequences
(Reynolds and McDonell, 2021). In November 2022, OpenAI launched a
conversational AI, called ChatGPT (Van Dis et al., 2023), built on GPT-3
technology. Following the success of GPT-3 and ChatGPT, several other
companies have ventured into the development of generative LLMs.
Notably, Google introduced PaLM2, and Meta unveiled LLaMA. In
March 2023, OpenAI released GPT-4, a powerful language model
trained on a substantial number of documents (Sanderson, 2023).
OpenAI is actively working on launching another model, GPT-5, which
is expected to be trained on an even larger and more extensive dataset.
Generative LLMs are in their infancy, aligning perfectly with the
definition of emerging technologies in innovation literature. These are
characterized by radical novelty, relatively rapid growth, coherence,
prominent impact, and a degree of uncertainty and ambiguity (Rotolo
et al., 2015). The radical novelty of LLMs is evident in the model’s ability
to understand and generate human-like text, pushing the boundaries of
natural language processing and artificial intelligence. This is not a mere
incremental improvement over existing technologies but a significant
leap forward, introducing capabilities previously thought to be exclusive
to human intelligence (Makridakis et al., 2023).
For what concern rapid growth, Generative LLMs reached millions of
users in just a few months (Burtsev et al., 2023), a pace of adoption that
is both remarkable and indicative of their transformative potential. The
rapid growth of Generative LLMs can be attributed to several factors.
Firstly, the technology itself is highly scalable. Once developed,
deploying these models to serve additional users often requires rela­
tively marginal increases in resources compared to the initial investment
in training and infrastructure. This scalability allows LLMs to be made
accessible to a broad audience swiftly. Moreover, the growth of LLMs is
pushed by the network effects in the technology ecosystem. As more
users engage with LLMs, more data becomes available, which, in turn,
can be used to train and refine the models, making them more accurate
and effective. This self-reinforcing cycle contributes to the rapid
improvement and adoption of LLMs, making their growth trajectory
3
Technovation 133 (2024) 103002
even faster. Finally, the versatility and broad applicability of LLMs have
powered their rapid adoption.
The last aspect is also linked to another characteristic of emerging
technology: prominent impact. In fact, these new generative models
have found applications across a wide spectrum of fields, such as sci­
entific writing (Else, 2023; Stokel-Walker, 2023), chemistry (Hocky and
White, 2022), journalism (Pavlik, 2023), psychology (Lee et al., 2022),
human resources management (Budhwar et al., 2023), healthcare (Sal­
lam, 2023), and education (Graham, 2022; Stokel-Walker, 2022). Other
reports have used these models in experiments, such as programming
code, writing songs, and writing cover letters (Chatterjee and Dethlefs,
2023).
While its potential is vast, there are also significant challenges and
uncertainty regarding its future development, governance, and broader
implications. Issues related to bias, fairness, accountability, and the
potential for misuse (Abadie et al., 2024; Ali et al., 2024) necessitate
careful consideration and ongoing research. This uncertainty nature is
typical of emerging technologies.
When considering the four distinct phases of the innovation process
identified by Salerno et al. (2015), namely, idea generation, idea se­
lection, development, and diffusion, generative LLMs have been
employed for diverse purposes. In the idea generation phase, Piller et al.
(2023) employed generative LLMs to enhance the creative process.
Moving to the idea selection phase, Hong et al. (2022) introduced sys­
tems for screening ideas to facilitate the early stages of technological
development, leveraging patent analysis and Word2Vec. In the context
of the development phase, Deloose et al. (2023) created systems that
classify failures within industrial contexts. As for the diffusion phase,
Arts et al. (2021) discovered that NLP technologies outperform other
approaches in assessing the impact of new technologies on the market.
In recent years, an increasing number of literature reviews have
explored the adoption of NLP and LLMs in innovation. Antons et al.
(2020) conducted a comprehensive review of the state-of-the-art NLP
systems employed in innovation research. Siddharth et al. (2022) delved
into the literature to examine NLP applications in design, shedding light
on their role in enhancing the creative and design processes. Just (2024)
reported deeper insights into how NLP approaches enrich intermedia­
tion in the early stages of innovation search. These reviews collectively
underscore the growing significance of NLP and LLMs in innovation
research and their capacity to revolutionise each of the innovation
phases.
3. Methodology
Fig. 2 presents the methodological steps for analysing the tasks
performed by ChatGPT. First, we collected tweets related to ChatGPT;
second, we used NER techniques to identify the tasks that users assigned
to ChatGPT; third, we normalised the identified tasks; and finally, we
clustered the tasks using BERTopic.
3.1. Twitter data collection
We collected 3,821,843 tweets using the Twitter API. Twitter queries
used for data extraction included the following keywords: ‘chat-gpt’,
‘chat gpt’, ‘chatgpt’, ‘#chatgpt’, and ‘#gpt3’. Other generative AI sys­
tems, such as AgentGPT and Google Bard, exist, and new ones may
emerge in the future. Nevertheless, given that OpenAI, with its product
ChatGPT, is the first mover in this field, we hypothesised that replicating
the success and popularity of ChatGPT could be challenging. For
instance, in May 2023, the tweets about AgentGPT (searched as
#agentgpt) and Google Bard (#googlebard) were only 2399 and 11,688,
respectively. For this reason, we assumed that our search strategy would
be robust. The data collection addressed tweets published from
November 3, 2022, when OpenAI’s ChatGPT was launched, to May 16,
2023. The dataset we collected from Twitter spanned a period of six
months. The dataset included only English tweets.
F. Chiarello et al. Technovation 133 (2024) 103002

Fig. 2. Flow diagram of the adopted methodology.

Twitter is recognised as a reliable tool for studying emerging tech­

Table 1
nologies (Kohl et al., 2018; Li et al., 2019, 2022). The temporal
Rules used for building the NER systems able to map the tasks.
perspective can vary widely, ranging from recent monthly analyses to
decades-long timeframes (Lacasa et al., 2003). Compared with these Technology Connection Action Verb Object
Element
studies, our research analysed 3,821,843 tweets, much larger than that
in other studies that adopted similar approaches: 3646 tweets in Li et al. gpt, (gpt), #gpt, to, explains, can, Optional Direct Object
(#gpt), gpt3, could, about, for, Adverb + Verb (excluding: me,
(2019), 29,272 tweets in Li et al. (2022), and 1,963,905 tweets in Kohl
(gpt3), #gpt3, may, might, + Optional you, it, him, her,
et al. (2018). This comparison illustrates that despite the relatively short (#gpt3),gpt-3, knows how, Word (to, us, them, this,
temporal window (about six months) of our study, our research aligns (gpt-3), gpt 3, (gpt knows how to, explains, can, that, those,
with other works that utilize Twitter to analyse emerging technologies. 3), #gpt-3, (#gpt- how to, can be could, about, these)
3), chatgpt, used to, can be for, may, might)
(chatgpt), used for, can be + Optional
3.2. Task identification with named entity recognition #chatgpt, used at, can be Article +
(#chatgpt), used in, can be Optional
chatgpt3, able to, can be Possessive
We analysed the Twitter data collected in Section 3.1 using NER (chatgpt3), chat able for, can be Adjective +
techniques to identify the tasks assigned to ChatGPT. We used NER, an gpt, (chat gpt), able at, can be Optional
NLP technique to sort specific entities in the text into specific categories. chat-gpt, (chat- able in, can be Adjective +
gpt), #chat-gpt, capable in, can be Optional
For a comprehensive understanding of the different NER techniques,
(#chat-gpt), chat- capable for, can Compound
refer to the work of Puccetti et al. (2023), section 3.2. gpt3, (chat-gpt3), be capable at, can
To the best of our knowledge, the literature on NER systems for #chat-gpt3, be capable of, can
extracting tasks performed by technology is limited. The NER methods (#chat-gpt3), chat help in, can help
aiming at categorising technical entities focus on extracting technolog­ gpt3, (chat gpt3), for, can help at,
#chatgpt3, can support in,
ical concepts, such as technologies (Giordano et al., 2021), functions (#chatgpt3), can support for,
(Yoon et al., 2013), chemical compounds (Ohms, 2021), and biomedical chatgpt-3, can support at,
entities (Huang et al., 2020). None of these works attempts to recognise (chatgpt-3), ai, how to help in,
the entity ‘task’ in the textual data. Only the work of Choi et al. (2021) artificial how to help for,
intelligence how to help at,
used rule-based NER to identify actions performed by humans in pro­
how to support in,
cedures. Nevertheless, they tailored their system to recognise tasks how to support
exclusively within a specific textual domain (procedures), within a for, how to
particular context (the nuclear field), and for a distinct case study. This support at, is
specific design limits the applicability of Choi et al.’s (2021) system to capable of, is
capable in, is
our Twitter dataset. Indeed, tweets have a conversational tone different capable to, is used
from the formal text structure of procedures. For these reasons, we to, is used for, is
developed a new rule-based NER instead of using state-of-the-art sys­ used in
tems, given that existing systems may not have been specifically trained
to identify the specific entities (‘task’) that are the focus of our research.
expressions, which can lead to incorrect task identification. Moreover,
We manually analysed 1000 tweets and then identified a recurring
for the clustering of similar tasks in the next step, we filtered the noisy
pattern for expressing a task in natural language:
ones with a manual revision of the clusters, reaching a theoretical pre­
[ChatGPT TECHNOLOGY] + [CONNECTION ELEMENT] + [ACTION cision of 100% of the overall process (i.e. only actual tasks performed by
VERB] + [OBJECT] ChatGPT were manually included in the analysis). Finally, we applied
the rule-based NER to the tweets.1 We extracted 526,861 tasks from 496,
This pattern indicates that users tend to express tasks for ChatGPT by 099 tweets, and we collected 232,449 unique tasks.
writing the name of the technology, followed by a variable sequence of The NER method has proven to be effective in achieving the desired
words (the ‘connection element’), an action verb (expressed through a outcomes. However, some limitations should be highlighted. Firstly, if
verb and other optional words, e.g. adverb, articles, and adjectives), and there is a shift in the type of entity being extracted (for instance, from
a direct object. Table 1 lists all the identified terms and sequences. We tasks to technology), it will be necessary to modify the extraction rules
created the rule-based approach by creating all possible combinations of accordingly. For guidance on extracting technology entities, one can
these elements. refer to Puccetti et al. (2023); for user entities, to Bonaccorsi et al.
We measured the precision and recall performance of the rule-based (2021); and for user needs, to Spada et al. (2023). Secondly, NER sys­
NER system (Tsai et al., 2006) using a sample of 1000 tweets that was tems that utilize machine learning and LLMs for text representation have
manually checked as a golden set. We specified whether a tweet dis­ been shown to outperform rule-based extraction methods across various
cussed a task performed by ChatGPT. The system had a precision of 0.59 NER tasks, as highlighted by Puccetti et al. (2023). However, adopting
and recall of 0.44. The lower recall performance did not negatively this approach requires the creation of a gold standard dataset, and the
affect our research, which aimed to elucidate the utilisation of LMMs, costs associated with this process are deemed to exceed the potential
not to create the most effective NER system for task identification. In benefits in terms of increased precision and recall for the system needed
other words, the low recall performance is not a problem given that the
macroscopic phenomena can still be observed. Regarding precision, the
pre-processing phase, described in Section 3.3, was designed to address
the challenges posed by informal language, misspellings, and colloquial 1
https://spacy.io, accessed 2 June 2023 6 February 2023.

4
F. Chiarello et al. Technovation 133 (2024) 103002

in the present work, as previously discussed.
3.3. Data pre-processing pipeline
We then processed the list of 232,449 tasks with a data pre-
processing pipeline for removing noise and normalising its text. The
process consisted of the following steps:
1. Lemmatisation;
2. Removal of adverbs as terms that do not provide any meaningful
information about the task;
3. Identification of the groups of tasks in our set that were synonyms
(terms written with a different spelling but having the same mean­
ing) using semantic similarity methods. These algorithms mapped
each task to a vector in a 1024-dimensional semantic space using the
BERT model stsb-distilbert-base2 (Devlin et al., 2018) and calculated
cosine similarity between each task pair. Tasks were grouped
together if their cosine similarity was greater than 0.92;
4. Automatic removal of tasks occurring less than six times;
5. Manual removal of entities that could not accurately be classified as
‘tasks performed by ChatGPT’ but were nonetheless extracted by our
algorithm. The entities list was manually screened by three of the
authors.
Subsequently, the list of tasks consisted of 31,747 unique tasks
identified in 262,192 tweets, representing 6.86% of the starting data­
base of tweets. The pre-processing steps used in the proposed method­
ology are largely used in text-mining techniques and have been proven
effective in the context of technology and innovation (Antons et al.,
2020).
require specifying the classes for grouping tasks a priori, information
that was not available before our study. Lastly, the semantic map
employed to present the results relies on dimensionality reduction via
the Uniform Manifold Approximation and Projection (UMAP) tech­
nique, as described by McInnes et al. (2018). This choice, too, is based on
empirical judgment. It’s worth noting that other visualization tech­
niques might reveal additional nuances of the data, offering further in­
sights into our findings.
4. Results and discussion
By applying the methodology outlined in Section 3, we identified
262,192 tweets that described a task (6.86% of the starting database of
tweets) and a total of 31,747 unique tasks. These tasks were then clus­
tered using the BERTopic algorithm into 389 topics; a topic consisted of
a group of semantically similar tasks (e.g. Writing a poem, Writing a haiku,
Generating a poem, Writing a sonnet, and Creating a poem). The label of the
topic was assigned based on the task with the greatest number of tweets.
Table 2 shows the top 10 topics, in terms of the number of tweets citing a
task in the topic, along with the total number of tweets, number of
different tasks, and top five tasks in terms of the number of tweets. In
this section, we explore each line of the table, establishing connections
with theories of innovation. To ensure brevity, we discuss some lines
collectively. The methodology for grouping tasks for discussion was
determined by the authors, who evaluated the task descriptions
Table 2
Top 10 topics of tasks mentioned in tweets.
# Topic Number of Number of Task (number of tweets citing
tweets tasks the task)

3.4. BERTopic 1 Write code 7549 578 write code (2,823), generate
code (889), explain code (118),
produce code (114), write good
We clustered the 31,747 tasks using the BERTopic algorithm code (94)
(Grootendorst, 2022). BERTopic is a clustering technique leveraging 2 Write 5452 258 write article (1,413), write tool
word embeddings and has excellent effectiveness for the topic modelling article (738), write something (692),
of tweets (Egger and Yu, 2022). The method has also been used in the write a text (408), write part
(159)
context of technology mapping and exploration (Jeon et al., 2023).
3 Write a 4569 149 write a poem (3,356), create
Details on the mathematical model and usages of the approach have poem poem (140), generate poem
been reported by Grootendorst (2022). (135), make poem (134), write a
In our study, the algorithm first generated document embeddings positive poem (102)
using the transformer all-MiniLM-L6-v2, which could map each task into 4 Answer 4244 260 answer questions (1,979), raise
questions question (157), responded to
a 384-dimensional dense vector space. Second, it reduced their dimen­ questions (139), answer some
sionality and clustered them based on semantic similarity (calculated search queries (130), answer
through the cosine similarity between vectors). Finally, it provided a follow-up question (119)
topic representation. In this way, we obtained a set of topics, with each 5 Write story 3647 220 write story (1,528), write a short
story (357), tell a story (155),
topic referring to a group of semantically similar tasks. We then analysed
generate story (133), write a
the distribution of these topics on a semantic map and the number of novel (127)
tweets associated with each topic for measuring popularity and 6 Give 3537 225 gives answers (1,272), give great
relevance. answers answers (594), generate answer
BerTopic proven to give the desired output. Anyway, the decision to (314), know answer (123), gives
better answers (121)
utilize the transformer model, all-MiniLM-L6-v2, and its corresponding
7 Write a 3510 174 write a tweet (1,638), create a
384-dimensional dense vector space is based on empirical evidence. As tweet tweet (579), write a Twitter
of now, there are no studies that definitively prove this approach to be thread (148), make a tweet (92),
the optimal solution for the task at hand. Furthermore, like the con­ reply to tweets (90)
8 Write an 3100 101 write an essay (2,270), wrote
straints encountered with rule-based NER, the performance of the
essay essays (114), writes smart essays
grouping task could be enhanced by having a golden set of similar tasks (112), generate essay (81),
for reference and switching from clustering to classification. However, generate essays (59)
the creation of such a labelled dataset falls outside the scope of this 9 Replace 2820 325 replace developer (177), replace
paper. We also believe that the potential improvement in performance developer human jobs (172), replace
programmers (168), replace
would not justify the costs associated with manually annotating the
writer (143), replace lawyer
data. Furthermore, adopting classification instead of clustering would (109)
10 Write an 2374 193 write an email (1,001), generate
email emails (232), draft an email (96),
2 write work emails (90), write
https://huggingface.co/sentence-transformers/stsb-distilbert-base,
blogs newsletter (57)
accessed 2 June 2023 6 February 2023.

5
F. Chiarello et al.
reviewing the whole set of tasks available in the dataset and considered
the characteristics of the document cited in the task, such as its length,
the technical nature of its content, and its specific field.
Lines 1 and 3 contain the topics Write code and Write a poem,
respectively, that properly represent the versatility of ChatGPT and its
span of uses. This variety of tasks demonstrated its widespread accep­
tance across different domains. This aligns with Diffusion of Innovations
(Rogers, 2004), indicating that ChatGPT is traversing the first stages of
the innovation adoption lifecycle. Write code shows that in the Infor­
mation Technology and Software Development sectors, AI-driven code
generation and optimisation is significantly streamlining development
processes, reducing the time and resources required for debugging and
optimising code. In parallel, the impact it will have on
creativity-intensive document production, such as poems, poses big
challenges for creative tasks.
Line 2 shows the topic Write article, a generic one that contains ab­
stract tasks (e.g. Write something). Thus, many users are describing tasks
with a high level of abstraction and are playing with ChatGPT to explore
its performance. They are having their first interaction with generative
LLMs through ChatGPT. ChatGPT’s purpose is to open up the power of
generative AI—already well-known by researchers (Radford et al.,
2018)—to a wider public. The high level of abstraction points to the
initial exploratory stage of technology adoption, reflecting the perceived
ease of use and curiosity-driven engagement, which are core aspects of
the Technology Acceptance Model (Davis, 1989).
The next topics on lines 4 and 6 are focused on one of the tasks that
ChatGPT has proven to be good at (i.e. Answer questions and Give An­
swers), so much so that it poses a threat to search engines, such as
Google. These tasks are so important that ChatGPT is perceived by users
as a potential game changer in the search engine market, which has been
a monopoly. Owing to a large investment in OpenAI, Microsoft is
seeking to gain market share in this context, threatening the leading
position of Google. Although ChatGPT is not a search engine, the
conversational interaction (i.e. without the need to write a query but
asking the engine questions) it uses shows a potentially new class of
products, indicating potential for disruptive innovation. By providing
conversational interaction, ChatGPT may redefine user expectations and
behaviours in information retrieval, challenging established players in
the market (Bower and Christensen, 1995).
Lines 5 and 8 give the results for the tasks Write story and Write essay,
respectively. These tasks pertain to two different types of content (the
first related to narrative writing, the second to technical writing) that
are longer than the ones described before. This shows the ability of
ChatGPT to cope with longer texts. ChatGPT (and all LLMs in general)
has some limitations in terms of the length of the output texts (Zhao
et al., 2021), which may be surmounted in the future. Notably, the po­
tential improvements in handling longer text outputs suggest incre­
mental innovations in the capabilities of transformer-based systems like
ChatGPT. Also, its ability to impact diverse fields, such as software
development, creative writing, and search engines, hints at radical in­
novations that could reshape entire industries (Dewar and Dutton,
1986). For example, in the Publishing and Education sectors, AI’s ability
to generate longer, complex texts can transform content creation. For
publishers, the technology introduces questions of authenticity and
copyright in AI-authored works. In education, it presents opportunities
for personalised learning materials but also challenges in assessing
students’ original work amidst AI-generated content.
Line 7 gives the topic Write a tweet, which reflects the impact that the
tool could have on social media. The fact that ChatGPT can generate
social media posts can destroy the content industry, by making content
creators into prompt creators. Impacts, which also emerge from other
tasks we analysed, can be negative in the production of disinformation
pieces or generally low-quality content. This diversity of tasks—from
creating social media content to writing stories and essays—shows how
users are adapting ChatGPT for their unique needs and aligns with the
concept of user-driven innovation, where the users themselves play a
6
Technovation 133 (2024) 103002
critical role in the development and application of new technologies
(Von Hippel, 1986). As more users engage with ChatGPT for a variety of
tasks, the better would network effects come into play. The value of the
AI system increases with the number of users, as it learns from diverse
input and improves its performance across tasks (Abrahamson and
Rosenkopf, 1997).
Finally, lines 9 and 10 give topics in the realm of office automation.
The topic Replace developer highlights a common perception among
users that AI could potentially substitute humans in performing tasks in
various fields (see topic 1). Indeed, AI can automate certain tasks.
However, AI is not intended to replace humans but rather augment their
capabilities. For example, the ability to generate, draft, and even
personalise emails can greatly enhance office productivity, allowing
workers to dedicate more time to tasks that require human creativity
and strategic thinking. The idea that AI can replace certain tasks but not
entire jobs is reminiscent of the job-task approach (Autor, 2013). This
theory suggests that automation is more likely to replace specific tasks
within a job rather than the entire job itself, leading to a change in jobs
and the skills required.
Fig. 3 shows the results of the topic modelling represented as a
scatter plot, offering a bird’s-eye view of the collected topics of tasks and
their similarities. In the figure, all the 31,747 extracted tasks are located
in a bi-dimensional space and grouped in topics represented by colours.
We performed dimensionality reduction through the Uniform Manifold
Approximation and Projection technique (McInnes et al., 2018). In
Fig. 3, each task is represented by a dot, and the topics of tasks also
present a label, indicating the main task that represents the group. The
distances between the tasks (and between the topics, which are clusters
of tasks) in the scatter plot are calculated through their semantic simi­
larity. If two topics are semantically similar, they are represented close
to each other in the graph. The grey dots represent the outliers topic,
which are 8156 tasks for which BERTopic could not find a suitable
cluster.
The map in Fig. 3 offers data-driven views of the different business
areas of development of tools such as ChatGPT, highlighting six
emerging areas of application: related to tasks able to improve pro­
ductivity (automation for repetitive office tasks), tasks belonging to
human resources (write a resume), programming (writing and explain­
ing code), social media (from tweets to blogs to jokes), education (pass
exams, write essays, generate academic paper), and search engines
(answering questions, replacing Google search).
5. Research agenda
Building on the empirical evidence of Section 4, we formulated a
series of research questions to guide further studies on the potential
impacts of generative LLMs on innovation management. To transition
from our empirical findings to a structured research agenda, we per­
formed an in-depth analysis of each topic (the top 10 shown in Table 2,
and the 379 in others). We adopted a dual-dimensional framework,
intertwining the six emerging areas of application shown in section 4
with the four main phases of the innovation process as defined by
Salerno et al. (2015). This approach allowed us to map a matrix that
captures the intersection of generative LLMs’ applications with the
stages of innovation, from ideation to market penetration. Table 3
synthesises the proposed research questions. Each emerging area is
linked to the empirical evidence by showing the two most related topics
(Table 2).
Scholars and practitioners have already engaged in discussions sur­
rounding some of the points in the identified agenda, as shown by
Mariani et al. (2023). Nonetheless, we expected the proposed research
agenda to contribute to the literature. First, previously identified chal­
lenges were unrelated to generative LLMs but were discussed for AI in
general. This specific type of technology was our focus, given its impact
in the technology development field and on society in general. Second,
the challenges that we identified in relation to generative LLMs emerged
F. Chiarello et al. Technovation 133 (2024) 103002

Fig. 3. Clusters of ChatGPT tasks mentioned by users on Twitter.

from a quantitative study. This will give the opportunity to the research the time needed to bring new software products to market. In turn,
community to monitor these challenges and compare them with others barriers to entry for software development become lower.
derived from different research approaches (e.g. qualitative ones). Social Media: Researchers working on the use of generative systems
for innovating the social media content production process should bal­
ance the potential of automated content generation with ethical con­
5.1. Business-area challenges
cerns, such as the spread of disinformation, filter bubbles, and
manipulation of public opinion. When studying novel business models,
Human Resources: HR is a knowledge-intensive area. Thus, it will be
researchers should ensure that the new approaches respect user privacy,
challenging to guide and monitor the impact that generative LLMs will
adhere to data protection regulations, and avoid intrusive data
have on the production of HR-related documents (e.g. job postings, job
collection.
profile descriptions, and curricula). Given also that this kind of docu­
Office Automation: The ability of generative AI to support specific
ment is private, researchers must monitor and help companies in
office tasks, especially knowledge-intensive and domain-specific ones,
adapting to the ever-evolving HR practices and regulations of AI-
will depend on how much companies can train generative LLMs on
generated solutions in HR. The dynamic nature of HR practices, legal
proprietary data. The lack of domain-specific data can limit the accuracy
regulations, and ethical standards can be expected to increase in the next
and effectiveness of these technologies in performing certain tasks,
decade, requiring continuous updates and fine-tuning of both HR prac­
which can hinder its adoption and impact. Scholars will need to inves­
tices and AI models.
tigate approaches for acquiring and sharing domain-specific data,
Programming: Elucidating the impact of generative LLMs on software
including data-sharing agreements and incentives for companies to
innovation will required multidisciplinary research that integrates the
share their data.
domains of management, innovation, and computer science. The inte­
Search Engines: Generative AI is progressively integrated with search
gration of generative LLMs could impact the industry of integrated
engines, which will pose an important challenge for innovation re­
development environments and other software development tools that
searchers who are monitoring the rapid evolution of this market. In this
could revolutionise the way developers work. AI-generated code sug­
business area more than the others, researchers will need to use various
gestions, real-time debugging, and code optimisation could become
methods ranging from qualitative strategies (e.g. case studies and in-
standard features in these tools, improving productivity and reducing

7
F. Chiarello et al. Technovation 133 (2024) 103002

Table 3
Opportunities for future research on generative LLMs in Innovation.
Idea generation Screening, Idea selection Development Diffusion, Market, Sales

Human How can generative LLMs aid in
Resources generating new HR policies and new
(Topics 9, 10) talent acquisition strategies?
How much will Generative LLMs
impact the production of HR-related
documents (e.g. resumes, job
postings, job descriptions)?
Programming To what extent can generative LLMs
(Topics 1, 9) be used for generating software
debugging ideas in brainstorming?
Will generative LLMs be able to
come out with novel software ideas?
Social Media How much will generative LLMs
(Topics 5, 7) change the way in which social
media content is generated?
How can generative LLMs
contribute to new strategies for
combating disinformation?
Office How much of the creative potential
Automation of generative LLMs will be disclosed
(Topics 2, 10) at an office automation level?
How can generative LLMs facilitate
the identification of new
opportunities for AI in office
automation?
Search Engines How will the combination of
(Topics 4, 6) generative LLMs and standard web-
search change the idea generation
process?
Will generative LLMs inspire the
creation of specialised search
engines or AI assistants?
Education Will generative LLMs support the
(Topics 8, 6) ideation of new educational
programs?
Will generative LLMs radically
change the way in which creativity
is taught?
How can generative LLMs streamline
HR processes for efficiency?
Will generative LLMs help the HR
function in the screening and selection
of candidates?
How can generative LLMs aid in the
selection of programming languages,
development frameworks, and
libraries?
How can generative LLMs support the
evaluation of software development
projects?
How much will generative LLMs
cooperate with people in the
assessment/selection of social media
content?
How can generative LLMs aid in the
selection of the most effective social
media channels?
To what extent will the performance of
idea selection and assessment be
implemented in the office automation
products based on generative LLMs?
How can generative LLMs assist in
selecting the most suitable office
automation tools?
Will generative LLMs enhance
standard search systems, making it
possible to move from information
search to information assessment?
How much can generative LLMs assist
in prioritising the most relevant search
results for users?
How will generative LLMs support
educators in the selection of
educational materials?
How can generative LLMs help in
evaluating the effectiveness of
educational activities?
Will generative LLMs create new
product opportunities in the context
of HR (e.g. CV analysis, candidate
selection software)?
How can generative LLMs help
create personalised employee
development plans?
What type of new programming
languages (high-abstraction
languages) will arise with the use of
generative LLMs?
Will generative LLMs speed up the
software prototyping and
development phases?
How will generative LLMs change
existing social media and enable
new social media to emerge?
How can generative LLMs support
the development of advanced
content moderation algorithms?
Will the use of generative LLMs in
everyday office automation tools
speed up product development?
How can generative LLMs help
improve communication and
collaboration in an automated
office environment?
Will AI-driven conversational
search interfaces substitute existing
ones?
Will generative LLMs make it
possible to develop more
personalised search engines?
Will generative LLMs develop
personalised educational content?
How can generative LLMs aid in
redesigning assessments to counter
AI-driven cheating and content
generation?
To what extent will the diffusion of
generative LLMs impact human
skills (support vs. substitution)?
How will upskilling and reskilling
be managed?
How will the software development
market respond to the automation
of many programming tasks in
terms of business modelling?
How can generative LLMs
contribute to improving products as
Integrated Development
Environments?
How will generative LLMs impact
the online advertising market?
How can generative LLMs impact
the role of content creators and
influencers?
How will generative LLMs impact
the market of office automation?
How much will this technology be
integrated in existing office
automation products?
Will the search engine market
switch from a monopoly to an
oligopoly?
How can generative LLMs influence
the evolution of online advertising
in the search engine market?
Will generative LLMs open new
markets in the education field?
How can generative LLMs affect
access to education and the digital
divide?

depth interviews) to empirical experimental designs (e.g. surveys, text
mining, and data analytics). Qualitative approaches could allow re­
searchers to explore the involvement of generative LLMs in the inno­
vation process, uncovering contextual factors and human—AI dynamics
that may be difficult to capture through quantitative approaches. In
contrast, quantitative studies could offer a broader perspective on the
impact of generative AI, enabling generalisation across various contexts
and industries while overcoming biases from expert judgments often
found in qualitative approaches.
Education: Innovation management researchers will face the chal­
lenge of helping the educational sector adapt to generative AI, to miti­
gate the barriers to the adoption of these methods (Kaplan and Haenlein,
2016). Research should elucidate the potential for digital divide issues
and develop strategies to ensure that all learners can benefit from
AI-driven educational innovations. The digital divide poses challenges
to adoption and use of generative LLMs, particularly in developing
countries with limited access to technology, infrastructure, and re­
sources. Insufficient internet infrastructure, varying digital literacy
levels, language barriers, and socioeconomic disparities are factors that
can hinder widespread adoption. Investigating these challenges and
factors could be valuable to bridge the digital divide and promote global
digital inclusion, thereby ensuring that the benefits of generative LLMs
are accessible and equitable.
5.2. Innovation-phase challenges
Idea generation: Innovation management researchers will need to
learn to integrate AI-generated ideas into the ideation process effec­
tively. This includes tackling challenges around managing the volume of
ideas, ensuring their relevance, and integrating them into existing
innovation processes (Verganti et al., 2020). To do so, they must address
two main challenges. First, researchers must ensure that LLMs-generated
ideas are free from cultural, social, and political biases—this requires a
comprehensive analysis of training data and continuous model fine
tuning. Second, to generate innovative ideas in various business areas,
researchers must train and adapt generative LLMs to comprehend
industry-specific language, concepts, and requirements, which may
involve specialised training data and domain expertise. Increasing the
ability in simulating the human creativity of machines can help re­
searchers understand creativity phenomena. LLMs must be made
explainable, and their decision, comprehensible (Guidotti et al., 2019).
However, as stated in the Introduction, access to generative LLMs’ en­
gine is limited to their website and API, owing to their black-box nature,
which also hinders explainability. The billions of parameters of LLMs
require significant computational power, posing challenges to scalabil­
ity and efficiency. There are currently no reliable tools to audit deep
learning systems.
Screening/Idea selection: Determining the quality and relevance of
LLMs-generated ideas in various business areas can be challenging.
Generative AI systems are more useful to exploring the solution space

8
F. Chiarello et al.
than to selecting the best ideas (Bouschery et al., 2023). To understand
how generative AI can be used also in the screening/idea selection, re­
searchers must develop methods to quantify and evaluate the quality of
ideas, considering factors such as feasibility, impact, and novelty. In this
context, decision-making frameworks may emerge, which can help
practitioners incorporate insights from AI. These, in turn, might require
new methods of idea valuation and risk assessment tailored to
AI-generated ideas. These frameworks will face the challenge of deter­
mining to balance human judgment and AI in idea selection. This in­
volves understanding when to rely on AI and when human intervention
is necessary.
Development: Researchers need to ensure that LLMs-generated solu­
tions can be seamlessly integrated in existing systems and workflows. To
foster product development, companies will need to share good prac­
tices in collaboration with AI. Here, scholars and practitioners may meet
the challenge of working with limited open databases of prompts. For
example, access to ChatGPT’s proprietary prompts and responses is
restricted by OpenAI’s terms of service. Although some websites are
collecting this information, the data stored remain limited and unlikely
to be made open to researchers. This limits research on generative LLMs
and impedes the convergence on best practices for product development
using generative AI. Lastly, as AI contributes more to product develop­
ment, questions about IP rights emerge. Researchers will face the chal­
lenge of navigating IP issues related to AI-generated designs or solutions,
potentially necessitating new approaches to IP.
Diffusion/Market/Sales: Innovation management researchers will
need to develop new methods for evaluating the market potential of AI-
generated innovations, which may require new market analysis tech­
niques and the theorisation of new business models. Generative LLMs
can create platforms around which developers can create a shared value
for customers (Trabucchi and Buganza, 2020). Researchers will also face
the challenge of managing the adoption and diffusion of AI innovations,
including understanding and overcoming barriers to adoption, devel­
oping strategies for effective market diffusion, and managing bundles of
different technologies to solve real-world industrial problems (Cho
et al., 2023). Finally, researchers interested in studying the diffusion of
generative AI technologies may focus on different characteristics of
companies adopting these solutions, such as firm size. As reported by
Mariani et al. (2023), the literature on innovation has highlighted the
importance of firm size in AI adoption, as it affects a company’s re­
sources and abilities to pursue innovative activities, as has been
emphasised by Chen and Hambrick (1995).
6. Conclusion and limitations
This study has explored the evolving landscape of generative Large
Language Models (LLMs), with a particular focus on ChatGPT, through
the analysis of over 3.8 million tweets. Our data-driven approach,
leveraging Named Entity Recognition (NER) and BERTopic algorithms,
has provided a granular understanding of the tasks assigned to genera­
tive LLMs, highlighting their versatility across various business areas.
Our findings indicate that ChatGPT, among other generative LLMs, is
being utilized in a wide spectrum of applications, ranging from pro­
gramming assistance to creative content generation. The identified
emerging areas of application—human resources, programming, social
media, office automation, search engines, and education—underscore
the potential of LLMs to revolutionise business processes and services.
Furthermore, this study contributes to the field of innovation manage­
ment by proposing a research agenda focused on exploring the inter­
section of generative LLMs with the innovation process. Our proposed
agenda outlines key research questions that span across the phases of
idea generation, screening and selection, development, and diffusion
and marketing.
Our results can assist managers in understanding how to implement
generative LMMs within their own innovation processes, taking into
account the unique characteristics of each business area. Moreover,
9
Technovation 133 (2024) 103002
policymakers can benefit from our research agenda, which offer pre­
liminary insights into tasks users request from generative AI. Under­
standing these tasks is vital for anticipating potential applications and
shaping technology adoption. With this knowledge, policymakers can
make informed decisions and design regulations supporting responsible
AI development and deployment, thereby ensuring an alignment be­
tween AI-driven innovation and societal goals. The research agenda can
also bridge the gap between academia and practice, generating action­
able insights and evidence-based recommendations for real-world
application.
Our results should be read considering the limitations of our
approach. Our method relied heavily on Twitter, which may have
introduced potential biases and limited the representativeness of the
sample. Twitter users may not fully capture the diversity of opinions of
the broader population. Furthermore, the prominence of certain topics
or trends on Twitter may not accurately reflect their importance in the
wider context of the research question, since they may be influenced by
trends and hype. To address this limitation in future work, researchers
could consider incorporating data from other sources (e.g. scientific
papers or alternative sources to Twitter, such as Reddit) to provide a
more comprehensive view.
Nonetheless, our study did not aim to establish an empirical setting
suitable for addressing the dynamic landscape of generative models.
Rather, our goal was to present a snapshot of the current situation,
thereby assisting researchers in conducting quantitative and rigorous
studies on an emerging technology that is currently discussed in quali­
tative terms. In particular, as newer models of generative LLMs emerge,
new tasks may also be introduced. Although we collected 31,747 tasks,
we do not assume that we have saturated the realm of possibilities for
generative systems. Future studies may demonstrate the expanse of such
a realm in the context of innovation management as well as quantify the
delta of the tasks performed by generative LLMs.
Funding
This work was partly funded by the ENCORE Project (ENriching
Circular use of OeR for Education) - European Union Erasmus+ Part­
nerships for Innovation: Alliances (ERASMUS-EDU-2021-PI-ALL-INNO)
(Project Number: 101055893) and PNRR - M4C2 - Investimento 1.3,
Partenariato Esteso PE00000013 – “FAIR - Future Artificial Intelligence
Research” - Spoke 1 “Human-centered AI,” funded by the European
Commission under the NextGeneration EU programme.
CRediT authorship contribution statement
Filippo Chiarello: Writing – review & editing, Writing – original
draft, Visualization, Validation, Supervision, Software, Resources,
Project administration, Methodology, Investigation, Funding acquisi­
tion, Formal analysis, Data curation, Conceptualization. Vito Giordano:
Writing – review & editing, Writing – original draft, Software, Meth­
odology, Investigation, Formal analysis, Data curation, Conceptualiza­
tion. Irene Spada: Writing – original draft, Methodology, Formal
analysis, Data curation, Conceptualization. Simone Barandoni: Writing
– original draft, Visualization, Software, Methodology. Gualtiero Fan­
toni: Writing – original draft, Validation, Supervision, Funding acqui­
sition, Conceptualization.
Data availability
We have shared the link to data and method in the paper, on an
anonymised Github page.
References
Abadie, A., Chowdhury, S., Mangla, S.K., 2024. A shared journey: experiential
perspective and empirical evidence of virtual social robot ChatGPT’s priori
F. Chiarello et al.
acceptance. Technol. Forecast. Soc. Change 201, 123202. https://doi.org/10.1016/j.
techfore.2023.123202.
Abrahamson, E., Rosenkopf, L., 1997. Social network effects on the extent of innovation
diffusion: a computer simulation. Organ. Sci. 8, 289–309.
Ali, O., Murray, P.A., Momin, M., Dwivedi, Y.K., Malik, T., 2024. The effects of artificial
intelligence applications in educational settings: challenges and strategies. Technol.
Forecast. Soc. Change 199, 123076. https://doi.org/10.1016/j.
techfore.2023.123076.
Ameye, N., Bughin, J., van Zeebroeck, N., 2023. How uncertainty shapes herding in the
corporate use of artificial intelligence technology. Technovation 127, 102846.
https://doi.org/10.1016/j.technovation.2023.102846.
Antons, D., Grünwald, E., Cichy, P., Salge, T.O., 2020. The application of text mining
methods in innovation research: current state, evolution patterns, and development
priorities. R D Manag. 50, 329–351. https://doi.org/10.1111/radm.12408.
Arts, S., Hou, J., Gomez, J.C., 2021. Natural language processing to identify the creation
and impact of new technologies in patent text: code, data, and new measures. Res.
Pol. 50, 104144 https://doi.org/10.1016/j.respol.2020.104144.
Autor, D.H., 2013. The ‘task approach’ to labor markets: an overview. J. Labour Mark.
Res. 46, 185–199.
Barredo Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S.,
Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R.,
Herrera, F., 2020. Explainable Artificial Intelligence (XAI): concepts, taxonomies,
opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115. https://
doi.org/10.1016/j.inffus.2019.12.012.
Bonaccorsi, A., Chiarello, F., Fantoni, G., 2021. Impact for whom? Mapping the users of
public research with lexicon-based text mining. Scientometrics 126, 1745–1774.
https://doi.org/10.1007/s11192-020-03803-z.
Bouschery, S.G., Blazevic, V., Piller, F.T., 2023. Augmenting human innovation teams
with artificial intelligence: exploring transformer-based language models. J. Prod.
Innovat. Manag. 40, 139–153. https://doi.org/10.1111/jpim.12656.
Bower, J.L., Christensen, C.M., 1995. Disruptive Technologies: Catching the Wave.
Bresnahan, T.F., Trajtenberg, M., 1995. General purpose technologies ‘Engines of
growth’. J. Econ. 65, 83–108. https://doi.org/10.1016/0304-4076(94)01598-T.
Budhwar, P., Chowdhury, S., Wood, G., Aguinis, H., Bamber, G.J., Beltran, J.R.,
Boselie, P., Lee Cooke, F., Decker, S., DeNisi, A., Dey, P.K., Guest, D., Knoblich, A.J.,
Malik, A., Paauwe, J., Papagiannidis, S., Patel, C., Pereira, V., Ren, S., Rogelberg, S.,
Saunders, M.N.K., Tung, R.L., Varma, A., 2023. Human resource management in the
age of generative artificial intelligence: perspectives and research directions on
ChatGPT. Human Res. Mgmt Journal. 33, 606–659. https://doi.org/10.1111/1748-
8583.12524.
Burtsev, M., Reeves, M., Job, A., 2023. The working limitations of Large Language
Models. MIT Sloan Manag. Rev. 65 (1), 1–5.
Chatterjee, J., Dethlefs, N., 2023. This new conversational AI model can be your friend,
philosopher, and guide and even your worst enemy. Patterns (N Y) 4, 100676.
https://doi.org/10.1016/j.patter.2022.100676.
Chen, M.J., Hambrick, D.C., 1995. Speed, stealth, and selective attack: how small firms
differ from large firms in competitive behavior. Acad. Manag. J. 38, 453–482.
https://doi.org/10.2307/256688.
Cho, J., Destefano, T., Kim, H., Kim, I., Paik, J.H., 2023. What’s driving the diffusion of
next-generation digital technologies? Technovation 119, 102477. https://doi.org/
10.1016/j.technovation.2022.102477.
Choi, Y., Nguyen, M.D., Kerr Jr, T.N., 2021. Syntactic and semantic information
extraction from NPP procedures utilizing natural language processing integrated
with rules. Nucl. Eng. Technol. 53, 866–878. https://doi.org/10.1016/j.
net.2020.08.010.
Dang, H., Mecke, L., Lehmann, F., Goller, S., Buschek, D., 2022. How to prompt?
Opportunities and challenges of zero-and few-shot learning for human-AI interaction
in creative applications of generative models. arXiv Preprint ArXiv:2209.01390.
https://doi.org/10.48550/arXiv.2209.01390.
Davis, F.D., 1989. Perceived usefulness, perceived ease of use, and user acceptance of
information technology. MIS Q. 13, 319–340. https://doi.org/10.2307/249008.
Deloose, A., Gysels, G., De Baets, B., Verwaeren, J., 2023. Combining natural language
processing and multidimensional classifiers to predict and correct CMMS metadata.
Comput. Ind. 145, 103830 https://doi.org/10.1016/j.compind.2022.103830.
Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018. Bert: pretraining of deep
bidirectional transformers for language understanding. arXiv Preprint ArXiv:
1810.04805. https://doi.org/10.48550/arXiv.1810.04805.
Dewar, R.D., Dutton, J.E., 1986. The adoption of radical and incremental innovations: an
empirical analysis. Manag. Sci. 32, 1422–1433.
Egger, R., Yu, J., 2022. A topic modeling comparison between lda, nmf, top2vec, and
bertopic to demystify twitter posts. Front. Sociol 7, 886498. https://doi.org/
10.3389/fsoc.2022.886498.
Else, H., 2023. Abstracts written by ChatGPT fool scientists. Nature 613, 423. https://
doi.org/10.1038/d41586-023-00056-7, 423.
Giordano, V., Chiarello, F., Melluso, N., Fantoni, G., Bonaccorsi, A., 2021. Text and
dynamic network analysis for measuring technological convergence: a case study on
defense patent data. IEEE Trans. Eng. Manag. 70, 1490–1503. https://doi.org/
10.1109/TEM.2021.3078231.
Graham, F., 2022. Daily briefing: will ChatGPT kill the essay assignment? Nature.
https://doi.org/10.1038/d41586-022-04437-2.
Grootendorst, M., 2022. BERTopic: neural topic modeling with a class-based TF-IDF
procedure. arXiv Preprint ArXiv:2203.05794. https://doi.org/10.48550/
arXiv.2203.05794.
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D., 2019.
A survey of methods for explaining black box models. ACM Comput. Surv. 51, 1–42.
https://doi.org/10.1145/3236009.
10
Technovation 133 (2024) 103002
Hannigan, T.R., Briggs, A.R., Valadao, R., Seidel, M.L., Jennings, P.D., 2022. A new tool
for policymakers: mapping cultural possibilities in an emerging AI entrepreneurial
ecosystem. Res. Pol. 51, 104315 https://doi.org/10.1016/j.respol.2021.104315.
Hocky, G.M., White, A.D., 2022. Natural language processing models that automate
programming will transform chemistry research and teaching. Dig. Dis. 1, 79–83.
https://doi.org/10.1039/D1DD00009H.
Hofmann, P., Keller, R., Urbach, N., 2019. Inter-technology relationship networks:
arranging technologies through text mining. Technol. Forecast. Soc. Change 143,
202–213. https://doi.org/10.1016/j.techfore.2019.02.009.
Hong, S., Kim, J., Woo, H.G., Kim, Y.C., Lee, C., 2022. Screening ideas in the early stages
of technology development: a word2vec and convolutional neural network
approach. Technovation 112, 102407. https://doi.org/10.1016/j.
technovation.2021.102407.
Huang, F., Kwak, H., An, J., 2023. Is ChatGPT better than human annotators? Potential
and limitations of ChatGPT in explaining implicit hate speech. arXiv Preprint ArXiv:
2302.07736. https://doi.org/10.48550/arXiv.2302.07736.
Huang, M.S., Lai, P.T., Lin, P.Y., You, Y.T., Tsai, R.T.H., Hsu, W.L., 2020. Biomedical
named entity recognition and linking datasets: survey and our recent development.
Briefings Bioinf. 21, 2219–2238. https://doi.org/10.1093/bib/bbaa054.
Jeon, E., Yoon, N., Sohn, S.Y., 2023. Exploring new digital therapeutics technologies for
psychiatric disorders using BERTopic and PatentSBERTa. Technol. Forecast. Soc.
Change 186, 122130. https://doi.org/10.1016/j.techfore.2022.122130.
Just, J., 2024. Natural language processing for innovation search – reviewing an
emerging nonhuman innovation intermediary. Technovation 129, 102883. https://
doi.org/10.1016/j.technovation.2023.102883.
Kaplan, A.M., Haenlein, M., 2016. Higher education and the digital revolution: about
MOOCs, SPOCs, social media, and the Cookie Monster. Bus. Horiz. 59, 441–450.
https://doi.org/10.1016/j.bushor.2016.03.008.
Kohl, C., Knigge, M., Baader, G., Böhm, M., Krcmar, H., 2018. Anticipating acceptance of
emerging technologies using twitter: the case of self-driving cars. J. Bus. Econ. 88,
617–642. https://doi.org/10.1007/s11573-018-0897-5.
Lacasa, I.D., Grupp, H., Schmoch, U., 2003. Tracing technological change over long
periods in Germany in chemicals using patent statistics. Scientometrics 57, 175–195.
https://doi.org/10.1023/A:1024133517484.
Lee, P., Fyffe, S., Son, M., Jia, Z., Yao, Z., 2022. A paradigm shift from ‘human writing’ to
‘machine generation’ in personality test development: an application of state-of-the-
art natural language processing. J. Bus. Psychol. 1–28 https://doi.org/10.1007/
s10869-022-09864-6.
Li, X., Wen, Y., Jiang, J., Daim, T., Huang, L., 2022. Identifying potential breakthrough
research: a machine learning method using scientific papers and Twitter data.
Technol. Forecast. Soc. Change 184, 122042. https://doi.org/10.1016/j.
techfore.2022.122042.
Li, X., Xie, Q., Jiang, J., Zhou, Y., Huang, L., 2019. Identifying and monitoring the
development trends of emerging technologies using patent analysis and Twitter data
mining: the case of perovskite solar cell technology. Technol. Forecast. Soc. Change
146, 687–705. https://doi.org/10.1016/j.techfore.2018.06.004.
Mariani, M.M., Machado, I., Magrelli, V., Dwivedi, Y.K., 2023. Artificial intelligence in
innovation research: a systematic review, conceptual framework, and future
research directions. Technovation 122, 102623. https://doi.org/10.1016/j.
technovation.2022.102623.
Makridakis, S., Petropoulos, F., Kang, Y., 2023. Large language models: their success and
impact. Forecasting 5 (3), 536–549. https://doi.org/10.3390/forecast5030030.
McInnes, L., Healy, J., Melville, J., 2018. Umap: Uniform manifold approximation and
projection for dimension reduction. arXiv Preprint ArXiv:1802.03426. https://doi.
org/10.48550/arXiv.1802.03426.
Mikolov, T., Chen, K., Corrado, G., Dean, J., 2013. Efficient estimation of word
representations in vector space. arXiv Preprint ArXiv:1301.3781. https://doi.org/
10.48550/arXiv.1301.3781.
Ohms, J., 2021. Current methodologies for chemical compound searching in patents: a
case study. World Patent Inf. 66, 102055 https://doi.org/10.1016/j.
wpi.2021.102055.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Lowe, R., 2022.
Training Language models to follow instructions with human feedback. arXiv
Preprint ArXiv:2203.02155. https://doi.org/10.48550/arXiv.2203.02155.
Ozcan, S., Suloglu, M., Sakar, C.O., Chatufale, S., 2021. Social media mining for ideation:
identification of sustainable solutions and opinions. Technovation 107, 102322.
https://doi.org/10.1016/j.technovation.2021.102322.
Pavlik, J.V., 2023. Collaborating with ChatGPT: considering the implications of
generative artificial intelligence for journalism and media education. Journal. Mass
Commun. Educat. 78, 84–93. https://doi.org/10.1177/10776958221149577.
Piller, F.T., Bouschery, S.G., Blazevic, V., 2023. Hybrid intelligence for innovation:
augmenting NPD teams with artificial intelligence and machine learning. The PDMA
Handbook of Innovation and New Product. Development 407.
Porter, A.L., Garner, J., Carley, S.F., Newman, N.C., 2019. Emergence scoring to identify
frontier R&D topics and key players. Technol. Forecast. Soc. Change 146, 628–643.
https://doi.org/10.1016/j.techfore.2018.04.016.
Puccetti, G., Giordano, V., Spada, I., Chiarello, F., Fantoni, G., 2023. Technology
identification from patent texts: a novel named entity recognition method.
Technology Identification from Patent Texts. Technol. Forecast. Soc. Change 186,
122160. https://doi.org/10.1016/j.techfore.2022.122160.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., 2018. Improving language
understanding by generative pretraining. https://cdn.openai.com/research-cover
s/language-unsupervised/language_understanding_paper.pdf.
Ranaei, S., Suominen, A., Porter, A., Carley, S., 2020. Evaluating technological
emergence using text analytics: two case technologies and three approaches.
Scientometrics 122, 215–247. https://doi.org/10.1007/s11192-019-03275-w.
F. Chiarello et al.
Reynolds, L., McDonell, K., 2021. Prompt programming for large language models:
beyond the few-shot paradigm. In: Extended. Abstracts of the 2021 CHI Conference
on Human Factors in Computing Systems, pp. 1–7. https://doi.org/10.1145/
3411763.3451760.
Rogers, E.M., 2004. A prospective and retrospective look at the diffusion model.
J. Health Commun. 9 (S 1), 13–19.
Rotolo, D., Hicks, D., Martin, B.R., 2015. What is an emerging technology? Res. Pol. 44,
1827–1843. https://doi.org/10.1016/j.respol.2015.06.006.
Salerno, M.S., Gomes, L.A., Da Silva, D.O., Bagno, R.B., Freitas, S.L.T.U., 2015.
Innovation processes: which process for which project? Technovation 35, 59–70.
https://doi.org/10.1016/j.technovation.2014.07.012.
Sallam, M., 2023. ChatGPT utility in healthcare education, research, and practice:
systematic review on the promising perspectives and valid concerns. Healthcare 11,
887. https://doi.org/10.3390/healthcare11060887.
Sanderson, K., 2023. GPT-4 is here: what scientists think. Nature 615, 773. https://doi.
org/10.1038/d41586-023-00816-5.
Siddharth, L., Blessing, L., Luo, J., 2022. Natural language processing in-and-for design
research. Des. Sci. 8, e21 https://doi.org/10.1017/dsj.2022.16.
Small, H., Boyack, K.W., Klavans, R., 2014. Identifying emerging topics in science and
technology. Res. Pol. 43, 1450–1467. https://doi.org/10.1016/j.
respol.2014.02.005.
Song, B., Luan, C., Liang, D., 2023. Identification of emerging technology topics (ETTs)
using BERT-based model and sematic analysis: a perspective of multiple-field
characteristics of patented inventions (MFCOPIs). Scientometrics 128, 5883–5904.
https://doi.org/10.1007/s11192-023-04819-x.
Spada, I., Barandoni, S., Giordano, V., Chiarello, F., Fantoni, G., Martini, A., 2023. What
users want: a Natural Language Processing approach to discover Users’Needs from
online reviews. Proceedings of the Design Society 3, 3879–3888. https://doi.org/
10.1017/pds.2023.389.
11
Technovation 133 (2024) 103002
Stokel-Walker, C., 2022. AI bot ChatGPT writes smart essays - should academics worry?
Nature. https://doi.org/10.1038/d41586-022-04397-7.
Stokel-Walker, C., 2023. ChatGPT listed as author on research papers: many scientists
disapprove. Nature 613, 620–621. https://doi.org/10.1038/d41586-023-00107-z.
Trabucchi, D., Buganza, T., 2020. Fostering digital platform innovation: from two to
multi-sided platforms. Creativ. Innovat. Manag. 29, 345–358. https://doi.org/
10.1111/caim.12320.
Tsai, R.T.H., Wu, S.H., Chou, W.C., Lin, Y.C., He, D., Hsiang, J., Sung, T.Y., Hsu, W.L.,
2006. Various criteria in the evaluation of biomedical named entity recognition.
BMC Bioinf. 7, 92. https://doi.org/10.1186/1471-2105-7-92.
Van Dis, E.A., Bollen, J., Zuidema, W., van Rooij, R., Bockting, C.L., 2023. ChatGPT: five
priorities for research. Nature 614 (7947), 224–226. https://doi.org/10.1038/
d41586-023-00288-7.
Verganti, R., Vendraminelli, L., Iansiti, M., 2020. Innovation and design in the age of
artificial intelligence. J. Prod. Innovat. Manag. 37, 212–227. https://doi.org/
10.1111/jpim.12523.
Von Hippel, E., 1986. Lead users: a source of novel product concepts. Manag. Sci. 32,
791–805.
Yoon, J., Park, H., Kim, K., 2013. Identifying technological competition trends for R&D
planning using dynamic patent maps: SAO-based content analysis. Scientometrics
94, 313–331. https://doi.org/10.1007/s11192-012-0830-6.
Zhao, K., Ding, H., Ye, K., Cui, X., Fu, Z., 2021. A latent variable model with hierarchical
structure and GPT-2 for long text generation. In: Artif. Neural Netw. Mach. Learn.–
ICANN. Proceedings of the Part V 30: 30th International Conference on Artificial
Neural Networks, Bratislava, Slovakia, 14–17 Sep, 2021 2021. Springer International
Publishing, pp. 297–308. https://doi.org/10.1007/978-3-030-86383-8_24.
Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Wen, J.R., 2023. A survey of
Large Language Models. arXiv Preprint ArXiv:2303.18223.