22/9/23, 12:10 Large language models (LLMs) vs.

ChatGPT

Get demo Get demo

Browse by topic

Artificial Intelligence

Emergent properties of Large

Language Models (LLMs) including
ChatGPT

by Amit Prakash, Co-founder and CTO

Published Feb 23, 2023, last updated Feb 22, 2023

In the last article, we learned how transformer neural networks work and how
ChatGPT is a transformer trained on language modeling tasks. We began talking
about how as these transformer-based language models get large, a very
interesting set of properties show up. This is true not just for ChatGPT but other
similar models such as Bloom and PaLM.
https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 1/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

InGet
thisdemo
module, weGetwill talk about what are these emergent properties and what are
demo
their implications for the real world. We will also look at how researchers from
OpenAI in designing ChatGPT went beyond training a transformer model by
training those models to learn from human feedback. Finally, we will look at some
of the limitations of ChatGPT as well as the implications of ChatGPT in the real
world.
While the GPT family of models has certainly led the way for training and exposing
larger and larger models there have been several other large language models
trained with billions of parameters (e.g. Bloom from HuggingFace, PaLM from
Google) that have produced some very interesting properties once they’ve
crossed a threshold of somewhere between 50 and 100 billion parameters. This is
a great document compiling research on the emergent properties of LLMs.

Let's look at the most interesting properties seen in GPT-3

Remember in the examples below the text with clear background is the prompt
and the text with green background is the response from GPT-3.

Zero-shot learning:
This is when a model successfully solves a problem that it was not explicitly trained
on. For example, you ask for completion for the text “what is 5 + 3" and the
response says 8.

Few-shot learning:
This is when the model doesn’t initially know how to solve a problem but works out
a solution based on a few examples. Note that all of this is happening through a

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 2/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

mechanism
Get demo that was just designed to predict the next words in a sequence of
Get demo
words. For example here is a prompt specifying how to detect sentiment:

Question answering:
This is when GPT composes information, as opposed to merely retrieving it like a
search engine:

Code generation:
One thing that has surprised a lot of computer scientists is how well GPT can
generate code based on instructions in natural language. It’s not perfect, but as an
assistive tool, it is proving to be a step change in the state of the art. You can find
blogs where someone has generated an entire e-commerce website with code
generated from GPT invocations.
https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 3/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

If you are not intoGet

Get demo generating
demo code, a very similar task that has taken the Internet by
storm is to ask GPT to generate prose or poetry in a particular style for a particular
topic. And its ability to compose arbitrary styles with arbitrary topics is really
uncanny:

Chain-of-thought-based reasoning:
Of all the interesting emergent properties of LLMs this one is perhaps the most
surprising and interesting. For certain complex reasoning tasks, where few-shot
examples don’t work, elaborating on the reasoning of how the problem was solved

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 4/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

enables Large Language

Get demo Get demo Models to solve them. Here is an example from this
paper.

Where LLMs fall short

While these capabilities are amazing and totally unprecedented, LLMs are not
perfect reasoning engines. In fact it is very easy to get a response from GPT and
other LLMs that is either plain wrong, misinformed, or downright unethical. Here are
some examples:

Basic arithmetic operations when large numbers are involved:

Hallucinating answers when it doesn’t know:

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 5/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

The
Getfollowing
demo question is a collatz hypothesis and is yet to be proven or disproved.
Get demo
We know that an LLM cannot suddenly produce a proof for it when it has eluded
the most elite mathematicians for more than half a century. Yet, GPT-3 has no
problems producing a nonsensical proof for it:

At ThoughSpot, we have been experimenting with GPTs ability to generate SQL

from Natural Language. While some of the intelligence it shows in terms of
resolving ambiguity based on world knowledge is completely uncanny, it is far
from perfect. In particular, once you are dealing with real-world schemas (as
opposed to simple toyish schemas) GPT cannot do the SQL translation on its own.

How ChatGPT is different than other LLMs

While there are a number of other LLMs trained using similar-sized data sets,
ChatGPT has simply run away with almost all the public attention. There are a few
reasons for it:

Reinforcement Learning from Human Feedback (RLHF):

As we have discussed, GPT-3 is simply a language model. While it does a good job
of responding to questions and reasoning tasks, it was not designed to do so.
Furthermore, it has been trained on textual data from the Internet which does
contain substantial amounts of hate-speech. So it is not surprising that with an
https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 6/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

adversarial
Get demo (or even
Get an arbitrary) prompt you can get responses that are in poor
demo
taste or offensive.
One way to fix this problem is by generating human-labeled training data that tells
the model what is a more desirable output. However, it would be impractical to
generate training data that is even a tiny fraction of original unsupervised training
data and hence less likely to make a dent. Instead, OpenAI researchers trained
another neural network from the labeled data to learn human preferences. Now,
this new neural network could be used at scale to fine-tune the GPT-3 models to
prefer one response over another. This technique is called reinforcement learning
for human feedback. OpenAI released a model called InstructGPT which was the
precursor to ChatGPT. This additional layer of safety allowed OpenAI to expose the
LLMs more freely.

The chat interface:

OpenAI did an extremely simple UX optimization in exposing ChatGPT, which
basically remembers previous exchanges with the user in the session and appends
that to prompt. This simple UX change made a huge difference in its usability for
the general public.

Free online service:

While models like Bloom are available to the public, there is no free resource where
you can go and simply play with it by typing your questions. Exposing ChatGPT to
the public comes at a significant compute cost to OpenAI, but it also attracts
mindshare and provides additional training data from all the interactions people are
having with it.

What is the impact of ChatGPT on society?

ChatGPT and other LLMs represent a substantial advancement in the state of the art
when it comes to AI’s ability to understand text, synthesize new text, compose
ideas, and do reasoning. In its current form it is deeply flawed and a lot of care
needs to be taken in its application. Without a doubt, there will be a big wave of
products coming soon ranging from assistive writing, code generation,
summarization, and more. I compare this to discovering a new element on the
periodic table in the sense that it will enable a lot of new things that were not
https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 7/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

possible
Get demobefore, but only by carefully studying its properties and mixing it with other
Get demo
elements carefully.
From my perspective, it is a significant advancement that will encourage society to
be even more invested in the advancement of AI and AI safety. A lot of new
products—including Sage, our AI-Powered Analytics experience—will be built
around this innovation and the status quo will be disrupted.

However, in general this innovation will bring more prosperity than harm and surely
reduce the amount of tedious work we do every day. I expect LLMs to be no
different. In the next article, we will explore the future of AI and trends to look out for
in a ChatGPT world.

Related articles

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 8/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

Get demo Get demo

Artificial Intelligence
10 best AI tools for business efficiency in 2023
Read more

Artificial Intelligence
LLM and AI advancements ignite a new chapter of FinOps
Read more

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 9/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

Get demo Get demo

Artificial Intelligence
Decision Intelligence: Using Data to Drive Better Outcomes
Read more

Start getting better insights

Sign up now and you'll be using ThoughtSpot in minutes.

Start free trial

ThoughtSpot is the AI-Powered Analytics company that lets everyone create personalized insights to drive decisions
and take action.

Product By Role
Product Business Leader
Sage Data Leader
https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 10/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

GetAnalyze
Auto demo Get demo Analyst
Visualize Product Leader
Connect Developer
Operationalize
By Department
Embed
Mobile Marketing
Govern & Secure Sales
New features ServiceNow Analytics
Pricing

Solutions About Us
Banking Team
Financial Services Customers
Retail & E-commerce Atlas Marketplace
Manufacturing & Logistics Partner Directory
Healthcare & Life Sciences Events
Media & Communications News
Procurement Careers
Public Sector Blog
Legal
Trust

Connect (800) 508-7008

Free Trial
Resources
Request Demo
Professional Services
Community
Documentation
Support
Contact Us

Stay in Touch
Get the latest from ThoughtSpot

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 11/12
22/9/23, 12:10 Large language models (LLMs) vs. ChatGPT

Get demo Get demo

©2023 ThoughtSpot Inc.

All Rights Reserved

Terms of Use Privacy Statement Cookie Policy

https://www.thoughtspot.com/data-trends/ai/large-language-models-vs-chatgpt 12/12