GPT-3 vs.

ChatGPT
How much have things progressed in 2 years?

Since I got access to GPT-3 in the summer of 2020 I have been astonished by how much it is capable of
doing. When OpenAI announced the availability of ChatGPT, I decided to compare how it performed on
some questions that had stumped GPT-3. This article shows the results of these comparisons between
the current level of GPT-3 and ChatGPT.

The King of France

I wanted to ask the question “Who is the king of France”, but ChatGPT baulks at questions about people
serving in political roles currently. For example, ask it who the current prime minister of Australia is,
and it demurs:

So, to get a head-to-head comparison, I needed to ask a question that is firmly rooted in the pre-2021
era: “who was the king of France in 1940”.

GPT-3 correctly names one of two heads of state for France in 1940 but incorrectly identifies Petain as
a king.

ChatGPT answers the question aptly by clarifying that France had no king in 1940 and correctly names
Lebrun, the head of state for the majority of 1940.
The first Russian on the Moon
We know that For All Mankind is a work of fiction, but will the AIs be able to answer the central
question of this series?

GPT-3 comes up with a mangled answer for this question:

ChatGPT handles the question perfectly:

What cars does a defunct company make?

As any Top Gear fan knows, British Leyland had its 15 years in the sun from the late 1960s until the
early 1980s and is currently defunct.

GPT-3 gives a list of cars that is mostly right (minus the Rover 200, which was a product of one post-
British Leyland successor brands), but stumbles by using the current tense in the answer:

ChatGPT is a bit passive-aggressive in referring to an earlier answer about British Leyland, and could
be more conclusive about there being not British Leyland products now. If Labour had won the 2017
election in the UK, it is possible that BL could have come back from the dead, but I don’t think that’s
the reason why ChatGPT is prevaricating.

A vice-regal riddle
The Commonwealth Realms (including Australia, New Zealand and Canada) have governors general to
represent the monarch. England, as a part of the UK, does not. How will the two systems answer a
question about an English governor general?

GPT-3 simply hallucinates an answer:

ChatGPT nails it like a constitution nerd. Perfect:

Pick a side
Finally, let’s compare how each system answers a question about who won a war with an ambiguous
outcome.

GPT-3 gives a valid, if biased, answer:

ChatGPT manages to provide a more nuanced answer, although surely the most famous battles of the
war happened “At Queenston Heights and Lundy’s Lane”, not New Orleans!
Conclusion
While it is not that hard to trip up ChatGPT, it is clearly a large improvement over GPT-3. The
comparisons in this article show the progress made in this field in a little over two years, and the
progress is impressive.