5 HAI 2024 AI-Index-Report 296-365

Artificial Intelligence CHAPTER 5:
Index Report 2024

Science and
Medicine
CHAPTER 5:
Artificial Intelligence
Index Report 2024 Science and
Medicine
Preview
Overview 298
Chapter Highlights 299
5.1 Notable Scientific Milestones 300

AlphaDev 300
FlexiCubes 301
Synbot 303
GraphCast 304
GNoME 305
Flood Forecasting 306
5.2 AI in Medicine 307

Notable Medical Systems 307
SynthSR 307
Coupled Plasmonic Infrared Sensors 309
EVEscape 310
AlphaMissence 312
Human Pangenome Reference 313
Clinical Knowledge 314
MedQA 314
Highlighted Research: GPT-4 Medprompt 315
Highlighted Research: MediTron-70B 317
Diagnosis 318
Highlighted Research: CoDoC 318
Highlighted Research: CT Panda 319
Other Diagnostic Uses 320
FDA-Approved AI-Related Medical Devices 321
Administration and Care 323
Highlighted Research: MedAlign 323
ACCESS THE PUBLIC DATA
Table of Contents 297

CHAPTER 5:
Medicine
Overview
This year’s AI Index introduces a new chapter on AI in science and medicine in recognition of
AI’s growing role in scientific and medical discovery. It explores 2023’s standout AI-facilitated
scientific achievements, including advanced weather forecasting systems like GraphCast
and improved material discovery algorithms like GNoME. The chapter also examines medical
AI system performance, important 2023 AI-driven medical innovations like SynthSR and
ImmunoSEIRA, and trends in the approval of FDA AI-related medical devices.

CHAPTER 5:
Medicine
Chapter Highlights
1. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance
scientific discovery. 2023, however, saw the launch of even more significant science-related AI applications—
from AlphaDev, which makes algorithmic sorting more efficient, to GNoME, which facilitates the process of
materials discovery.
2. AI helps medicine take significant strides forward. In 2023, several significant medical systems
were launched, including EVEscape, which enhances pandemic prediction, and AlphaMissence, which assists in
AI-driven mutation classification. AI is increasingly being utilized to propel medical advancements.
3. Highly knowledgeable medical AI has arrived. Over the past few years, AI systems have shown
remarkable improvement on the MedQA benchmark, a key test for assessing AI’s clinical knowledge. The
standout model of 2023, GPT-4 Medprompt, reached an accuracy rate of 90.2%, marking a 22.6 percentage
point increase from the highest score in 2022. Since the benchmark’s introduction in 2019, AI performance on
MedQA has nearly tripled.
4. The FDA approves more and more AI-related medical devices. In 2022, the FDA approved 139
AI-related medical devices, a 12.1% increase from 2021. Since 2012, the number of FDA-approved AI-related medical
devices has increased by more than 45-fold. AI is increasingly being used for real-world medical purposes.

Artificial Intelligence Chapter 5: Science and Medicine
Index Report 2024 5.1 Notable Scientific Milestones
This section highlights significant AI-related scientific breakthroughs of 2023 as chosen by the AI Index Steering Committee.
5.1 Notable Scientific Milestones

AlphaDev fundamental sorting algorithms on short sequences
AlphaDev discovers faster sorting algorithms such as Sort 3, Sort 4, and Sort 5 (Figure 5.1.1). Some
AlphaDev is a new AI reinforcement learning system of the new algorithms discovered by AlphaDev have
that has improved on decades of work by scientists been incorporated into the LLVM standard C++ sort
and engineers in the field of computational algorithmic library. This marks the first update to this part of
enhancement. AlphaDev developed algorithms with the library in over 10 years and is the first addition
fewer instructions than existing human benchmarks for designed using reinforcement learning.
AlphaDev vs. human benchmarks when optimizing for algorithm length

Source: Mankowitz et al., 2023 | Chart: 2024 AI Index report
120 115
AlphaDev Human benchmarks
100
80
Instruction length
66
63
60
46
42
40 37
33
31
28 28 27
21
20 17 18
0
Sort 3 Sort 4 Sort 5 VarSort3 VarSort4 VarSort5 VarInt
Algorithm
Figure 5.1.1
Table of Contents Chapter 5 Preview 300

FlexiCubes quality. FlexiCubes addresses some of these

3D mesh optimization with FlexiCubes limitations by employing AI for gradient-based
3D mesh generation, crucial in computer graphics, optimization and adaptable parameters (Figure
involves creating a mesh of vertices, edges, and 5.1.2). This method allows for precise, localized
faces to define 3D objects. It is key to video games, mesh adjustments. Compared to other leading
animation, medical imaging, and scientific visualization. methods that utilize differentiable isosurfacing for
Traditional isosurface extraction algorithms often mesh reconstruction, FlexiCubes achieves mesh
struggle with limited resolution, structural rigidity, and extractions that align much more closely with the
numerical instabilities, which subsequently impacts underlying ground truth (Figure 5.1.3).
Sample FlexiCubes surface reconstructions

Source: Nvidia, 2023
Figure 5.1.2

Select quantitative results on 3D mesh reconstruction

Source: Shen et al., 2023 | Chart: 2024 AI Index report
80.67%
80%
70%
63.34%
60%
55.22%
52.37%
50.20%
50% 48.66%
IN>5◦ (%) ↓
40%
34.87%
30%
20%
10%
0%
MCSDF DChermite NDCSDF MC DMTet(64) DMTet(80) FlexiCubes
Algorithm/method evaluated at 64³
Figure 5.1.3

Synbot Synbot design

Source: Ha et al., 2023
AI-driven robotic chemist for
synthesizing organic molecules
Synbot employs a multilayered system,
comprising an AI software layer for
chemical synthesis planning, a robot
software layer for translating commands,
and a physical robot layer for conducting
experiments. The closed-loop feedback
mechanism between the AI and the
robotic system enables Synbot to develop
synthetic recipes with yields equal to
or exceeding established references
Figure 5.1.4
(Figure 5.1.4). In an experiment aimed at
synthesizing M1 [4-(2,3-dimethoxyphenyl)- the mid-80% reference range and completed the synthesis
1H-pyrrolo[2,3-b]pyridine], Synbot in significantly less time (Figure 5.1.5). Synbot’s automation
developed multiple synthetic formulas of organic synthesis highlights AI’s potential in fields such as
that achieved conversion yields surpassing pharmaceuticals and materials science.
Reaction kinetics of M1 autonomous optimization experiment, Synbot vs. reference

Source: Ha et al., 2023 | Chart: 2024 AI Index report
100%, Synbot #1
100%, Synbot #2
100% 100%, Synbot #4
85%, Reference
80%
Conversion yield (%)
60%
40%
20%
0%
0 3 6 9 12 15 18 21 24
Time (hours)
Figure 5.1.5

GraphCast and more. Figure 5.1.7 compares the performance

More accurate global weather forecasting of GraphCast with the current industry state-of-the-
with GraphCast art weather simulation system: the High Resolution
GraphCast is a new weather forecasting system Forecast (HRES). GraphCast posts a lower root mean
that delivers highly accurate 10-day weather squared error, meaning its forecasts more closely
predictions in under a minute (Figure 5.1.6). Utilizing correspond to observed weather patterns. GraphCast
graph neural networks and machine learning, can be a valuable tool in deciphering weather patterns,
GraphCast processes vast datasets to forecast enhancing preparedness for extreme weather events,
temperature, wind speed, atmospheric conditions, and contributing to global climate research.
GraphCast weather prediction

Source: DeepMind, 2023
Figure 5.1.6
Ten-day z500 forecast skill: GraphCast vs. HRES

Source: Lam et al., 2023 | Chart: 2024 AI Index report
GraphCast HRES (06z/18z) HRES (00z/12z)
800
700
600
RMSE (m²/s²) ↓
500
400
300
200
100
0
1 2 3 4 5 6 7 8 9 10
Lead time (days)
Figure 5.1.7

GNoME Sample material structures

Source: Merchant et al., 2023
Discovering new materials with GNoME
The search for new functional materials is key to
advancements in various scientific fields, including
robotics and semiconductor manufacturing. Yet this
discovery process is typically expensive and slow.
Recent advancements by Google researchers have
demonstrated that graph networks, a type of AI
model, can expedite this process when trained on
large datasets. Their model, GNoME, outperformed
the Materials Project, a leading method in materials
discovery, by identifying a significantly larger
number of stable crystals (Figure 5.1.8). GNoME has
unveiled 2.2 million new crystal structures, many
Figure 5.1.8
overlooked by human researchers (Figure 5.1.9 and
Figure 5.1.10). The success of AI-driven projects like
GNoME highlights the power of data and scaling in
speeding up scientific breakthroughs.
GNoME vs. Materials Project: stable crystal count GNoME vs. Materials Project: distinct prototypes
Source: Merchant et al., 2023 | Chart: 2024 AI Index report Source: Merchant et al., 2023 | Chart: 2024 AI Index report
1,000,000
GNoME Material Project GNoME Material Project
20,000
100,000
Stable crystal count
Distinct prototypes
10,000 10,000
1,000
2 3 4 5 6 0
2 3 4 5 6
Unique elements Unique elements
Figure 5.1.9 Figure 5.1.10

Flood Forecasting A team of Google researchers has used AI to develop

AI for more accurate and reliable flood forecasts highly accurate hydrological simulation models
New research introduced in 2023 has made that are also applicable to ungauged basins.1 These
significant progress in predicting large-scale flood innovative methods can predict certain extreme flood
events. Floods, among the most common natural events up to five days in advance, with accuracy that
disasters, have particularly devastating effects in matches or surpasses current state-of-the-art models,
less developed countries where infrastructure for such as GloFAS. The AI model demonstrates superior
prevention and mitigation is lacking. Consequently, precision (accuracy of positive predictions) and recall
developing more accurate prediction methods that (ability to correctly identify all relevant instances)
can forecast these events further in advance could across a range of return period events, outperforming
yield substantial positive impacts. the leading contemporary method (Figure 5.1.11).2 The
model is open-source and is already being used to
predict flood events in over 80 countries.
Predictions of AI model vs. GloFAS across return periods

Source: Nearing et al., 2023 | Chart: 2024 AI Index report
1.00 1.00
AI model GloFAS
0.80 0.80
Precision (median) ↑
Recall (median) ↑
0.60 0.60
0.40 0.40
0.20 0.20
0.00 0.00
1 (N=3,649) 2 (N=3,675) 5 (N=3,416) 10 (N=3,087) 1 (N=3,682) 2 (N=3,691) 5 (N=3,597) 10 (N=3,321)
Return period
Figure 5.1.11
1 An ungauged basin is a watershed for which there is insufficient streamflow data to model hydrological flows.
2 A return period (recurrence interval) measures the likelihood of a particular hydrological event recurring within a specific period. For example, a 100-year flood means there is a 1% chance of
the event being equaled or exceeded in any given year.

Index Report 2024 5.2 AI in Medicine
AI models are becoming increasingly valuable in healthcare, with applications for detecting polyps to aiding clinicians
in making diagnoses. As AI performance continues to improve, monitoring its impact on medical practice becomes
increasingly important. This section highlights significant AI-related medical systems introduced in 2023, the current
state of clinical AI knowledge, and the development of new AI diagnostic tools and models aimed at enhancing
hospital administration.
5.2 AI in Medicine
Notable Medical Systems SynthSR generations
Source: Iglesias et al., 2023
This section identifies significant AI-related

medical breakthroughs of 2023 as chosen by the
AI Index Steering Committee.
SynthSR
Transforming brain scans for advanced analysis
SynthSR is an AI tool that converts clinical brain
scans into high-resolution T-1 weighted images
(Figure 5.2.1). This advancement addresses the issue
of scan quality variability, which previously limited
the use of many scans in advanced research. By
transforming these scans into T1-weighted images,
known for their high contrast and clear brain
Figure 5.2.1
structure depiction, SynthSR facilitates the creation
of detailed 3D brain renderings. Experiments using
SynthSR demonstrate robust correlations between
observed volumes at both scan and subject levels,
suggesting that SynthSR generates images closely
resembling those produced by high-resolution T1
scans. Figure 5.2.2 illustrates the extent to which
SynthSR scans correspond with ground-truth
observations across selected brain regions. SynthID
significantly improves the visualization and analysis
of brain structures, facilitating neuroscientific
research and clinical diagnostics.

SynthSR correlation with ground-truth volumes on select brain regions

Source: Iglesias et al., 2023 | Chart: 2024 AI Index report
Subject level
0.91 0.93 0.91 0.99 0.89 0.90
(n=41)
Scan and subject level
Scan level (ablated

0.79 0.79 0.76 0.99 0.74 0.54
segmentation task)
Scan level
0.79 0.83 0.77 0.99 0.76 0.60
(n=435)
W Co Su Ve Hip Am
hit rt b co n tric po yg
em ica rt ca d
att lg ica les m ala
er ray lg pu
ma ray s
tte ma
r tte
r
Brain region
Figure 5.2.2

Coupled Plasmonic Infrared Sensors ImmunoSEIRA detection principle and the setup
Source: Kavungal et al., 2023
Coupled plasmonic infrared sensors for the
detection of neurodegenerative diseases
Diagnosis of neurodegenerative diseases such as
Parkinson’s and Alzheimer’s depends on fast and
precise identification of biomarkers. Traditional
methods, such as mass spectrometry and ELISA, are
useful in that they can focus on quantifying protein
levels; however, they cannot discern changes in
structural states. This year, researchers uncovered a
new method for neurodegenerative disease diagnosis
that combined AI-coupled plasmonic infrared sensors
that use Surface-Enhanced Infrared Absorption
(SEIRA) spectroscopy with an immunoassay
technique (ImmunoSEIRA; Figure 5.2.3). In tests that
compared actual fibril percentages with predictions Figure 5.2.3
made by AI systems, the accuracy of the predictions

was found to very closely match the actual reported
percentages (Figure 5.2.4).
Deep neural network predicted vs. actual brils percentages in test samples
Source: Kavungal et al., 2023 | Chart: 2024 AI Index report
100%
80%
Predicted brils concentration (%)
60%
40%
20%
0%
0% 25% 40% 50% 60% 75% 100%

Actual brils concentration (%)
Figure 5.2.4

EVEscape
Forecasting viral evolution for pandemic of viruses (Figure 5.2.5). EVEscape evaluates
preparedness viral escape independently of current strain data
Predicting viral mutations is vital for vaccine design predicting 50.0% of observed SARS-CoV-2 mutations,
and pandemic minimization. Traditional methods, outperforming traditional lab studies which predicted
which rely on real-time virus strain and antibody data, 46.2% and 32.3%, as well as a previous model, which
face challenges during early pandemic stages due predicted only 24% of mutations (Figure 5.2.6).
to data scarcity. EVEscape is a new AI deep learning This performance highlights EVEscape’s potential
model trained on historical sequences and biophysical as a valuable asset for enhancing future pandemic
and structural information that predicts the evolution preparedness and response efforts.
EVEscape design
Source: Thadani et al., 2023
Figure 5.2.5

EVEscape vs. other models on SARS-CoV-2 RBD mutation prediction

Source: Thadani et al., 2023 | Chart: 2024 AI Index report
10 months
Pandemic start
17 months
50% 50%, EVEscape (prepandemic)
46%, Later experimental scans (pandemic ab + sera)
40%
Predicted mutations (%)
32%, Earlier experimental scans (pandemic ab)

30%
24%, Previous model
20%
10%
0%
2020-Jan 2020-Jul 2021-Jan 2021-Jul 2022-Jan 2022-Jul 2023-Jan

Pandemic date
Figure 5.2.6

AlphaMissence Hemaglobin subunit beta (HBB)

Source: Google DeepMind, 2023
Better classification of AI mutations
Scientists still do not fully understand which
genetic mutations lead to diseases. With millions of
possible genetic mutations, determining whether
a mutation is benign or pathogenic requires labor-
intensive experiments.
In 2023, researchers from Google DeepMind

unveiled AlphaMissense, a new AI model
that predicted the pathogenicity of 71 million
missense variants. Missense mutations are
genetic alterations that impact the functionality
of human proteins (Figure 5.2.7) and can lead
to various diseases, including cancer. Of the 71
million possible missense variants, AlphaMissense
classified 89%, identifying 57% as likely benign and
32% as likely pathogenic, while the remainder were
categorized as uncertain (Figure 5.2.8). In contrast,
human annotators have only been able to confirm Figure 5.2.7
the nature of 0.1% of all missense mutations.
AlphaMissense predictions
Source: Google DeepMind, 2023 | Chart: 2024 AI Index report
Likely benign Likely pathogenic Uncertain
Prediction category 57% 32% 11%
0% 20% 40% 60% 80% 100%

% of variants classi ed
Figure 5.2.8

Human Pangenome Reference Graph genome for the MHC region of the genome
Source: Google Research, 2023
Using AI to map the human genome
The human genome is a set of molecular instructions
for a human. The first human genome draft was
released in 2000 and updated in 2022. However,
the update was somewhat incomplete. It did not
incorporate various genetic mutations, like blood
type, and did not as completely map diverse ancestry
groups. Therefore, under the existing genome
reference, it would be difficult to detect diseases or
find cures in certain groups of people.
In 2023, the Human Pangenome Research Consortium,

comprising 119 scientists from 60 institutions, used AI
to develop an updated and more representative human
genome map (Figure 5.2.9). The researchers achieved
remarkable accuracy, annotating a median of 99.07% Figure 5.2.9
of protein-coding genes, 99.42% of protein-coding This latest version of the genome represents the most
transcripts, 98.16% of noncoding genes, and 98.96% comprehensive and genetically diverse mapping of the
of noncoding transcripts, as detailed in Figure 5.2.10. human genome to date.
Ensembl mapping pipeline results

Source: Liao et al., 2023 | Chart: 2024 AI Index report
99.07% 99.42% 98.16% 98.96%
100%
80%
% of identi ed (median)
60%
40%
20%
0%
Protein-coding genes Protein-coding transcripts Noncoding genes Noncoding transcripts
Genes and transcripts Figure 5.2.10

Clinical Knowledge
Evaluating the clinical knowledge of AI models remarkable improvement, with the leading system,
involves determining the extent of their medical GPT-4 Medprompt, achieving an accuracy rate of
expertise, particularly knowledge applicable in a 90.2%—an increase of 22.6 percentage points from
clinical setting. the top score in 2022 (Figure 5.2.11). Since MedQA’s
inception, AI capabilities on this benchmark have
MedQA nearly tripled, showcasing the rapid improvements of
Introduced in 2020, MedQA is a comprehensive clinically knowledgeable AI systems.
dataset derived from professional medical board
exams, featuring over 60,000 clinical questions
designed to challenge doctors.
AI performance on the MedQA benchmark has seen

MedQA: accuracy
Source: Papers With Code, 2023 | Chart: 2024 AI Index report
90% 90.20%
80%
70%
Accuracy (%)
60%
50%
40%
2019 2020 2021 2022 2023
Figure 5.2.11

Highlighted Research:
GPT-4 Medprompt
Although LLMs exhibit impressive Moreover, as noted earlier, GPT-4 Medprompt was the first to
general knowledge, it is commonly surpass the 90% accuracy mark on the MedQA benchmark.
assumed that significant fine-tuning This breakthrough not only underscores GPT-4 Medprompt’s
is required for them to excel at exceptional and potentially clinically useful medical
specialized knowledge, such as capabilities but also demonstrates that fine-tuning may not
answering medical questions. Fine- always be necessary for adapting models to specialized
tuning entails training an LLM on domains. Prompt engineering has shown to be a promising
domain-specific data. alternative strategy.
Research from Microsoft in late 2023

has overturned this assumption. GPT-4 vs. Med-PaLM 2 answering a medical question
This study employed prompt Source: Nori et al., 2023
engineering to direct GPT-4 toward

achieving remarkable performance
on the MultiMedQA benchmark
suite, a group of four challenging
medical benchmarks (Figure 5.2.12).
GPT-4 Medprompt exceeded the
performance of the top 2022 model,
Flan-PaLM 540B, in the multiple-
choice sections of several renowned
medical benchmarks, including
PubMedQA, MedMCQA, and MMLU,
by 3.0, 21.5, and 16.2 percentage
points, respectively. It also exceeded
the performance of the then state-of-
the-art Med-PaLM 2 (Figure 5.2.13). Figure 5.2.12

GPT-4 Medprompt (cont’d)
Model performance on MultiMedQA sub-benchmarks

Source: Nori et al., 2023 | Chart: 2024 AI Index report
MMLU MedMCQA PubMedQA MedQA
100%
94.25%
89.88% 90.20%
86.50% 87.37%
81.80% 81.40% 82.00%
78.02% 79.00% 79.10%
80%
75.20%
72.30% 72.40%
67.60%
60% 57.60%
Accuracy (%)
40%
20%
0%
Flan-PaLM 540B Med-PaLM 2 GPT-4 GPT-4 Medprompt
2022 2023
Figure 5.2.13

MediTron-70B
GPT-4 Medprompt is an impressive system; PaLM 2 (both closed models), it represents
however, it is closed-source, meaning its weights a significant improvement over the state-of-
are not freely available to the broader public for the-art results from 2023 and surpasses other
use. New research in 2023 has also sought to open-source models like Llama 2 (Figure 5.2.14).
advance the capabilities of open-source medical MediTron-70B’s score on MedQA is the highest
LLMs. Among this new research, MediTron-70B yet achieved by an open-source model. If medical
stands out as particularly promising. This model AI is to reach its fullest potential, it is important
achieves a respectable 70.2% accuracy on the that its capabilities are widely accessible. In this
MedQA benchmark. Although this is below the context, MediTron represents an encouraging
performance of GPT-4 Medprompt and Med- step forward.
Performance of select models on MedQA

Source: Chen et al., 2023 | Table: 2024 AI Index report
Model Release date Access type Score on MedQA
GPT-4 Medprompt November 2023 Closed 90.20%

Med-PaLM 2 April 2023 Closed 86.20%
MediTron-70B November 2023 Open 70.20%
Med-PaLM December 2022 Closed 67.20%
Llama 2 July 2023 Open 63.80%
Figure 5.2.14

Diagnosis
AI tools can also be used for diagnostic purposes including, for example, in radiology or cancer detection.
CoDoC
AI medical imaging systems demonstrate robust (the ability to accurately identify those without it).
diagnostic capabilities, yet there are instances Specifically, across four medical datasets, CoDoC’s
where they overlook diagnoses that clinicians sensitivity surpasses clinicians’ by an average of
catch, and vice versa. This observation suggests 4.5 percentage points and a standalone AI model’s
a logical integration of AI systems and clinicians’ by 6.5 percentage points (Figure 5.2.15). In terms
diagnostic abilities. In 2023, researchers unveiled of specificity, CoDoC outperforms clinicians by
CoDoC (Complementarity-Driven Deferral to an average of 2.7 percentage points across tested
Clinical Workflow), a system designed to discern datasets and a standalone predictive model by 5.7
when to rely on AI for diagnosis and when to defer percentage points. Moreover, CoDoC has been
to traditional clinical methods. CoDoC notably shown to reduce clinical workflow by 66%. These
enhances both sensitivity (the ability to correctly findings suggest that AI medical systems can be
identify individuals with a disease) and specificity integrated into clinical workflows, thereby enhancing
diagnostic accuracy and efficiency.
CoDoC vs. standalone predictive AI system and clinical readers: sensitivity
Source: Dvijotham et al., 2023 | Chart: 2024 AI Index report
CoDoC Clinician(s) Standalone predictive AI model
100% 96.70% 96.70%

90.50%89.40%90.90%
86.70%
80%
72.60%
64.90%
62.70%
Sensitivity (%)
60% 56.90%
50.00%48.30%
40%
20%
0%
UK mammography dataset US mammography dataset 1 US mammography dataset 2 TB dataset
Breast cancer detection TB detection
Task and dataset Figure 5.2.15

CT Panda
PANDA
Pancreatic ductal adenocarcinoma (PDAC) is a particularly
detection
lethal cancer, often detected too late for surgical intervention. Source:
Cao et al., 2023
Screening for PDAC in asymptomatic individuals is Figure 5.2.16
challenging due to its low prevalence and the risk of false
positives. This year, a Chinese research team developed
PANDA (pancreatic cancer detection with artificial
intelligence), an AI model capable of efficiently detecting
and classifying pancreatic lesions in X-rays (Figure 5.2.16). In
validation tests, PANDA surpassed the average radiologist in
sensitivity by 34.1% and in specificity by 6.3% (Figure 5.2.17).
In a large-scale, real-world test involving approximately
20,000 patients, PANDA achieved a sensitivity of 92.9% and
a specificity of 99.9% (Figure 5.2.18). AI medical tools like
PANDA represent significant advancements in diagnosing
challenging conditions, offering cost-effective and accurate
detection previously considered difficult or prohibitive.
PANDA vs. mean radiologist on multicenter validation PANDA performance on real-world multi-scenario
(6,239 patients) validation (20,530 patients)
Source: Cao et al., 2023 | Chart: 2024 AI Index report Source: Cao et al., 2023 | Chart: 2024 AI Index report
99.90%
100%
35% 34.10% 92.90%
30% 80%
Score for lesion detection
Performance di�erence
25%
60%
20%
15% 40%
10%
6.30% 20%
5%
0% 0%
Sensitivity Speci�city Sensitivity Speci city

Other Diagnostic Uses

New research published in 2023 highlights how AI can be used in other diagnostic contexts. Figure 5.2.19
summarizes some of the findings.
Additional research on diagnostic AI use cases

Source: AI Index, 2024
Research Use case Findings
Schopf et al., 2023 Breast cancer The authors conducted a meta-review of the literature exploring mammography-image-based
AI algorithms. They discovered that predicting future breast cancer risk using only
mammography images achieves accuracy that is comparable to or better than traditional risk
assessment tools.
Dicente Cid et al., 2023 X-ray interpretation The researchers developed two open-source neural networks, X-Raydar and X-Raydar-NLP,
for classifying chest X-rays using images and free-text reports. They found that these
automated classi cation methods perform at levels comparable to human experts and
demonstrate robustness when applied to external data sets.
Figure 5.2.19

FDA-Approved AI-Related Medical Devices

The U.S. Food and Drug Administration (FDA) Figure 5.2.20 illustrates the number of AI medical
maintains a list of AI/ML-enabled medical devices devices approved by the FDA over the past decade.
that have received approval. The devices featured In 2022, a total of 139 AI-related medical devices
on this list meet the FDA’s premarket standards, received FDA approval, marking a 12.1% increase from
which include a detailed review of their effectiveness the total approved in 2021. Since 2012, the number of
and safety. As of October 2023, the FDA has not these devices has increased by more than 45-fold.
approved any devices that utilize generative AI or are
powered by LLMs.
Number of AI medical devices approved by the FDA, 2012–22

Source: FDA, 2023 | Chart: 2024 AI Index report
139
140
124
120
107
Number of AI medical devices
100
80 77
63
60
40
26
20 18
6 5
3 3
0
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 5.2.20
3 The FDA last updated the list in October 2023, meaning that the totals for 2023 were incomplete. Consequently, the AI Index limited its data presentation to include only information
up to 2022.

Figure 5.2.21 illustrates the specialties associated with FDA-approved medical devices. Of the 139 devices
approved in 2022, a significant majority, 87.1%, were related to radiology. The next most common specialty was
cardiovascular, accounting for 7.2% of the approvals.
Number of AI medical devices approved by the FDA by specialty, 2012–22

Source: FDA, 2023 | Chart: 2024 AI Index report
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Radiology 2 0 5 0 11 15 39 51 94 105 121
Cardiovascular 0 0 0 1 4 6 9 12 7 11 10
Neurology 0 0 1 0 1 1 4 4 0 2 2
Gastroenterology and urology 0 0 0 0 0 1 1 1 0 3 1

Hematology 0 1 0 0 0 2 2 1 3 0 1
Microbiology 0 2 0 0 0 0 0 2 1 0 0
Medical specialty
General hospital 0 0 0 1 0 0 0 2 0 0 0
General and plastic surgery 0 0 0 0 1 0 2 1 0 1 0
Ophthalmic 0 0 0 1 0 0 2 1 1 1 2
Clinical chemistry 0 0 0 1 1 0 2 1 0 0 1
Anesthesiology 0 0 0 1 0 1 0 0 1 0 0
Pathology 1 0 0 0 0 0 0 0 0 1 0
Ear nose and throat 0 0 0 0 0 0 0 0 0 0 1
Dental 0 0 0 0 0 0 1 0 0 0 0
Orthopedic 0 0 0 0 0 0 1 0 0 0 0
Obstetrics and gynecology 0 0 0 0 0 0 0 1 0 0 0
Figure 5.2.21

Administration and Care

AI tools also hold the potential to enhance medical administration efficiency and elevate the standard of patient care.
MedAlign
Despite significant advances benchmark with 983 questions and instructions and 303 clinician
in AI for healthcare, existing responses, drawn from seven different medical specialties (Figure
benchmarks like MedQA and 5.2.22). MedAlign is the first extensive EHR-focused benchmark.
USMLE, focused on knowledge-
The researchers then tested various existing LLMs on MedAlign. Of
based questions, do not fully
all LLMs, a GPT-4 variant using multistep refinement achieved the
capture the diverse tasks
highest correctness rate (65.0%) and was routinely preferred over
clinicians perform in patient
other LLMs (Figure 5.2.23). MedAlign is a valuable milestone toward
care. Clinicians often engage
using AI to alleviate administrative burdens in healthcare.
in information-intensive tasks,
such as creating tailored
MedAlign workflow
diagnostic plans, and spend a Source: Fleming et al., 2023
significant proportion of their

working hours on administrative
tasks. Although AI has the
Figure 5.2.16
potential to streamline these
processes, there is a lack of
suitable electronic health
records (EHR) datasets for
benchmarking and fine-tuning
medically administrative LLMs.
This year researchers have
made strides to address this
gap by introducing MedAlign:
a comprehensive EHR-based Figure 5.2.22

MedAlign (cont’d)
Evaluation of model performance: human vs. COMET ranks
Source: Fleming et al., 2023 | Chart: 2024 AI Index report
Human ranks COMET ranks
GPT-4 (32k + MR) 48% 56% 73% 71% 82% GPT-4 (32k + MR) 50% 52% 66% 63% 79%
GPT-4 (32k) 52% 58% 72% 74% 81% GPT-4 (32k) 50% 51% 63% 58% 77%
Model A (winner)
Model A (winner)
GPT-4 (2k) 44% 42% 67% 70% 76% GPT-4 (2k) 48% 49% 66% 61% 79%
Vicuña-13B (2k) 27% 28% 33% 50% 63% Vicuña-13B (2k) 34% 37% 34% 49% 70%
Vicuña-7B (2k) 29% 26% 30% 50% 64% Vicuña-7B (2k) 37% 42% 39% 51% 71%
MPT-7B-Instruct (2k) 18% 19% 24% 37% 36% MPT-7B-Instruct (2k) 21% 23% 21% 30% 29%
GPT-4 (32k + MR)
GPT-4 (32k)
GPT-4 (2k)
Vicuña-13B (2k)
Vicuña-7B (2k)
MPT-7B-Instruct (2k)
GPT-4 (32k + MR)
GPT-4 (32k)
GPT-4 (2k)
Vicuña-13B (2k)
Vicuña-7B (2k)
MPT-7B-Instruct (2k)
Model B (loser) Model B (loser)
Figure 5.2.23

Artificial Intelligence CHAPTER 6:
Index Report 2024
Education
CHAPTER 6:
Index Report 2024 Education
Preview
Overview 327
Chapter Highlights 328
6.1 Postsecondary CS and AI Education 329

United States and Canada 329
CS Bachelor’s Graduates 329
CS Master’s Graduates 331
CS PhD Graduates 333
CS, CE, and Information Faculty 336
Europe 344
Informatics, CS, CE, and IT Bachelor’s Graduates 344
Informatics, CS, CE, and IT Master’s Graduates 347
Informatics, CS, CE, and IT PhD Graduates 351
AI-Related Study Programs 355
Total Courses 355
Education Level 356
Geographic Distribution 357
6.2 K–12 CS and AI Education 359

United States 359
State-Level Trends 359
AP Computer Science 361
Highlight: Access Issues 363
Highlight: ChatGPT Usage Among
Teachers and Students 364
ACCESS THE PUBLIC DATA

CHAPTER 6:
Overview
This chapter examines trends in AI and computer science (CS) education, focusing on who is
learning, where they are learning, and how these trends have evolved over time. Amid growing
concerns about AI’s impact on education, it also investigates the use of new AI tools like
ChatGPT by teachers and students.
The analysis begins with an overview of the state of postsecondary CS and AI education in the
United States and Canada, based on the Computing Research Association’s annual Taulbee
Survey. It then reviews data from Informatics Europe regarding CS education in Europe. This
year introduces a new section with data from Studyportals on the global count of AI-related
English-language study programs.
The chapter wraps up with insights into K–12 CS education in the United States from Code.org
and findings from the Walton Foundation survey on ChatGPT’s use in schools.

CHAPTER 6:
Chapter Highlights
1. The number of American and Canadian CS bachelor’s graduates continues to rise, new
CS master’s graduates stay relatively flat, and PhD graduates modestly grow. While the
number of new American and Canadian bachelor’s graduates has consistently risen for more than a decade, the
number of students opting for graduate education in CS has flattened. Since 2018, the number of CS master’s and
PhD graduates has slightly declined.
2. The migration of AI PhDs to industry continues at an accelerating pace. In 2011, roughly

equal percentages of new AI PhDs took jobs in industry (40.9%) and academia (41.6%). However, by 2022, a
significantly larger proportion (70.7%) joined industry after graduation compared to those entering academia
(20.0%). Over the past year alone, the share of industry-bound AI PhDs has risen by 5.3 percentage points,
indicating an intensifying brain drain from universities into industry.
3. Less transition of academic talent from industry to academia. In 2019, 13% of new AI faculty in the
United States and Canada were from industry. By 2021, this figure had declined to 11%, and in 2022, it further dropped
to 7%. This trend indicates a progressively lower migration of high-level AI talent from industry into academia.
4. CS education in the United States and Canada becomes less international. Proportionally
fewer international CS bachelor’s, master’s, and PhDs graduated in 2022 than in 2021. The drop in international
students in the master’s category was especially pronounced.
5. More American high school students take CS courses, but access problems remain.
In 2022, 201,000 AP CS exams were administered. Since 2007, the number of students taking these exams has
increased more than tenfold. However, recent evidence indicates that students in larger high schools and those in
suburban areas are more likely to have access to CS courses.
6. AI-related degree programs are on the rise internationally. The number of English-language,
AI-related postsecondary degree programs has tripled since 2017, showing a steady annual increase over the past
five years. Universities worldwide are offering more AI-focused degree programs.
7. The United Kingdom and Germany lead in European informatics, CS, CE, and IT graduate
production. The United Kingdom and Germany lead Europe in producing the highest number of new informatics,
CS, CE, and information bachelor’s, master’s, and PhD graduates. On a per capita basis, Finland leads in the
production of both bachelor’s and PhD graduates, while Ireland leads in the production of master’s graduates.

Artificial Intelligence Chapter 6: Education
Index Report 2024 6.1 Postsecondary CS and AI Education
This section provides an overview of postsecondary education in CS and AI, highlighting graduation statistics across
North America and Europe for various degrees including bachelor’s, master’s, and PhDs. It also covers information on
AI-related courses offered in English.
6.1 Postsecondary CS and AI Education

CS Bachelor’s Graduates
United States and Canada Over the past decade, the total number of new CS
This subsection presents an analysis of data from the bachelor’s graduates in North America has steadily
Computing Research Association’s Taulbee Survey, risen, increasing more than threefold, with a 7.9%
which evaluates the state of CS and AI postsecondary year-over-year rise from 2021 to 2022 (Figure 6.1.1).
education in the United States and Canada. The
survey covers 297 PhD-granting CS departments
across the United States and Canada.1
New CS bachelor’s graduates in the United States and Canada, 2010–22

Source: CRA Taulbee Survey, 2023 | Chart: 2024 AI Index report
35,666
35,000
33,059
31,835
30,000 28,527
Number of new CS bachelor’s graduates
26,709
25,000
22,343
20,000 18,954
15,256
15,000
12,228
11,049 10,776
10,000 9,008 9,286
5,000
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.1
1 It is important to note that not all PhD-granting departments targeted in the survey provided responses. Out of the 297 departments targeted, only 182 responded, yielding an overall
response rate of 61%.

For the first time in almost eight years, the proportion the data. The decline is also partially attributable to
of international students among CS bachelor’s international travel restrictions that were imposed
graduates in American and Canadian universities during the COVID-19 pandemic, affecting the ability
declined, falling from 16.3% in 2021 to 15.2% in of international students to study in the United States
2022 (Figure 6.1.2). This decline likely reflects the and Canada. Despite this recent drop, the overall
increased difficulty of obtaining study visas during trend over the last decade shows a steady increase in
the early years of the Trump administration, an the proportion of international students.
impact that is only now beginning to manifest in
New international CS bachelor’s graduates (% of total) in the United States and Canada, 2010–22
16%
15.20%
New international CS bachelor’s graduates (% of total)
12%
8%
4%
0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.2

CS Master’s Graduates
AI courses are commonly included in CS master’s the number appears to have leveled out since 2018 and
degree programs. While the total number of new CS slightly decreased, by 2.5%, last year (Figure 6.1.3). This
master’s graduates from American and Canadian leveling is a reflection of the decline in international
universities more than doubled over the past decade, master’s students shown in the following graph.
New CS master’s graduates in the United States and Canada, 2010–22

16,000 15,532
14,853 15,068
14,735 14,696
14,000
13,037
Number of new CS master’s graduates
12,000
11,239
9,933
10,000
8,000 7,462 7,488

7,205
6,851 6,611
6,000
4,000
2,000
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.3

In 2022, American and Canadian universities experienced a notable decrease in international CS master’s students.
This downward trend began around 2017, but the decline was most pronounced last year, at 14.8 percentage points
(Figure 6.1.4). Currently, the split between international and domestic CS master’s graduates is roughly even.
New international CS master’s graduates (% of total) in the United States and Canada, 2010–22
80%
New international CS master’s graduates (% of total)
60%
50.40%
40%
20%
0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.4

CS PhD Graduates
For the first time in a decade, there has been a significant increase in the number of new CS PhD graduates at
American and Canadian universities. In 2022, the number of CS PhD graduates reached 2,105, the highest since
2010 (Figure 6.1.5).
New CS PhD graduates in the United States and Canada, 2010–22

2,105
1,991 1,997
2,000 1,929 1,940
1,888 1,860 1,893
1,834
1,772 1,782 1,780 1,787
Number of new CS PhD graduates
1,500
1,000
500
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.5

While the proportion of international students among CS PhD graduates has risen over the past decade, there was
a slight decrease in this proportion in the last year, dropping from 68.6% in 2021 to 65.9% in 2022 (Figure 6.1.6).
New international CS PhD graduates (% of total) in the United States and Canada, 2010–22
70%
65.90%
60%
New international CS PhD graduates (% of total)
50%
40%
30%
20%
10%
0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.6

Where do newly minted AI PhDs choose to work However, by 2022, a significantly larger proportion
after graduating? Following a trend highlighted in last (70.7%) joined industry after graduation compared
year’s AI Index report, a growing share of AI doctoral to those entering academia (20.0%). The percentage
recipients are pursuing careers in industry (Figure 6.1.7 of new AI PhDs going into government roles has
and Figure 6.1.8). In 2011, around the same percentage remained relatively low and steady at around 0.7%
took jobs in industry (40.9%) as in academia (41.6%). over the past five years.
Employment of new AI PhDs (% of total) in the Employment of new AI PhDs in the United States and
United States and Canada by sector, 2010–22 Canada by sector, 2010–22
Source: CRA Taulbee Survey, 2023 | Chart: 2024 AI Index report Source: CRA Taulbee Survey, 2023 | Chart: 2024 AI Index report
Academia 362
70% 70.71%, Industry
350
Government
Industry 79
60%
300
New AI PhD graduates (% of total)
Number of new AI PhD graduates
281
50% 249
250 238 84
219
40% 201
65
200 73
178 61
30% 154 154
63
150 60
134 132 136 280
47 123
20% 19.95%, Academia 72 43
63 51
100 42 195
180
162
10% 153
134
116
50 101
85
76 74 77
0% 0.76%, Government 64
0
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2 The sums in Figure 6.1.7 do not add up to 100, as there is a subset of new AI PhDs each year who become self-employed, unemployed, or report an “other” employment status in the CRA
survey. These students are not included in the chart.

CS, CE, and Information Faculty

To better understand trends in CS and AI education, it is helpful to examine data on CS faculty. Last year, the total
number of CS, CE, and information faculty in American and Canadian universities increased 7.2% (Figure 6.1.9).
Since 2011, the increase is 42.4%.
Number of CS, CE, and information faculty in the United States and Canada, 2011–22
10,000
Tenure Track Teaching Professors Other Instructors Research Postdoc
9,000 8,738
531
8,149
7,858 7,976
8,000 7,657 522
Number of CS, CE, and information faculty
530
7,362 668 919
653
7,000 6,806 6,887 691 426
6,629 465 736 861
6,478 589
6,314 649 432 617
6,138 689 494 1,252
6,000 602 766 390
432 1,150
656 895 1,183
529 1,180 831
515 676 1,122
447 1,014
5,000 661 863
669 487
4,000
3,000 5,733
5,059 5,214 5,252 5,231 5,310
4,536 4,549 4,548 4,711 4,786
4,366
2,000
1,000
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.9

In 2022, the United States had 7,084 CS faculty members, with the majority (65.7%) on the tenure track (Figure
6.1.10). The total number of American CS faculty has risen 4.4% since 2021 and 45.0% since 2011.
Number of CS faculty in the United States, 2011–22

8,000
Tenure Track Teaching Professors Other Instructors Research Postdoc
7,084
7,000 6,789
6,654 453
6,430 6,533 428
424
6,098 531
518
6,000 5,729 736
5,637 567 382 693
426 618
5,256 491
5,202 535 408 436
534
5,068 947
Number of CS faculty
5,000 4,885 521 509 396

364
899
592 946
947 671 715
522
460 455 903
491 826
387
4,000 550 679
521 421
3,000
4,482 4,657
4,366 4,384 4,390
2,000 3,971 4,176
3,725 3,880
3,455 3,564 3,559
1,000
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.10

Last year, 915 new faculty were hired across CS, CE, and information disciplines in North America, a decade high.
455 of these positions were tenure track. (Figure 6.1.11).
New CS, CE, and information faculty hires in the United States and Canada, 2011–22
Total Tenure-Track 915

878
860
800
Number of new CS, CE, and information faculty hires
800 765
749 749
733
710
691
600 583 572

543
455
422
396 406
400 374
348 358
320 324
294
249 258
218
200
0
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.11

In 2022, 43% of new faculty appointments came from other academic positions, indicating a “churn” within
the academic workforce (Figure 6.1.12). Since these “new” faculty members vacated positions elsewhere, their
previous roles will eventually need to be filled. Additionally, the proportion of faculty transitioning from industry
in 2022 fell to 7% from 11% in the previous year and 13% in 2019.
Source of new faculty in American and Canadian CS, CE, and information departments, 2018–22
New PhD From Postdoc From Other Academic From Industry
100%
7%
13% 13% 11% 11%
80%
Source of new faculty
34% 34% 43%

34%
41%
60%
16% 15%
15%
16%
40%
17%
20% 38% 39% 40%

34%
29%
0%
2018 2019 2020 2021 2022
Figure 6.1.12

The reasons for faculty positions remaining unfilled were turned down. This trend appears to reflect an
have varied over the past decade. In 2011, 37% of failed increasingly competitive market for new CS faculty.
searches were due to no offer being made, while 34% However, it remains unclear whether this indicates
were because the offer made was declined (Figure heightened competition with other academic positions
6.1.13). In contrast, in 2022, only 15% ended with no or with industry positions.
offer being made, while 55% involved offers that
Reason why new CS, CE, and information faculty positions remained un lled (% of total), 2011–22
Didn’t �nd a person who met our hiring goals Technically vacant, not �lled for admin reasons Other
O�ers turned down Hiring in progress
Reason faculty positions remained un�lled (% of total)
100%
10% 18%
10% 23% 23%
25% 26% 22%
31% 28% 27%
80% 17%
18%
5% 6%
10%
55%
6%
60% 45% 12%
34%
40%
36% 55%
40% 52% 51% 56% 53%
43%
44%
20% 37% 37% 37%

26% 26%
16% 14% 14% 13% 14% 15%
8%
0%
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.13

In 2022, North American departments in CS, CE, and information disciplines experienced a significant increase
in faculty departures, totaling 405, compared to 303 in 2021 (Figure 6.1.14). Of these losses, 38.5% left for other
academic positions, while 16.3% moved to nonacademic roles, maintaining a trend consistent with previous years.
Faculty losses in American and Canadian CS, CE, and information departments, 2011–22
Died Took academic position elsewhere Remained, but changed to part-time Unknown
450
Retired Took nonacademic position Other
405
400
350
327
312 66
303 303
300
270
Faculty losses
23 43 37
250 246
232 237 234 34 46
221 33
213
42
200 156
36 24
44 26 139
27 32 110
126
150 113
34 77 89
62 85
74 86
100 52
112
94 90 103 100
50 89 94 91
67 74 65 80
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.14

Since 2015, the increase in median nine-month salaries for full professors has slightly fallen below U.S. inflation
rates, whereas median salaries for assistant and associate professors have seen slight increases above inflation.
In 2022, a full professor’s salary was 3.2% higher than in 2021, which did not keep pace with the 7% U.S. inflation
rate, and 16.4% higher than in 2015, still below the 19% inflation increase over those years (Figure 6.1.15).
Median nine-month salary of CS faculty in the United States, 2015–22

Full Professor Associate Professor Assistant Professor 181.61

180 176.01
168.87 170.57
164.54
Median salary of CS faculty (in thousands of U.S. dollars)
158.97 159.96
160 156.02
140 134.08
127.47
121.55 123.71
117.5 119.48 119.03
120 113.95 114.07
111.67 109.23
105.45 107.55
101.16 103.01
99.12
100
80
60
40
20
0
2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.15

In 2022, the proportion of international hires among new tenure-track faculty in CS, CE, and information
disciplines significantly increased to 19.3% from 13.2% the previous year (Figure 6.1.16). This marked the second-
highest percentage recorded in the past decade, only surpassed by 2013.
New international CS, CE, and information tenure-track faculty hires (% of total) in the United States and
Canada, 2010–22
25%
New international tenure-track faculty hires (% of total)
20%
19.30%
15%
10%
5%
0%
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.1.16

Europe Informatics, CS, CE, and IT Bachelor’s

Graduates
Data on European CS graduates comes from In 2022, the United Kingdom led with the highest
Informatics Europe, an academic and research number of new graduates in informatics, CS, CE,
community that, among other goals, monitors the and IT at the bachelor’s level, totaling approximately
state of informatics education in Europe.3 Informatics 25,000 (Figure 6.1.17).5 Germany and Turkey followed
Europe gathers data on graduates in informatics, closely. Most countries in the sample saw an increase
CS, CE, computing, and information technology in graduates in these fields compared to a decade
(IT) disciplines from statistical offices of European ago, though there were exceptions like Poland, Spain,
governments.4 and the Czech Republic (Figure 6.1.18).
New informatics, CS, CE, and IT bachelor’s graduates by country in Europe, 2022
Source: Informatics Europe, 2023 | Chart: 2024 AI Index report
United Kingdom 25,040
Germany 19,920
Turkey 13,826
Poland 9,241
Italy 8,508
Netherlands 6,877
Romania 6,740
Spain 6,650
Portugal 3,696
Finland 2,970
Czech Republic 2,670
Norway 2,340
Ireland 1,995
Austria 1,900
Switzerland 1,789
0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 22,000 24,000
Number of new informatics, CS, CE, and IT bachelor’s graduates
Figure 6.1.17
3 There is no singular term for CS education that is used uniformly across European countries. Across Europe, CS education can be reflected in terms such as informatics, computer science
(CS), computer engineering (CE), computing, information technology (IT), information and communication technology (ICT), and information science and technology (IST). The full list of
subject names (and English translations) that Informatics Europe uses to identify informatics studies programs can be found at the following link.
4 Readers are cautioned against making per capita comparisons between the CRA North American data and the European CS graduate data detailed in subsequent sections, as the European
data is collected from national statistical offices and boasts broader coverage.
5 Note that not all countries for which the AI Index has data are visualized in the figures in this section. To access the complete data, please view the public data associated with this chapter.
Moreover, the year label refers to the year in which an academic year ends. For example, the figures visualizing new graduates for 2022 reflect the number of graduates reported for the
2021/2022 academic year. For the sake of visual simplicity, the Index opts to focus on the year in which students graduated.

Percentage change of new informatics, CS, CE, and IT bachelor’s graduates by country in Europe,
2012 vs. 2022
Norway 153%
Turkey 118%
Switzerland 117%
Netherlands 88%
Portugal 72%
Romania 70%
Ireland 66%
Austria 59%
Italy 48%
United Kingdom 39%
Germany 28%
Finland 23%
Spain -14%
Poland -15%
Czech Republic -21%
−20% 0% 20% 40% 60% 80% 100% 120% 140% 160%

% change of new informatics, CS, CE, and IT bachelor’s graduates
Figure 6.1.18
Finland (53.4), Norway (42.6), and the Netherlands (38.6) lead in the number of new bachelor’s graduates in
informatics CS, CE, and IT per 100,000 inhabitants (Figure 6.1.19). On a per capita basis, most sampled European
countries have seen increases in the total number of informatics, CS, CE, and IT bachelor’s graduates (Figure 6.1.20).

New informatics, CS, CE, and IT bachelor’s graduates per 100,000 inhabitants by country in Europe, 2022
Finland 53.38
Norway 42.63
Netherlands 38.61
Ireland 38.41
United Kingdom 37.36
Romania 35.38
Portugal 35.31
Estonia 34.78
Lithuania 33.42
Poland 25.14
Bulgaria 25.14
Czech Republic 24.66
Germany 23.61
Latvia 22.73
Austria 20.87
0 10 20 30 40 50
New informatics, CS, CE, and IT bachelor’s graduates (per 100,000 inhabitants)
Figure 6.1.19
Percentage change of new CS, CE, and Information bachelor’s graduates per 100,000 inhabitants by country
in Europe, 2012 vs. 2022
Norway 133%
Romania 78%
Netherlands 77%
Portugal 72%
Bulgaria 67%
Ireland 48%
Austria 47%
United Kingdom 33%
Estonia 25%
Germany 22%
Finland 20%
Poland -12%
Czech Republic -23%
Latvia -33%
−40% −20% 0% 20% 40% 60% 80% 100% 120% 140%

% change of new informatics, CS, CE, and IT bachelor’s graduates (per 100,000 inhabitants)
Figure 6.1.20

Informatics, CS, CE, and IT Master’s Graduates graduates (Figure 6.1.21). In the last decade, Germany
Similar to bachelor’s graduates, the United Kingdom (259%), Turkey (197%), and Spain (194%) have seen the
leads Europe in producing new master’s graduates in greatest percentage growth in new informatics, CS, CE,
informatics, CS, CE, and IT, with approximately 20,000 and IT master’s graduates (Figure 6.1.22).
New informatics, CS, CE, and IT master’s graduates by country in Europe, 2022
Germany 11,092
Poland 4,271
Italy 3,883
Spain 3,214
Turkey 2,677
Romania 2,200
Netherlands 1,953
Ireland 1,620
Czech Republic 1,589
Finland 1,458
Austria 1,327
Switzerland 1,202
Portugal 1,109
Norway 770
0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000
Number of new informatics, CS, CE, and IT master’s graduates
Figure 6.1.21

Percentage change of new informatics, CS, CE, and IT master’s graduates by country in Europe,
2012 vs. 2022
Germany 259%
Turkey 197%
Spain 194%
Norway 191%
Switzerland 148%
Netherlands 147%
Ireland 140%
United Kingdom 139%
Portugal 61%
Finland 60%
Austria 54%
Italy 46%
Poland 8%
Czech Republic -36%
−30% 0% 30% 60% 90% 120% 150% 180% 210% 240% 270%
% change of new informatics, CS, CE, and IT master’s graduates
Figure 6.1.22

Per capita metrics paint a somewhat similar picture. Ireland has the most informatics, CS, CE, and IT master’s
graduates on a per capita basis (31.2), followed by the United Kingdom (29.8) and Estonia (27.4) (Figure 6.1.23).
On a per capita basis, Germany (243%) has also seen the greatest growth of informatics, CS, CE, and IT master’s
graduates in the last decade (Figure 6.1.24).
New informatics, CS, CE, and IT master’s graduates per 100,000 inhabitants by country in Europe, 2022
Ireland 31.19
United Kingdom 29.79
Estonia 27.38
Finland 26.20
Czech Republic 14.68
Austria 14.57
Norway 14.03
Switzerland 13.64
Germany 13.15
Denmark 12.66
Poland 11.62
Romania 11.55
Netherlands 10.96
Bulgaria 10.70
Portugal 10.59
0 5 10 15 20 25 30
New informatics, CS, CE, and IT master’s graduates (per 100,000 inhabitants)
Figure 6.1.23

Percentage change of new informatics, CS, CE, and IT master’s graduates per 100,000 inhabitants
by country in Europe, 2012 vs. 2022
Germany 243%
Estonia 189%
Norway 167%
Netherlands 132%
United Kingdom 128%
Switzerland 127%
Ireland 113%
Bulgaria 80%
Portugal 61%
Finland 56%
Denmark 56%
Austria 43%
Poland 11%
Czech Republic -38%
−30% 0% 30% 60% 90% 120% 150% 180% 210% 240%

% change of new informatics, CS, CE, and IT master’s graduates (per 100,000 inhabitants)
Figure 6.1.24

Informatics, CS, CE, and IT PhD Graduates 6.1.25). In the last decade, Turkey has seen the greatest
The United Kingdom (1,060) and Germany (910) also growth in new CS, CE, and information PhD graduates
produced the most informatics, CS, CE, and IT PhD (Figure 6.1.26).
graduates in 2022, followed by Italy (581) (Figure
New informatics, CS, CE, and IT PhD graduates by country in Europe, 2022
Germany 910
Italy 581
Spain 479
Turkey 298
Czech Republic 150
Netherlands 120
Switzerland 116
Finland 114
Austria 96
Portugal 72
Ireland 65
Romania 47
Estonia 26
Bulgaria 24
0 100 200 300 400 500 600 700 800 900 1,000 1,100
Number of new informatics, CS, CE, and IT PhD graduates
Figure 6.1.25

Percentage change of new informatics, CS, CE, and IT PhD graduates by country in Europe, 2012 vs. 2022
Turkey 173%
Estonia 86%
Bulgaria 60%
United Kingdom 18%
Spain 14%
Italy 12%
Switzerland 12%
Portugal 3%
Germany 1%
Ireland -4%
Finland -18%
Czech Republic -19%
Austria -31%
−40% −20% 0% 20% 40% 60% 80% 100% 120% 140% 160% 180%
% change of new informatics, CS, CE, and IT PhD graduates
Figure 6.1.26

Finland has the greatest number of new informatics, capita basis, the growth rate of new informatics, CS,
CS, CE, and IT PhD graduates per capita. For every CE, and IT PhDs has been relatively marginal in several
100,000 inhabitants, it has 2.1 informatics, CS, CE, major European countries such as the United Kingdom,
and IT PhD graduates (Figure 6.1.27). Estonia slightly Portugal, and Switzerland (Figure 6.1.28).
trails (1.9), as does the United Kingdom (1.6). On a per
New informatics, CS, CE, and IT PhD graduates per 100,000 inhabitants by country in Europe, 2022
Finland 2.05
Estonia 1.90
United Kingdom 1.58
Czech Republic 1.39
Switzerland 1.32
Ireland 1.25
Germany 1.08
Austria 1.05
Spain 1.00
Italy 0.99
Portugal 0.69
Netherlands 0.67
Latvia 0.64
Bulgaria 0.37
Turkey 0.35
0.00 0.30 0.60 0.90 1.20 1.50 1.80 2.10

New informatics, CS, CE, and IT PhD graduates (per 100,000 inhabitants)
Figure 6.1.27

Percentage change of new informatics, CS, CE, and IT PhD graduates per 100,000 inhabitants by country
in Europe, 2012 vs. 2022
Turkey 142%
Bulgaria 81%
Estonia 79%
Italy 14%
United Kingdom 12%
Spain 10%
Portugal 3%
Switzerland 2%
Germany -4%
Latvia -8%
Ireland -15%
Finland -20%
Czech Republic -22%
Austria -36%
−40% −20% 0% 20% 40% 60% 80% 100% 120% 140%

% change of new informatics, CS, CE, and IT PhD graduates (per 100,000 inhabitants)
Figure 6.1.28

AI-Related Study Programs Total Courses

A study program, or degree program, comprises a
Tracking the number of AI-related courses provides
series of courses designed to enable students to earn
insight into the educational interest in AI. This section
a relevant qualification, such as a degree or diploma.
highlights data from Studyportals, an international
The number of English-language AI-related study
platform monitoring English-language university
programs has tripled since 2017, demonstrating a
study programs worldwide. Their portal encompasses
consistent yearly increase over the last five years
information on over 200,000 courses at more than
(Figure 6.1.29). This trend indicates a steadily growing
3,750 educational institutions across 110 countries.6
educational interest in AI.
Number of AI university study programs in English in the world, 2017–23

Source: Studyportals, 2023 | Chart: 2024 AI Index report
3.00
Number of AI university study programs in English (in thousands)
2.52
2.50
2.32
2.16
2.00
1.64
1.50 1.45
1.17
1.00 0.88
0.50
0.00
2017 2018 2019 2020 2021 2022 2023
Figure 6.1.29
6 Currently, Studyportals, the company supplying data on AI university study programs, tracks only English-language AI courses. In the coming years, the Index plans to extend its coverage to
include non-English programs.

Education Level
Broken down by educational level, the majority of AI study programs are offered at the master’s level (55.0%),
followed by the bachelor’s level (39.8%), and finally at the PhD level (5.3%) (Figure 6.1.30).
AI university study programs in English (% of total) by education level, 2023

60%
54.97%
AI university study programs in English (% of total)
50%
39.77%
40%
30%
20%
10%
5.27%
0%
Bachelor’s Master’s PhD
Figure 6.1.30

Geographic Distribution the sample, there was a greater number of AI university

In 2023, the United Kingdom had the greatest study programs in 2023 than in 2022. Malta, the United
number of English-language AI study programs (744) Kingdom, and Cyprus had the greatest number of
(Figure 6.1.31). Next was the United States (667) and English-language AI university study programs per
Canada (89). For virtually every country included in capita in 2023 (Figure 6.1.32).7
Number of AI university study programs in English by geographic area, 2022 vs. 2023
United Kingdom 744 (+12%)

665
United States 667 (+6%)

630
Canada 89 (+2%)
87
Australia 84 (+1%)
83
Germany 80 (+54%)
52
France 69 (+15%)
60
Netherlands 59 (+7%)
55
India 45 (+2%)
44
Ireland 44 (+5%)
42
Spain 43 (+13%)
38
Italy 40 (+21%)
33
Malaysia 39 (+8%)
36
United Arab Emirates 38 (+36%)

28
2023
Sweden 37 (+16%)
32 2022
China 32 (-16%)
38
0 100 200 300 400 500 600 700 800
Number of AI university study programs in English
Figure 6.1.31
7 Although the United Kingdom has fewer universities overall compared to the United States, it likely reports a higher number of AI study programs for several reasons. Firstly, Studyportals
has slightly greater coverage of the United Kingdom than the United States in its data. Secondly, the structure of higher education in the United States tends to be more generalist compared
to the United Kingdom, meaning students studying AI might be enrolled in broader computer science programs that are not explicitly identified as AI study programs.

AI university study programs in English per 100,000 inhabitants by geographic area, 2022 vs. 2023
Malta 2.03 (+32.15%)

1.54
United Kingdom 1.10 (+11.50%)

0.99
Cyprus 0.98 (-11.56%)

1.11
Ireland 0.85 (+2.05%)

0.83
Lithuania 0.52 (+13.31%)

0.46
United Arab Emirates 0.40 (+34.63%)

0.30
Sweden 0.35 (+14.86%)

0.31
Finland 0.34 (+11.45%)

0.31
Netherlands 0.33 (+5.94%)

0.31
Australia 0.32 (-0.45%)

0.32
Norway 0.27 (+5.90%)

0.26
Latvia 0.27 (+66.02%)

0.16
Iceland 0.26 (-2.97%)

0.27
2023
Canada 0.22 (-0.86%)
0.22 2022
United States 0.20 (+5.87%)
0.19
0.00 0.50 1.00 1.50 2.00
Number of AI university study programs in English (per 100,000 inhabitants)
Figure 6.1.32

Index Report 2024 6.2 K–12 CS and AI Education
This section presents trends in high school CS education in the United States as a representation of K–12 AI education.
6.2 K–12 CS and AI Education

United States States requiring that all high schools offer a foundational CS course,
2023
Source: Code.org, 2023 | Chart: 2024 AI Index report
Data on the state of K–12
AK ME
CS education in the United
States comes from Code. VT NH MA
org, an education innovation
nonprofit dedicated to WA MT ND SD MN WI MI NY CT RI
ensuring that every school
OR ID WY NE IA IL IN OH PA NJ
includes CS as part of its core
K–12 education. CA NV UT CO KS MO KY WV DC MD DE
State-Level Trends AZ NM OK AR TN VA NC
In 2023, 30 American states
TX LA MS AL GA SC
required that all high schools
offer a foundational course in HI FL Yes
No
CS (Figure 6.2.1).
Figure 6.2.1
The percentage of public
Public high schools teaching foundational CS (% of total in state),
schools offering CS courses 2023
varies significantly from state
AK ME
to state (Figure 6.2.2). The 51% 66%
top three states in terms of VT NH MA
76% 81% 83%
percentage of CS offerings
are Maryland (99%), WA MT ND SD MN WI MI NY CT RI
47% 34% 47% 44% 28% 56% 55% 58% 84% 86%
Arkansas (99%), and Nevada
(96%); the bottom three are 64% 38% 63% 50% 84% 54% 91% 62% 68% 89%
Minnesota (28%), Montana CA NV UT CO KS MO KY WV DC MD DE
45% 96% 77% 54% 36% 50% 79% 78% 45% 99% 40%
(34%), and Louisiana (35%).
AZ NM OK AR TN VA NC
36% 50% 62% 99% 64% 75% 71%
TX LA MS AL GA SC
54% 35% 78% 95% 71% 94%
HI FL
77% 40% Figure 6.2.2

K–12 CS education is expanding in the United States (Figure 6.2.3). In 2017, only a few states supported high
school CS programs. Now, approximately two-thirds of states require that CS be taught in high schools,
allocate funding for it, and have developed state plans for CS education.
Changes over time in state-level US K–12 CS education

36, Allocate funding

35
30 30, Adopted state plan

29, Require CS course
25
Number of U.S. states
20
15
10
0
2017 2018 2019 2020 2021 2022 2023
Figure 6.2.3

AP Computer Science 201,000 exams were administered, marking an 11.1%

The state of K–12 CS education in the United States increase from 2021 (Figure 6.2.4). Since 2007, the
can also be observed by analyzing trends in the total number of AP CS exams administered has increased
number of AP CS exams. In 2022, approximately 8
more than tenfold.
Number of AP computer science exams taken, 2007–22

201.61
200
Number of AP computer science exams taken (in thousands)
179.19 181.04
158.56
150
130.90
99.87
100
54.38
50 46.34
37.33
29.55
24.78
19.39 19.83 20.96 19.39 21.14
0
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Figure 6.2.4
8 There are two types of AP CS exams: Computer Science A and Computer Science Principles. Data on computer science exams taken includes both exams. AP CS Principles was initially
offered in 2017.

In 2022, California (33,262), Number of AP computer science exams taken, 2022

Texas (20,901), and Florida
(16,248) were the leading AK ME
157 287
states in terms of the number
VT NH MA
of AP CS exams taken (Figure 114 526 6,470
6.2.5). On the other end, WA MT ND SD MN WI MI NY CT RI
4,755 39 100 40 1,811 2,223 4,919 15,189 3,241 752
Montana (39), South Dakota
(40), and North Dakota (100) OR ID WY NE IA IL IN OH PA NJ
891 514 107 530 645 9,778 2,979 3,968 6,230 10,433
are the states where the
CA NV UT CO KS MO KY WV DC MD DE
fewest exams were taken. 33,262 2,476 632 3,077 345 1,398 1,832 326 403 7,796 616
Per capita, Maryland (126.5), 1,881 445 629 1,500 2,347 6,142 6,880
New Jersey (112.7), and TX LA MS AL GA SC
20,901 987 791 2,983 8,195 2,103
Massachusetts (92.7) ranked
highest in the number of AP HI FL
720 16,248
CS exams taken (Figure 6.2.6). Figure 6.2.5
Number of AP computer science exams taken per 100,000 inhabitants,

2022
AK ME
21.41 20.66
VT NH MA
17.62 37.60 92.66
WA MT ND SD MN WI MI NY CT RI
61.08 3.47 12.84 4.40 31.69 37.74 49.03 77.21 89.81 68.75
21.02 26.51 18.40 26.93 20.16 77.71 43.60 33.74 48.03 112.66
CA NV UT CO KS MO KY WV DC MD DE
85.20 77.92 18.69 52.68 11.75 22.63 40.61 18.38 60.06 126.48 60.42
25.54 21.06 15.65 49.24 33.30 70.77 64.32
TX LA MS AL GA SC
69.60 21.51 26.91 58.79 75.09 39.81
HI FL
50.02 73.04 Figure 6.2.6

Highlight:
Access Issues Schools o ering foundational CS courses by size,

2023
90%
90%
Code.org data suggests that factors such
as school size and location significantly 80%
75%
influence CS education accessibility. 70%
Large schools (over 1,200 students)
60%
are 15 percentage points more likely to
offer CS courses than medium-sized % of schools 50%
41%
schools (500–1,200 students), with the 40%
gap widening further when compared

30%
to small schools (under 500 students)
20%
(Figure 6.2.7). Similarly, students in
suburban districts have better access 10%
to CS courses than their counterparts in 0%

Small Medium Large
both urban and rural areas (Figure 6.2.8).
Figure 6.2.7
Schools o ering foundational CS courses by

geographic area, 2023
70% 67.21%
60%
54.73% 54.62%
50%
% of schools
40%
30%
20%
10%
0%
Urban Suburban Rural
Figure 6.2.8

Highlight:
ChatGPT Usage Among Teachers and Students

The introduction of generative AI ChatGPT usage rate among American K–12 teachers,
2023
tools, including ChatGPT, has sparked Source: Impact Research, 2023 | Chart: 2024 AI Index report
significant debate regarding their 63%
potential applications in education. 60%
Some individuals have raised concerns 51%

50%
that these tools could be misused
for plagiarism, potentially prompting
40%
Usage rate
a reevaluation of the ways in which

American students may be taught. 30%
This year, Impact Research, funded by

20%
the Walton Family Foundation, carried
out a series of surveys on American 10%
teachers’ and educators’ perceptions
and use of ChatGPT.9 The surveys 0%
2023-Mar 2023-Jul
revealed that a majority of K–12 teachers Figure 6.2.9
in the United States are already utilizing ChatGPT usage purposes among American K–12
teachers, 2023
ChatGPT, with usage increasing over Source: Impact Research, 2023 | Chart: 2024 AI Index report
the year: In March 2023, 51% of teachers

reported having used ChatGPT at least Coming up with
30%
creative class ideas
once, and by July 2023, that figure had
risen to 63% (Figure 6.2.9). Among the
teachers who reported using ChatGPT,
Usage purpose
30% employed it for lesson planning, Lesson planning 30%

another 30% for generating new creative
class ideas, and 27% for enhancing their
background knowledge (Figure 6.2.10).
Building background
27%
knowledge
0% 10% 20% 30%

Usage rate
Figure 6.2.10
9 To learn more about the surveys, including their methodologies, please visit the following links: March 2023 and July 2023.

Highlight:
ChatGPT Usage Among Teachers and Students (cont’d)

Both teachers and students have ChatGPT perceptions among educational users, 2023
Source: Impact Research, 2023 | Chart: 2024 AI Index report
overwhelmingly positive attitudes toward
Teachers Students
ChatGPT. According to the March 2023
survey, 88% of the teachers believe
that ChatGPT has a positive impact, a 88%
ChatGPT has had
sentiment echoed by 79% of the students a positive impact
79%
surveyed (Figure 6.2.11). Furthermore,
Statement
76% of teachers and 65% of students feel

that ChatGPT is important to incorporate
into the educational process. This recent
76%
data indicates that tools like ChatGPT ChatGPT is important
to incorporate
are poised to become a staple in the 65%
American educational landscape in the

foreseeable future. 0% 20% 40% 60% 80%
% of respondents that “agree”
Figure 6.2.11

5 HAI 2024 AI-Index-Report 296-365

Uploaded by

Copyright:

Available Formats

You might also like

5 HAI 2024 AI-Index-Report 296-365

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

5 HAI 2024 AI-Index-Report 296-365

Uploaded by

Copyright:

Available Formats

Artificial Intelligence CHAPTER 5:

Index Report 2024

5.1 Notable Scientific Milestones 300

5.2 AI in Medicine 307

ACCESS THE PUBLIC DATA

Table of Contents 297

Table of Contents 298

Table of Contents 299

5.1 Notable Scientific Milestones

AlphaDev vs. human benchmarks when optimizing for algorithm length

Table of Contents Chapter 5 Preview 300

FlexiCubes quality. FlexiCubes addresses some of these

Sample FlexiCubes surface reconstructions

Table of Contents Chapter 5 Preview 301

Select quantitative results on 3D mesh reconstruction

Table of Contents Chapter 5 Preview 302

Synbot Synbot design

Reaction kinetics of M1 autonomous optimization experiment, Synbot vs. reference

Table of Contents Chapter 5 Preview 303

GraphCast and more. Figure 5.1.7 compares the performance

GraphCast weather prediction

Ten-day z500 forecast skill: GraphCast vs. HRES

GraphCast HRES (06z/18z) HRES (00z/12z)

Table of Contents Chapter 5 Preview 304

GNoME Sample material structures

Table of Contents Chapter 5 Preview 305

Flood Forecasting A team of Google researchers has used AI to develop

Predictions of AI model vs. GloFAS across return periods

Table of Contents Chapter 5 Preview 306

This section identifies significant AI-related

Table of Contents Chapter 5 Preview 307

SynthSR correlation with ground-truth volumes on select brain regions

Scan level (ablated

Table of Contents Chapter 5 Preview 308

made by AI systems, the accuracy of the predictions

0% 25% 40% 50% 60% 75% 100%

Table of Contents Chapter 5 Preview 309

Table of Contents Chapter 5 Preview 310

EVEscape vs. other models on SARS-CoV-2 RBD mutation prediction

46%, Later experimental scans (pandemic ab + sera)

32%, Earlier experimental scans (pandemic ab)

24%, Previous model

2020-Jan 2020-Jul 2021-Jan 2021-Jul 2022-Jan 2022-Jul 2023-Jan

Table of Contents Chapter 5 Preview 311

AlphaMissence Hemaglobin subunit beta (HBB)

In 2023, researchers from Google DeepMind

the nature of 0.1% of all missense mutations.

Likely benign Likely pathogenic Uncertain

Prediction category 57% 32% 11%

0% 20% 40% 60% 80% 100%

Table of Contents Chapter 5 Preview 312

In 2023, the Human Pangenome Research Consortium,

Ensembl mapping pipeline results

Table of Contents Chapter 5 Preview 313

AI performance on the MedQA benchmark has seen

2019 2020 2021 2022 2023

Table of Contents Chapter 5 Preview 314

Research from Microsoft in late 2023