0 Useful AI Stuff For Right Now

‣ Things that are top of mind for me right now.

14
‣ Everything else in this preso is looking backwards, to help me look forwards.
‣ If you want to do that too, then stick around!

AESTHETICS
Source: @johnmaeda DESIGNINTECH.REPORT
 Every year I run a survey related to the Design in Tech Report. But this year I’ve been a little
busy with all-things GPT-universe. That said, I got a lot of great contributions!

Thank you to Henrik, Frederik, Sean-Wood, Kai, Kim, Samuel, Henry, Ryan, Chris, Colin, Agnieszka, Bob, Gary, Agu, Dave, Moisés, Me, Ankita, Darius, Rachel, Aldo, PChia, Heather, Justine, Miguel, Florentin, Juliana, Amanda, Sam, Gaby,
Kerry, Chloe, Duncan, Jan, Harshika, Arum, Gael, Raffaela, Ghizlene, Marcel, Narasimha, Eelko, Jason, Debk, Andrea, Ashish, André, Bill, Maria, Art, Puneet, Vidhi, Matt, Lauren, Brian, Sridhar, Gaurang, Julie, Ocean, Axel, Sithembiso, Rodolfo,
Sanmitra, Sébastien, Bruce, Jason, Maria, Carlo, Bs, Angelina, Silvia, Filippo, Manisha, Roland-Philippe, Andrés, Ame, Lucas, Marin, Mark, Cristin, Quang, Juhan, Pat, Luiza, Zyo, Chrys, Mea, Rupert, Ellie, Richard, Paolo, Sean, Guy, Kaladhar,
Sebastian, Alessio, Surya, Richard, Andre, Dani, Vivek, Christina, Jesper, Ranjeet, Kshipra, Betsongeorge, Jodean, Marc, Paul, Akshay, Roman, Dylan, Naomi, George, Paul for contributing to the 2023 survey!

How likely will the design profession need to What prominent Design × What *obscure* Design ×
change given the emergence of Large
Language Model AI systems like ChatGPT?
AI tool out there is top of AI tool out there is top of
mind for you? mind for you?
‣ Adobe Tools ‣ Pinecone | Weaviate | Qdrant: Vector DB CONTEXT
‣ Uizard: Instant UI
Average = 8 ‣ Figma + AI Plugins
‣ NVIDIA Gaugan: Universe Painter
15 10 = “highly likely” ‣ DALL-E ‣ Vizcom: 3D From Prompts
‣ Midjourney ‣ Autodraw: Early Google Experiment
‣ Wombo: Lip Sync
‣ Runway
Someone I know has ‣ Synesthesia: Avatar Spokesperson
#
‣ D-ID ‣ Luma AI: Instant 3D
been directly affected Good luck to everyone out
there who will find their next
by the recent spate of step in life. I believe in you! ‣ Copy.ai ‣ Deepl: Language translator
layoffs in tech ... —<3 John
‣ Elevenlabs ‣ Tome: Instant Presos
‣ Lyric Studio: Instant Music
‣ Stable Diffusion
‣ Playground: IRL Observability
65% ‣ Hugging Face *Anything* ‣ Photofeeler: Judge Your Photo
‣ ChatGPT / ChatGDP / Chat GBT ‣ Edward Teaching Bot

As Figma AI plug-ins go, diagram.com has some really cool ones. My favorite obscure find was the Pokemon
generator. For ready-to-eat prompts go to promptbase or lexica.art which are prompt marketplaces.
AESTHETICS
Source: @johnmaeda Annual Design in Tech Survey Respondees | futurepedia.io has amore of these kinds of finds for you to forage through … DESIGNINTECH.REPORT
 N E

N
E W S 1992
W
S

AESTHETICS
TO DA I J I T E M P L E | N A R A , J A PA N 2 0 0 1 DESIGNINTECH.REPORT
 materials
Ò DESIGN Ó 1992

IS DEFINED BY
M AT E R I A L S S E L E C T I O N

AESTHETICS
DESIGNINTECH.REPORT
 “A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT”

HISTORY OF PRETRAINED FOUNDATION MODELS

Unified
Graph PFMs
Graph Graph NLP CV
CV CV Graph GCC UNIFIED-IO
CV NLP NLP
NLP DeepWalk VGAE GraphCL OFA
NLP GPT-3 MoCo
LINE BiGAN node2vec BigBiGAN DGI SUGAR FLAVA
Context CoVE RoBERTa BART SimCLR
GloVe NAT GraphSage DeepCluster MoCL Gato
Skip-Gram Exemplar-CNN Shared LSTM GPT-2 PET BEiT
LM-LSTM Counting RotNet BEiT-3
FastText BERT SimCSE MAE
ChatGPT

18
2014 2016 2018 2020 2022
Unified IO
https://github.com/allenai/unified-io-inference

OFA (Once For All)

https://github.com/mit-han-lab/once-for-all

FLAVA (A Foundational Language And Vision Alignment Model)

https://github.com/facebookresearch/multimodal/blob/main/examples/flava/README.md
= Large Language Model = Unified PFM
Gato
https://www.deepmind.com/publications/a-generalist-agent

BEiT (Bidirectional Encoder representation from Image Transformers)

https://www.microsoft.com/en-us/research/publication/beit-bert-pre-training-of-image-transformers/

BB8 (Bouncing Ball) Just checking if you’re still awake …

https://en.wikipedia.org/wiki/BB-8

Learn about many of the models out there at https://huggingface.co/docs/transformers/model_doc/flava

AESTHETICS
Source: https://arxiv.org/pdf/2302.09419.pdf DESIGNINTECH.REPORT
 The end of the year seemed to go really fast for me. Same for you, too?

SEP 19 OCT 17 OCT 24

19

AESTHETICS
Source: Sonya Huang https://twitter.com/sonyatweetybird/status/1582040028015837187?lang=en DESIGNINTECH.REPORT
 It helps to squint a little to understand a few things. Because there’s a lot to take in rn.

The app you build depends upon What you build will change as there
the model(s) you have available. are more models available to use.

KINDS OF MODELS
pre-2020 2030?

APP
LAYER
Green means “look
20 KINDS OF to be surprised”
MODELS
MODEL
LAYER

AESTHETICS
Source: Sequoia https://www.sequoiacap.com/article/generative-ai-a-creative-new-world/ DESIGNINTECH.REPORT
 Simon’s Scissors: Context and Cognition

THE PRETRAINED
FOUNDATIONAL MODEL

i t i on
og n
C THE PROMPT YOU
FEED THE MODEL

21

“ Human rational behavior is shaped by a

scissors whose blades are the structure
of task environments and the
computational capabilities of the actor.”
—Herbert Simon, “Invariants of Human Behavior”
in Annual Review of Psychology, 41, 1990, p.1-19

Scissors Photo by Matt Artz on Unsplash

AESTHETICS
Source: Dan Lockton, “Simon’s Scissors and Ecological Psychology in Design for Behavior Change” in SSRN Electronic Journal, Aug 2012. DESIGNINTECH.REPORT
To build apps with models takes a little time to settle into your mind. It’s not as easy as
saying to yourself as a dev, “Hey. This is similar to XYZ.” Because it’s not.

% % % %
Machine Learning

Data Hire ML Build Wrap Your

Warehouse Expert Model Endpoint Product

HARD

i t i on
g n
Co LLM Prompt Vector DB + Your
API Design Other Data Product

EASY EASY EASY $

“DevTools for language
models — predicting the
future” —@unusualvc

New models are being built with plenty of determination across the world. The blade that matters to the non-
model builders is where a lot is happening. OSS langchain and llamaindex are great for design-devs to try out!
AESTHETICS
Source: Unusual VC https://www.unusual.vc/post/devtools-for-language-models DESIGNINTECH.REPORT
 Prompt Engineering 101

1. Select a 2. Create 3. Add in 4. Tune the

large language the core useful model’s
model** prompt context settings
NEWER ‣ Davinci ‣ Make your wish. ‣ Give one example of ‣ Set the temperature
($$$) how you’d like your high to make it more
‣ Curie ‣ Be specific about the
outcome, length, prompt to behave. random; lower the
‣ Babbage
OLDER format, and style. ‣ Give multiple examples temperature to make it
23
‣ Ada more deterministic.
($) ‣ Use precision in how of how you want your
you prompt. prompt to behave. ‣ Alternatively, adjust
The OpenAI universe also ‣ When you don’t give the “p” parameter to
includes Codex (for ‣ When referring to text
guidance with broaden or narrow the
generating computer to act upon, delimit it
code) and the new examples, that’s “zero range of words that will
with “”” (3 double
ChatGPT model
shot.” The other cases be generated in
quote marks) or ###
are called “one shot” response.
(3 hash sign marks).
and “few shot.” ‣ Give it more tokens
when you need to work
with longer prompt or
longer responses.

AESTHETICS
Source: https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api DESIGNINTECH.REPORT
 Prompt Engineering 201

A. Shorten an B. Go for a C. Use “chain

often-used two-fer or of thought”
prompt three-fer prompting

‣ You can reduce** the ‣ Sometimes with one ‣ Getting a model to use
amount of tokens that prompt you can get our own human
get used by finding a multiple things done. tendency to “think out
similar but shorter loud” has been shown
24
version of a prompt. to produce more
‣ This way if you’re using accurate outcomes.
the prompt a lot, you’re ‣ When prompting, give
not using up as many an explicit “thinking out
tokens in a cumulative loud” explanation for
sense. how a problem is to be
solved with your
**One DiT survey responder related reasoning.
suggested using Quillbot.

Playing with the OAI tokenizer https://platform.openai.com/tokenizer gives you valuable insights on how it “eats”
text. And learn more about how “chain of thought” prompting works via https://arxiv.org/abs/2201.11903 asap.
AESTHETICS
Source: https://arxiv.org/abs/2201.11903 DESIGNINTECH.REPORT
 When I mentioned the Herbert Simon scissors metaphor to Sam Schillace, I recall him
quickly saying, “And what did our parents teach us? Don’t run with scissors.”

i t i on
g n
Co

25
& &

&

Scissors Photo by Matt Artz on Unsplash

AESTHETICS
Source: Sam Schillace https://sundaylettersfromsam.substack.com/p/ai-isnt-a-feature-of-your-product DESIGNINTECH.REPORT
 Stanford HAI “On the Opportunities and Risks of Foundation Model” (2022)

2. Ultimately, all of this modeling eventually gets deployed to real

people. And the implications at the scale of millions are huge.
MACHINE LEARNING

DEEP LEARNING Data Data

Creation Curation Training Adaptation Deployment

FOUNDATION MODELS

Emergence of “how”

learning
features functionalities
(
26 Homogenization of algorithms
architectures models

TIME
)
1. While new capabilities emerge, our reliance on a more
homogenized approach creates a single point of failure.
'
S
E
C Large uncharted data sets are a Downstream apps easier to
U
Foundational models are a
R source of robustness and are an security choke point and also an develop but “dual use” (use for
I increased risk of poisoning increased attack surface unintended purposes) risk ↑
T
Y

It’s the most comprehensive take on the pros and cons of pretrained foundation models as a ~160 page “weapon
of understanding” with 30 pages of references — if you’re new to this space, it’s a godsend.
AESTHETICS
Source: https://arxiv.org/pdf/2108.07258.pdf DESIGNINTECH.REPORT
 For my introductory AI course, I had Professor Joseph Weizenbaum as the instructor.
His significance was lost on me until many years later I realized that he was a GOAT.

“ I knew of course that people form all sorts of emotional bonds to

machines, for example, to musical instruments, motorcycles, and cars.
And I knew from long experience that the strong emotional ties many
programmers have to their computers are often formed after only short
exposures to their machines.

What I had not realized is that extremely short

exposures to a relatively simple computer
program could induce powerful delusional
27
thinking in quite normal people.

This insight led me to attach new importance to questions of the

relationship between the individual and the computer, and hence to
resolve to think about them.”

—Joseph Weizenbaum (1976)

There’s a fab article on Dr. Weizenbaum on 99% Invisible https://99percentinvisible.org/episode/the-eliza-effect/

AESTHETICS
Source: Computer Power and Human Reason (1976) DESIGNINTECH.REPORT
 Judgment Call kicked off a nuanced approach to talking about AI in product companies.
And Professor Helen Armstrong’s 2021 book on AI for designers is certainly timely.

28

#
More card toolkit
recommendations courtesy
of data and trust guru CEO/
Founder Sarah Gold of
projectsbyif.com (UK)

Judgment Call (2019) was authored by Stephanie Ballard, Karen M. Chappell, Kristin Kennedy to use value
sensitive design and design fiction to surface ethical concerns related to technology.
AESTHETICS
Source: Mira Lane / Helen Armstrong / Sarah Gold DESIGNINTECH.REPORT
 It’s easy to get GPT-bots to “pretend you’re a …” for everything ranging from a JS console to
a math teacher. Critical thinking is key right now; David Karam has “Culture Club” for you.

“ LLMs give us the

opportunity to explore
29 concepts from multiple
perspectives in simulation.”
—David Karam

Other approaches recommended by DiT survey responders are storyfit.com for simulate audience reaction,
realeyesit.com webcam reaction aggregation, and synthetic health which is an open source patient generator.
AESTHETICS
Source: https://github.com/f/awesome-chatgpt-prompts / David Karam https://medium.com/@posttool/a-productive-narrative-around-ai-a47325563c3f DESIGNINTECH.REPORT
 Ruskin & Morris famously led the Arts & Crafts Movement in the late 1800s as a reaction to
the Industrial Revolution’s introduction of new fabrication technologies.

“ Morris was not entirely against the

use of machines, but felt that the
division of labour – a system
30 designed to increase efficiency, in
which the manufacture of an object
was broken into small, separate
tasks, meaning individuals had a very
weak relationship with the results of
their labour – was a move in the
wrong direction.”
—V&A

A useful take on the plight of the Luddites in relationship to what’s happening now with AI fears is https://
www.smithsonianmag.com/innovation/when-robots-take-jobs-remember-luddites-180961423/
AESTHETICS
Source: https://www.vam.ac.uk/articles/arts-and-crafts-an-introduction / William Morris, made by Ada Phoebe Godman (1877) DESIGNINTECH.REPORT
 Just like in the Arts & Crafts Movement, we should expect to see *critical making* happen.
Like this brand new animation just released by design technologist Masashi Kawamura

31

AESTHETICS
Source: @masakawa https://www.youtube.com/watch?v=DpefYPLH67A DESIGNINTECH.REPORT
 How Will Foundation Models Impact Design Thinking?

This is one hypothetical way:

Generate possible
3
solutions

Summarize customer Visualize solution

1 4
support or field logs journeys and examples
Empathize Ideate
32

Define Prototype

Test

Categorize common Simulate customer

2 5
problems reaction

AESTHETICS
Source: https://web.stanford.edu/class/me113/d_thinking.html DESIGNINTECH.REPORT
 How Will Foundation Models Impact Classical Design?

This is one hypothetical way: Generate options with AI

Imagine Explore Test options with AI

Examine Experiment
Perceive Practice

The
33 Creative
Process
Use AI audience Share Create

Sketch with AI
Reflect
Assess
Revise

Chat with AI critic

AESTHETICS
Source: https://www.kennedy-center.org/education/resources-for-educators/classroom-resources/articles-and-how-tos/articles/collections/arts- DESIGNINTECH.REPORT
integration-resources/arts-integration-and-universal-design-for-learning/
 In this new age of partnering with PFMs, we need to keep in mind what we don’t enjoy doing
versus what we really love to do. Remove the drudgery while increasing your joy.

* +
“DOES NOT SPARK JOY” “DOES SPARK JOY”

• Calendar Management • Decision Making

• Making Powerpoint Decks • Talking With People
34
• Cleaning Data • Learning With Others
• Low-priority E-mails • Mentoring / Coaching
• Competitive Research • Relationship Building
• Summarizing • Delivering Presentations
• Monitoring / Alerts • Crafting Strategy
•… •…

I run a poll regularly in large groups of what people prefer not to do at work versus what they really love doing at
work. It turns out there’s tons of things that don’t spark joy in the workplace. Ya think? <wink>
AESTHETICS
Source: @johnmaeda DESIGNINTECH.REPORT
 Simplifying the Laws of Simplicity ...

2005

Quiz: Which cookie will a child choose?

AESTHETICS
DESIGNINTECH.REPORT
 2005

Quiz: Which cookie will a child choose?

AESTHETICS
DESIGNINTECH.REPORT
 2005

Which pile of laundry will a child choose?

AESTHETICS
DESIGNINTECH.REPORT
 COOKIE LAUNDRY

2005

want more want less

Simplicity is about living life with more enjoyment and less pain.
(enjoy/consume) (work/transform)
AESTHETICS
DESIGNINTECH.REPORT
Think of how to use AI So that you have more
to get rid of things you time to do more of
don’t really want to do. what you love to do.

2023

LAUNDRY COOKIE

* +
“DOES NOT SPARK JOY” “DOES SPARK JOY” AESTHETICS
DESIGNINTECH.REPORT
 Useful AI Stuff For Right Now
‣ I’ve used two sci-fi movie metaphors when struggling to understand this stuff
1. Arrival (2016)
40
Directed by Denis Villeneuve
Story by Ted Chiang
It’s about communicating with an
alien race within a set of strict
limitations in how long you can
talk with the alien. The mystery
is grounded in discovering the
language of the alien through
trial and error.
Spacetime Language = Prompts
Source: @johnmaeda @IMDB
2. Black Panther (2018)
Directed by Ryan Coogler
Story by Ryan Coogler + Joe Cole
It’s about the king of an African
nation called Wakanda — which is
the most technologically advanced
country on earth. Their early
access to an alien material with
special properties that defy
physics shaped their history.
Vibranium = Cognition , Blade
AESTHETICS
DESIGNINTECH.REPORT