701 Prompt Engineering

Introduction to Prompt Engineering
q Motivation
q Prompt Engineering Project
q GPT-3 Playground
q Examples
q Strategies
PE:I-1 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

What is Prompt Engineering about?
q Using existing pre-trained language models to solve a broad range of tasks.

Might be classification tasks or generative tasks
(where the task is to generate an output sequence).
q Justification: Language models generalize well and can be applied to tasks

which they were not trained on.
Just need to ask the right way!
q Might allow for tasks that were not possible before
q Benefits: No (re-)training, no finetuning
q Behaviour of the model is almost completely controlled by the model input

(i.e. the prompt).
Must optimize the input prompt to receive optimal results (prompt
engineering).

Disclaimer
q May be seen as orthogonal to the approaches that you are used to

Doesn’t replace the classic DL approaches that we studied before
q Many aspects stay the same

– Careful problem analysis
– Data aquisition and exploration
– Rigorous and metrics-based evaluation
q May seem like alchemy

But didn’t Feature Engineering and DL Model Engineering feel like alchemy at
first, too?

Disclaimer
q New problems arise

– Extremely hard to detect train-test leakage
Using canary strings in benchmark tasks
– Non-rigorous evaluation leads to wrong beliefs about performance
– Model availability and computation cost
– Blackbox models (even more extreme than in classic DL!)
– Undetected bias

Relevance to term projects
q Provides a way to quickly demonstrate language-based tasks

Without actually designing and training a model
q Forces you to think about model inputs and outputs
q Helps to visualize the text data in the single processing steps

Actually see what is happening!
q Might actually be used in the term projects for some pipeline steps
E.g. for refining the data
q Next week, we will talk about making prompt engineering accessible to

software through APIs

Large Language Models (LLM)
q Transformer-based language models
q Typically billions of parameters
q Built to generate an output sequence based on an input sequence
q Many publicly available models

e.g. Open Pre-trained Transformers (OPT)
q Some only through APIs

e.g. Generative Pre-trained Transformer 3 (GPT-3)

Prompt Engineering Project
Workflow
One mini project for each student.
1. Find an interesting prompt task

i.e. a question that should be solved using prompt engineering
Examples follow. (Please do not choose exactly the tasks from the lab.)
2. To make sure that you do engineering and not just running some arbitrary
prompt:
q Choose an interesting aspect to investigate further
e.g. How well do different prompt engineering strategies work?
q If the quality of the results can be measured objectively, a plot would be
useful

Prompt Engineering Project
Deliverable: Slides
To present the results to your fellow students, please prepare exactly 3 slides:
1. Short title of the project + short problem description in max 2 sentences
2. Example prompt that aims to solve the problem + output

also specify the used model
3. Plot or chart or . . . to document your engineering investigation
Please send the slides as a PDF to niklas.deckers@uni-leipzig.de until 12.06.2022,

22:00,
and prepare for a 2-minute presentation in class.

Prompt Engineering Examples
GPT-3 Playground
https://beta.openai.com/playground/
q Basic usage
q Engine selection
q Stop sequences

Basic examples
q Q&A
q Magic Spells
q Summarize for a 2nd grader
q Analogy extraction
q Group Name Generation

More examples
q https://beta.openai.com/examples
q https://github.com/google/BIG-bench/tree/main/bigbench/
benchmark_tasks#readme

More ideas related to web data
q Argument generation
q Generating replies to reddit posts from a specific subreddit

r/explainlikeimfive, r/AmItheAsshole, r/changemyview, r/WrongAnswersOnly
...
q Generating/understanding/explaining humour
q Your own ideas?

Prompt Engineering Strategies
General advice
q Do not think that prompts should “command the machine” (imperatives)
q Do not think that “the machine is talking to you”
q Rather think about the output as the “most probable completion”

Depends on the data the LLM had been trained on, beware of internet
language!
q This might also lead to mere reproduction of training data
q Classification tasks are also possible

Computing which of the given options is more probable to be the completion

Defining context/identity/intent
q Defining context allows the LLM to adjust to a specific task
q Specifying the intent allows to e.g. prevent insulting language
q May significantly improve result quality

Few-shot learning
q Idea: The LLM has been trained on a too broad scope, so provide it with
successful examples
q Also useful to enforce a certain output format
q Cover a broad range of samples for your task
q Samples must have high quality
q Remember: Computation cost increases linearly

Instructions
q Originally, LLMs did not perform well with instructions

This aligns with our expectations from the concept of the “most probable
completion”
q However, modern LLMs are often optimized to follow specified instructions
q Might help to specify/clarify the task explicitly

Intermediate steps
q Splitting the task up into multiple intermediate steps
q May be represented by multiple execution steps
q Should also be reflected in the few-shot learning samples

Allowing and expecting edge cases
q Allow for special values as a result (e.g. exception statements)

Can be specified in the context/instruction and included in few-shot learning
samples
q For classification-like tasks, cover a broad scope of classes (e.g. neutral )
q Expect outputs that don’t match your desired pattern

701 Prompt Engineering

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

701 Prompt Engineering

Uploaded by

Copyright:

Available Formats

Introduction to Prompt Engineering

q Prompt Engineering Project

PE:I-1 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Using existing pre-trained language models to solve a broad range of tasks.

q Justification: Language models generalize well and can be applied to tasks

q Might allow for tasks that were not possible before

q Benefits: No (re-)training, no finetuning

q Behaviour of the model is almost completely controlled by the model input

PE:I-2 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q May be seen as orthogonal to the approaches that you are used to

q Many aspects stay the same

q May seem like alchemy

PE:I-3 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q New problems arise

PE:I-4 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Provides a way to quickly demonstrate language-based tasks

q Forces you to think about model inputs and outputs

q Helps to visualize the text data in the single processing steps

q Next week, we will talk about making prompt engineering accessible to

PE:I-5 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Transformer-based language models

q Typically billions of parameters

q Built to generate an output sequence based on an input sequence

q Many publicly available models

q Some only through APIs

PE:I-6 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

One mini project for each student.

1. Find an interesting prompt task

PE:I-7 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

1. Short title of the project + short problem description in max 2 sentences

2. Example prompt that aims to solve the problem + output

3. Plot or chart or . . . to document your engineering investigation

Please send the slides as a PDF to niklas.deckers@uni-leipzig.de until 12.06.2022,

PE:I-8 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

PE:I-9 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Summarize for a 2nd grader

q Group Name Generation

PE:I-10 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

PE:I-11 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Generating replies to reddit posts from a specific subreddit

q Your own ideas?

PE:I-12 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Do not think that prompts should “command the machine” (imperatives)

q Do not think that “the machine is talking to you”

q Rather think about the output as the “most probable completion”

q This might also lead to mere reproduction of training data

q Classification tasks are also possible

PE:I-13 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Defining context allows the LLM to adjust to a specific task

q Specifying the intent allows to e.g. prevent insulting language

q May significantly improve result quality

PE:I-14 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Also useful to enforce a certain output format

q Cover a broad range of samples for your task

q Samples must have high quality

q Remember: Computation cost increases linearly

PE:I-15 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Originally, LLMs did not perform well with instructions

q However, modern LLMs are often optimized to follow specified instructions

q Might help to specify/clarify the task explicitly

PE:I-16 Prompt Engineering © NIKLAS DECKERS/WEBIS 2022

q Splitting the task up into multiple intermediate steps

q May be represented by multiple execution steps