Prompt Engineering

sudhanshu.bhargav@gmail.com
5VOKZCXGRH

The art and science of designing effective LLM prompts

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Agenda

● Large Language Models - An Introduction

● Models & Deployments in Azure Open AI
● Prompt Engineering Fundamentals
sudhanshu.bhargav@gmail.com
5VOKZCXGRH

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Operationalizing Generative AI

1
Task Definition, Metrics
Prepare Context Data Curation
Data Examples Curation
Gold Examples Curation

2
sudhanshu.bhargav@gmail.com
5VOKZCXGRH
3

Devise Evaluate Internal APIs

Approve Integrate External APIs
Prompts Prompts Guardrails
Databases

Prompt Engineering, Configuring Tools, Variability

Debugging
4

Deploy &
This file is meant for personal use by sudhanshu.bhargav@gmail.com only.
Monitor
Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
LLMs are trained using language modeling, that is, predicting the next word in a sequence.

positive?
Masked Sample
sudhanshu.bhargav@gmail.com
The movie was awesome. Overall, the experience was
5VOKZCXGRH negative?

magical?

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
LLMs are trained using language modeling, that is, predicting the next word in a sequence.

positive
Masked Sample
sudhanshu.bhargav@gmail.com
The movie was awesome. Overall, the experience was
5VOKZCXGRH negative

magical

positive
This file is meant for personal use by sudhanshu.bhargav@gmail.com only. Ground Truth
Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
LLMs are trained using language modeling, that is, predicting the next word in a sequence.

positive
Masked Sample
sudhanshu.bhargav@gmail.com
The movie was awesome. Overall, the experience was
5VOKZCXGRH negative

magical

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
LLMs are trained using language modeling, that is, predicting the next word in a sequence.

Vocabulary

positive p = .03

sudhanshu.bhargav@gmail.com
negative p = .00001
The movie is a visually stunning, action-packed, and
5VOKZCXGRH

emotionally resonant thrill ride that will leave you on the

edge of the seat from the beginning to end. Overall, the
experience was magical.
…

magical p = .83

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
LLMs are trained using language modeling, that is, predicting the next word in a sequence.

Vocabulary

positive p = .03

sudhanshu.bhargav@gmail.com
negative p = .00001
The movie is a visually stunning, action-packed, and
5VOKZCXGRH

emotionally resonant thrill ride that will leave you on the

edge of the seat from the beginning to end. Overall, the
experience was magical.
…
match

magical p = .83

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
LLMs are trained using language modeling, that is, predicting the next word in a sequence.
Original sentence
The movie was awesome. Overall, the experience was positive.

Training Samples
positive
The movie was awesome. Overall, the experience was positive.
sudhanshu.bhargav@gmail.com
The movie was awesome. Overall, the experience was positive.
5VOKZCXGRH negative
…
The movie was awesome. Overall, the experience was positive. movie

The movie was awesome. Overall, the experience was positive. magical
The movie was awesome. Overall, the experience was positive.
Vocabulary
The movie was awesome. Overall, the experience was positive.

⁞ This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models (LLMs)
During inference, the LLM predicts the next word in the input sequence.
Input word = prompt
The

Output, word-by-word
positive
The movie was awesome. Overall, the experience was positive.
sudhanshu.bhargav@gmail.com
The movie was awesome. Overall, the experience was positive.
5VOKZCXGRH negative
…
The movie was awesome. Overall, the experience was positive. movie

The movie was awesome. Overall, the experience was positive. magical
The movie was awesome. Overall, the experience was positive.
Vocabulary
The movie was awesome. Overall, the experience was positive.

⁞ This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models
Over the last 2 years, LLMs (e.g., Open AI GPT) have evolved to be AI assistants

GPT (117M parameters) InstructGPT

First model to be trained in Instruction-tuned models
a “generative” mode by understand human inputs as
masking portions of input instructions; path to ChatGPT is
text from left-to-right paved
sudhanshu.bhargav@gmail.com
5VOKZCXGRH

2019 2020 2023

2018 2022
GPT-2 (1.5B parameters) GPT-3 (175B parameters) GPT-4 (1T parameters?)
The era of prompting begins. Large scale foundation models Era of completely closed
Models are relatively small, are born. Prompting is shown models begins; API
open-source and fine-tuning to use
This file is meant for personal induce robust performance
by sudhanshu.bhargav@gmail.com only.
access only
is possible on NLP tasks
Sharing or publishing the contents in part or full is liable for legal action.
 Large Language Models

How are LLMs trained today?

Reinforcement
Foundation Supervised SFT GPT
Pretraining Learning with
model fine-tuning model Assistant
Human feedback
sudhanshu.bhargav@gmail.com
5VOKZCXGRH

Data
Large corpus of Curated high quality Ranking of prompt
internet data prompt-response pairs responses on quality

This file is meant for personal use by sudhanshu.bhargav@gmail.com only. Source: Karpathy (2023)
Sharing or publishing the contents in part or full is liable for legal action.
 Models and Deployments

Open AI base
Model model hosted on
Azure

sudhanshu.bhargav@gmail.com
5VOKZCXGRH

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Models and Deployments

Copy of base model +

Model dedicated hosting + rate limit Deployment
(tokens per minute)

sudhanshu.bhargav@gmail.com
5VOKZCXGRH

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Tokens

Token = Sequence of characters found in text

sudhanshu.bhargav@gmail.com
5VOKZCXGRH

Parses sentences
GPT-3 as tokens

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Prompt Engineering*

Prompt = Specific set of instructions sent to a LLM to accomplish a task

Engineering = Iteratively deriving a specific prompt for the task

Max tokens One-off

prompt
sudhanshu.bhargav@gmail.com
5VOKZCXGRH Prompt
Temperature
Parameters Structure

Top P
Reusable
template
Frequency Penalty
This file is meant for personal use by sudhanshu.bhargav@gmail.com only.
Sharing or publishing the contents in part or full is liable*Also
for legalreferred
action. to as in-context learning
 Prompt Engineering*

Prompt = Specific set of instructions sent to a LLM to accomplish a task

Engineering = Iteratively deriving a specific prompt for the task

Length of input + output Max tokens

sudhanshu.bhargav@gmail.com
Prompt
More temperature
5VOKZCXGRH = More
randomness in response
Temperature
Parameters Structure

More Top P = More tokens

selected for completion Top P

More FP = Less chance of Frequency

tokens repeating Penalty (FP)
This file is meant for personal use by sudhanshu.bhargav@gmail.com only.
Sharing or publishing the contents in part or full is liable*Also
for legalreferred
action. to as in-context learning
 Prompt Engineering

Components of a prompt template

System Message
Clear instructions explaining the task that the LLM should accomplish.
Should include the expected format of user input and output and a chain
sudhanshu.bhargav@gmail.com
of thought to accomplish the task.
5VOKZCXGRH

Few Shot Examples GPT

Prediction
Input - Output pairs delineating the expected response for exemplar API
inputs. Outputs need not be golden.

User Input
Input presented in the format mentioned in the system message

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Pricing

Pricing is per 1000 tokens in the prompt + completion

sudhanshu.bhargav@gmail.com
5VOKZCXGRH

This file is meant for personal use by sudhanshu.bhargav@gmail.com only.

Sharing or publishing the contents in part or full is liable for legal action.
 Summary
Trained on
language
modeling Instruction
Foundation fine-tuned
SFT
Models RLHF Assistant
Models
Train

sudhanshu.bhargav@gmail.com
5VOKZCXGRH
System Message,
LLMs Templates Chain of Thought,
Few Shot Examples

Infer Prompts

Max Length,
Parameters Temperature
This file is meant for personal use by sudhanshu.bhargav@gmail.com only.
Sharing or publishing the contents in part or full is liable for legal action.