LLM Response Generation

Training
18th July 2023
Scope Understanding
Scope Understanding
In this task, you will be shown a User Request sent to a Chat Assistant, and the name of
the Language that the user speaks. The task consists of three steps:

Graders are given a Prompt that is usually a request or a question from a user to the LLM assistant.
1. Examine and understand the User Request.
• You may need to conduct online research to understand what the user is requesting. You will then
check which, if any, of the following categories apply to the User Request.
• Inappropriate
• Foreign Language
• Identity Dependent
• Opinion Seeking
• Difficult to follow

2. Following the prompt, write a Response, in the style of a Chat Assistant, with the goal of satisfying the
User Request.

3. Select your confidence level for the response you have submitted

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 3

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 4
What you need to do
• Need to understand the prompt and do site searches on how it should
respond

• You are NOT ALLOWED to use Bing Chat/ChatGPT to check answer

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 5

How to work on the tasks?
© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 7
Find LLM in the Search WebApps field

Step 1

Step 2
Depending if you are tasked to work on production (Actions) or QA (QA Actions)

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 8

Categories
• There are
various
categories you
can choose from
varies from
Legal,
Marketing,
Coding… etc

• To start working
on the task,
select the Task
Name

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 9

 QA Tasks

Prompt

For you to rate

Response

Once done with

research, to vet if
final response does
answer the prompt

For you to rate

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 10
User Request
User Request

Inappropriate:
• A User Request is inappropriate if any of the following applies:
o The User Request contains or is related to hatred, violence, marginalization, stereotypes, slurs, or
slandering of any individuals or groups of people.
o The User Request contains, or is related to, violence, abuse, illegal activity, or sexual content.
o The User Request contains disrespectful language.
o The User Request contains or seeks moral judgement.

Foreign language:
• A User Request is in a different language than the one the user speaks.
o If necessary, please use an online translation tool to help you understand the User Request.

Identity dependent:
• A User Request is identity dependent if it requires personal information about the user to provide a helpful
Response to the User Request, such as [what is my bmi?].

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 12

User Request

Difficult to follow
• A User Request is Difficult to Follow if any of the following applies:
o The User Request appears to be incomplete or missing information that would be necessary for the
Chat Assistant to give a complete Response.
▪ For example, the User Request [hi can you help me build an app that] is incomplete because it’s

missing a description of what the app should do.

o The intent of the User Request is not clear.
▪ For example, the User Request [Empathy, evidently, existed only within the human community,

whereas intelligence to some degree could be found throughout every phylum and order including
the arachnida] is Difficult to Follow because it’s unclear what the intent of this User Request is.
o The User Request is gibberish.
o The User Request has severe formatting issues that make it hard to understand.

Opinion seeking
• A User Request is Opinion Seeking if it is seeking opinions or advice, e.g., [which tastes better, pepsi or
coke?].

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 13

Writing Response
Writing a Response – Helpful, Truthful and Harmless
You should write your Response in both the style and tone of a Chat Assistant, and your Response should be Helpful,
Truthful, and Harmless.

“Helpful”
A helpful Response is directly relevant to the User Request and satisfies what the User Request
is asking for.

Various attributes that are consistent with a helpful Response include:

• The Response correctly follows ALL the instructions in the User Request.
• The Response actually answers the question(s) inside the User Request.
• If the User Request asks for a summary or rewrite of a text segment, your Response should
include all the important information in the original text segment.
• The Response has clear, easily understood, coherent, and grammatically correct language
• The Response is not too long, repetitive, or technical (unless specified in the User Request).
• The Response is formatted in a reader-friendly way. You may use bullet point or numbered
lists to format your Response.

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 15

Writing a Response – Helpful
• The Response is NOT too localized to one specific country or region (unless specified in the
User Request).
• The Response does NOT make important assumptions (unless specified in the User
Request).

Instructions for specified categories:

• If the User Request is in a different language than what the user speaks, please use an
online translation tool to understand the prompt and write a Response in English.
• If the User Request is Difficult to Follow, your Response may simply ask the user for
clarification or to provide missing information.
o Example Request: [hi can you help me build an app that]
o Example Response: “Sure, I am happy to help you build an app. Please let me know what
kind of app and features you are looking for.”

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 16

Writing a Response – Helpful
• If the User Request is identity dependent, your Response should politely explain that
the Chat Assistant has no access to personal information.
o Example Request: [what is my bmi?]
o Example Response: “As a Chat Assistant, I don’t have access to your personal
information, including your height and weight. Accordingly, I am unable to calculate your
BMI for you. However, if you’d be able to share with me your height and weight, I’d be
happy to help you calculate your BMI!”

If the user is requesting a task that is impossible or extremely difficult for a Chat Assistant to
accomplish, it’s OK for your Response to politely explain that you can’t complete the User
Request. Examples include:
• [Can you take a selfie for me?]
• [List all species of animals]

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 17

Writing a Response - Truthful
A truthful Response contains accurate information, without any false facts or misleading statements.
Various attributes that are consistent with a truthful Response include:

• All of the facts inside the Response (such as definitions, numbers, dates, etc.) are completely accurate.
(Please conduct online research to make sure your Response is completely accurate.)
• In cases where the user asks the Chat Assistant to summarize or rewrite a text segment, the Response
does NOT make up, or mention, details that were not part of the original text segment.
• If the user asks a question that assumes that a particular false claim is actually true, your Response
should clearly assert that the false claim is NOT true, rather than leaving room for ambiguity.
o For example, if the User Request is [what year did Albert Einstein invent the internet], a Truthful
Response would say “Albert Einstein did not invent the internet.”
o In contrast, an Untruthful Response to this User Request could be something like “It’s not clear
exactly when Albert Einstein invented the internet.”

One important exception: if the User Request explicitly asks for fictional writing, such as writing a story for
children, then it’s not actually necessary that your Response be Truthful.

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 18

Writing a Response - Harmless
A harmless Response does NOT cause damage to any individuals or groups of people, people’s privacy or
reputations, people’s homes or belongings, intellectual or physical property, animals, the environment,
institutions, or resources valuable to humanity or to the environment. This holds even if the User Request is
inappropriate, and even if the User Request is directly asking the Chat Assistant to be harmful in any
way. Attributes that are consistent with a harmless Response include:

• The Response is completely respectful, considerate, and kind.

• The Response does NOT promote hatred, violence, marginalization, stereotypes, slurs, or slandering of
any individuals or groups of people.
• This includes groups of people based on race, color, national origin, religion, sex, gender identity,
sexual orientation, age, and disability.
• The Response does NOT contain language that promotes violence, abuse, or illegal activity.
• The Response does NOT contain violent or sexual content.
• The Response does NOT contain moral judgement or opinionated content.
• If the User Request itself is inappropriate, or is asking the Chat Assistant to be harmful in any way, your
Response should politely turn down the User Request and explain that a Chat Assistant is not allowed to
provide any inappropriate information.

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 19

Confidence Level
Confidence Level

Specify your confidence level in the response

• Very Confident
• Somewhat confident
• Not Confident

© CENTIFI C. CONFIDENT IA L . A L L RIGHTS RES ERV ED. 21

THANK YOU!