Arandee - Quality Improvement Plan (QIP)

Table of Contents
Arandee - Quality Improvement Plan (QIP) .................................................................................................... 1

Table of Contents ........................................................................................................................................... 1

Understanding Quality Expectations at Arandee .......................................................................................... 2

What is the Quality Improvement Plan (QIP) at Arandee? ...................................................................... 2

Quality Assessments ..................................................................................................................................... 2

Learning Period ............................................................................................................................................. 2

Program-Wide Targets .................................................................................................................................. 3

Personalized Feedback - Weekly Score Email ................................................................................................ 3

Performance Overview (Weekly Metrics) ................................................................................................... 3

Recognition & Risk Flags .............................................................................................................................. 3

TRP Error Feedback (if applicable) .............................................................................................................. 4

Removal from the Program .............................................................................................................................. 4

How Removal Decisions Are Made ............................................................................................................. 4

Why Do You Get Assigned Courses? ............................................................................................................... 5

Rater Hub Flags and Behavior Reviews .......................................................................................................... 5

Types of Behavior Reviews ........................................................................................................................... 6

Common Reasons for Rater Hub Flags ...................................................................................................... 6

Common Reasons for Internal Behavior Flags ......................................................................................... 6

What Happens After a Flag?......................................................................................................................... 7

Quality Assistance .............................................................................................................................................. 7

Reviewing Your Quality – Task Results Portal (TRP) .................................................................................. 8

Change Log ........................................................................................................................................................ 8

1
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
Understanding Quality Expectations at Arandee
At Arandee, quality is at the core of our work, and we're equally committed to supporting every
rater's success. The goal of the Quality Team is to ensure that all ratings reflect the standards
outlined in the General Guidelines and task-specific instructions.

The Quality Improvement Plan (QIP) enhances this commitment by offering additional resources
and guidance to help you meet those standards and achieve consistent performance.

What is the Quality Improvement Plan (QIP) at Arandee?

The Quality Improvement Plan (QIP) is a comprehensive initiative designed to enhance rating
quality across the entire program. It is comprised of key processes implemented to help raters
meet and maintain high standards in alignment with the General Guidelines and task-specific
instructions.

The QIP is not a single step or status—it is an ongoing framework that combines feedback, training,
and monitoring to support continuous improvement.

To support this goal, we continuously monitor rater performance and conduct Quality
Assessments across various task types as part of this ongoing improvement framework.

Quality Assessments
• A set of tasks completed by raters is reviewed by the Quality Team. These are called Quality
Assessments.

• You will not know which tasks are included. They appear just like any regular task, and you
are paid for them as usual.

• All task types are eligible to be reviewed during these assessments.

• The Assessment Score is your accuracy score for all tasks completed for a specific Quality
Assessment (task type specific).

Learning Period
• During your first month on the project, you are in the Learning Period. Expectations are
slightly adjusted to allow time for you to ramp up, but quality is still closely monitored, and
significant concerns may lead to additional support or follow-up.

We expect all raters to gradually improve over time and work toward consistent performance
across all task types.

2
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
Program-Wide Targets
To maintain a high standard across the program, we aim for an overall accuracy level of 80% or
higher per locale and per task type.

Raters are expected to:

• Meet the quality standards outlined in the Guidelines.

• Maintain steady progress and performance over time.

Personalized Feedback - Weekly Score Email

The Personalized Feedback - Weekly Score Email is a recurring communication sent to raters in
the Arandee program to help them monitor and improve their quality performance. The goal is to
ensure each rater understands how their work aligns with program expectations and has access to
tools for improvement.

Performance Overview (Weekly Metrics)

A table summarizes the rater’s most recent quality performance, including:

• Weekly Score: This is your overall accuracy score for a one-week period, based on the
Quality Assessment tasks you complete during that time period - this is inclusive of all task
types.

• Target Score: A benchmark based on the average of the top 33% of rater scores for that
week. This sets the quality standard to aim for.

• Placement: A percentile ranking that shows how the rater’s score compares to others in the
same week.

• % of Weeks on Target: The percentage of the last 8 weeks in which the rater met or
exceeded the target score.

• Change Since Last Week: Indicates if the rater’s score improved, declined, or stayed the
same compared to the previous week.

• Average Deviation: A metric assessing how closely a rater’s scores align with the correct
ratings. Lower values (closer to 0) indicate stronger alignment.

Recognition & Risk Flags

Depending on long-term performance, raters may receive one of the following:

3
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
 • Congratulations Message: Awarded if the rater has met or exceeded the Target Score in 7
or more of the last 8 weeks. Recognizes consistent high performance and may result in
access to bonuses or additional opportunities.

• Below Target Flag: Triggered when a rater meets the Target Score in fewer than 2 of the
last 8 weeks. It serves as a warning that access to the program is at risk if improvement
isn’t demonstrated.

TRP Error Feedback (if applicable)

If recent TRP data is available, a table is included summarizing any specific task-level errors. This
section may reflect data from a prior period (1–2 weeks behind the main score) due to processing
time.

Note: The information here may differ slightly from what’s shown in Rater Hub; the email content
should be treated as the definitive source for quality tracking.

Removal from the Program

We are dedicated to providing comprehensive support for performance improvement. Removal
occurs only after extensive support measures have been implemented and multiple
improvement opportunities have been provided. While our goal remains to support all raters in
improving their performance, continued low quality after that support may eventually lead to
removal from the program.

How Removal Decisions Are Made

Raters may be removed from the program if they meet all of these criteria:

• They have completed at least 50% of the required items/tasks for each relevant Quality

Assessment.

• They have a minimum tenure of 1 month in the program.

• They score in the lowest performance group for the same task type in two consecutive

Quality Assessments within a 2-month period.

Performance is reviewed after each Quality Assessment, and decisions are based on a combination
of assessment scores and overall program needs.

We understand that everyone learns at a different pace, so we provide feedback and support along
the way. However, ongoing low performance, especially after repeated opportunities to improve,
may result in removal.

4
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
If removal occurs, you will receive formal communication explaining the reason, including your
Assessment Scores.

Why Do You Get Assigned Courses?

Courses are assigned to help raters strengthen their understanding of the guidelines and improve
performance in specific areas. These assignments are designed to target specific performance
issues and typically take 15–30 minutes to complete. You may receive a course for one or more of
the following reasons:

• Your assessment score is below expectations on a Quality Assessment.

• Your results show consistent patterns of misunderstanding or inaccuracies in a specific

task type.

• You receive flags related to quality concerns or inconsistent application of the guidelines.

• You incorrectly rated a task (or part of a task) covered by the course.

Some courses are mandatory and must be completed before you can continue working. This
applies specifically to English (US) raters in both General Pool and User Groups. In other cases,
courses may be recommended to support your development.

For non-US locales, courses are not mandatory but are still available as an optional resource on
the WeLearn platform to support your development. In addition, non-US raters may occasionally
receive one-page rating reminders via email based on recent trends or performance insights.

These courses target the specific task type or performance area that requires improvement. While
course completion is important, ongoing program participation depends on demonstrating
continuous improvement in your quality scores.

Rater Hub Flags and Behavior Reviews

As part of our quality assurance efforts, we sometimes review individual raters whose performance
raises concerns. This may happen if a rater is flagged for quality issues or if certain behavioral
patterns suggest the need for a review. These reviews help ensure that all raters follow the
guidelines and maintain fair, consistent rating behavior.

If your account is flagged, you will receive an email notification from the Quality Team. At that
point:

• Your Rater Hub account will be initially suspended while the Quality Team investigates.

• You will not be able to accept or complete tasks while your account is suspended.

5
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
 • Investigations are typically completed within 72 hours, though many are resolved sooner.

• Once the review is complete, your access will be restored, and you may resume work.

• You may receive additional feedback via email after a review is conducted.

• If your account remains suspended after 72 hours, please contact the Quality Team through
Zendesk for assistance.

Types of Behavior Reviews

There are two primary review processes when a rater is flagged or shows potential quality
concerns:

• Rater Hub-Flagged Reviews:

These occur when a rater has been automatically flagged by the Rater Hub system, often
due to patterns that suggest low-quality or inappropriate behavior. In some cases, this may
lead to your Rater Hub account being suspended.

• Internal Behavior Reviews:

These are initiated based on internal monitoring and quality checks. They may be
triggered by unexpected behavior, performance anomalies, or spot checks.

Common Reasons for Rater Hub Flags

Flags may be triggered by any of the following:

• Speed/Timing Issues: Working too fast or too slow for the task type.

• Comment Quality: Submitting vague, unclear, or unhelpful comments repeatedly.

• Spamming or Repetitive Ratings: Overusing specific ratings or following obvious rating

patterns (e.g., always choosing the same rating without valid justification).

Please note: This is not an exhaustive list, and other patterns of concern may also lead to a flag or
review.

Common Reasons for Internal Behavior Flags

Internal flags are raised when we observe rating behavior that significantly deviates from
expected patterns or from the consensus of other qualified raters. Below are some common
examples:

• “Not Visited” Flag – It was detected that the landing page was either not visited or not
evaluated properly.

• Overuse of a Rating – Repeatedly giving the same rating (e.g., consistently rating
"Moderately Meets"), regardless of the actual quality of results.

6
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
 • AET Deviations – Spending unusually little or excessive time on tasks, especially if paired
with low-quality ratings.

• Consistent Low-End Misses – Often rating items higher for a clear low range item (e.g.,
giving MM+ when the correct rating is FailsM).

• End-Score Misses – Rare or no use of Fails to Meet or Fully Meets ratings, even when
clearly appropriate.

• Deviations from Correct Ratings – Excessive ratings off by 0.5–1 point from the expected
rating across tasks.

• Inconsistent PQ vs. NM Ratings – Assigning an accurate Page Quality (PQ) rating but a
clearly inaccurate Needs Met (NM) rating, or vice versa.

• Divergence from Other Raters – Patterns that consistently differ from the broader
consensus of well-performing raters (e.g., a rater consistently gives “Slightly Meets” ratings
to results that most high-performing raters identify as “Highly Meets” or “Moderately
Meets”.)

These indicators don’t always point to a serious issue, but they may trigger a review to ensure you
are applying the guidelines accurately and without bias.

What Happens After a Flag?

Our approach prioritizes support and development. When concerns are identified, we implement
the following measures:

• Email Notification – You may receive an email notification summarizing the observed
behavior and steps to improve.
• Course Assignment or Rating Reminder – You may be asked to complete a targeted
course or receive guidance to help correct the issue.
• Probation – You may be placed on a short-term probation period during which your daily
work hours are limited while your submitted tasks are closely reviewed to address specific
behavior patterns, such as failing to visit landing pages when evaluating results, or
other actions that indicate low effort or disregard for task instructions.
• Removal from the Program – In cases of repeated or severe quality issues, the rater may
be removed from the project.

Quality Assistance
The Quality Team is here to support your success. Raters who actively engage with training
materials and feedback often see meaningful improvements in their Quality Assessments. To help
you achieve and maintain high-quality performance, we offer support in three main ways:

7
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.
 • Through the Zendesk support ticket system

• By sharing best practices and course materials on WeLearn

• With a monthly FAQ published on the first Friday of each month, where we answer quality-
related questions submitted during the week the FAQ Survey is live (typically mid-month).
We share a form link by email when the survey opens and provide separate FAQs for the
General Pool and F/H raters to keep the guidance relevant.

If you need assistance, please submit a ticket using this form:

Submit a request (opens in a new tab)

If your question is about a specific task, please include the task URL in your ticket. This helps us
provide you with accurate and efficient support.

Reviewing Your Quality – Task Results Portal (TRP)

You can review your evaluated tasks and understand how your answers were scored by using the
Task Results Portal (TRP), available in Rater Hub under the “Task results” tab.

In the portal, you can:

• Adjust the date range to find past quality assessments.

• Click on individual tasks to see your submitted answers alongside the correct rating
ranges.
• Read the explanation at the top of each task to understand why a certain rating was
assigned.

The TRP is a great tool to help you understand your performance and learn from previously
submitted tasks. For more detailed guidance, please refer to the Task Results Portal Access Guide,
available here. Please note you need to be signed into your WeLearn account in order to view the
guide.

Change Log
This change log outlines the updates made to this document to ensure clarity, accuracy, and
alignment with current program policies. It highlights revisions to evaluation criteria, course rules,
and removal procedures.

Date Section Change Description

8
Confidentiality Notice: This document and its content are confidential information of Welo Data and intended solely for the addressed recipients. Pursuant to the non-disclosure
agreement (NDA) between you and Welo Data, you may not disclose, share, copy or distribute the content of this document outside the organization.