Transcription QA Guidelines

The project is designed to collect audio and transcription of phone conversations. Phone
conversations are collected through our app Robson: participants register for the project, call their
friends/family through the app, and get paid per hour of phone conversation. After we approve the
audio collected (making sure that speakers are audible and speak required language throughout the
entire call) we transcribe the audio through our app as well. The audio/transcription will be used to
develop speech recognition/transcription technologies.

Phone conversations must pass AUDIO QA to be eligible for payment. See Audio QA guidelines for
further instructions. After the phone conversation audio is processed (split into two audio files and
segmented) a transcriber transcribes the audio verbatim.

Robson Workflow

Follow the QA invitation link and log in with your Robson profile. Click on “QA” in the blue list of
tabs. (Similar to audio QA) Choose an available QA task which is in QA status “New” and Task Status
“In Review”.

Once you click on the QA Task ID, the task will open. Click on Task No. 1 in the left panel and you will
see this:
Each task has only one number (no. 1), but you will see several segments in one task. Each task is
max. 10 minutes long. Play the audio of each segment (grey box) and check the transcription. On the
left you see the transcription of the transcriber.

The right text field has the same transcription in it, but you can adapt and change it to the correct
transcription.

Please adjust the transcription and decide whether the transcriber did a decent job for the
transcription.

You either “PASS” the segment, if the majority of the transcription was correct or you “FAIL” it if the
transcriber did an insufficient job. Transcribers will only be paid for “PASSED” segments. The default
setting for a segment is on “PASSED”.

You see the guidelines on the right, please make sure to read them before you start!

PASS a segment.
If you only had to change a few things, like adding a tag, fix punctuation and add some words that
were missed, keep the default setting of “pass”.

FAIL a segment.
Only fail a segment when you feel as if you had to transcribe the segment again almost from scratch
and most of the words have been missed or obvious tags (like initialisms, foreign languages) have
not been used. Ask your project manager for a check if you are unsure whether to fail a segment
before submitting QA.

DISCARD a segment.

Discard a segment, if there is no sound or only a non-speech sound, but no speech. The transcriber
might have already discarded the segment. No transcription is needed for a discarded segment. You
still must pass/fail the segment depending on what the transcriber filled in.

Please make sure to PASS or FAIL each single segment, even if it was discarded! You will not be
able to submit QA if you do not click PASS or FAIL for each segment.
TOPICS.

Once you have checked and passed/failed all segments for one task, click “Audio topics” in the left
panel. Check whether the topics chosen by the transcriber match the audio. If you disagree,
click/unclick topics. Click on “Update topics” after you adjusted the list.

Once all transcription and the audio topics have been checked and adjusted, submit QA by choosing
“QA Submit”. Please finalize your marking by choosing PASS or FAIL.
Pass a task if 70% of the Segments has been marked as PASSED.
Fail a task is more than 30% of the segments have been marked as Failed.

You will not be able to adjust any transcription or audio topics once you have submitted the task,
so make sure everything is correct.

When you click "Submit QA", a pop-up will appear and invite you to provide some feedback to the
person who completed the task. Select one or several feedback items among the provided options,
depending on the errors you observed while completing QA. The transcriber will see them in the
web platform alongside the total of minutes that were passed and failed.

NOTE: In Robson if a QA decides to fail an entire transcription task, it will be released back into the
pool and anyone else can work on the task except for the original transcriber.