PERK Corp.

Final Testing Detailed Experiments

Lab Report

Pranav Pannala, Evan Colenbrander, Ryan Miller, Kaleb Ryan

1
 Abstract

Our first two experiments focused on the audio portions of our product. We
tested different accents and different distances to determine how effective our
product would be under those two different conditions. We also tested the male
and female versions of each accent. In order to test the experiment we used a
computer and our circuit prototype. These two devices remained the same
throughout the entire test to not introduce any changing variables and the device
playing the audio was also set at the same volume level for the entire test. For
the distance test we recorded each audio file three times and then calculated the
average accuracy. After each trial, we moved the computer back 1 additional foot
until we reached the 16 foot mark. For the accent test we placed the devices 2
feet apart. We then tested the male and female versions of four different accents
three times each and calculated the average accuracy. The first result from our
findings was that the female voices worked better than the male voices, but there
wasn’t a significant difference. This is because the males' voices are lower pitch
so they blend in more with the background noise. This made it difficult for the
microphone to pick up on the lower pitched voices compared to the higher
pitched female voices. The second result from our tests determined that there
wasn't a significant difference with any of the different accents. The final finding
from our tests was that our product works fine as far as 13 feet away from the
microphone. We were not able to test outside of this range due to the length of
our cables but this will not be a significant issue because users will not be
communicating with the product from very far away.

2
 Table of Contents

1. Abstract

2. Table of Contents

3. Introduction

4. Test Plan and Set-Up

5. Data

6. Analysis / Conclusions

7. References

3
 Introduction

Purpose:

This report details our findings for our first two tests: the distance and accent
tests. The goal of our tests is to determine if the speech to text capabilities of the
RoboBuddy are accurate enough for the product to be successful. We will analyze the
effects of speaking to the RoboBuddy from increasingly farther distances and the
effects of speaking to the RoboBuddy using different accents. In the future, we plan on
conducting one more test to look at the physical properties of the product. For this test
we will stress test ABS plastic to determine how strong the product will be.

Distance Test Background:

Research has determined that the average standard size bedroom in the United
States is 11 feet by 12 feet.1 For our distance test, we will test the maximum possible
distance that would fit in a 11 by 12 foot rectangle. This value would be the diagonal of
the 11 by 12 foot space which is approximately 16 feet. This will be the maximum value
we test for the distance test.

Voice recordings Background:

For the accent test we will use 8 different recordings. These recordings have
been selected to represent accents from North America, Europe, Asia, and Australia. In
addition to each of the accents we will also be testing the male and female variants.

1 https://www.doorwaysmagazine.com/average-bedroom-size/#:~:text=We%20determined%20that%20the%20average,14%20feet%20by%2016%20feet

4
The different voices we have selected are listed below:
● American Accent
○ English (United States) Zoe
○ English (United States) Evan
● English Accent
○ English (Great Britain): Malcolm
○ English (Great Britain): Kate
● Indian Accent
○ English (India): Rishi
○ English (India): Veena
● Australian Accent
○ English (Australia): Karen
○ English (Australia): Lee

Text to Speech Background:

For each accent, we used an online website with a text to speech field to
generate the audio recordings.2 We decided to use the default message from the
website because it was a decent length with varying sentence structures.

Default Message:

“Hi, how can I help you? Great, no problem. Your transaction is complete. Have a great
day. Feel free to type anything else, and even choose another voice.”

2 https://www.nuance.com/omni-channel-customer-engagement/voice-and-ivr/text-to-speech.html#!

5
 Test Plan

Test 1: Accent Test

Purpose:
The United States is a very diverse country so our product needs to be compatible with all
different types of accents. By testing each accent, we can determine if our product will be able
to be used by all different types of consumers.

Experimental Setup:
We will test different audio recordings of regional accents to determine if the RoboBuddy is able
to pick up all different accents. We will test male and female versions of the
American, English, Indian, and Australian accents.

Setup Sketch:

Materials:
● 8 Audio Files: 4 different accents male/female
● Raspberry Pi
● Computer
● Recording device: spreadsheet on computer
● Web Speech API
● Visual Studio Code

Dependent Variables:
● Accent of voice
● Male or female

Independent Variables:
● Distance from the microphone
● Speaker volume
● Speaker and microphone type

6
Procedure:
1. Setup the experiment by loading the accent audio files onto a phone. Load the
RoboBuddy code on a raspberry pi and set up the two devices to be exactly two feet
apart.
2. Play the first audio recording. Repeat the recording three times and record the accuracy
for each test.
3. Determine the accuracy by dividing the number of correct words from the total number of
words said. Look at the text to speech field on the computer to see which words the
RoboBuddy incorrectly recorded.
4. Repeat the second step seven more times for each of the other audio files.
5. Determine the averages across each test by adding up the accuracies for each trial and
dividing by three.
6. Analyze the results to determine if the RoboBuddy is compatible with different accents.
7. Determine if a better speaker or speech to text system will be needed for the final
prototype.

Test 2: Distance Test

Purpose:
We must determine if the RoboBuddy can effectively work at longer distances. We also need to
determine the effective range of the microphone and speaker system. Our distance test will be
able to give us results to form conclusions on both of these questions.

Experimental Setup:
This test will be used to determine how effective the RoboBuddy will be in larger rooms. In order
to eliminate as many changing variables, only two audio files will be used. We will have a male
and female American accent to test.

Setup Sketch:

7
Materials:
● Tape measure
● 2 Audio Files: 1 male and 1 female
● Raspberry Pi
● Computer
● Recording device: spreadsheet on computer

Dependent Variables:
● Distance from the speaker
● Male / female audio file versions

Independent Variables:
● Distance from the microphone
● Speaker volume
● Audio files used
● Speaker and microphone type

Procedure:
1. Setup the experiment by loading the two audio files onto a phone. Load the RoboBuddy
code on a raspberry pi and set up the two devices to be exactly one foot apart.
2. Play the first audio recording. Repeat the recording three times and record the accuracy
for each test.
3. Determine the accuracy by dividing the number of correct words from the total number of
words said. Look at the text to speech field on the computer to see which words the
RoboBuddy incorrectly recorded.
4. Move the phone one foot farther from the microphone. Repeat steps 2-3 using the same
two audio files. Continue to move the phone back until the accuracy falls below 50%.
5. Determine the averages across each test by adding up the accuracies for each trial and
dividing by three.
6. Analyze results to determine the effectiveness of the RoboBuddy and determine if any
improvements need to be made for the microphone and speaker system.

8
 Data

Test 1: Accent Test

Voice Test 1 Test 2 Test 3 Average

American “hi how can I “hi how can I “hi how can I 99% accuracy
Accent (Male) help you great help you great help you grade
no problem your no problem your no problem your
transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

100% 100% 96%

American “hi how can I “hi how can I “hi how can I 100% accuracy
Accent (Female) help you great help you great help you great
no problem your no problem your no problem your
transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

100% 100% 100%

English Accent “how can I help “hi how can I “hi how can I 99% accuracy
(Male) you great no help you great help you great
problem your no problem your no problem your
transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

9
 96% 100% 100%

English Accent “hi how can I “hi how can I “hi how can I 96% accuracy
(Female) help you great help you great help you great
no problem you no problem you no problem you
will transaction will transaction will transaction
is complete have is complete have is complete have
a great day feel a great day feel a great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

96% 96% 96%

Indian Accent “hi how can I “hi how can I “hi how can I 100% accuracy
(Male) help you great help you great help you great
no problem your no problem your no problem your
transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

100% 100% 100%

Indian Accent “hi how can I “hi how can I “hi how can I 100% accuracy
(Female) help you great help you great help you great
no problem your no problem your no problem your
transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

100% 100% 100%

10
Australian “no problem “hi how can I “hi how can I 88% accuracy
Accent (Male) you'll transaction help you great help you great
is complete have no problem no problem
a great day feel you'll transaction you'll transaction
free to type is complete have is complete have
anything else a great day feel a great day feel
and even free to type free to type
choose another anything else anything else
voice” and even and even
choose another choose another
71% voice” voice”

96% 96%

Australian “how can I help “how can I help “how can I help 93% accuracy
Accent (Female) you write no you write no you write no
problem your problem your problem your
transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and even and even and even
choose another choose another choose another
voice” voice” voice”

93% 93% 93%

Test 2: Distance Test

Female Voice: Zoe
Distance (feet) Trial 1 Trail 2 Trail 3 Average

5’ “Hi, how can I “Hi, how can I “how can I help 99% accuracy
help you? Great, help you? Great, you? Great, no
no problem. no problem. problem. Your
Your transaction Your transaction transaction is
is complete. is complete. complete. Have
Have a great Have a great a great day. Feel
day. Feel free to day. Feel free to free to type
type anything type anything anything else,
else, and even else, and even and even
choose another choose another choose another
voice.” voice.” voice.”

100% 100% 96%

10’ “hi how can I “hi how can I “Hi how can I 89% accuracy
help you great help you great help you great
no problem your no problem your no problem your

11
 transaction is transaction is transaction is
complete have a complete have a complete have a
great day feel great day feel great day feel
free to type free to type free to type
anything else anything else anything else
and” and even juice and”
another voice”
86% 86%
96%

13’ “Hi, how can I “Hi, how can I “Hi, how can I 99% accuracy
help you? Great, help you? Great, help you? Great,
no problem. no problem. no problem.
Your transaction Your transaction Your transaction
is complete. is complete. is complete.
Have a great Have a great Have a great
day. Feel free to day. Feel free to day. Feel free to
type anything type everything type anything
else, and even else, and even else, and even
choose another choose another choose another
voice.” voice.” voice.”

100% 96% 100%

Male Voice: Evan

Distance (feet) Trial 1 Trail 2 Trail 3 Average

5’ “How can I help “How can I help “How can I help 94% accuracy
you? Great, no you? Great, no you? Great,
problem. Your problem. Your Your transaction
transaction is transaction is is complete.
complete. Have complete. Have Have a great
a great day. Feel a great day. Feel day. Feel free to
free to type free to type type anything
anything else, anything else, else, and even
and even and even choose another
choose another choose another voice.”
voice.” voice.”
89%
96% 96%

10’ “How can I help “How can I help “How can I help 67%
you? Great, no you? Great, no you? Great, no
problem. Your problem. Your problem. Your
transaction is transaction is transaction is
complete. Have complete. Have complete. Have
a great day.” a great day. Feel a great day.”
free to type
57% anything else, 57%

12
 and even
choosing.”

86%

13’ “how can I help “how can I help “how can I help 72%
you Greg no you great no you Greg no
problem your problem your problem your
transaction is transaction is transaction is
complete have a complete. Have complete have a
great day feel a great day feel great day”
free to type free to type
anything else anything else 54%
voice” voice”

79% 82%
*There were varying amounts of background noise during the experiments, the voice recognition
was much more effective when tested while there was little to no background noise.

Test 1 Rankings (based on average percentage):

1. American Female - 100%
2. Indian Female - 100%
3. Indian Male - 100%
4. American Male - 99%
5. English Male - 99%
6. English Female - 96%
7. Australian Female - 93%
8. Australian Male - 88%

Test 1 Setup:
The setup consisted of two devices placed exactly 2 feet apart. One device was used to play the
audio recordings and the other device had the RoboBuddy interface loaded.

Test 2 Setup:
The setup for our second test used the same two devices from the first test. One device was
placed on the table and was moved farther from the microphone after each trial.

13
 Conclusions

Test 1:

Our results show us that the product works best when used with almost every accent.
Compared to the previous tests with the computer microphone, the real microphone performed
significantly better. When looking at the graphs for both the male and female voices, the new
microphone clearly performs better with an average accuracy never falling below 88%. Because
the tests gave us very high accuracy results, we do not expect any of these accents to have
issues. The only accent that raised concerns in the last test was the English accent. This time,
the English accent accuracy was raised to 99% and 96% so the previous test results are no
longer a concern.

Original Female Accent Test: Updated Female Accent Test:

Original Male Accent Test: Updated Male Accent Test:

In contrast to our last test where particular accents had higher accuracies, our data this
time shows all accents maintaining high accuracies across all trials. Also contrasting our old
data, the male and female voices both had comparable accuracies. Except for one outline trial
with the Australian male accent, each trial scored over a 93% average. With our past results, we
concluded that the male voices tended to be much deeper compared to the higher pitched
female voices. This made the male audio harder to pick up because it blended in more with the

14
background noises. This time, this was not as much of an impact because both the male and
female voices were successful. Our findings from the second accent test raise no noticeable
issues that must be addressed and the tests confirm that our RoboBuddy speech recognition
system works as intended. The new microphone showed significant improvements over the old
trials.

Test 2:

When looking at the results for the second test, the accuracy for the female voice scored
a 99% at both 5 and 13 feet and scored 89% at 89 feet. The male voice scored a 94% at 5 feet
and then a 67% and 72% at 10 and 13 feet. Most users would not be standing on the farthest
possible side of a room when using the product so this data tells us that the product will be
functional at standard distances. When the user starts to move farther away from the
microphone the range results can drop but we do not believe this will be a significant issue
because most conversations will happen in a closer range.

Original Distance Test: Updated Distance Test:

When looking at the male vs female data we can see that the male voice did perform
worse than the female voice. Again, this is due to the male voice being a lower pitch compared
to the female voice. Because the RoboBuddy is being used with children we can expect the
children to have accuracies that align more with the female voice than the deep adult male
voice.

Overall Conclusion:

The overall takeaway from our tests is that the product works accurately with all the
accents up to thirteen feet away.

15
 References

1. https://www.doorwaysmagazine.com/average-bedroom-size/#:~:text=We%20determined%20that

%20the%20average,14%20feet%20by%2016%20feet

2. https://www.nuance.com/omni-channel-customer-engagement/voice-and-ivr/text-to-speech.html#!

3. https://en.wikipedia.org/wiki/British_Americans#:~:text=0.6%25%20of%20the%20total%20U.S.%

20population.

16