UHRS Web Entry Instructions

1.  Required – Sign up for a Windows Live ID if you do not already have one.

Appen will create your user account on UHRS using the Windows Live ID with same
email that you used for your contract.

If you currently do not have a Windows Live ID account associated with that email
address, sign up for one by going to https://signup.live.com/ . (Please note that you
do not need to sign up for any other Microsoft service like Hotmail to get a Windows
Live ID) Remember to create your Windows Live ID with the email you provided in
your contract under the Windows Live ID section.
2. Using Internet Explorer only, go to the UHRS Web Entry Link:

                 https://prod.uhrs.playmsn.com/Judge/Views/Login.aspx 

3.  Bookmark or make this address a Favorite (Ctrl+D) 

4.  Click Sign in with LiveID and enter your username and password.

Your Windows Live ID (WLID) will be formatted like an email address (for

example, johnsmith@internet.com).
If you do not recall your WLID password, you can reset it yourself
athttps://account.live.com/ResetPassword.aspx.
If you any difficulty logging in, please send an email with complete details, including
the username and password you are using along with a screenshot
tocrowdhelp@appen.com.
5.  Click on MarketPlace 

6.  Click on View Guidelines, Preview and then Start Judging.  Be sure to review the
guidelines for any crowdsourcing tasks.  You will be evaluated on the quality of your
ratings, and judges who do not comply with project guidelines may no longer be
considered eligible to work on crowdsourcing tasks.  For complete details, refer to
your contract and Statement of Work. 
7.  Some tasks may require a Qualification Test.  You must successfully pass the
Qualification Test before the Start Judging link will be active.

8.  Once you have worked on a task in the MarketPlace, it will appear in My
HitApps.  You can select tasks from either the MarketPlace (new tasks) or My
HitApps (tasks you have already worked on) when available. 

9.  Start judging!  The ratings pane and the landing page may appear side by side or
stacked on top of each other as in the examples below.  Select the rating for the
page, click Submit, and the next page should be loaded automatically. 

10.  Make sure that you are not clicking the Skip icon.  This will advance to the next
query, and your rating will not be counted. 

11.  When you are done rating, be sure to Logout by clicking the small, gray arrow
on the ratings pane, and then click Logout on the dashboard. 

12.  Click on My Report to view a summary of judgments completed and earning

totals.  The ‘My Report’ feature is very useful for keeping track of HitApps, valid
judgments completed, and earning totals. Reports can be viewed by selecting a date
range and then clicking on "generate report".  To calculate anticipated earnings, be
sure to run the report in line with the invoice cycle dates.
 Troubleshooting and Error Codes
IF YOU ARE UNABLE TO LOG IN: 
Make sure your Windows Live ID has been registered at www.live.com/signup
If you do not recall your WLID password, you can reset it yourself
athttps://account.live.com/ResetPassword.aspx
 Log out of any WLID you may be logged into Internet Explorer:

 Go to: www.live.com and logout if the page shows that you are logged into
an an account.
 Be sure that the “Keep me signed in” box is unchecked, as you may be
logged into other sites with your WLID unknowingly.
If you are still not able to login, contact Crowd Help (crowdhelp@appen.com) and
provide the Windows Live ID that you are attempting to use so we can update your
profile if necessary.
IF YOU ARE HAVING ISSUES WITH PAGES NOT LOADING CORRECTLY:
Error 503 indicates you are logged in through the tool bar which is incompatible with
Web Entry.  To process Crowdsourcing tasks, you must be logged in through UHRS
Web Entry via the link https://prod.uhrs.playmsn.com/Judge/Views/Login.aspx
Clear the toolbar/IE application cache:

 Click on ‘Start’ then select ‘Run’, hit enter.

 Type in CMD to launch a command prompt.
 Input the following into the prompt, rundll32 dfshim CleanOnlineAppCache
(You can copy and paste this line using your mouse).
 Hit enter to run the command, nothing will pop up, it should move to a
new line. This is normal behavior.
 Close the window after running the command.
Re-Sync System Time:

 Right click the clock in the bottom right hand corner, select Adjust
Date/Time.
 Once the new window appears, please click on the “Internet Time” tab,
click the “Change Setting” button.
 In the Server drop-down list please select another server, for example:
time.nist.gov.
 Click the “Update Now”, now click OK.
 Restart your computer.
If you are having difficulty getting the links to open in the tool or if you are receiving
the 404 file error you may need to disable your add-ons.  Complete the steps:

 Open Internet Explorer.

 Click on the programs tab.
 Click on Manage Add-ons.
 Disable any add-ons or toolbars.
Xml malformatted error:

 IE Settings and click in Manage add-ons. 

 Enable XML DOM Document 3.0

Understanding Spam
What is a Spam score?

The client uses Spam scores to insure the judgments they are collecting are useful. 
The spam test works by giving you “test” hits mixed in with the regular hits.  Spam
hits are very simple and easy to get correct.  They are not designed to trick a
consultant into a poor judgment.  If too many Spam hits are failed, the judge may
become auto-disabled or manually blocked from the task.  Your Spam score will be
reported as a percentage of spam hits correct and can be found in the My Reports
section in UHRS.

What is the difference between Spam and RTA scores?

The RTA score is based upon “real time audits” that help you understand what a
good judgment is for the task. RTAs are designed to help reinforce the guideline and
assist you in your future judgments

You will know when you have encountered an RTA because instant feedback will be
provided after you click the “submit” button. This differs from spam hits, as with
spam hits you will not see immediate feedback and you will not be aware that you
have completed a spam hit.

Your RTA score is based upon how many RTA hits you answered correctly vs. the
total number of RTA hits that have been presented to you.  The spam hits are mixed
into the set of judgment scenarios (hits) and you will see your score in the My Report
feature in UHRS.  The spam score is calculated on how many of the spam set
questions you answer correctly.

Why have I had a drop is Spam score when I have done well on the same HitApp
before?

In order for the data to be useful, judgments need to be reliable.  To promote the
data quality, the client has recently tightened Spam thresholds on some HitApps.  If
you find you are not doing well on a task, be sure to re-read the Guidelines carefully
before attempting the task again.  It is very possible that the guidelines have been
updated since your last review or that new spam hits are targeting an area of the
guideline that you need to review.

Also, keep in mind that simple HitApps often have a stricter Spam requirement.  So
be careful not to get too comfortable or “click happy” as you may become disabled
quickly.

I didn’t pass the qualification for the HItApp, can you reset the qualification test for
me?

Unfortunately, Appen does not have the ability to reset or add qualification for the
HitApps.  Nor are we able to provide feedback, clarification, or examples on the
crowdsourcing tasks.  All HitApps are designed and controlled by the client and are
meant to be completed independently by the consultant.  Our best advice is to
carefully read the guidelines for each task before attempting qualification and apply
critical thinking to your judgments.

I have been working on a HitApp that offers incentive money for a good spam
score.  Why am I not earning the incentive when MyReports says I have met the
Spam requirement?

Often we see incentive descriptions like this; For every 100 hits with a Spam score of
“X” or more will result in earning an additional $”X”.  What is important to
understand is that incentive earnings are based on your rolling 14 day Spam score,
and they are not based on your average Spam score for all judgments on the HitApp.
Incentive earnings will start and stop as your rolling 14-day Spam score achieves or
falls below the Spam score target on the HitApp.  If you notice you are not earning
incentive on a particular day, you can check your 14-day Spam score.  For example,
to see the Spam score used on April 15th’s incentive calculation, you could pull a
report for April 2nd through April 15th to see what score you achieved and if you
met the minimum to receive incentive earnings.

We understand that incentives can be confusing, and we are currently working with
the client to modify incentive descriptions and provide more detail within UHRs to
help consultants understand them better.