Georg Zoeller,

BioWare

Game Development
Telemetry
Georg Zoeller,
Lead Technical Designer,
BioWare Austin

georg@gulbsoft.org

About http://www.linkedin.com/in/georgzoeller

Georg Zoeller
• 7 player characters with unique content and quests.
• 4 male, 3 female player appearances.

• 10 dedicated story testers.

• 9 male and 1 female.

Guess which appearances rarely got used?

Once upon a time….

…in the ‘Jade Empire’ QA department
Dragon Age: Origins
• ~800,000 words of dialog voiced by
• 144 voice actors in English alone.
• 180+ area layouts.
• 3 classes, 3 races, 2 gender options.
• 10 companion characters.
• 6 unique, hour long Origin Stories.
• 300+ abilities
• 300,000+ lines of script code.
• 3,000 Cutscenes, 5,500 animations, 20,000 models
• New IP, World, Rules, Engine.

Full Path coverage is hard to achieve, and even harder to verify.

QA resources are finite and sometimes volatile.

Planning and managing for full path coverage is very time

intensive.

The Problem.
 Telemetry -
n. The science and technology of automatic measurement
and transmission of data by wire, radio, or other means
from remote sources, as from space vehicles, to
receiving stations for recording and analysis.

Game Development Telemetry –

… automatic measurement and transmission of data from

game executable, build pipeline and development tools
for recording, analysis and workflow improvement!

…not from space, yet

Definition.
Customer facing Developer facing
 Understand how customers  Understand how developers use
experience the game. tools and pipeline.

 Identify and react to issues post  Identify and react to pipeline

launch. and workflow issues .

 Understand the market, learn for  Measure stability, productivity

future games. and efficiency of development.

 Already quite popular.  Is Business Intelligence.

Same idea, different goals.

2nd Generation homegrown telemetry system.

Serving Dragon Age: Origins, Mass Effect2, SWTOR

On Dragon Age: Origins alone:

• 1,100,000+ games tracked on 1,141 different machines.

• Tracked any game of DA:O in the entire EA network.
• 250 GB of data gathered in ~38,000,000 datapoints
• 50,000+ bugs, 4597 unique users on 582 machines.

Let’s have a look…

All your data….

Introducing ‘SkyNet’
• Know who is running which build, language, executable.
• Drill down into any running game…
Chronological / spatial event visualization.
Aggregated spatial visualization (hmp)
• Raw data is great…
... if your day starts with SELECT socks, pants FROM closet;

• Engineers demand different detail than managers.

• Executive summaries are … graphs.

• Best solution we found: Drill-down

Data Visualization
Why presentation makes or breaks your system.
• Report view provides the next level of detail.

• Aggregates information for leads and intermediate managers.

• Reports are nothing more than SQL statements.

• But can be exported and subscribed to.

• Allow further drilldown…

• This view is interesting to programmers, not so much managers.

• Incoming callstacks are timestamped, sanitized and hashed.

• Tells us the rough# of unique crashes in the system

•Freshly introduced crashes file an ‘investigation task’ to

engineering QA.

• Who can retrieve the full details by drilling down…

 Business Intelligence – Great for evaluation outsourcer performance
Now let’s use the telemetry data to improve other workflow...

Industry Typical Bugtracker interface

• Lots of fields
• Some mandatory
• Many freeform.

Result:

• Fields stay empty

• Random values, Typos
• Database pollution.

Smart Bugs
When bugs learned to listen…
Telemetry enabled interface.

• Retrieves fields from the server.

• No typos in database fields.
• Fewer mandatory fields.
• Links to session history, RunInfo, etc.
• Location aware bugs.

• Suggests who should get the bug.

• Warns of duplicates based on text, build and location.
• Vacation Tracker Aware.
• Smart Categories.

But wait, there is more…

Better?.
• 95% adoption rate for new interface after 3 months.

• Significant Bug count increase (especially non QA).

• Significant Bug quality increase.

• Bugs always have platform and build #.
• Hardware information in each bug.

• Much cleaner bug database, spatial bug information.

• Spatial information == great for finding dupes.

• Increased overall QA capacity by drafting everyone to help.

• Much improved reporting…

Results.
Achievements work!
• They influence player behavior through reward.
• Teach about deeper aspects of the game.
• Yield valuable telemetry for developers.
• Inspire some gamers to go to great length...

… and many developers are gamers, so:

• Yes, they work for (some) developers too.
• Can be used to influence how testers play.
• Yield valuable balance telemetry for design.
• Inspire some developers to play the game. A lot.

Achievements
For developers?
 Doubled hours of playtime  Some people stayed
for a month after ‘addicted’ and had to get
introduction them all.

 Increased bugs filed by  Reward desirable

both QA and engineers. behavior…

 Got people competing and

excited about the game.
Morale boosters.

 Taught users about more

obscure game features.

Results.
• Telemetry systems don’t mix well!
• Different goals, different needs.
• Different technical requirements

• Start small, don’t over-engineer.

• Be Non Invasive.

• Reuse test hooks

• Make it dead simple to add events.

• Create an API and they will come…

Getting Started
Practical tips and pitfalls.
 • C# Server
• Receives Packets (udp)
• Tracks clients.

• Mangles packets into DB.

ASP.Net WebApp
DB

SkyNet API

Tools and
Sensors

Our Tech.
 UDP vs TCP?
◦ Lightweight, non blocking.
◦ We use one way communication.
◦ Telemetry data may get lost.

 Server software
◦ C# is RAD.
◦ Runtime-Compiled ‘packet handlers’
◦ Dynamically routes events into tables by type.

 Protocol
◦ Character delimited string.
◦ Only defined in packet handlers.
◦ No support for fragmentation.

More Tech Talk.

 Game Client / Server
◦ Callstacks, Warnings, etc.
◦ Game Events, Movement, etc.
◦ Savegames, Screenshots
◦ Performance Timings

 Toolset
◦ Startup time, Usage patterns.
◦ Callstacks, Warnings, etc.

 Wikis / Sharepoint.
 Bugtracker
 Landesk
 IP enabled coffee maker.
 Twitter
◦ Goes well with timeline of PR

Useful data sources.

Usage patterns.
•One database that indexes information from any available
datasource in the company.

• Makes it searchable and viewable.

•And retains historical information so we can understand,

optimize and learn.

•Push information back into the game (e.g. warn users when
playing a broken build).

• Something like this…

Endgame
Where do we go from here?
 Questions, Comments, Feedback?

If you have further questions -or end up implementing your

own development telemetry system, I’d like to hear from
you – especially if you have ideas for additional data
sources.

Georg Zoeller
gdc@gulbsoft.org

Thank you.
For attending this presentation.