Measuring Performance, JavaScript Games, and Distributed Testing

John Resig http://ejohn.org - http://twitter.com/jeresig 

Measuring Performance 

Analyzing Performance
✦

Optimizing performance is a huge concern: Faster code = happy users! Measure execution time Loop the code a few times Measure the difference: ✦ (new Date).getTime();

✦ ✦ ✦ 

Stack Profiling
✦ ✦ ✦

jQuery Stack Profiler Look for problematic methods and plugins http://ejohn.org/blog/deep-profilingjquery-apps/ 

Accuracy of JavaScript Time
We’re measuring the performance of JavaScript from within JavaScript!

http://ejohn.org/blog/accuracy-of-javascript-time/ 

15ms intervals ONLY! Error Rate of 50-750%! 

Performance Tools
✦ ✦

How can we get good numbers? We have to go straight to the source: Use the tools the browsers provide. Tools: ✦ Firebug Profiler ✦ Safari Profiler ✦ (Part of Safari 4) ✦ IE 8 Profiler

✦ 

Firebug Profiler 

Safari 4 Profiler 

IE 8 Profiler 

FireUnit
✦

A simple JavaScript test suite embedded in Firebug. http://fireunit.org/

✦ 

FireUnit Profile Data
fireunit.getProfile();
{ "time": 8.443, "calls": 611, "data":[ { "name":"makeArray()", "calls":1, "percent":23.58, "ownTime":1.991, "time":1.991, "avgTime":1.991, "minTime":1.991, "maxTime":1.991, "fileName":"jquery.js (line 2059)" }, // etc.

http://ejohn.org/blog/function-call-profiling/ ]} 

Complexity Analysis
✦ ✦

Analyze complexity rather than raw time jQuery Call Count Profiler (uses FireUnit)
Method Calls 542 592 754 610 495 887 23772 307 6n 6n 8n 6n 5n 9n 2n+n2 3n Big-O

.addClass("test"); .addClass("test"); .removeClass("test"); .removeClass("test"); .css("color", "red"); .css({color: "red", border: "1px solid red"}); .remove(); .append("<p>test</p>"); 

Complexity Analysis
✦

Reducing call count helps to reduce complexity Results for 1.3.3:
Method Calls 298 3n 5n 2n Big-O

✦

.remove();

.html("<p>test</p>"); 507 .empty(); 200

http://ejohn.org/blog/function-call-profiling/ 

Why JavaScript Games Are HARD to Build 

Browser-Based Games
✦ ✦

No installation required (Will be able to play at work!) 

Why not Flash?
✦

JavaScript works everywhere ✦ Desktop ✦ iPhone / Mobile Device ✦ Wii ✦ OLPC JavaScript is an open technology

✦ 

Goals of a game
✦ ✦ ✦ ✦

Should be Multiplayer Can’t be casually cheated Work well everywhere Addictive 

Why Multiplayer?
✦

Incredibly addictive ✦ No longer “alone” ✦ Enticed to continue playing 

3 Styles of Games
✦ ✦ ✦

Strategy Intelligence Accuracy 

Strategy Games
✦ ✦ ✦ ✦

Require a lot of time to think Generally last over a couple hours or days Replayability Hard to cheat 

WarFish

✦

http://warfish.net/ 

Nile Online

✦

http://www.playnileonline.com/ 

Strategy Games
✦

Very server-side heavy ✦ Most logic hidden from the user Hard to cheat ✦ Casual cheaters can’t change values ✦ Dedicated cheaters have to write full AI

✦ 

Intelligence Games
✦ ✦ ✦

Player’s intelligence/knowledge challenged Games could be quick or slow-paced Easy to cheat ✦ Casual cheaters can open dictionary / encyclopedia for answers 

Word Twist

✦

http://wordtwist.org/ 

Babble

✦

http://playbabble.com/ 

Iron Sudoku

✦

http://ironsudoku.com/ 

Speed/Accuracy Games
✦ ✦ ✦

Require low latency Fast-paced and addictive JavaScript completely fails ✦ Garbage Collection cycles freeze the browser None, or few, Accuracy-centric JavaScript games Experienced coders can easily cheat ✦ (A bot to hit the keys at the right time)

✦

✦ 

Guitar Hero

http://ejohn.org/apps/hero/ 

Guitar Hero
✦

Heavily dependent upon accuracy ✦ (Hit the right notes at the right time) Garbage collection cycles absolutely kill the game Rendering the play area is also difficult ✦ And impossible in all browsers. ✦ (Use HTML 5 Canvas Element)

✦

✦ 

Failures on All Ends
✦ ✦ ✦ ✦

Strategy: Slow, secret server-side code Intelligence: Easily cheatable Accuracy: Too hard to implement Optimal solution would be a combination of the above. JavaScript games can’t be like other games, have to be unique.

✦ 

What can make a fun game?
✦ ✦ ✦ ✦

Quick play Points High Scores Head-to-head competition 

Wordsplay

✦ ✦

Real-time Boggle Head-to-head with other players http://www.wordsplay.net/ 

Tringo

✦

Tetris + Bingo (based on a Second Life game) http://ejohn.org/tringo/ 

DeepLeap

✦ ✦

Fast-paced Scrabble-like game Speed + Intelligence + Strategy http://deepleap.org/ 

vs. Cheating
✦

All words are recorded with exact time information Game is “played back” on the server to verify score integrity using Server-Side JS

✦

✦

This data can be used to simulate a multiplayer experience! 

DeepLeap
✦

Works in multiple languages ✦ Dictionaries pulled from OpenOffice, can build a Spanish version in < 5sec Players can challenge each other head-tohead Score multiplier (carry over from Guitar Hero)

✦

✦ 

Distributed Testing 

Choose Your Browsers 

Cost / Benefit

IE 7

IE 6

FF 3

Safari 3

Opera 9.5

Cost

Benefit

Draw a line in the sand. 

Graded Support

Yahoo Browser Compatibility 

Browser Support Grid
IE Previous Current Next
6.0

Firefox
2.0

Safari
3.0

Opera Chrome
9.5

7.0

3.0

3.2

9.6

1.0

8.0

3.1

4.0

10.0

2.0

jQuery Browser Support 

Browser Support Grid
IE Previous Current Next
6.0

Firefox
2.0

Safari
3.0

Opera Chrome
9.5

7.0

3.0

3.2

9.6

1.0

8.0

3.1

4.0

10.0

2.0

jQuery 1.3 Browser Support 

The Scaling Problem
✦

The Problem: ✦ jQuery has 6 test suites ✦ Run in 11 browsers ✦ (Not even including multiple platforms!) All need to be run for every commit, patch, and plugin. JavaScript testing doesn’t scale well.

✦

✦ 

Distributed Testing
✦ ✦ ✦

Hub server Clients connect and help run tests A simple JavaScript client that can be run in all browsers ✦ Including mobile browsers!

✦ TestSwarm 

FF 3.5 FF 3.5 FF 3.5 FF 3 FF 3

IE 6 Op 9

IE 6 IE 6

IE 7 TestSwarm IE 7

Test Suite

Test Suite

Test Suite 

Manual Testing
✦

Push tests to users who follow pre-defined steps Answer ‘Yes’/’No’ questions which are pushed back to the server. An effective way to distribute manual test load to dozens of clients.

✦

✦ 

TestSwarm.com
✦ ✦ ✦

Incentives for top testers (t-shirts, books) Will be opening for alpha testing very soon Help your favorite JavaScript library become better tested! http://testswarm.com

✦ 

Questions
✦

Contact: ✦ John Resig ✦ http://ejohn.org/ ✦ http://twitter.com/jeresig