Benchmarking in jRuby for non-Java-ists

Eric Lubow
CTO, SimpleReach
http://eric.lubow.org
@elubow

Funny Image

(Pause for laughter)

SimpleReach is Hiring
Isnt everyone?
elubow@simplereach.com

SimpleReach

Innovation in SMB advertising

10k potential impressions for $10
Targeted ads that dont look like ads
Mongo, Redis, Hadoop
Rails 3, Sinatra, Ruby, jRuby
Large Datasets / Real Time Bidding/Processing
Bare metal and cloud integration

Why Benchmark (jRuby)?

Know my application in MRI, but not jRuby
Need to find problem areas in jRuby app (50ms)
Extensions in Java vs. C Extensions

MRI vs. jRuby

Comparisons
MRI 1.8.7-p302
Written in C
Pros
C extensions

Cons
No Java integration
No Garbage collection
Static Compilation

jRuby 1.5.3
Java (Java Virtual Machine)
Runs Java code
Pros
Optimizations
Hotspotting
Forced Compile vs. JIT
Garbage Collection

Benefits of years of Java dev

Cons
Warm Up time

Hotspots
Dynamic profiling is the path to excellent
performance. HotSpot has the benefit of profile
data from the running application to inform
itself. It also has the ability to de-optimize code.
When Hotspot does an optimization it puts a
cheap guard in front of the optimization to make
sure the rationale for the optimization still holds
true. If the guard ever fails then it de-optimizes
back to a slow path. Thomas Enebo (EngineYard)

Hotspots

JSON Comparisons

JSON Pure
MRI

jRuby

Rehearsal -----------------------------------------parse: 0.255000 0.000000 0.255000 ( 0.255000)

--------------------------------- total: 0.255000sec

Rehearsal -----------------------------------------parse: 0.001000 0.000000 0.001000 ( 0.001000)

--------------------------------- total: 0.001000sec

user system total

real
parse: 0.017000 0.000000 0.017000 ( 0.017000)

user system total

real
parse: 0.001000 0.000000 0.001000 ( 0.000000)
Rehearsal ------------------------------------------encode: 0.001000 0.000000 0.001000 ( 0.001000)
---------------------------------- total: 0.001000sec

Rehearsal ------------------------------------------encode: 0.036000 0.000000 0.036000 ( 0.035000)

---------------------------------- total: 0.036000sec
user system total
real
encode: 0.006000 0.000000 0.006000 ( 0.006000)

user system total

real
encode: 0.000000 0.000000 0.000000 ( 0.000000)

MRI Ruby-1.8.7-p302 w/Yajl

Code

Results

json = File.new('bid1.json', 'r')

parser = Yajl::Parser.new
parser.on_parse_complete = lambda {|obj|}
100_000.times { parser.parse(json); }

elubow@beacon json$ ruby parse.rb

Rehearsal -----------------------------------------parse: 0.240000 0.120000 0.360000 ( 0.588048)
--------------------------------- total: 0.360000sec

Benchmark.bmbm do |x|
x.report("parse:") { 100_000.times { parser.parse(json); } }
end

user system total

real
parse: 0.230000 0.110000 0.340000 ( 0.466025)
Rehearsal ------------------------------------------encode: 0.230000 0.120000 0.350000 ( 0.686670)
---------------------------------- total: 0.350000sec

parsed = parser.parse(json)
100_000.times { parsed.to_json }
Benchmark.bmbm do |x|
x.report("encode:") { 100_000.times { parsed.to_json } }
end

user system total

real
encode: 0.240000 0.100000 0.340000 ( 0.486075)

MRI Ruby-1.8.7-p302 w/Yajl

Code

Results

require 'yajl'
parser = Yajl::Parser.new
parser.on_parse_complete = lambda {|obj|}
100_000.times { parser.parse(json); }

elubow@beacon json$ ruby parse.rb

Rehearsal -----------------------------------------parse: 0.000000 0.000000 0.000000 ( 0.000159)
--------------------------------- total: 0.000000sec

Benchmark.bm do |x|
x.report("parse:") { parser.parse(json) }
end

parsed = parser.parse(json)
100_000.times { parsed.to_json }
Benchmark.bm do |x|
x.report("encode:") { parsed.to_json }
end

user system total

real
parse: 0.000000 0.000000 0.000000 ( 0.000093)
Rehearsal -----------------------------------------encode: 0.000000 0.000000 0.000000 ( 0.000088)
--------------------------------- total: 0.000000sec
user system total
real
encode: 0.000000 0.000000 0.000000 ( 0.000063)

jRuby-1.5.3 w/ JSON-jRuby
Code

Results

json = File.read('bid1.json')
100_000.times { JSON.parse(json) }

elubow@beacon json$ jruby --server -JDjruby.compile.fastops=true -S parse.rb

Rehearsal -----------------------------------------parse: 12.433000 0.000000 12.433000 ( 12.433000)
-------------------------------- total: 12.433000sec

Benchmark.bmbm do |x|
x.report("parse:") { 100_000.times {
JSON.parse(json) } }
end
parsed = JSON.parse(json)
100_000.times { parsed.to_json }
Benchmark.bmbm do |x|
x.report("encode:") { 100_000.times {
parsed.to_json } }
end

user system total

real
parse: 11.735000 0.000000 11.735000 ( 11.735000)
Rehearsal ------------------------------------------encode: 2.840000 0.000000 2.840000 ( 2.840000)
---------------------------------- total: 2.840000sec
user system total
real
encode: 2.868000 0.000000 2.868000 ( 2.868000)

jRuby-1.5.3 w/ JSON-jRuby
Code

Results

json = File.read('bid1.json')
100_000.times { JSON.parse(json) }

elubow@beacon json$ jruby --server -JDjruby.compile.fastops=true -S parse.rb

Rehearsal -----------------------------------------parse: 0.001000 0.000000 0.001000 ( 0.000000)
--------------------------------- total: 0.001000sec

Benchmark.bmbm do |x|
x.report("parse:") { JSON.parse(json) }
end
parsed = JSON.parse(json)
100_000.times { parsed.to_json }
Benchmark.bmbm do |x|
x.report("encode:") { parsed.to_json }
end

user system total

real
parse: 0.000000 0.000000 0.000000 ( 0.000000)
Rehearsal ------------------------------------------encode: 0.001000 0.000000 0.001000 ( 0.001000)
---------------------------------- total: 0.001000sec
user system total
real
encode: 0.000000 0.000000 0.000000 ( 0.000000)

JSON Comparisons
Yajl
C extension
Not a drop in replacement

JSON-jRuby
Java extension
Drop in replacement

elubow@beacon json$ ruby parse.rb

Rehearsal -----------------------------------------parse: 0.240000 0.120000 0.360000 ( 0.588048)
--------------------------------- total: 0.360000sec

elubow@beacon json$ jruby --server -JDjruby.compile.fastops=true -S parse.rb

Rehearsal -----------------------------------------parse: 12.433000 0.000000 12.433000 ( 12.433000)
-------------------------------- total: 12.433000sec

user system total

real
parse: 0.230000 0.110000 0.340000 ( 0.466025)
Rehearsal ------------------------------------------encode: 0.230000 0.120000 0.350000 ( 0.686670)
---------------------------------- total: 0.350000sec
user system total
real
encode: 0.240000 0.100000 0.340000 ( 0.486075)

user system total

real
parse: 11.735000 0.000000 11.735000 ( 11.735000)
Rehearsal ------------------------------------------encode: 2.840000 0.000000 2.840000 ( 2.840000)
---------------------------------- total: 2.840000sec
user system total
real
encode: 2.868000 0.000000 2.868000 ( 2.868000)

JSON Comparisons (100k)

Yajl
elubow@beacon json$ ruby parse.rb
Rehearsal -----------------------------------------parse: 0.240000 0.120000 0.360000 ( 0.588048)
--------------------------------- total: 0.360000sec
user system total
real
parse: 0.230000 0.110000 0.340000 ( 0.466025)
Rehearsal ------------------------------------------encode: 0.230000 0.120000 0.350000 ( 0.686670)
---------------------------------- total: 0.350000sec
user system total
real
encode: 0.240000 0.100000 0.340000 ( 0.486075)

JSON-jRuby
elubow@beacon json$ jruby --server -JDjruby.compile.fastops=true -S parse.rb
Rehearsal -----------------------------------------parse: 12.433000 0.000000 12.433000 ( 12.433000)
-------------------------------- total: 12.433000sec
user system total
real
parse: 11.735000 0.000000 11.735000 ( 11.735000)
Rehearsal ------------------------------------------encode: 2.840000 0.000000 2.840000 ( 2.840000)
---------------------------------- total: 2.840000sec
user system total
real
encode: 2.868000 0.000000 2.868000 ( 2.868000)

JSON Comparisons (1x)

Yajl
elubow@beacon json$ ruby parse.rb
Rehearsal -----------------------------------------parse: 0.000000 0.000000 0.000000 ( 0.000159)
--------------------------------- total: 0.000000sec
user system total
real
parse: 0.000000 0.000000 0.000000 ( 0.000093)
Rehearsal -----------------------------------------encode: 0.000000 0.000000 0.000000
( 0.000088)
--------------------------------- total: 0.000000sec
user system total
real
encode: 0.000000 0.000000 0.000000 ( 0.000063)

JSON-jRuby
elubow@beacon json$ jruby --server -JDjruby.compile.fastops=true -S parse.rb
Rehearsal -----------------------------------------parse: 0.001000 0.000000 0.001000 ( 0.000000)
--------------------------------- total: 0.001000sec
user system total
real
parse: 0.000000 0.000000 0.000000 ( 0.000000)
Rehearsal ------------------------------------------encode: 0.001000 0.000000 0.001000 (
0.001000)
---------------------------------- total: 0.001000sec
user system total
real
encode: 0.000000 0.000000 0.000000 (
0.000000)

Mongo

Mongo

Mongo + BSON
Code

Results

require 'mongo'
require 'benchmark'

@db = Mongo::Connection.new('localhost',
27017).db(simplereach_website_development)
100_000.times { @db[ads].find({}).to_a }
Benchmark.bm do |x|
x.report(find:") { @db[ads'].find({}).to_a }
end

Ruby 1.8.7 + bson:

270ms
Ruby 1.8.7 + bson_ext: 13ms
JRuby 1.6dev + bson:
20ms
JRuby 1.6dev + bson_ext: 20ms

Conclusions
Still have lots of testing to do
Discovered JSON conversions werent a large
problem
Discovered BSON conversions are
C extensions
FFI (Foreign Function Interface)

jRuby gets better with every release

jRuby gets better with every Java release

References
http://www.engineyard.com/blog/2009/j-isfor-jvm-why-the-j-in-jruby/
http://eric.lubow.org/2010/ruby/jruby/jsonbenchmarks-in-jruby/
http://www.infoq.com/presentations/enebojruby

Questions?
Eric Lubow
http://eric.lubow.org
eric@lubow.org