Last Comment Bug 793913 - (Robohornet) Robohornet benchmark performance
(Robohornet)
: Robohornet benchmark performance
Status: RESOLVED INCOMPLETE
: meta, perf
Product: Core Graveyard
Classification: Graveyard
Component: Tracking (show other bugs)
: Trunk
: All All
: -- normal with 4 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: chris hofmann
:
Mentors:
http://www.robohornet.org/
Depends on: 793940 793941 793942 626021 793954 793956 793957
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-24 17:16 PDT by timbugzilla
Modified: 2016-07-15 12:13 PDT (History)
43 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description timbugzilla 2012-09-24 17:16:44 PDT
Robohornet is a browser benchmark designed to test performance in web app bottlenecks, especially "DOM, <canvas> API methods, SVG".

Paul Irish from Google's Chrome team is in charge of it. He blogged on it here:

http://paulirish.com/2012/a-browser-benchmark-that-has-your-back-robohornet/

The Github page is here:

https://github.com/robohornet/robohornet

Tom's Hardware ran it on desktop and mobile browsers. Firefox 15 came last on all desktop platforms (win7, win8, ubuntu 12.04.1, OSX 10.8.2). Firefox was not tested on Android.
Comment 1 timbugzilla 2012-09-24 17:19:46 PDT
Tom's Hardware benchmark tests:

http://www.tomshardware.com/reviews/robohornet-web-browser-performance,3303.html
Comment 2 Boris Zbarsky [:bz] (still a bit busy) 2012-09-24 17:39:57 PDT
Can someone put together a list of the tests we seem to do particularly badly on here?
Comment 3 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-09-24 18:05:53 PDT
I wish I knew.  The test seems to go on forever in the SVG resizing benchmark.
Comment 4 timbugzilla 2012-09-24 18:11:34 PDT
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #3)
> I wish I knew.  The test seems to go on forever in the SVG resizing
> benchmark.

Indeed. 

In the mean time, the Tom's Hardware article has test breakdowns. For Win 7:

Chrome 21: http://media.bestofmicro.com/0/D/353821/original/Chrome21Win7.PNG

FF 15: http://media.bestofmicro.com/0/H/353825/original/Firefox15Win7.PNG

"offsetHeight triggers reflow" stands out.
Comment 5 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-09-24 18:25:04 PDT
So, without the SVG benchmark:

http://hg.mozilla.org/mozilla-central/rev/9d285bedbc1f
Benchmark 	Status 	Time 	Baseline 	Weight 	Index
Add Rows to Table	Completed successfully 	54.43ms	49.68ms	6.72%	6.14
Add Columns to Table	Completed successfully 	93.88ms	45.29ms	5.04%	2.43
Descendant Selector	Completed successfully 	64.67ms	87.19ms	6.72%	9.06
2D Canvas toDataURL	Completed successfully 	603.39ms	304.56ms	6.72%	3.39
2D Canvas clearRect	Completed successfully 	10.71ms	16.73ms	6.72%	10.50
innerHTML Table	Completed successfully 	752.05ms	283.53ms	6.72%	2.53
Table scrolling	Completed successfully 	2182.34ms	964.37ms	6.72%	2.97
Resize columns	Completed successfully 	3794.38ms	2739.90ms	6.72%	4.85
SVG resize	Skipped	-	502.73ms	6.72%	-
ES5 Property Accessors	Completed successfully 	231.82ms	89.57ms	3.36%	1.30
Calculate primes	Completed successfully 	148.63ms	96.83ms	3.36%	2.19
Argument instantiation	Completed successfully 	30.18ms	315.13ms	6.72%	70.20
Animated GIFS	Completed successfully 	101.05ms	127.69ms	0.84%	1.06
offsetHeight triggers reflow	Completed successfully 	4537.98ms	723.69ms	10.08%	1.61
DOM Range API	Completed successfully 	109.09ms	103.79ms	3.36%	3.20
Write to localStorage	Completed successfully 	46.87ms	49.84ms	6.72%	7.15
Read from localStorage	Completed successfully 	34.73ms	33.93ms	6.72%	6.57
Raw score	98880.88


Chrome 21.0.1180.89 m

Benchmark	Status	Time	Baseline	Weight	Index
Add Rows to Table	Completed successfully	42.70ms	49.68ms	6.72%	7.82
Add Columns to Table	Completed successfully	57.05ms	45.29ms	5.04%	4.00
Descendant Selector	Completed successfully	55.55ms	87.19ms	6.72%	10.55
2D Canvas toDataURL	Completed successfully	273.30ms	304.56ms	6.72%	7.49
2D Canvas clearRect	Completed successfully	16.43ms	16.73ms	6.72%	6.84
innerHTML Table	Completed successfully	272.09ms	283.53ms	6.72%	7.01
Table scrolling	Completed successfully	1055.66ms	964.37ms	6.72%	6.14
Resize columns	Completed successfully	1584.73ms	2739.90ms	6.72%	11.62
SVG resize	Skipped	-	502.73ms	6.72%	-
ES5 Property Accessors	Completed successfully	22.40ms	89.57ms	3.36%	13.44
Calculate primes	Completed successfully	120.48ms	96.83ms	3.36%	2.70
Argument instantiation	Completed successfully	243.89ms	315.13ms	6.72%	8.69
Animated GIFS	Completed successfully	380.46ms	127.69ms	0.84%	0.28
offsetHeight triggers reflow	Completed successfully	495.35ms	723.69ms	10.08%	14.73
DOM Range API	Completed successfully	113.44ms	103.79ms	3.36%	3.08
Write to localStorage	Completed successfully	38.80ms	49.84ms	6.72%	8.64
Read from localStorage	Completed successfully	20.87ms	33.93ms	6.72%	10.93
Raw score	30692.69

"offsetHeight triggers reflow" and "ES5 Property Accessors" are by far the worst performers on our side, we're 10x slower than Chrome.  We're 3x slower on the toDataURL benchmark and the innerHTML table benchmark.

On the bright side, we're 8x faster than Chrome on the argument instantiation benchmark, 3x faster on animated GIF scrolling, and nominally faster on clearRect and DOM range.
Comment 6 Boris Zbarsky [:bz] (still a bit busy) 2012-09-24 18:32:50 PDT
Add Columns to Table is not great for us either.

OK, I'll file some bugs blocking this.  Chickens coming home to roost with the JS folks on the ES5 accessor stuff.  ;)
Comment 7 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-09-24 21:07:16 PDT
The canvas clearRect test is totally bogus.  It clears a canvas it never even bothered to draw to, so it just tests how fast we can call into C++ and return back to JS.
Comment 8 K. Gadd (:kael) 2012-09-24 23:51:49 PDT
As I mentioned to them on their github issue tracker, their accesors test might be a lie. It very often fails to make any use of the values it reads from the accessors, so an aggressive optimizer could entirely eliminate the loops. V8 might be smart enough to do this (in my experience V8 is *not* 10x faster than SpiderMonkey at ES5 accessors)
Comment 9 Joel Maher ( :jmaher) 2012-09-25 07:39:22 PDT
our svg tests need an developer contact and are probably outdated, this might be a good solution to update our svg tests.
Comment 10 Nicholas Nethercote [:njn] 2012-09-25 16:47:27 PDT
> It very often fails to make any use of the values it reads
> from the accessors, so an aggressive optimizer could entirely eliminate the
> loops.

This is a *classic* microbenchmark pitfall.

Fun fact:  V8's "regexp" test runs many RegExp.exec() calls but never inspects their return values (which is an array).  So if your JS engine can detect that you can avoid creating that return value (which is an array, and so expensive to create) and run the test much faster.
Comment 11 Joel Maher ( :jmaher) 2012-10-17 05:56:04 PDT
would we want to run this in talos?
Comment 12 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-10-17 06:12:36 PDT
Not with the current quality of the benchmark.

Note You need to log in before you can comment on or make changes to this bug.