691414 - Regression in CSS3 MazeSolver test from FF6.x to FF7

Status: RESOLVED WORKSFORME

(Core :: DOM: Core & HTML, defect)

Description

[Randell Jesup [:jesup] (needinfo me)]

According to TomsHardware's browser shootout, MazeSolver regressed by 10 seconds to ~85 seconds.

There are two known bugs on this test, though those aren't the source of the regression (putting them in as blockers, though they're really more see-alsos).

Comment 1

[Boris Zbarsky [:bzbarsky]]

Need regression range, if possible.

Comment 2

[AndreiD[QA]]

I tried to find a regression range but cannot point out exactly when the so called 'performance issue' between 6 and 7 appears. 
This is because on the same Firefox build and the same maze (20x20) I got different timing values (i.e. 9.2, 10.3, 11 even 13 seconds). This applies for all releases and channels starting Firefox 6.
Although, you can find the results I obtained when trying to find the regression range between 6 and 7, using harthur's tool mozregression and other builds.
Build    Time (Maze 20x20)
Fx6         9.2 s
2011-08-13  13 s
2011-08-15  11 s
2011-08-18  12 s
2011-08-25  33 s
2011-09-27  13 s
Fx7         9.2 s
Fx8         11 s
Fx9         11 s
Fx10        10 s

Finally, I'm not sure that there was a change in behavior regarding the performance on this website between Fx6 and Fx10.

Comment 3

[(mostly gone) XtC4UaLL [:xtc4uall]]

(In reply to AndreiD[QA] from comment #2)
Confirming this Oberservations.
FWIW, Fx4 -> ~8,1s vs. FxNightly -> ~7,6s (3 Runs each).

-> resolving.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Note: Tom's hardware clearly didn't use the 20x20 maze, probably they used 30x30.   On my Xeon 20x20 (nightly, local build) is 5.0-5.9s; 30x30 is 70-72s... (Tom reported 85s).  Chrome is 4.0s on 30x30 (~18x as fast), but repeating the test gives 1.2-1.3s for 30x30(!!)  20x20 is 2.8s, but re-runs are 0.7s.(!)

Freakily, Re-running the maze doesn't take the same path - it takes a much faster path....  and the 30x30 mazes for chrome and Mozilla are different!  the 20x20's are the same, but solve too fast to be good tests.

On 20x20, Mozilla takes 5 to 5.9s on repeats - but 14s for initial runs.  20x20 may be much less repeatable and subject to distortion. 

And of course hitting "reload" next to the size changes mazes.  If you look carefully, you can get the same maze on both.  The test source should be tweaked to show and allow setting of the seed.  Different mazes take dramatically different times to solve, in addition to the change in behavior on re-run of the same maze.

All these could have affected Tom's hardware, so I agree there's no reason to go any deeper on this result.  It's still ridiculously slow. :-(