We should have an eideticker test that measures the amount of time to fully render a site like cnn.com in the browser when an instance of the browser is not currently running. This represents the user story of someone clicking a link in another android application (e.g. twitter or gmail).
We should try to generate comparative numbers for the stock browser and chrome for this under android 4.2, to make sure we're competitive. Most existing eideticker tests are hard to do cross browser, but this is one we pretty safely can do, so we should probably do it since there seems to be a lot of interest in getting this kind of data.
Most of the work for this is already done, though a few minor things may have broken since I haven't run this test in a while. The todo list looks something like this:
1. Add support to the dashboard for using a "baseline" number for stock and chrome, so we're not constantly generating new numbers for them every day (which is wasteful)
2. Add a startup variant of cnn.com.
3. Fix any random bugs preventing this from working.
4. Add this test to the dashboard!
See bug 758338, which discusses comparing network perf of firefox vs. chrome on mobile.
Pushed first round of work on this:
(more to come)
Dashboard results here:
The short summary is that we're quite bad in comparison to the competition. Like 3 or 4 times slower. I'll file a bug about this.
I need to at least update the dashboard to show baseline results appropriately, as well as add this test to the nightly schedule. Probably also want to do a profiling run of this on fennec, so we can see where any bottlenecks are.
Filed bug 857359 about the bad performance.
Added this to the default run for the samsung gn and lg g2x and dashboard now displays baseline results. Calling this done, if anything else is desired please file a follow up bug. :)