From dev-tree-management Digest, Vol 56, Issue 117 Subject: <Regression> Fx-Team - Robocop Pan Benchmark - Android 2.2 (Native) - 1.92e+03% Regression: Fx-Team - Robocop Pan Benchmark - Android 2.2 (Native) - 1.92e+03% increase --------------------------------------------------------------------------------------- Previous: avg 63678.342 stddev 6663.786 of 12 runs up to revision 072341c2c66f New : avg 1285005.750 stddev 440599.791 of 12 runs since revision aed7628fff59 Change : +1221327.408 (1.92e+03% / z=183.278) Graph : http://mzl.la/12tlszE Changeset range: http://hg.mozilla.org/integration/fx-team/pushloghtml?fromchange=072341c2c66f&tochange=aed7628fff59
2000% !! I hope a human can actually feel the regression.
Given what I've been looking into in bug 907720, I don't know that we can actually trust this. Running testCheck locally, I noticed that the test would be panning and scrolling the about:home content, not the test page. Sometimes this would result in a crash (bug 907720), but sometimes the test would act like it was run successfully. Just now I decided to try running testPan locally to see what happens, and indeed, the same thing happened. So, I think we need to look into making sure the tests are accurate before investigating a possible regression. (A regression in tPan doesn't even really make sense, since the fig merge shouldn't have changed the behavior of web content.) I can own this, since I'm already looking into these talos tests.
Assignee: nobody → margaret.leibovic
In bug 907720 I tried landing a fix to make sure we're hiding the HomePager before running the testPan benchmark, but I had to back it out because it was causing testPan to fail too frequently. This makes me continue to suspect that these numbers aren't measuring what we think they're measuring.
Cc'ing lucasr, in case he wants to help look into this as part of his fig performance work :)
http://graphs.mozilla.org/graph.html#tests=[[174,64,20]]&sel=1378296269705.0615,1378837999314,833333.3333333321,11666666.666666666&displayrange=7&datatype=running The graph doesn't indicate a regression IMO. It indicates a completely unreliable test, or completely inconsistent behavior. I'd guess it's the former.
However, looking at the bigger picture, (last 30 days), something definitely changes. http://graphs.mozilla.org/graph.html#tests=[[174,64,20]]&sel=none&displayrange=30&datatype=running Until Aug 21 it was nicely consistent at around 5000 (if you zoom enough), and then it started getting wild progressively, with a peak on Aug-26 - Sep-1, and then it kept very noise since.
Looking at data from the last few days, it looks like the patches in bug 913683 did fix this issue. So should we close this bug?
There is much less noise since Sept 21 -- looks much better. And the values seem pretty close to pre-Aug 22 values. I would just watch the results for another day or two to be sure.
(In reply to Geoff Brown [:gbrown] (PTO Sep 26 - summit) from comment #8) > There is much less noise since Sept 21 -- looks much better. And the values > seem pretty close to pre-Aug 22 values. I would just watch the results for > another day or two to be sure. There was a blip around Sept 25, but things have returned back to normal since then. I'm going to close this bug now, since it seems like the change for bug 913683 fixed this issue.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.