908823 - Talos regression trobopan 2000% on Android, Aug 22 2013

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

From dev-tree-management Digest, Vol 56, Issue 117

Subject: <Regression> Fx-Team - Robocop Pan Benchmark - Android 2.2
        (Native) -        1.92e+03%

Regression: Fx-Team - Robocop Pan Benchmark - Android 2.2 (Native) - 1.92e+03% increase
---------------------------------------------------------------------------------------
    Previous: avg 63678.342 stddev 6663.786 of 12 runs up to revision 072341c2c66f
    New     : avg 1285005.750 stddev 440599.791 of 12 runs since revision aed7628fff59
    Change  : +1221327.408 (1.92e+03% / z=183.278)
    Graph   : http://mzl.la/12tlszE

Changeset range: http://hg.mozilla.org/integration/fx-team/pushloghtml?fromchange=072341c2c66f&tochange=aed7628fff59

Comment 1

[Mark Finkle (:mfinkle) (use needinfo?)]

2000% !!  I hope a human can actually feel the regression.

Comment 2

[:Margaret Leibovic]

Given what I've been looking into in bug 907720, I don't know that we can actually trust this.

Running testCheck locally, I noticed that the test would be panning and scrolling the about:home content, not the test page. Sometimes this would result in a crash (bug 907720), but sometimes the test would act like it was run successfully. Just now I decided to try running testPan locally to see what happens, and indeed, the same thing happened. So, I think we need to look into making sure the tests are accurate before investigating a possible regression. (A regression in tPan doesn't even really make sense, since the fig merge shouldn't have changed the behavior of web content.)

I can own this, since I'm already looking into these talos tests.

Comment 3

[:Margaret Leibovic]

In bug 907720 I tried landing a fix to make sure we're hiding the HomePager before running the testPan benchmark, but I had to back it out because it was causing testPan to fail too frequently.

This makes me continue to suspect that these numbers aren't measuring what we think they're measuring.

Comment 4

[:Margaret Leibovic]

Cc'ing lucasr, in case he wants to help look into this as part of his fig performance work :)

Comment 5

[Avi Halachmi (:avih)]

http://graphs.mozilla.org/graph.html#tests=[[174,64,20]]&sel=1378296269705.0615,1378837999314,833333.3333333321,11666666.666666666&displayrange=7&datatype=running

The graph doesn't indicate a regression IMO. It indicates a completely unreliable test, or completely inconsistent behavior. I'd guess it's the former.

Comment 6

[Avi Halachmi (:avih)]

However, looking at the bigger picture, (last 30 days), something definitely changes.

http://graphs.mozilla.org/graph.html#tests=[[174,64,20]]&sel=none&displayrange=30&datatype=running

Until Aug 21 it was nicely consistent at around 5000 (if you zoom enough), and then it started getting wild progressively, with a peak on Aug-26 - Sep-1, and then it kept very noise since.

Comment 7

[:Margaret Leibovic]

Looking at data from the last few days, it looks like the patches in bug 913683 did fix this issue. So should we close this bug?

Comment 8

[Geoff Brown [:gbrown]]

There is much less noise since Sept 21 -- looks much better. And the values seem pretty close to pre-Aug 22 values. I would just watch the results for another day or two to be sure.

Comment 9

[:Margaret Leibovic]

(In reply to Geoff Brown [:gbrown] (PTO Sep 26 - summit) from comment #8)
> There is much less noise since Sept 21 -- looks much better. And the values
> seem pretty close to pre-Aug 22 values. I would just watch the results for
> another day or two to be sure.

There was a blip around Sept 25, but things have returned back to normal since then. I'm going to close this bug now, since it seems like the change for bug 913683 fixed this issue.