866113 - Talos Regression tcheck2 26% on Android 2.2, Apr 26

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect)

Description

[Geoff Brown [:gbrown]]

From dev-tree-management Digest, Vol 52, Issue 171

Message: 3
Date: Fri, 26 Apr 2013 00:21:40 -0000
From: [e-mail]
To: [e-mail]
Subject: <Regression> Mozilla-Inbound - Robocop Checkerboarding Real
        User        Benchmark - Android 2.2 (Native) - 26.3%
Message-ID:
        <[e-mail]>
Content-Type: text/plain; charset="us-ascii"

Regression: Mozilla-Inbound - Robocop Checkerboarding Real User Benchmark - Android 2.2 (Native) - 26.3% increase
-----------------------------------------------------------------------------------------------------------------
    Previous: avg 2.804 stddev 0.162 of 12 runs up to revision f50572bf5e49
    New     : avg 3.542 stddev 0.213 of 12 runs since revision 7f68735fc8da
    Change  : +0.738 (26.3% / z=4.554)
    Graph   : http://mzl.la/13ymzMK

Changeset range: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=f50572bf5e49&tochange=7f68735fc8da

Changesets:
  * http://hg.mozilla.org/integration/mozilla-inbound/rev/9defb9661f84
    : Adam Roach [:abr] <[e-mail]> - Bug 864982 - Remove legacy err_msg/buginf/notice_msg logging r=jesup
    : http://bugzilla.mozilla.org/show_bug.cgi?id=864982

  * http://hg.mozilla.org/integration/mozilla-inbound/rev/7f68735fc8da
    : Joel Maher <[e-mail]> - Bug 865780 - deploy a new talos.zip to capture the fennec preference, datazilla output, and mozlog fix. r=armenzg
    : http://bugzilla.mozilla.org/show_bug.cgi?id=865780

Bugs:
  * http://bugzilla.mozilla.org/show_bug.cgi?id=865780 - deploy a new talos.zip to capture the fennec preference, datazilla output, and mozlog fix
  * http://bugzilla.mozilla.org/show_bug.cgi?id=864982 - Remove legacy err_msg/buginf/notice_msg logging in SIPCC

Comment 1

[Geoff Brown [:gbrown]]

The Talos update rev: 

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=7f68735fc8da

has check2=3.25

The previous rev:

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=9defb9661f84

has check2=2.11

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

what is interesting here is I added 3 things to talos:
1) datzilla debug output string (only affects post runtime)
2) cleanup of noisy/debug flags, could affect runtime of the python harness (i.e. not really the test), and on mobile this shouldn't affect anything
3) the preference to turn off the dynamic toolbar: bug 865742

Of all of these, I suspect 3 is the culprit, although there is a chance the other changeset caused a regression as well.

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

CC'ing Chris in case he has any ideas.

I am not sure how much we worry about a 2.11 to 3.25 change on this metric.

Comment 4

[Chris Lord [:cwiiis]]

So I guess it would affect results if not enough time was left for the preference change to propagate? This would also explain the intermittent oranges for robocop, I guess.

I think this is worth investigating, but in objective terms, a 1.0 difference in this test means very little in perceptible terms.

Comment 5

[Chris Lord [:cwiiis]]

I would suggest comparing a push that has the toolbar default to off + remove the pref change in talos, with a plain push.

Comment 6

[Kevin Brosnan [Ex-Mozilla]]

23+ ship has sailed. Need to re-triage this.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

This bug is past the statute of limitations for tcheck2 regressions (we have a new baseline that we've been living with for almost a year, nobody cares about this bug anymore, etc.). At any rate it wasn't caused by a code change, it was a change in the talos.zip harness so any regression should not affect real-user experience.