777357 - Apparent regression in CNN benchmark between 2012-07-22 and 2012-07-23

Status: RESOLVED DUPLICATE of bug 785069

(Firefox for Android Graveyard :: Toolbar, defect, P5)

Description

[William Lachance (:wlach)]

http://wrla.ch/eideticker/dashboard/#/cnn/checkerboard seems to indicate a regression between 2012-07-22 and 2012-07-23. I will run a quick manual test to verify this.

Comment 1

[William Lachance (:wlach)]

Yeah, there definitely seems to be a regression here:

http://people.mozilla.com/~wlachance/cnn-benchmark/metric.html?data=metric-test-1343419943.79.json#/checkerboard

The build on the 24th corresponds to: 100108:3613cbdc3481
The build on the 23rd corresponds to: 100084:82b6c5885345

Looking at the changelog, nothing is particularly jumping out at me. I guess we'll have to try profiling against a buncha tinderbox builds.

Comment 2

[William Lachance (:wlach)]

Attachment: Results of running Eideticker against CNN tests on various tinderbox builds

Here's some results from running internal telemetry against CNN.com for the regression range I noted. Not seeing where the issue might be.

Comment 3

[William Lachance (:wlach)]

I'm pretty stumped about where to take this next. If you look at the dashboard, the overall trend does seem to be "up", but it's not a strong enough difference to easily pinpoint the difference. I wonder if we need to try a harder test here?

Comment 4

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/9b6f5ce1065a

Comment 5

[Ed Morley [:emorley]]

Typoed bug number in 9b6f5ce1065a, seems to be bug 777537.

Comment 6

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Somewhere between : 
https://tbpl.mozilla.org/?rev=ef20925bc2a5
https://tbpl.mozilla.org/?rev=a26e751bfb54
https://tbpl.mozilla.org/?rev=d78729026fb9

Comment 7

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Oh oops.  need to go back a bit further.  

My guess is : http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-22-03-05-55-mozilla-central-android/
will not show the issue.

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-23-03-06-06-mozilla-central-android/
probably will show the issue.

https://tbpl.mozilla.org/?rev=d78729026fb9&jobname=Android%20Debug%20mozilla-central%20build

Is there an easy way to test builds : 
using revision: mozilla-central/57499abe6479
using revision: mozilla-central/19fe7ecd0af8

My guess is something in : 
https://tbpl.mozilla.org/?rev=19fe7ecd0af8&jobname=Android%20Debug%20mozilla-central%20build

But I am not sure.  I can't set up Eideticker on this machine because I don't have HDMI in.  Pinging :wlach for further possible investigation?

Comment 8

[William Lachance (:wlach)]

(In reply to Naoki Hirata :nhirata from comment #7)
> Oh oops.  need to go back a bit further.  
> 
> My guess is :
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-22-03-
> 05-55-mozilla-central-android/
> will not show the issue.
> 
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-23-03-
> 06-06-mozilla-central-android/
> probably will show the issue.
> 
> https://tbpl.mozilla.org/?rev=d78729026fb9&jobname=Android%20Debug%20mozilla-
> central%20build
> 
> Is there an easy way to test builds : 
> using revision: mozilla-central/57499abe6479
> using revision: mozilla-central/19fe7ecd0af8
> 
> My guess is something in : 
> https://tbpl.mozilla.org/?rev=19fe7ecd0af8&jobname=Android%20Debug%20mozilla-
> central%20build
> 
> But I am not sure.  I can't set up Eideticker on this machine because I
> don't have HDMI in.  Pinging :wlach for further possible investigation?

This is almost certainly a duplicate of bug 785069. It looks like the regression is much more obvious with the robocop benchmark in that bug, so that's probably what I'd use to try and reproduce it.

Comment 9

[Kevin Brosnan [Ex-Mozilla]]

There is still a need for a regression range from one (or both) of the two bugs.

Comment 10

[Aaron Train [:aaronmt]]

Is this bug even still an issue? I don't think QA has the Eideticker setup or resources to look into a regression range here. William, can you take a stab at doing so if it deems necessary still?

Comment 11

[William Lachance (:wlach)]

(In reply to Aaron Train [:aaronmt] from comment #10)
> Is this bug even still an issue? I don't think QA has the Eideticker setup
> or resources to look into a regression range here. William, can you take a
> stab at doing so if it deems necessary still?

As I mention above, this is almost certainly a duplicate of bug 785069, which both has a more obvious regression and can be reproduced without eideticker. It looks like the ball is in David Zbarsky's court there, I'll take the liberty of doing a ping check there.

If we find the cause of the regression and a fix in that bug, I'd be happy to verify the improvement on eideticker using my setup.

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

David Zbarsky was an intern who is back in school.  I am not sure if we will get a response from him about this bug.

Comment 13

[William Lachance (:wlach)]

(In reply to Naoki Hirata :nhirata from comment #12)
> David Zbarsky was an intern who is back in school.  I am not sure if we will
> get a response from him about this bug.

Regardless, it looks like bug 785069 is the place to look for regressions. Let's see what happens there, it looks like :kbrosnan has been doing some good work tracking down the offending changeset.

Comment 14

[Kevin Brosnan [Ex-Mozilla]]

This slipped down my list of things to look at. This test does not seem to be running currently http://graphs.mozilla.org/graph.html#tests=[[204,63,20]]&sel=none&displayrange=365&datatype=running

Pulling regression window for the time being.

Comment 15

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Was that for tcheck2? I'm not sure what's wrong with the link in Comment 14, but this one shows current results for tcheck2: http://graphs.mozilla.org/graph.html#tests=[[201,63,20]]&sel=none&displayrange=365&datatype=running

Comment 16

[Brad Lassey [:blassey] (use needinfo?)]

filter on [mass-p5]

Comment 17

[Kartikaya Gupta (email:[e-mail])]

Per comment 11, comment 13.