Apparent regression in CNN benchmark between 2012-07-22 and 2012-07-23

RESOLVED DUPLICATE of bug 785069

Status

()

Firefox for Android
Toolbar
P5
normal
RESOLVED DUPLICATE of bug 785069
6 years ago
2 years ago

People

(Reporter: wlach, Assigned: snorp)

Tracking

({perf, regression})

unspecified
Firefox 17
ARM
Android
perf, regression
Points:
---

Firefox Tracking Flags

(fennec+)

Details

Attachments

(1 attachment)

http://wrla.ch/eideticker/dashboard/#/cnn/checkerboard seems to indicate a regression between 2012-07-22 and 2012-07-23. I will run a quick manual test to verify this.

Updated

6 years ago
tracking-fennec: --- → ?
Yeah, there definitely seems to be a regression here:

http://people.mozilla.com/~wlachance/cnn-benchmark/metric.html?data=metric-test-1343419943.79.json#/checkerboard

The build on the 24th corresponds to: 100108:3613cbdc3481
The build on the 23rd corresponds to: 100084:82b6c5885345

Looking at the changelog, nothing is particularly jumping out at me. I guess we'll have to try profiling against a buncha tinderbox builds.
Created attachment 647524 [details]
Results of running Eideticker against CNN tests on various tinderbox builds

Here's some results from running internal telemetry against CNN.com for the regression range I noted. Not seeing where the issue might be.
Assignee: nobody → wlachance
I'm pretty stumped about where to take this next. If you look at the dashboard, the overall trend does seem to be "up", but it's not a strong enough difference to easily pinpoint the difference. I wonder if we need to try a harder test here?
Assignee: wlachance → snorp
tracking-fennec: ? → 17+
Keywords: regression, regressionwindow-wanted

Comment 4

6 years ago
https://hg.mozilla.org/mozilla-central/rev/9b6f5ce1065a
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 17

Comment 5

6 years ago
Typoed bug number in 9b6f5ce1065a, seems to be bug 777537.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

6 years ago
Keywords: perf
Oh oops.  need to go back a bit further.  

My guess is : http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-22-03-05-55-mozilla-central-android/
will not show the issue.

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-23-03-06-06-mozilla-central-android/
probably will show the issue.

https://tbpl.mozilla.org/?rev=d78729026fb9&jobname=Android%20Debug%20mozilla-central%20build

Is there an easy way to test builds : 
using revision: mozilla-central/57499abe6479
using revision: mozilla-central/19fe7ecd0af8

My guess is something in : 
https://tbpl.mozilla.org/?rev=19fe7ecd0af8&jobname=Android%20Debug%20mozilla-central%20build

But I am not sure.  I can't set up Eideticker on this machine because I don't have HDMI in.  Pinging :wlach for further possible investigation?
(In reply to Naoki Hirata :nhirata from comment #7)
> Oh oops.  need to go back a bit further.  
> 
> My guess is :
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-22-03-
> 05-55-mozilla-central-android/
> will not show the issue.
> 
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/07/2012-07-23-03-
> 06-06-mozilla-central-android/
> probably will show the issue.
> 
> https://tbpl.mozilla.org/?rev=d78729026fb9&jobname=Android%20Debug%20mozilla-
> central%20build
> 
> Is there an easy way to test builds : 
> using revision: mozilla-central/57499abe6479
> using revision: mozilla-central/19fe7ecd0af8
> 
> My guess is something in : 
> https://tbpl.mozilla.org/?rev=19fe7ecd0af8&jobname=Android%20Debug%20mozilla-
> central%20build
> 
> But I am not sure.  I can't set up Eideticker on this machine because I
> don't have HDMI in.  Pinging :wlach for further possible investigation?

This is almost certainly a duplicate of bug 785069. It looks like the regression is much more obvious with the robocop benchmark in that bug, so that's probably what I'd use to try and reproduce it.

Updated

5 years ago
Depends on: 785069

Updated

5 years ago
Keywords: regressionwindow-wanted
There is still a need for a regression range from one (or both) of the two bugs.
Keywords: regressionwindow-wanted
Is this bug even still an issue? I don't think QA has the Eideticker setup or resources to look into a regression range here. William, can you take a stab at doing so if it deems necessary still?
(In reply to Aaron Train [:aaronmt] from comment #10)
> Is this bug even still an issue? I don't think QA has the Eideticker setup
> or resources to look into a regression range here. William, can you take a
> stab at doing so if it deems necessary still?

As I mention above, this is almost certainly a duplicate of bug 785069, which both has a more obvious regression and can be reproduced without eideticker. It looks like the ball is in David Zbarsky's court there, I'll take the liberty of doing a ping check there.

If we find the cause of the regression and a fix in that bug, I'd be happy to verify the improvement on eideticker using my setup.
David Zbarsky was an intern who is back in school.  I am not sure if we will get a response from him about this bug.
Flags: needinfo?(wlachance)
(In reply to Naoki Hirata :nhirata from comment #12)
> David Zbarsky was an intern who is back in school.  I am not sure if we will
> get a response from him about this bug.

Regardless, it looks like bug 785069 is the place to look for regressions. Let's see what happens there, it looks like :kbrosnan has been doing some good work tracking down the offending changeset.
Flags: needinfo?(wlachance)
tracking-fennec: 17+ → +
This slipped down my list of things to look at. This test does not seem to be running currently http://graphs.mozilla.org/graph.html#tests=[[204,63,20]]&sel=none&displayrange=365&datatype=running

Pulling regression window for the time being.
Keywords: regressionwindow-wanted
Was that for tcheck2? I'm not sure what's wrong with the link in Comment 14, but this one shows current results for tcheck2: http://graphs.mozilla.org/graph.html#tests=[[201,63,20]]&sel=none&displayrange=365&datatype=running
filter on [mass-p5]
Priority: -- → P5
Per comment 11, comment 13.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 785069
You need to log in before you can comment on or make changes to this bug.