1026798 - 1.74e+03 Robopan Android 4.0.4 regression on fx-team (v33) June 12th from push 578bf8d6363c

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

here is a graph showing the regression:
http://graphs.mozilla.org/graph.html#tests=[[174,64,29]]&sel=1402512218000,1402685018000&displayrange=7&datatype=running

I did some retriggers to make sure there is no spiders in the tegra farm causing odd numbers:
https://tbpl.mozilla.org/?tree=Fx-Team&fromchange=98d801fe3f5d&tochange=ca66ccfcd177&jobname=Android%204.0%20Panda%20fx-team%20talos%20remote-trobopan

Here is the pushlog:
http://hg.mozilla.org/integration/fx-team/pushloghtml?fromchange=431dd53007b7&tochange=578bf8d6363c

I see in bug 1019318 there is talk about robopan numbers- maybe something didn't get checked in?  I suspect this is easier to fix or explain why the test as it is has become invalid in some way.

btw, I believe this hits the award for the largest number posted from the alert regression email tool, an e+03 is pretty slick and worth some bragging points!

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

oh, upon further looking, this appears to be fixed a couple of days later- the graph show that- does that make sense to you Margaret?

Comment 2

[:Margaret Leibovic]

(In reply to Joel Maher (:jmaher) from comment #1)
> oh, upon further looking, this appears to be fixed a couple of days later-
> the graph show that- does that make sense to you Margaret?

Yup, I backed out the original patch in bug 1019735 when I got the email about this regression, then I landed a fixed version.

Comment 3

[Avi Halachmi (:avih)]

(In reply to :Margaret Leibovic from comment #2)
> Yup, I backed out the original patch in bug 1019735 when I got the email
> about this regression, then I landed a fixed version.

\o/

However, it should still be noted that the robopan (or was it robocheck?) numbers are all over the place even without the regression in this bug. Their "normal" values are kinda uniform between 25000 and 100000. Yes, a range of x4 factor.

I talked to kats recently about revisiting this test (he wrote it, and theoretically it does impressive measuring), so maybe it's time to nudge him a bit more? ;)

Comment 4

[Kartikaya Gupta (email:[e-mail])]

IIRC we were only discussion robocheck before. I'm not entirely convinced that robopan uses a good algorithm [1] to come up with its number.

[1] http://mxr.mozilla.org/mozilla-central/source/build/mobile/robocop/FennecNativeDriver.java?rev=523e67cf4b9e#129