746215 - Invert tcheckerboard (and tcheckerboard2) output and make it cumulative instead of average

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

Right now the output means 1.0 is no checkerboarding and 0.0 is all checkerboarding. We should flip this so that when viewing the graphs on the graph server the y-axis scale will get more detailed as we drive down checkerboarding and we can better see smaller regressions.

Also it just seems more reasonable for a test called "checkerboarding" to measure checkerboarding as opposed to not-checkerboarding.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

good call.  We should be able to change the test cases themselves and ignore the rest of the automation and buildbot scripts for this.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

We might also want to change the scale from [0,1] to [0,100] instead. Right now when I click on a data point on the graph server it always reports the score as "0", which I assume is because it doesn't like fractional values. Scaling the values up would avoid this.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Attachment: Patch

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 617007 [details] [diff] [review]
Patch

Review of attachment 617007 [details] [diff] [review]:
-----------------------------------------------------------------

this looks fine!  I assume this will work for both tcheckerboard tests?

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Yeah I pushed it to try at https://tbpl.mozilla.org/?tree=Try&rev=2050322f4a9a and it looked fine.

Comment 8

[Kartikaya Gupta (email:[e-mail])]

Attachment: Patch (v2)

New patch that also reports the sum instead of average. The "inversion" still applies, so that lower numbers are better.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 618351 [details] [diff] [review]
Patch (v2)

Review of attachment 618351 [details] [diff] [review]:
-----------------------------------------------------------------

looks good, thanks for pushing these patches through.

Comment 10

[Kartikaya Gupta (email:[e-mail])]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c978a9f2d1d8

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/c978a9f2d1d8