748817 - Modify tcheckerboard and tcheck2 to provide a cumulative checkerboard score instead of an average

Status: RESOLVED DUPLICATE of bug 746215

(Firefox for Android Graveyard :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

:wlach pointed out that computing the average checkerboarded area per frame may not be the best way to measure checkerboarding. If you have run 1 which has 10 frames, one of which is checkerboard, that will give a different result from run 2 which has 11 frames with one checkerboard. Really both of these have the same amount of checkerboard, so the score should be the same but with our current metric we'd get 0.90 for the first one and 0.91 for the second one. Eideticker just takes the sum of all the checkerboard across all the frames which guards against this issue. Should we change tcheckerboard and tcheck2 to also do this?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

interesting.  Would this be a change inside of testCheckerboard and testCheck2?  

Currently Talos runs each test 5 times and then take an average of the numbers produced.  I want to make sure we are talking about the same thing here.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Yes, this would change the code in FennecNativeDriver's stopCheckerboardRecording method to sum the checkerboard percentages instead of averaging them.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

During the sync-up today we decided to do this, so I'm going to dupe it to bug 746215 and do it all as one patch.