982132 - B2G startup tests still sometimes generating wrong results

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect, P1)

Description

[William Lachance (:wlach)]

Even with the new better startup analysis based on sobel-convolved frames, we're still getting some outliers for the startup tests. e.g. on the 9th here:

http://eideticker.wrla.ch/b2g/#/inari/b2g-contacts-startup/timetostableframe

Even before this patch, we compared each frame against an average of the 5 previous to make sure we weren't measuring two outliers against each other. Kyle suggested we could do something similar with the frame we're measuring. So measure the 5 previous against the 5 next:

"I am only concerned with the amount of noise in your results, and I wonder if this is the noisiest sample.  You can reduce the noise further simply by comparing the past mean (of five values) with a future mean (of five values)."

For the case of the end of capture (where we don't have 5 subsequent frames of data), we should probably just stop 5 frames before the end.

Comment 1

[William Lachance (:wlach)]

This is an easy fix, will try to get to it this week.

Comment 2

[William Lachance (:wlach)]

Attachment: Use subsequent samples to calculate entropy difference

I think this should work a bit better than what's currently in there. Let me know if you want me to explain the logic behind this patch further.

Comment 3

[William Lachance (:wlach)]

The earlier patch had a few logic errors in it, which I fixed after testing. Pushed here:

https://github.com/mozilla/eideticker/commit/3613f1673b11f3579c8da1b4c023b21884baba87

Comment 4

[Wil Clouser [:clouserw]]

A rough glance looks like this really cleaned up the outliers - everywhere, but particularly noticeable on the marketplace.  Thanks, it's looking great.