976121 - Spurious regression reported on contacts startup test

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

The videos show that the startup time is about the same, but the graphs show something different:

http://eideticker.wrla.ch/b2g/#/inari/b2g-contacts-startup/timetostableframe

I think doing a Sobel pass before calculating the entropy should fix the calculation.

Comment 1

[William Lachance (:wlach)]

Attachment: Use sobelized entropy diffs for app startup measurement;r=mchang

Will hopefully be more accurate than the previous approach, by filtering out
all but the edges.

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8381063 [details] [diff] [review]
Use sobelized entropy diffs for app startup measurement;r=mchang

Review of attachment 8381063 [details] [diff] [review]:
-----------------------------------------------------------------

So I think this should do the trick. This actually paves the way for automatic checkerboarding detection as well, where we can calculate the entropy over the entire capture. :P

before:

https://plot.ly/~WilliamLachance/33

after:

https://plot.ly/~WilliamLachance/31

As you can see, the entropy differences (we take a large difference of entropy in the capture to mean something changed) are much spikier if we only compare the sobelized (edge detected) capture.

Comment 3

[William Lachance (:wlach)]

Attachment: Fixed issue where it was impossible to turn off sobelization

Comment 4

[Mason Chang [Inactive] [:mchang]]

Comment on attachment 8381069 [details] [diff] [review]
Fixed issue where it was impossible to turn off sobelization

Review of attachment 8381069 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.

::: src/videocapture/videocapture/entropy.py
@@ +22,5 @@
> +    samples_probability = [float(h) / histogram_length for h in histogram]
> +    entropy = -sum([p * math.log(p, 2) for p in samples_probability if p != 0])
> +
> +    return entropy
> +

Nit: Delete Line

@@ +63,5 @@
> +        if len(prev_samples) > num_samples:
> +            prev_samples = prev_samples[1:]
> +
> +    return entropy_diffs
> +

Nit: delete line

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This caused eideticker-ci to regress due to missing requirements. I have now installed gfortran, libblas-dev, and liblapack-dev to the box.

Comment 6

[William Lachance (:wlach)]

So it looks like we're now saying that we're done too early (e.g. contacts says it's finished when only the app chrome is loaded and not the contacts), presumably because I set the threshold for change too high this time.

I think I need to do something like calculate the threshold dynamically based on the standard deviation of variables in the capture, or something. I'll do up a quick patch to add the values to the metadata so I can measure this.

Comment 7

[William Lachance (:wlach)]

Added a small patch to store sobelized entropies so I can experiment on the live data.

(I can also use this data to generate new metrics for the days on which it was collected, once we figure out what the metric should be...)

https://github.com/mozilla/eideticker/commit/839455d5b87dd0be38aa38ad1e5828759f7de8b4

Comment 8

[William Lachance (:wlach)]

Attachment: Use standard deviation to calculate threshold for stable frame automatically

So the previous value was set too high. Instead of using an arbitrary value, let's calculate it dynamically using the standard deviation... according to this article:

http://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

... if we the amount of noise follows approximately a normal distribution, we can use 3x the standard deviation to distinguish large changes. I just ran this algorithm on the existing captures, and all the numbers were in the right ballpark.

Comment 9

[Mason Chang [Inactive] [:mchang]]

Comment on attachment 8383338 [details] [diff] [review]
Use standard deviation to calculate threshold for stable frame automatically

Review of attachment 8383338 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good!

Comment 10

[William Lachance (:wlach)]

Attachment: Actually do sobelization preprocessing step

So my previous patch didn't actual enable the new behaviour for the dashboard. Ooops. This patch corrects that, and I tested it to make sure it works properly.

Comment 11

[William Lachance (:wlach)]

Pushed: https://github.com/mozilla/eideticker/commit/d1e7b2258bcf939e459dc4f767159f127d5494e0

Comment 12

[William Lachance (:wlach)]

This is finally look good for all tests. Yay!