915730 - Add responsiveness metrics to all Android tests with actions

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

We can measure this for all existing Android tests, and it's just another small addition to the datapoint for each test.

This is a prelude to doing the same thing for FirefoxOS.

Comment 1

[William Lachance (:wlach)]

Attachment: 0001-Bug-915370-Store-all-non-video-capture-data-acquired.patch

Various cleanup to make this possible/easier.

Comment 2

[William Lachance (:wlach)]

Attachment: The actual patch

Comment 3

[Geoff Brown [:gbrown]]

Comment on attachment 804098 [details] [diff] [review]
The actual patch

Review of attachment 804098 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great. I'm eager to see some initial results.

::: src/eideticker/eideticker/metrics.py
@@ +1,3 @@
> +import videocapture
> +
> +INPUT_RESPONSE_THRESHOLD = 10000

Why 10000? My naive instinct would be to use something smaller. Do you expect much noise (framediffs without an action, or independent of action)?

@@ +18,5 @@
> +        if first_non_sleep_action:
> +            framediffs = videocapture.get_framediff_sums(
> +                capture, threshold=difference_threshold)
> +            for (i, framediff) in enumerate(framediffs):
> +                t = i/60.0

Is there an existing constant that could be used instead of "60.0"?

Comment 4

[William Lachance (:wlach)]

(In reply to Geoff Brown [:gbrown] from comment #3)
> ::: src/eideticker/eideticker/metrics.py
> @@ +1,3 @@
> > +import videocapture
> > +
> > +INPUT_RESPONSE_THRESHOLD = 10000
> 
> Why 10000? My naive instinct would be to use something smaller. Do you
> expect much noise (framediffs without an action, or independent of action)?

I noticed some small variations before the real response in taskjs, but I agree 10000 is probably too high. Set it to 4096 for now -- we can always adjust later.

> @@ +18,5 @@
> > +        if first_non_sleep_action:
> > +            framediffs = videocapture.get_framediff_sums(
> > +                capture, threshold=difference_threshold)
> > +            for (i, framediff) in enumerate(framediffs):
> > +                t = i/60.0
> 
> Is there an existing constant that could be used instead of "60.0"?

We should probably make the fps a property of the capture. Changed that before committing.

Comment 5

[William Lachance (:wlach)]

Pushed: 

https://github.com/mozilla/eideticker/commit/92f084a5dea830a2089cf1937944dd174e65c45e
https://github.com/mozilla/eideticker/commit/f9c0714c45fddc7aae3ac8690020795fb9af5fd2