739415 - Need more comprehensive talos pan/zoom checkerboarding tests

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

We need a talos checkerboarding measurement test that incorporates at least the following:

- scroll on one axis followed by scroll on another axis
- pinch zoom
- double-tap zoom
- multi-fling panning with different velocities on each fling (like a real user would do)

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Attachment: (1/3) Add ability to log exceptions to robocop log

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Attachment: (2/3) Add ability to have other data files with test cases

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Attachment: (3/3) Add another checkerboarding test case that replays actual user motion events

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 609786 [details] [diff] [review]
(1/3) Add ability to log exceptions to robocop log

Review of attachment 609786 [details] [diff] [review]:
-----------------------------------------------------------------

oh, this could give us error messages from tests, I like it.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 609787 [details] [diff] [review]
(2/3) Add ability to have other data files with test cases

Review of attachment 609787 [details] [diff] [review]:
-----------------------------------------------------------------

looks pretty straightforward, not fully sure what an asset it though.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 609788 [details] [diff] [review]
(3/3) Add another checkerboarding test case that replays actual user motion events

Review of attachment 609788 [details] [diff] [review]:
-----------------------------------------------------------------

can we doccument what testCheck2 is really measuring as well.  Will 5 runs be adequate for getting an average to report?

::: mobile/android/base/tests/testCheck2.java.in
@@ +17,5 @@
> +
> +        mDriver.startCheckerboardRecording();
> +        // replay the events
> +        try {
> +            mer.replayEvents(getAsset("testcheck2-motionevents"));

would we want to load the events and parse then before we start the checkerboardrecording?  maybe:
mer.loadEvents(getAsset("testcheck2-motionevents"));
mDriver.startCheckerboardRecording();
try {
    mer.replayEvents();
}

Comment 7

[Kartikaya Gupta (email:[e-mail])]

(In reply to Joel Maher (:jmaher) from comment #6)
> 
> can we doccument what testCheck2 is really measuring as well.  Will 5 runs
> be adequate for getting an average to report?
> 

Ok, will do. And yes, 5 runs should be fine for an average. I've been running this and testCheck with 10 iterations locally but I haven't been seeing that much variation in the numbers so 5 runs should be fine.

> would we want to load the events and parse then before we start the
> checkerboardrecording?  maybe:
> mer.loadEvents(getAsset("testcheck2-motionevents"));
> mDriver.startCheckerboardRecording();
> try {
>     mer.replayEvents();
> }

I considered doing this but I didn't really want to load the entire file and create all the MotionEvent objects together because if we had really large event files then that would eat up a lot of memory and potentially affect the test run by leaving less memory for the java code to run in. The current approach might suffer from small I/O latencies as it's reading the file, but since all the events are timestamped anyway it shouldn't make a huge difference with respect to the fling velocities and general viewport movement that results.

Also the way I implemented the "real-time" delay between injecting events means that if a few events are delayed and then there's a time gap between events, the injection will "catch up" to where it's supposed to be, rather than maintaining that delay all the way through.

Comment 8

[Kartikaya Gupta (email:[e-mail])]

https://hg.mozilla.org/integration/mozilla-inbound/rev/bfd56e7aa7c1
https://hg.mozilla.org/integration/mozilla-inbound/rev/fcf521827011
https://hg.mozilla.org/integration/mozilla-inbound/rev/5fb45a56e2af

Comment 10

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/5fb45a56e2af

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/bfd56e7aa7c1
https://hg.mozilla.org/mozilla-central/rev/fcf521827011