882936 - Add a simple eideticker "responsiveness" test

Status: RESOLVED WORKSFORME

(Testing Graveyard :: Eideticker, defect)

Description

[Geoff Brown [:gbrown]]

We want to start measuring "responsiveness" using eideticker.

As a simple first step, the following test is proposed:
 - load a page (a. simple page; b. large/complex page)
 - click on page and drag
 - measure and report time between first drag event sent and first motion in page

Comment 1

[William Lachance (:wlach)]

The biggest obstacle I can see to this is that currently we communicate actions from the eideticker test harness to the device over wifi (using sutagent), which can introduce latency (guessing 50ms-100ms). This isn't that big a deal for the tests we have now, but for something like the above it was make the results pretty erratic. In order to really make this work, we'd have to find a way around that.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Can we use adb over USB instead?

Comment 3

[William Lachance (:wlach)]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #2)
> Can we use adb over USB instead?

Not really unfortunately. :(

1. Some devices (such as the galaxy nexus) can't do HDMI out and be connected to adb over usb simulatenously.
2. Other devices (such as the lg-p999) don't have this problem, but also don't get enough power over the usb connection to stay up reliably over days/weeks.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Brad also suggested having the SUT agent on the device dump out the time it receives the command and using that as the action-start time. That way the network latency should be eliminated, and as long as clock skew is accounted for it should be reasonably accurate.

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

Karen,
comment 0 describes a panning responsiveness test as the thing to be worked on. I think we want both a panning responsiveness test and a zooming responsiveness test. The questions to you are 1) which one is the higher priority? and 2) are those two tests sufficient?

Comment 6

[Karen Rudnitski [:kar]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #5)
>1) which one is the higher priority? and 2) are those two tests sufficient?

1) panning is first priority, followed by zooming.

2) can do do 'page load' as well?

Comment 7

[William Lachance (:wlach)]

(In reply to Karen Rudnitski [:kar] from comment #6)

> 2) can do do 'page load' as well?

Maybe. ;) What particular action(s) would you want to measure the responsiveness of in a page load test? We already have various eideticker tests which attempt to measure page load times.

Comment 8

[Karen Rudnitski [:kar]]

I'm thinking similar to how we measure load times on fennec so we can have at least one 'common' benchmark to compare FxOS browser load times against. I may suggest adding another reference page that would be more commonly accessed by FxOS browsers, but I just want parity with fennec page loading to start.

(unless you're asking me which action to determine when the page has loaded?)

Comment 9

[William Lachance (:wlach)]

So this bug is (as far as I am aware) about adding eideticker responsiveness tests (defined as tests that measure the time from "input event received" to "visible action observed") for Fennec specifically.

If we want to add pageload tests for the FxOS browser we can definitely file a new bug to work on that. It would probably be a good idea to measure that.

(In reply to Karen Rudnitski [:kar] from comment #8)
> I'm thinking similar to how we measure load times on fennec so we can have
> at least one 'common' benchmark to compare FxOS browser load times against.
> I may suggest adding another reference page that would be more commonly
> accessed by FxOS browsers, but I just want parity with fennec page loading
> to start.
> 
> (unless you're asking me which action to determine when the page has loaded?)

Comment 10

[Geoff Brown [:gbrown]]

(In reply to William Lachance (:wlach) from comment #9)
> So this bug is (as far as I am aware) about adding eideticker responsiveness
> tests (defined as tests that measure the time from "input event received" to
> "visible action observed") for Fennec specifically.

That was my intention, exactly; I should have stated that more explicitly in comment 0.

Comment 11

[Karen Rudnitski [:kar]]

of course - that makes total sense (I think I was muddling up some different programs in my head so obviously wasn't quite thinking clearly when I responded earlier). 

One more question then, though around page loads:
 Can we measure when a page 'appears' to have loaded (the screen fills up as expected and I can start doing something with it) vs when a page has properly and truly fully loaded (and therefore when the 'Reader' icon appears)? I see these as two different events and I'd like some way of putting a number to them.

Let's hope that sounds intelligent and you understand my ask :)

Comment 12

[William Lachance (:wlach)]

(In reply to Karen Rudnitski [:kar] from comment #11)

>  Can we measure when a page 'appears' to have loaded (the screen fills up as
> expected and I can start doing something with it) vs when a page has
> properly and truly fully loaded (and therefore when the 'Reader' icon
> appears)? I see these as two different events and I'd like some way of
> putting a number to them.

So in fact the reader icon appearing has nothing to do with when the page is loaded -- we ignore that piece of information.

We consider a page to be finished loading when all the content is displayed. You can see the breakdown in the framediff view:

http://eideticker.wrla.ch/framediff-view.html?title=Frame%20Difference%20Time%20for%20nytimes%20pageload%20after%20startup%20%282013-08-27%29&video=videos/video-1377677013.24.webm&framediff=framediffs/framediff-1377677019.95.json

On this graph, x represents time in capture, y represents represents differences between frames -- when the graph flatlines, it means there is no difference between frames -- we consider the page fully loaded when there are no further major changes between frames in the capture.

So in fact waiting for the reader icon to display would in fact require writing a new test (not sure if eideticker is the best framework to do that or not).

Hope that makes sense.

Comment 13

[Geoff Brown [:gbrown]]

Responsiveness measures are now collected for several tests and can be viewed from the dashboard. For example, http://eideticker.mozilla.org/#/samsung-gn/cnn/timetoresponse.