992392 - Ignore status bar when running Marketplace tests (and maybe others)

Status: RESOLVED WONTFIX

(Testing Graveyard :: Eideticker, defect)

Description

[Wil Clouser [:clouserw]]

If you look at the output for one of the Marketplace videos you can see the Marketplace finishes rendering midway through, but the deferred assets/updates are still being checked in the background so the traffic icon in the status bar is flashing and preventing the end of the test.

Example: http://eideticker.mozilla.org/b2g/videos/video-1396619596.27.webm

Is it possible to ignore the statusbar region of the screen?  This may also be applicable in other tests, but I think the Marketplace is the only one doing network traffic.  Thanks.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We could achieve this by modifying the capture area depending on the test. It doesn't appear that this affects all test runs though, for example http://eideticker.mozilla.org/b2g/detail.html?id=a4158d5abd2f11e38620f0def1767b24#/framediffsums has a timetostableframe of 10.43 (we should indicate that value on the detail view) and you can see that the status bar continues to fluctuate after this time. It's could be that the difference does not breach the threshold.

Comment 2

[Wil Clouser [:clouserw]]

Yeah, it's hard to say.  There is a lot of jitter on http://eideticker.mozilla.org/b2g/#/inari/b2g-marketplace-startup/timetostableframe which may be real, or it may be that icon, sometimes.  I think the best way to be consistent would be to ignore it.  How much work is modifying the capture area?

Comment 3

[William Lachance (:wlach)]

It might be possible to make the capture area not include this area by changing the getdimensions script to load an app that includes the status bar (instead of a webpage that encompasses everything). This would not work for tests which hide the status bar, so we'd probably have to vary this behaviour per run or something.

I suspect bug 982132 is the culprit for your particular bug: one little status indicator shouldn't make that big a difference in the entropy level of the capture. I'd fix that first, then possibly revisit this later.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

With bug 982132 resolved do we still need to exclude the status bar from the capture area? I'd rather not introduce having to run the getdimensions script before individual tests, but we could store the capture areas with and without the status bar and pick the appropriate one for the test. It's possible that the status bar should even be excluded in the majority of cases.

Comment 5

[William Lachance (:wlach)]

(In reply to Dave Hunt (:davehunt) from comment #4)
> With bug 982132 resolved do we still need to exclude the status bar from the
> capture area? I'd rather not introduce having to run the getdimensions
> script before individual tests, but we could store the capture areas with
> and without the status bar and pick the appropriate one for the test. It's
> possible that the status bar should even be excluded in the majority of
> cases.

I don't think the status bar makes an appreciable difference in the amount of entropy in the capture. I'm inclined to WONTFIX this one for now, as the effort involved is far greater than any potential gains.

Comment 6

[William Lachance (:wlach)]

Resolving. If we find evidence my assumption above is wrong, we can reopen.