1533304 - Investigating tp6m-6 raptor test for raptor-tp6m-bbc-geckoview

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

This test is failing for Android 8.0 Pixel2 opt: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4661634472cedab54112b0389024fa43622d830f&selectedJob=231224968

Comment 1

[Florin Strugariu [:Bebe]]

try build without ttfi
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dc61d1bfd936209a5812b96186d658e06edfb55d

Comment 2

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I tried running this on Pixel2 with different parameters changed (timeout, startup delay, TTFI, run_at) but still couldn't get rid of the TIMEOUT fail.
Every time the test fails there is a js error:

Error: permission denied to access object
onload https://static.bbc.co.uk/bbcdotcom/...

I used the --debug-mode to see if window.performance.timing.* is available when this error is received, and it is! (screenshot attached)
I searched for permission denied to access object and this pointed me to something that might have to do with nss certificates?
https://searchfox.org/mozilla-central/search?q=Permission+denied+to+access+object&case=false&regexp=false&path=

Comment 3

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Screenshot from 2019-03-18 11-11-46.png

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Do you see the same JavaScript error when the test passes? :acreskey have you seen this before when running tests?

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

davehunt, I don't see this error when the tests passes. If you want the log of any test in the spreadsheet (https://docs.google.com/spreadsheets/d/1NcKPjWAl7roU04o8j7tZBCM2-_-hRl2vlPWi2P_Lbos/edit#gid=0) I saved all of them.

Comment 6

[Andrew Creskey [:acreskey]]

davehunt, I haven't seen this error before.

Possibly window.performance is undefined when we store this in perfData?
https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/measure.js#6

We could test this by instead of storing window.performance in perfData we could verify that it's defined and use it directly in measure.js.

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

:acreskey, I replaced perfData with a function perfData returning window.performance but I am getting the same result with a startup delay of 120s and page timeout of 90s.

Comment 8

[Andrew Creskey [:acreskey]]

Hmm...

Alexandru, do you know if it's window or window.performance that's not defined at this point?

As a test we may have to hold off on the measuring until the undefined object becomes defined.

In Bug 1533059 I'm proposing that the raptor web extension isn't injecting itself at a stable time. But note that we haven't found a solution that solves all pages there either.

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I am analyzing this thing, I am trying to find out at which point becomes window or window.performance defined. Or rather to find a dynamic logic that will wait for it to become defined no matter what point in time it is.
I will look into Bug 1533059, thanks.

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Screenshot from 2019-03-19 10-52-45.png

Bad news. When the test fails, for some reason the execution never gets into the overloading of window.onload and apparently it's not blocked by the undefined state of window.performance.
Ignore the undefined, put it by mistake before the assignment.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu from comment #10)

Created attachment 9051934 [details]
Screenshot from 2019-03-19 10-52-45.png

Bad news. When the test fails, for some reason the execution never gets into the overloading of window.onload and apparently it's not blocked by the undefined state of window.performance.
Ignore the undefined, put it by mistake before the assignment.

Is this a race condition between Raptor and the website for defining window.onload? If the website gets there first, Raptor preserves the onload and defines it's own, but if Raptor get there first the website would be replacing Raptor's. Can you experiment with :acreskey's patch in https://hg.mozilla.org/try/rev/880f845a9c3a1a7d4adb664e06b3bcbfb35e9dba to see if using the load event listener instead of the onload helps?

Comment 12

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Fixed after adding document_end and event listener:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d500a6f51ebbc4dd65d87d7dec86e2c19c2f43e

Comment 13

[Marian Raiciof [:marauder]]

Attachment: Bug 1533304 - remove disabled attribute for raptor-tp6m-bbc-geckoview

Comment 14

[Marian Raiciof [:marauder]]

Push To Try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fe19d2ae347f0db0715c695a658cdc25008c47d5

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11979156a93e
remove disabled attribute for raptor-tp6m-bbc-geckoview r=davehunt

Comment 16

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/11979156a93e