1623159 - Raptor Browsertime warm page load tests are failing on geckoview and fenix

Status: RESOLVED FIXED

(Testing :: Performance, defect, P2)

Description

[Greg Mierzwinski [:sparky]]

There seems to be many failures occurring when we run warm page load tests with raptor browsertime.

See here for Fenix ones: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=da2777f7da6321ea3dfc78259d679d7718b36b22

See here for Geckoview ones: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=44cb8e457167909185ef891a81f2f3ecbe895b8f

This bug is for fixing these errors.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Is this related to bug 1617615 and bug 1617617?

Comment 2

[Greg Mierzwinski [:sparky]]

Yes, I've added them as blockers to this bug.

Comment 3

[Greg Mierzwinski [:sparky]]

So I have a possible patch in this push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9c52e3a2274360c16557476ac99685a24394ed4c

When testing locally, I noticed that the pages don't get reloaded when the same url is used - I see this in Chrome as well. The only change I made is that we now navigate to "about:blank" before each page cycle.

I think this is fine as a solution, but I'd like to know what everyone else thinks? https://hg.mozilla.org/try/rev/7c239e6d1a970d66dd0bd168804ed17a74cf22dc

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Why can't we just run each cycle in it's own tab? That will make sure that claimed memory will be freed, and we always start from a nearly clean state.

Comment 5

[Greg Mierzwinski [:sparky]]

Good point! I'll try that.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Note that WebDriver spec has the NewWindow end-point, which you might want to utilize here. But not sure how this works via Browsertime...

Comment 7

[Greg Mierzwinski [:sparky]]

I can get access to the driver, but I'm hitting this issue on Fenix:

15:02:32     INFO -  [2020-03-26 11:02:32] ERROR: [browsertime.command.measure] No page that could be associated with the error:openTab() not supported in firefox preview
15:02:32     INFO -  [2020-03-26 11:02:32] ERROR: [browsertime] UnsupportedOperationError: openTab() not supported in firefox preview
15:02:32     INFO -      at Object.throwDecodedError (/home/sparky/mozilla-source/mozilla-central/tools/browsertime/node_modules/selenium-webdriver/lib/error.js:550:15)
15:02:32     INFO -      at parseHttpResponse (/home/sparky/mozilla-source/mozilla-central/tools/browsertime/node_modules/selenium-webdriver/lib/http.js:565:13)
15:02:32     INFO -      at Executor.execute (/home/sparky/mozilla-source/mozilla-central/tools/browsertime/node_modules/selenium-webdriver/lib/http.js:491:26)
15:02:32     INFO -      at process._tickCallback (internal/process/next_tick.js:68:7)

This is the code I am calling: context.selenium.driver.switchTo().newWindow("about:blank", "Testing Tab");

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Damn I forgot that we haven't fixed bug 1559120 yet. So it will not work for Fenix yet. As such loading about:blank after each cycle seems to be reasonable.

Comment 9

[Tarek Ziadé (:tarek)]

It seems fine to me since we don't measure it. Andrew what do you think?

Comment 10

[Andrew Creskey [:acreskey]]

I think that's a very good solution.

Comment 11

[Greg Mierzwinski [:sparky]]

Ok, thanks for your input everyone! I'm going to submit a patch to make this change and I'll file a follow-up bug for the new tab idea :whimboo mentioned when it's available on Fenix.

Comment 12

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1623159 - Navigate to about:blank before starting pageload tests. r?#perftest,whimboo

This patch fixes an issue in warm pageload tests where the browser doesn't navigate to the page that was requested since the browser was already open on that page. To get around this, we navigate to about:blank, and then to the rquested url.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/876a51b67aca
Navigate to about:blank before starting pageload tests. r=perftest-reviewers,whimboo,Bebe

Comment 14

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/876a51b67aca