Closed Bug 1604527 Opened 1 year ago Closed 10 months ago

Run raptor-browsertime page-load visual metrics on android in CI

Categories

(Testing :: Performance, enhancement, P2)

Version 3
enhancement

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1574182

People

(Reporter: rwood, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(4 files)

Ideally we should run browsertime android visual metrics tests on Fenix, Fennec v68, and GVE in CI.

Assignee: nobody → rwood
Status: NEW → ASSIGNED

Try run with browsertime video recording enabled running on GVE looks good (see videos in the browsertime-results.tgz):

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3f69988401cab50bdde9e6147b46d3a1f6eb29df&selectedJob=282155916

So now a push trying out running the actual vismetrics job with btime android GVE:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=c009672cdadbdcebb5281a4d4f9aa91b73237b10

(In reply to Robert Wood [:rwood] from comment #2)

So now a push trying out running the actual vismetrics job with btime android GVE:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=c009672cdadbdcebb5281a4d4f9aa91b73237b10

The good news is the vismets task does run for btime android GVE page-load (see try link above). The bad news is there are alot of '0' replicates; so much so that some of the final values are '0' (videoRecordingStart, and VisuallyComplete), i.e:

Perfherder:
amazon-cold ContentfulSpeedIndex opt: 410.43
amazon-cold FirstVisualChange opt: 360
amazon-cold LastVisualChange opt: 425.71
amazon-cold PerceptualSpeedIndex opt: 378.5
amazon-cold SpeedIndex opt: 390.64
amazon-cold videoRecordingStart opt: 0
amazon-cold VisuallyComplete opt: 0

Hey :barret, does the run-visual-metrics.py (or some of the parameters sent into it) need to be tweaked for android maybe?

https://searchfox.org/mozilla-central/rev/ac43d7e69a435ede789827f20ab309da6f04c09b/taskcluster/ci/visual-metrics-dep/kind.yml#45

https://searchfox.org/mozilla-central/source/taskcluster/docker/visual-metrics/run-visual-metrics.py

Flags: needinfo?(brennie)

:rwood, not that I know of. that jobs just runs the visual metrics script. Maybe some of the command line parameters in the job description need to be tweaked for android? Im not sure if there are any differences. :acreskey or :denispal maight now if we require different flags for mobile.

Flags: needinfo?(dpalmeiro)
Flags: needinfo?(brennie)
Flags: needinfo?(acreskey)

Thanks, right these are the cmd line args we're passing into the script now (and which are forwarded on to visualmetrics.py):

command: /builds/worker/bin/run-visual-metrics.py --browsertime-results /builds/worker/fetches/browsertime-results.tgz -- --orange --perceptual --contentful --force --renderignore 5 --json --viewport

Video recorded by btime on GVE in CI. Looks like the page is already loaded when the video recording has began...?

Cool, video recording with btime on Fenix works in CI as raptor is currently turning off webrender on Fenix by default, you'll see the videos in the browsertime-results.tgz archive:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=7d84aa2995f9324f1b86979778a8ec3d1bd7a7d0&selectedJob=282431117

Video recorded by btime on Fenix in CI (webrender disabled). It has an orange screen first before loading the page.

Since video recording seems to work on Fenix (webrender disabled) here's a push to try including the actual btime Fenix vismetrics job:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3749a27726b3e900961e59c9596e0e2fa8445bbf

(In reply to Robert Wood [:rwood] from comment #9)

Since video recording seems to work on Fenix (webrender disabled) here's a push to try including the actual btime Fenix vismetrics job:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3749a27726b3e900961e59c9596e0e2fa8445bbf

Resulting data from the btime Fenix (webrender disabled) vismetrics task (doesn't have the '0' replicates issue that is seen on GVE):

Perfherder:
amazon-cold ContentfulSpeedIndex opt: 1035.43
amazon-cold FirstVisualChange opt: 962.86
amazon-cold LastVisualChange opt: 2242.86
amazon-cold PerceptualSpeedIndex opt: 1098.14
amazon-cold SpeedIndex opt: 1034.21
amazon-cold videoRecordingStart opt: 2017.14

If videoRecordingStart is 0, then there is no orange screen in the video and something went wrong. Perhaps, this is related to bug 1565225, so try setting layout.display-list.retain.chrome to false.

Flags: needinfo?(dpalmeiro)

(In reply to Denis Palmeiro [:denispal] from comment #11)

If videoRecordingStart is 0, then there is no orange screen in the video and something went wrong. Perhaps, this is related to bug 1565225, so try setting layout.display-list.retain.chrome to false.

Thanks Denis! Also just adding here what you noted on slack as a reminder for myself, since 'firefox.windowRecorder' was also on:

If you set --firefox.windowRecorder to false, browsertime will use the android screenrecorder instead. I find this is usually more stable, but on lower end devices like G5 it can introduce overhead.

(In reply to Robert Wood [:rwood] from comment #14)

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=500d299746b19db9ce6b3395df95aea1dceab7d1

Added the vismet fenix job to the try run (didn't select it in mach try fuzzy)

This first patch will enable browsertime Fenix and GVE page-load jobs (amazon, youbute) with visual metrics turned on, and webrender turned off. Tier3, and on GP2 aarch64 PGO only so we won't overload the device queue.

Still need to tweak GVE (no orange screen) and also eventually enable webrender when video recording works with webrender enabled (Bug 1601004).

Keywords: leave-open
Depends on: 1606131
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0f6acdafe0eb
Enable raptor-browsertime with visual metrics on Fenix (webrender off) and GVE for central and try only r=sparky
Regressions: 1605885
No longer regressions: 1605885

(In reply to Cosmin Sabou [:CosminS] from comment #19)

Please take a look over these raptor failures after this bug was merged to central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=20b3a68275d5143ff754b4b7e92e624bed5a565e&searchStr=raptor&selectedJob=282816485

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=282816485&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=282816480&repo=mozilla-central

I don't see how this bug caused those failures; those failures involve conditioned profiles on raptor fennec; this patch enabled some browsertime android visual metrics jobs, unrelated to raptor fennec. Thanks.

Flags: needinfo?(rwood)

Note that raptor is not turning on webrender for GVE or Fenix in CI. Try push adding the browsertime cmd line argument --firefox.windowRecorder=false, to see if using the android recorder (instead of the firefox window recorder) fixes the recording issue (no orange screen) on GVE:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=fbf2cddd17e7a960bae46aa01f79a0f086bdc21f

Even with --firefox.windowRecorder=false (to use the android video recorder instead of the firefox one), on GVE there is still no orange screen (although it shows the about:blank first, which is better than the firefox windowRecorder example in Comment 6).

Try push using the default firefox.windowRecorder but setting layout.display-list.retain.chrome to false as Denis suggested in Comment 11:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=7f1b20a3a13ed5f244a9547a42eb855684cb7f09

Depends on: 1601414

(In reply to Robert Wood [:rwood] from comment #23)

Try push using the default firefox.windowRecorder but setting layout.display-list.retain.chrome to false as Denis suggested in Comment 11:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=7f1b20a3a13ed5f244a9547a42eb855684cb7f09

No change - the resulting browsertime recordings on GVE look similar to the one in Comment 6.

It might be worth running this locally and observing what happens with the orange screen and the pageload.
Is the orange screen showing up at all on GVE?

Flags: needinfo?(acreskey)

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #25)

It might be worth running this locally and observing what happens with the orange screen and the pageload.
Is the orange screen showing up at all on GVE?

Hey Andrew, when I run raptor-browsertime with video recording locally on GVE and watch the page-load, no there is no orange screen appearing during the page-load (or in the resulting recordings of course).

(In reply to Robert Wood [:rwood] from comment #26)

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #25)

It might be worth running this locally and observing what happens with the orange screen and the pageload.
Is the orange screen showing up at all on GVE?

Hey Andrew, when I run raptor-browsertime with video recording locally on GVE and watch the page-load, no there is no orange screen appearing during the page-load (or in the resulting recordings of course).

Hmm, and from this thread it sounds like webrender is already being disabled.

I'm running gve through regular mozilla browsertime and I'm seeing the orange.

Rob, can you provide me with the ./mach command to try this locally and I'll see if I notice anything off?

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #27)

I'm running gve through regular mozilla browsertime and I'm seeing the orange.

Rob, can you provide me with the ./mach command to try this locally and I'll see if I notice anything off?

Interesting, and yes webrender is disabed - at least enable_webrender=False inside Raptor. This is the command I am using locally:

./mach raptor --test amazon --app=geckoview --binary="org.mozilla.geckoview_example" --browser-cycles 2 --browsertime --cold --browsertime-video

And this is the actual browsertime command that raptor is calling:

raptor-main Info: browsertime cmd: /Users/rwood/.mozbuild/node/bin/node /Users/rwood/mozilla-central/tools/browsertime/node_modules/browsertime/bin/browsertime.js /Users/rwood/mozilla-central/testing/raptor/raptor/../browsertime/browsertime_pageload.js --browser firefox --android --firefox.binaryPath /Users/rwood/.mozbuild/node/bin/node --firefox.android.package org.mozilla.geckoview_example --firefox.android.activity org.mozilla.geckoview_example.GeckoViewActivity --browsertime.page_cycles 1 --browsertime.url https://www.amazon.com --browsertime.page_cycle_delay 1000 --browsertime.post_startup_delay 30000 --firefox.profileTemplate /var/folders/ty/60lmmg9565b_51cfpchkzbkw0000gn/T/tmpy98qTB --skipHar --video true --visualMetrics false --timeouts.pageLoad 60000 --timeouts.script 60000 -vvv --resultDir /Users/rwood/mozilla-central/testing/mozharness/build/blobber_upload_dir/browsertime-results/amazon-cold -n 2

Also I am using a GVE build from treeherder's Android 4.0 API 16 (OPT) on my GP2, i.e.:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=9d401beea71d16526843d414e0f17415ce606a8c&searchStr=android%2C4.0%2Capi%2C16&selectedJob=283559709

I'm seeing the same behaviour -- no orange screen.
Rob, I noticed the --visualMetrics false in that command -- is that wanted?
It doesn't resolve the missing orange though.

I so see that gfx.webrender.force-disabled is set to true at runtime.

Browsertime is definitely trying to add the orange screen:

19:58:57     INFO -  [2020-01-06 14:58:57] INFO: [browsertime.command.measure] Testing url https://www.amazon.com iteration 1
19:58:57     INFO -  [2020-01-06 14:58:57] DEBUG: [browsertime.video] Add orange color

and in fact, once I was navigating to about:config while raptor was waiting for the browser to settle and in that case I could see the orange screen.

Given that the amazon.com page appears almost instantly after the navigation, I'm wondering if there is simply no time to draw the orange?

Same thing on the youtube.com page.

I'm not that familiar with the orange screen mechanism, but I would have expected to see it here, at least briefly, before it's removed:

19:58:57     INFO -  1578340737658	webdriver::server	DEBUG	-> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"\n      (function() {\n        const orange = document.createElement('div');\n        orange.id = 'browsertime-orange';\n        orange.style.position = 'absolute';\n        orange.style.top = '0';\n        orange.style.left = '0';\n        orange.style.width = Math.max(document.documentElement.clientWidth, document.body.clientWidth) + 'px';\n        orange.style.height = Math.max(document.documentElement.clientHeight,document.body.clientHeight) + 'px';\n        orange.style.backgroundColor = '#DE640D';\n        orange.style.zIndex = '2147483647';\n        document.body.appendChild(orange);\n        document.body.style.display = '';\n      })();","args":[]}
19:58:57     INFO -  1578340737728	webdriver::server	DEBUG	<- 200 OK {"value":null}
19:58:59     INFO -  1578340739778	webdriver::server	DEBUG	-> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"return document.documentURI;","args":[]}
19:58:59     INFO -  1578340739815	webdriver::server	DEBUG	<- 200 OK {"value":"about:blank"}
19:58:59     INFO -  1578340739816	webdriver::server	DEBUG	-> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"(function() {\n          const orange = document.getElementById('browsertime-orange');\n          if (orange) {\n            orange.parentNode.removeChild(orange);\n          }\n          window.requestAnimationFrame(function(){\n            window.requestAnimationFrame(function(){\n              window.location=\"https://www.amazon.com\";\n            });\n          });\n        })();","args":[]}
19:58:59     INFO -  1578340739844	webdriver::server	DEBUG	<- 200 OK {"value":null}
19:59:00     INFO -  1578340740349	webdriver::server	DEBUG	-> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"return document.documentURI;","args":[]}
19:59:00     INFO -  1578340740383	webdriver::server	DEBUG	<- 200 OK {"value":"https://www.amazon.com/"}
19:59:00     INFO -  [2020-01-06 14:59:00] DEBUG: [browsertime] Waiting for script pageCompleteCheck at most 300000 ms

As for getting raptor-browsertime (with visual metrics) running on fennec (v68) in production, here's a try push:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=4c563d3372177a1a049c4cf06f42faf4f7d1bfe4

Locally (with ^ patch applied) I can run raptor-browsertime on fennec v68 with:

./mach raptor --test amazon --app fennec --binary org.mozilla.fennec_aurora --activity org.mozilla.gecko.BrowserApp --browser-cycles 2 --browsertime --cold

It does run successfully and grabs the page-load measurements (using the mitm recordings) however it looks like it isn't able to get 'timeToContentfulPaint' on fennec v68:

20:22:52 INFO - results.MissingResultsError: Browsertime cycle missing timeToContentfulPaint measurement

And also there's a webdriver error when trying to shutdown fennec:

20:18:51 INFO - [2020-01-06 15:18:51] DEBUG: [browsertime] Telling browser to quit.
20:18:51 INFO - 1578341931322 webdriver::server DEBUG -> DELETE /session/33182f50-f282-4ec0-adb4-818852dfbd44
20:18:51 INFO - 1578341931325 webdriver::server DEBUG <- 500 Internal Server Error {"value":{"error":"unsupported operation","message":"Only supported in Firefox","stacktrace":"WebDriverError@chrome://marionette/content/error.js:175:5\nUnsupportedOperationError@chrome://marionette/content/error.js:493:5\nassert.that/<@chrome://marionette/content/assert.js:428:13\nassert.firefox@chrome://marionette/content/assert.js:97:57\nGeckoDriver.prototype.quit@chrome://marionette/content/driver.js:3459:10\ndespatch@chrome://marionette/content/server.js:305:40\nexecute@chrome://marionette/content/server.js:275:16\nonPacket/<@chrome://marionette/content/server.js:248:20\nonPacket@chrome://marionette/content/server.js:249:9\n_onJSONObjectReady/<@chrome://marionette/content/transport.js:503:20\n"}}

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #29)

I'm seeing the same behaviour -- no orange screen.
Rob, I noticed the --visualMetrics false in that command -- is that wanted?
It doesn't resolve the missing orange though.

Thanks Andrew! Yes, my understanding is we don't enable --visualMetrics on the browsertime command line b/c we are doing our own visual metrics analysis in the *-vismet treeherder/tc task. On Fenix in production (webrender disabled) where there is an orange screen (and visual metrics appear to be working great) I verified the same --visualMetrics false is on the browsertime cmd line there too.

i.e. https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=btime%2Candroid&revision=989bbd83ccd5a6fbc706b8eb6635a471596c6a43&selectedJob=283670183

Depends on: 1607891
Depends on: 1609173
Assignee: rwood → gmierz2

This is more of a meta-bug so I'm unassigning myself from it.

Assignee: gmierz2 → nobody
Status: ASSIGNED → NEW
Priority: P1 → P2

Greg: Is there anything to do here that isn't already covered by bug 1574182? If not, let's close this and reduce the number of meta bugs for Browsertime on Android.

Flags: needinfo?(gmierz2)

I don't think so, we can always reopen this is needed.

Status: NEW → RESOLVED
Closed: 10 months ago
Flags: needinfo?(gmierz2)
Resolution: --- → DUPLICATE
Duplicate of bug: browsertime-ci-android
You need to log in before you can comment on or make changes to this bug.