Run raptor-browsertime page-load visual metrics on android in CI
Categories
(Testing :: Performance, enhancement, P2)
Tracking
(Not tracked)
People
(Reporter: rwood, Unassigned)
References
Details
Attachments
(4 files)
Ideally we should run browsertime android visual metrics tests on Fenix, Fennec v68, and GVE in CI.
|
||
Updated•5 years ago
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 1•5 years ago
|
||
Try run with browsertime video recording enabled running on GVE looks good (see videos in the browsertime-results.tgz):
|
|
Reporter | |
Comment 2•5 years ago
|
||
So now a push trying out running the actual vismetrics job with btime android GVE:
|
|
Reporter | |
Comment 3•5 years ago
•
|
||
(In reply to Robert Wood [:rwood] from comment #2)
So now a push trying out running the actual vismetrics job with btime android GVE:
The good news is the vismets task does run for btime android GVE page-load (see try link above). The bad news is there are alot of '0' replicates; so much so that some of the final values are '0' (videoRecordingStart, and VisuallyComplete), i.e:
Perfherder:
amazon-cold ContentfulSpeedIndex opt: 410.43
amazon-cold FirstVisualChange opt: 360
amazon-cold LastVisualChange opt: 425.71
amazon-cold PerceptualSpeedIndex opt: 378.5
amazon-cold SpeedIndex opt: 390.64
amazon-cold videoRecordingStart opt: 0
amazon-cold VisuallyComplete opt: 0
Hey :barret, does the run-visual-metrics.py (or some of the parameters sent into it) need to be tweaked for android maybe?
https://searchfox.org/mozilla-central/source/taskcluster/docker/visual-metrics/run-visual-metrics.py
|
||
Comment 4•5 years ago
|
||
:rwood, not that I know of. that jobs just runs the visual metrics script. Maybe some of the command line parameters in the job description need to be tweaked for android? Im not sure if there are any differences. :acreskey or :denispal maight now if we require different flags for mobile.
|
|
Reporter | |
Comment 5•5 years ago
|
||
Thanks, right these are the cmd line args we're passing into the script now (and which are forwarded on to visualmetrics.py):
command: /builds/worker/bin/run-visual-metrics.py --browsertime-results /builds/worker/fetches/browsertime-results.tgz -- --orange --perceptual --contentful --force --renderignore 5 --json --viewport
|
|
Reporter | |
Comment 6•5 years ago
•
|
||
Video recorded by btime on GVE in CI. Looks like the page is already loaded when the video recording has began...?
|
|
Reporter | |
Comment 7•5 years ago
|
||
Cool, video recording with btime on Fenix works in CI as raptor is currently turning off webrender on Fenix by default, you'll see the videos in the browsertime-results.tgz archive:
|
|
Reporter | |
Comment 8•5 years ago
•
|
||
Video recorded by btime on Fenix in CI (webrender disabled). It has an orange screen first before loading the page.
|
|
Reporter | |
Comment 9•5 years ago
|
||
Since video recording seems to work on Fenix (webrender disabled) here's a push to try including the actual btime Fenix vismetrics job:
|
|
Reporter | |
Comment 10•5 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #9)
Since video recording seems to work on Fenix (webrender disabled) here's a push to try including the actual btime Fenix vismetrics job:
Resulting data from the btime Fenix (webrender disabled) vismetrics task (doesn't have the '0' replicates issue that is seen on GVE):
Perfherder:
amazon-cold ContentfulSpeedIndex opt: 1035.43
amazon-cold FirstVisualChange opt: 962.86
amazon-cold LastVisualChange opt: 2242.86
amazon-cold PerceptualSpeedIndex opt: 1098.14
amazon-cold SpeedIndex opt: 1034.21
amazon-cold videoRecordingStart opt: 2017.14
|
||
Comment 11•5 years ago
|
||
If videoRecordingStart is 0, then there is no orange screen in the video and something went wrong. Perhaps, this is related to bug 1565225, so try setting layout.display-list.retain.chrome to false.
|
|
Reporter | |
Comment 12•5 years ago
|
||
(In reply to Denis Palmeiro [:denispal] from comment #11)
If videoRecordingStart is 0, then there is no orange screen in the video and something went wrong. Perhaps, this is related to bug 1565225, so try setting layout.display-list.retain.chrome to false.
Thanks Denis! Also just adding here what you noted on slack as a reminder for myself, since 'firefox.windowRecorder' was also on:
If you set --firefox.windowRecorder to false, browsertime will use the android screenrecorder instead. I find this is usually more stable, but on lower end devices like G5 it can introduce overhead.
|
|
Reporter | |
Comment 13•5 years ago
|
||
|
|
Reporter | |
Comment 14•5 years ago
|
||
|
|
Reporter | |
Comment 15•5 years ago
•
|
||
(In reply to Robert Wood [:rwood] from comment #14)
Added the vismet fenix job to the try run (didn't select it in mach try fuzzy)
|
|
Reporter | |
Comment 16•5 years ago
•
|
||
This first patch will enable browsertime Fenix and GVE page-load jobs (amazon, youbute) with visual metrics turned on, and webrender turned off. Tier3, and on GP2 aarch64 PGO only so we won't overload the device queue.
Still need to tweak GVE (no orange screen) and also eventually enable webrender when video recording works with webrender enabled (Bug 1601004).
|
||
Comment 17•5 years ago
|
||
|
||
Comment 18•5 years ago
|
||
| bugherder | ||
|
||
Comment 19•5 years ago
|
||
Please take a look over these raptor failures after this bug was merged to central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=20b3a68275d5143ff754b4b7e92e624bed5a565e&searchStr=raptor&selectedJob=282816485
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=282816485&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=282816480&repo=mozilla-central
|
|
Reporter | |
Comment 20•5 years ago
|
||
(In reply to Cosmin Sabou [:CosminS] from comment #19)
Please take a look over these raptor failures after this bug was merged to central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=20b3a68275d5143ff754b4b7e92e624bed5a565e&searchStr=raptor&selectedJob=282816485
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=282816485&repo=mozilla-central
https://treeherder.mozilla.org/logviewer.html#?job_id=282816480&repo=mozilla-central
I don't see how this bug caused those failures; those failures involve conditioned profiles on raptor fennec; this patch enabled some browsertime android visual metrics jobs, unrelated to raptor fennec. Thanks.
|
|
Reporter | |
Comment 21•5 years ago
|
||
Note that raptor is not turning on webrender for GVE or Fenix in CI. Try push adding the browsertime cmd line argument --firefox.windowRecorder=false, to see if using the android recorder (instead of the firefox window recorder) fixes the recording issue (no orange screen) on GVE:
|
|
Reporter | |
Comment 22•5 years ago
|
||
Even with --firefox.windowRecorder=false (to use the android video recorder instead of the firefox one), on GVE there is still no orange screen (although it shows the about:blank first, which is better than the firefox windowRecorder example in Comment 6).
|
|
Reporter | |
Comment 23•5 years ago
|
||
Try push using the default firefox.windowRecorder but setting layout.display-list.retain.chrome to false as Denis suggested in Comment 11:
|
|
Reporter | |
Comment 24•5 years ago
•
|
||
(In reply to Robert Wood [:rwood] from comment #23)
Try push using the default firefox.windowRecorder but setting
layout.display-list.retain.chrometo false as Denis suggested in Comment 11:
No change - the resulting browsertime recordings on GVE look similar to the one in Comment 6.
|
||
Comment 25•5 years ago
|
||
It might be worth running this locally and observing what happens with the orange screen and the pageload.
Is the orange screen showing up at all on GVE?
|
|
Reporter | |
Comment 26•5 years ago
•
|
||
(In reply to Andrew Creskey [:acreskey] [he/him] from comment #25)
It might be worth running this locally and observing what happens with the orange screen and the pageload.
Is the orange screen showing up at all on GVE?
Hey Andrew, when I run raptor-browsertime with video recording locally on GVE and watch the page-load, no there is no orange screen appearing during the page-load (or in the resulting recordings of course).
|
|
||
Comment 27•5 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #26)
(In reply to Andrew Creskey [:acreskey] [he/him] from comment #25)
It might be worth running this locally and observing what happens with the orange screen and the pageload.
Is the orange screen showing up at all on GVE?Hey Andrew, when I run raptor-browsertime with video recording locally on GVE and watch the page-load, no there is no orange screen appearing during the page-load (or in the resulting recordings of course).
Hmm, and from this thread it sounds like webrender is already being disabled.
I'm running gve through regular mozilla browsertime and I'm seeing the orange.
Rob, can you provide me with the ./mach command to try this locally and I'll see if I notice anything off?
|
|
Reporter | |
Comment 28•5 years ago
•
|
||
(In reply to Andrew Creskey [:acreskey] [he/him] from comment #27)
I'm running
gvethrough regular mozilla browsertime and I'm seeing the orange.Rob, can you provide me with the ./mach command to try this locally and I'll see if I notice anything off?
Interesting, and yes webrender is disabed - at least enable_webrender=False inside Raptor. This is the command I am using locally:
./mach raptor --test amazon --app=geckoview --binary="org.mozilla.geckoview_example" --browser-cycles 2 --browsertime --cold --browsertime-video
And this is the actual browsertime command that raptor is calling:
raptor-main Info: browsertime cmd: /Users/rwood/.mozbuild/node/bin/node /Users/rwood/mozilla-central/tools/browsertime/node_modules/browsertime/bin/browsertime.js /Users/rwood/mozilla-central/testing/raptor/raptor/../browsertime/browsertime_pageload.js --browser firefox --android --firefox.binaryPath /Users/rwood/.mozbuild/node/bin/node --firefox.android.package org.mozilla.geckoview_example --firefox.android.activity org.mozilla.geckoview_example.GeckoViewActivity --browsertime.page_cycles 1 --browsertime.url https://www.amazon.com --browsertime.page_cycle_delay 1000 --browsertime.post_startup_delay 30000 --firefox.profileTemplate /var/folders/ty/60lmmg9565b_51cfpchkzbkw0000gn/T/tmpy98qTB --skipHar --video true --visualMetrics false --timeouts.pageLoad 60000 --timeouts.script 60000 -vvv --resultDir /Users/rwood/mozilla-central/testing/mozharness/build/blobber_upload_dir/browsertime-results/amazon-cold -n 2
Also I am using a GVE build from treeherder's Android 4.0 API 16 (OPT) on my GP2, i.e.:
|
|
||
Comment 29•5 years ago
|
||
I'm seeing the same behaviour -- no orange screen.
Rob, I noticed the --visualMetrics false in that command -- is that wanted?
It doesn't resolve the missing orange though.
I so see that gfx.webrender.force-disabled is set to true at runtime.
Browsertime is definitely trying to add the orange screen:
19:58:57 INFO - [2020-01-06 14:58:57] INFO: [browsertime.command.measure] Testing url https://www.amazon.com iteration 1
19:58:57 INFO - [2020-01-06 14:58:57] DEBUG: [browsertime.video] Add orange color
and in fact, once I was navigating to about:config while raptor was waiting for the browser to settle and in that case I could see the orange screen.
Given that the amazon.com page appears almost instantly after the navigation, I'm wondering if there is simply no time to draw the orange?
|
|
||
Comment 30•5 years ago
|
||
Same thing on the youtube.com page.
I'm not that familiar with the orange screen mechanism, but I would have expected to see it here, at least briefly, before it's removed:
19:58:57 INFO - 1578340737658 webdriver::server DEBUG -> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"\n (function() {\n const orange = document.createElement('div');\n orange.id = 'browsertime-orange';\n orange.style.position = 'absolute';\n orange.style.top = '0';\n orange.style.left = '0';\n orange.style.width = Math.max(document.documentElement.clientWidth, document.body.clientWidth) + 'px';\n orange.style.height = Math.max(document.documentElement.clientHeight,document.body.clientHeight) + 'px';\n orange.style.backgroundColor = '#DE640D';\n orange.style.zIndex = '2147483647';\n document.body.appendChild(orange);\n document.body.style.display = '';\n })();","args":[]}
19:58:57 INFO - 1578340737728 webdriver::server DEBUG <- 200 OK {"value":null}
19:58:59 INFO - 1578340739778 webdriver::server DEBUG -> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"return document.documentURI;","args":[]}
19:58:59 INFO - 1578340739815 webdriver::server DEBUG <- 200 OK {"value":"about:blank"}
19:58:59 INFO - 1578340739816 webdriver::server DEBUG -> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"(function() {\n const orange = document.getElementById('browsertime-orange');\n if (orange) {\n orange.parentNode.removeChild(orange);\n }\n window.requestAnimationFrame(function(){\n window.requestAnimationFrame(function(){\n window.location=\"https://www.amazon.com\";\n });\n });\n })();","args":[]}
19:58:59 INFO - 1578340739844 webdriver::server DEBUG <- 200 OK {"value":null}
19:59:00 INFO - 1578340740349 webdriver::server DEBUG -> POST /session/8f330118-4d84-4f0b-af2c-2352fd46d41e/execute/sync {"script":"return document.documentURI;","args":[]}
19:59:00 INFO - 1578340740383 webdriver::server DEBUG <- 200 OK {"value":"https://www.amazon.com/"}
19:59:00 INFO - [2020-01-06 14:59:00] DEBUG: [browsertime] Waiting for script pageCompleteCheck at most 300000 ms
|
|
Reporter | |
Comment 31•5 years ago
•
|
||
As for getting raptor-browsertime (with visual metrics) running on fennec (v68) in production, here's a try push:
Locally (with ^ patch applied) I can run raptor-browsertime on fennec v68 with:
./mach raptor --test amazon --app fennec --binary org.mozilla.fennec_aurora --activity org.mozilla.gecko.BrowserApp --browser-cycles 2 --browsertime --cold
It does run successfully and grabs the page-load measurements (using the mitm recordings) however it looks like it isn't able to get 'timeToContentfulPaint' on fennec v68:
20:22:52 INFO - results.MissingResultsError: Browsertime cycle missing timeToContentfulPaint measurement
And also there's a webdriver error when trying to shutdown fennec:
20:18:51 INFO - [2020-01-06 15:18:51] DEBUG: [browsertime] Telling browser to quit.
20:18:51 INFO - 1578341931322 webdriver::server DEBUG -> DELETE /session/33182f50-f282-4ec0-adb4-818852dfbd44
20:18:51 INFO - 1578341931325 webdriver::server DEBUG <- 500 Internal Server Error {"value":{"error":"unsupported operation","message":"Only supported in Firefox","stacktrace":"WebDriverError@chrome://marionette/content/error.js:175:5\nUnsupportedOperationError@chrome://marionette/content/error.js:493:5\nassert.that/<@chrome://marionette/content/assert.js:428:13\nassert.firefox@chrome://marionette/content/assert.js:97:57\nGeckoDriver.prototype.quit@chrome://marionette/content/driver.js:3459:10\ndespatch@chrome://marionette/content/server.js:305:40\nexecute@chrome://marionette/content/server.js:275:16\nonPacket/<@chrome://marionette/content/server.js:248:20\nonPacket@chrome://marionette/content/server.js:249:9\n_onJSONObjectReady/<@chrome://marionette/content/transport.js:503:20\n"}}
|
|
Reporter | |
Comment 32•5 years ago
|
||
(In reply to Andrew Creskey [:acreskey] [he/him] from comment #29)
I'm seeing the same behaviour -- no orange screen.
Rob, I noticed the--visualMetrics falsein that command -- is that wanted?
It doesn't resolve the missing orange though.
Thanks Andrew! Yes, my understanding is we don't enable --visualMetrics on the browsertime command line b/c we are doing our own visual metrics analysis in the *-vismet treeherder/tc task. On Fenix in production (webrender disabled) where there is an orange screen (and visual metrics appear to be working great) I verified the same --visualMetrics false is on the browsertime cmd line there too.
|
||
Updated•5 years ago
|
|
||
Comment 33•5 years ago
|
||
This is more of a meta-bug so I'm unassigning myself from it.
|
|
||
Comment 34•5 years ago
|
||
Greg: Is there anything to do here that isn't already covered by bug 1574182? If not, let's close this and reduce the number of meta bugs for Browsertime on Android.
|
|
||
Comment 35•5 years ago
|
||
I don't think so, we can always reopen this is needed.
|
||
Updated•5 years ago
|
Description
•