1539111 - Add support for video streaming tests in Raptor

Status: RESOLVED INCOMPLETE

(Testing :: Raptor, enhancement, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

For both bug 1539090, and bug 1539092 we want to test media streaming performance with Raptor. As of now only page-load, and benchmark tests exist. Maybe the streaming tests could be run via the benchmark test type by implementing our own custom benchmark class. Otherwise a new test type should be used.

For the latter option it would be good to have an argument to define the number of iterations to improve measuring the performance/quality.

The documentation at https://wiki.mozilla.org/Performance_sheriffing/Raptor would need an update for this addition.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

For now I'm trying to setup a simple HTML page which can be used to drive the video streaming tests. This page will contain an embedded video via an iframe, which in case of YouTube is the default option when right click the video and select Copy embeded video.

The problem now is that both domains are different and that I don't have access to the video element inside the frame from the outer benchmark site. I wonder if it would be possible to use something like X-Frame-Options but which doesn't require a local web server to allow accessing the video element even it has a different origin. So far I wasn't lucky to find anything.

Olli, is something like that possible? Thanks a lot!

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

And a note through... we don't want to embed the video element itself, but have to use the YouTube JS client to play the video.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

Cross domain accesses aren't possible in web pages (except similar-origin when playing with .domain).
Not sure why one needs to access the video element, but if that is needed, you need to have some privileged JS running in the parent document, or parent document needs to communicate with some privileged code which then queries the needed information from iframe.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Olli Pettay [:smaug] from comment #3)

Not sure why one needs to access the video element, but if that is needed, you need to have some privileged JS running in the parent document, or parent document needs to communicate with some privileged code which then queries the needed information from iframe.

It's needed to retrieve the media statistics like dropped frames and such. So it means the only two ways to get this done (as I can see it now) is by using Marionette/geckodriver which runs privileged JS code in a framescript, or maybe by allowing the Raptor WebExtension to inject code in the YouTube page to retrieve the values and report it to Raptor.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As pointed out by Olli there is also a YouTube postMessage API: https://developers.google.com/youtube/iframe_api_reference

Sadly it is not helpful in retrieving the video statistics. So it's not an option for me.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Actually the Raptor webextension already injects a measuring script into various pages:
https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/measure.js

I assume that I can use that one and retrieve video performance metrics by just adding the code I need.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The video streaming tests we can make use of can be found at:
https://ytlr-cert.appspot.com/2019/main.html?test_type=playbackperf-test#1555317292453

As agreed in the last video streaming meeting we will make use of that page. Therefore Tarek will setup an AWS instance which duplicates that website so that we have a copy of it under our own control. We may not have the full videos but trim those to 20s.

For now I will setup a benchmark test in Raptor which will make use of the above page. Later we can switch the instances.

I hope that the current test page will allow us to easily trigger the wanted tests, and to get all the results. If not I might have to contribute some patches.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks to Tarek we have our own clone of the testing page now:

http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With the YT test page I have problems with geckoview_example on FireTV stick to auto-play the video when the testharness runs video.play(). The same works fine in Firefox, so I wonder if there is an additional preference or setting for geckoview to stop blocking autoplay media files? At least the testing/profiles/common/user.js states it should be all fine.

Test page:
http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test&command=run:8

Common user prefs:
https://searchfox.org/mozilla-central/rev/d143f8ce30d1bcfee7a1227c27bf876a85f8cede/testing/profiles/common/user.js#53-58

Chris, could you help out? Thanks

Comment 10

[Chris Pearce [:cpearce (Not reading bugmail)]]

media.autoplay.default=0 should be disabling autoplay blocking in GeckoView. Toggling that pref in about:config unblocks autoplay for me in GVE. Are you sure your GVE harness is loading that user.js pref file?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #10)

media.autoplay.default=0 should be disabling autoplay blocking in GeckoView. Toggling that pref in about:config unblocks autoplay for me in GVE. Are you sure your GVE harness is loading that user.js pref file?

Oh, you are right. It looks like that this pref somehow doesn't get set to 0 when I run my PoC for streaming tests through Raptor, even it included the common user.js preferences. I will have a look why this isn't correctly used.

Also when I start geckoview_example via adb shell, modify the pref, and force-stop the application after a while, the preference isn't retained. It looks like that the app doesn't auto-save preferences regularly but only during a regular quit.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #10)

media.autoplay.default=0 should be disabling autoplay blocking in GeckoView. Toggling that pref in about:config unblocks autoplay for me in GVE. Are you sure your GVE harness is loading that user.js pref file?

As seen in bug 1528236 comment 4 the user.js file as generated is not completely obeyed by geckoview_example, and a lot of preferences are missing after its shutdown in prefs.js. If you have ideas please let me know over on the other bug, otherwise I'm going to further explore why that is happening.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1539111 - [raptor] Temporarily disable the proxy server.

Depends on D29048

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1539111 - [raptor] WIP: Add streaming tests to Raptor

Depends on D29049

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #12)

(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #10)

media.autoplay.default=0 should be disabling autoplay blocking in GeckoView. Toggling that pref in about:config unblocks autoplay for me in GVE. Are you sure your GVE harness is loading that user.js pref file?

As seen in bug 1528236 comment 4 the user.js file as generated is not completely obeyed by geckoview_example, and a lot of preferences are missing after its shutdown in prefs.js. If you have ideas please let me know over on the other bug, otherwise I'm going to further explore why that is happening.

The underlying problem can be found on bug 1547717. It's GeckoRuntimeSettings which is blocking this preference from being set via prefs.js or user.js.

nalexander pointed me to the experimental API for webextensions, which I could use here to allow getting/setting of browser preferences.

Comment 16

[Tarek Ziadé (:tarek)]

About the live site feature

https://searchfox.org/mozilla-central/source/testing/raptor/raptor/manifest.py#41-54

https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#446

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As discussed in the streaming meeting today, the YT playback performance tests will no longer be a specific video streaming test type, but included as a benchmark test (see bug 1548845). The type of test as covered by this bug will be used for extended streaming tests in the near future.