1527620 - Add youtube streaming tests

Status: RESOLVED FIXED

(Core :: Audio/Video, enhancement, P2)

Comment 1

[Tarek Ziadé (:tarek)]

Adding a Youtube streaming test is done through the following steps:

• build a proxy server that’s able to fully replay videos stream with various combination of quality changed during playback
• package the proxy server as a standalone python app containing all data
• integrate the app in the taskcluster/tooltool chain
• build a test fixture in media/ that starts the proxy locally
• create a test that verifies that a video stream played correctly (end reached)
• provide a simple semi-automated way to update the recorded video

Extended features:

• full automation of the proxy recording
• add metrics to track the streaming progress

Comment 2

[Tarek Ziadé (:tarek)]

Pushed a first prototype of the proxy server at https://github.com/tarekziade/youtube-test

Comment 3

[Tarek Ziadé (:tarek)]

I can push the test in the tree at this point. Since it's a very specific python-based test, I was thinking about creating

dom/media/tests/streams/

It roughly consists of porting the https://github.com/tarekziade/youtube-test package in there, that will provide a simple function to run a stream test against Youtube.

Since it's a tool rather than a test, maybe it could be somewhere else but I am not sure where...

the layout could be:

• dom/media/tests/streams/yytest/<the package>
• dom/media/tests/streams/test_youtube_one.py
• dom/media/tests/streams/test_youtube_two.py
• dom/media/tests/streams/<more tests>

And in the future, it could contain tests for other video platforms.

Jya?

Comment 4

[Jean-Yves Avenard [:jya]]

dom/media/test as base directory

dom/media/tests is an exception that shouldn't be there (currently its used for webrtc)

I would suggest to have an extra folder path to show that it's external , but "streams" is probably enough

Comment 5

[Tarek Ziadé (:tarek)]

Attachment: Bug 1527620 - Add youtube streaming tests - r?whimboo

Work in progress but should already work.

To run the Youtube streaming test:

./mach stream-test dom/media/test/streams/test_youtube.py

I still have a ton of things to clean up and fix,
but this is the reference patch I will update iften
if you need to run a stream test - so don't use the github
repo anymore.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So we have those Youtube streaming tests currently running as part of the Python tests, which works locally but will fail in CI because this job doesn't depend on a build job. Also the purpose of those tests are not to test the Python code, but the media performance in Firefox. As such I feel that the Python tests suite is not the best one.

All the other media tests seem to use the mochitest harness, and having those tests hooked up under that too would be good. But I wonder if that is possible due to the requirements we have here, and which are the following:

• Firefox build (dependent on the build job)
• mozproxy to setup and run a proxy server (mitmproxy)
• pre-downloaded video files as served through the proxy for playback later (could be a dependency task)
• custom preferences (proxy, new tab, autoplay - could go into testing/profiles)
• submission of recorded data to perfherder

I wonder if we could generate a special media mochitest job for the streaming tests, if the Mochitest harness could handle all the requirements as given above (Tarek, if I forgot one please add it).

Geoff, what do you think about the harness selection? Sounds that reasonable to do as Mochitest? I assume that I would have to add the mozproxy integration for it, and maybe some more code for the results submission. If it's not doable we could still use a job based on Marionette.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Geoff, please see my last comment. I just forgot to set the flag. Thanks!

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #6)

Geoff, what do you think about the harness selection? Sounds that reasonable to do as Mochitest? I assume that I would have to add the mozproxy integration for it, and maybe some more code for the results submission. If it's not doable we could still use a job based on Marionette.

Mochitest does so much, I would be hesitant to add more special-case code to it; I think you can make it work, but it doesn't feel like a great match.

When I hear about mitmproxy, I think of raptor, especially if results are being submitted to perfherder. Might that be appropriate? (:rwood can tell you more about raptor)

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, that what we finally also came up on IRC earlier today before I had to quickly leave and haven't had the time to update this bug. I will check tomorrow if Raptor would be appropriate and how to get the tests running with that harness.

Comment 10

[Tarek Ziadé (:tarek)]

Since Raptor is using the same reqs and roughly doing the same work to run the test (calling mozproxy, etc) that would work well, as long as we stick with the current Python-based model that injects the test using marionette

Comment 11

[Tarek Ziadé (:tarek)]

try https://treeherder.mozilla.org/#/jobs?repo=try&revision=ccf99da65e6cebb6cf0dadce1c6c8c3205892a94

Comment 12

[Tarek Ziadé (:tarek)]

As suggested by Jean-Yves, let's make sure this test catches https://bugzilla.mozilla.org/show_bug.cgi?id=1513511

Comment 13

[Tarek Ziadé (:tarek)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0f7e2646085cd5d76e62966f23181c5c61c1f593

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/304b957afe29
Add youtube streaming tests - r=whimboo

Comment 15

[Narcis Beleuzu [:NarcisB]]

Backed out for marionette-media failures on test_youtube.py

Backout link: https://hg.mozilla.org/integration/autoland/rev/cf6c7c082dc3a53f2c054567a7089c710982ba02
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237530622&repo=autoland&lineNumber=845

Comment 16

[Tarek Ziadé (:tarek)]

Looking into this (worked well on my mach try prior to landing...)

Comment 17

[Tarek Ziadé (:tarek)]

Triggering a bunch of MnMsto make sure this time we don't have any intermittent

https://treeherder.mozilla.org/#/jobs?repo=try&revision=43cdf6fdde680ac668a5ff513758b3eb3f790ee9

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a54e5e2802d7
Add youtube streaming tests - r=whimboo

Comment 19

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/a54e5e2802d7