1618566 - [raptor][YTP] Change the development server URL with the production server URL for Youtube playback tests once 2020 tests have been stabilized

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Marian Raiciof [:marauder]]

I am working on https://phabricator.services.mozilla.com/D62546
and for the moment i am using a development server URL to be able to run the tests.
This has to be changed to production server once we have all in place.

Comment 1

[Marian Raiciof [:marauder]]

INI files test_url values:

• raptor-youtube-playback.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-hfr-test&raptor=true&exclude=1,2&muted=true&command=run

• raptor-youtube-playback-sfr-av1.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-sfr-av1-test&raptor=true&exclude=1,2&muted=true&command=run

• raptor-youtube-playback-sfr-h264.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run

• raptor-youtube-playback-sfr-vp9.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run

• raptor-youtube-playback-widevine-hfr.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-widevine-hfr-test&raptor=true&exclude=1,2&muted=true&command=run

• raptor-youtube-playback-widevine-sfr-h264.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-widevine-sfr-h264-test&raptor=true&exclude=1,2&muted=true&command=run

• raptor-youtube-playback-widevine-sfr-vp9.ini
  -- DEV = http://ec2-52-214-163-93.eu-west-1.compute.amazonaws.com/testing_2020/perf-youtube-playback/2019/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run
  -- PROD = http://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-widevine-sfr-vp9-test&raptor=true&exclude=1,2&muted=true&command=run

---

I added a new domain "://.amazonaws.com/*" , after the line below:
https://searchfox.org/mozilla-central/rev/b2ccce862ef38d0d150fcac2b597f7f20091a0c7/testing/raptor/webext/raptor/manifest.json#70
that should be removed once we have all we need on the production server

Comment 2

[Marian Raiciof [:marauder]]

The dot has been removed from the test names on branch 2020 and is harder to read.
I opened this issue on youtube's github repo, maybe they will add it back:
https://github.com/youtube/js_mse_eme/issues/48

Another solution is to create a new patch where the dot can be added in output.py when we do the parsing for youtube tests results, by changing the _sub variable using some regexes :
https://searchfox.org/mozilla-central/rev/b2ccce862ef38d0d150fcac2b597f7f20091a0c7/testing/raptor/raptor/output.py#1218

Or we can do that in mozilla's github repo for YTP - on raptor branch:
https://github.com/mozilla/perf-youtube-playback/tree/raptor
after the changes for 2020 will be merged.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This bug is currently blocked and cannot be worked on. So lets get it out of the P1's for now.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Greg and Dave, I want to remind that someone has to continue the work here. Right now we operate on a dev server that is not maintained by IT. If something breaks it will cause test bustage across all the trees.

So what needs to be done:

a) Figuring out if we want a 2019/2020 subfolder in the Youtube playback repository

1. File a bug so that IT can update the media files
2. Update the raptor branch on the Youtube playback repo with the 2020 branch
3. The production server will pick it up - manually check that all is working
4. Switch the Raptor tests over to the production server

I'm happy to assist / help out with the process but someone has to do the actual work.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This was discussed yesterday. Greg's planning to pick this up in July.

Comment 6

[Greg Mierzwinski [:sparky]]

Jira ticket for this: https://jira.mozilla.com/browse/FXP-1095

Comment 7

[Greg Mierzwinski [:sparky]]

The jira issue is solved, we are now able to use the 2020 prod URL. There's an issue in the prod server though: https://github.com/mozilla/perf-youtube-playback/issues/22

Once that's resolved, we can migrate the widevine tests. For now, all the non-widevine tests can be migrated.

Comment 8

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1618566 - Use prod server instead of dev for youtube-playback tests. r?#perftest

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/20240998bb95
Use prod server instead of dev for youtube-playback tests. r=perftest-reviewers,Bebe

Comment 10

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/20240998bb95