1554967 - Run raptor-youtube-playback jobs for Chrome in CI

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It would be nice to have the playback performance tests running against Chromium to get comparison numbers with Firefox on desktop and mobile.

Given that also Chrome/Chromium has blocking auto-play turned on, we will have to run those tests muted.

We could already get the tests enabled by specifying the same test_url as for android, or we wait until bug 1554966 has been done, so we know if we have to run with audio enabled.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a enhancement, but please change it back in case of error.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:jmaher do you have any concerns over turning on additional tests for Chromium? We could consider running these once/week.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

a few jobs once/week isn't going to kill things- especially on a weekend :)

Comment 5

[Marian Raiciof [:marauder]]

Attachment: Bug 1554967 - Run raptor-youtube-playback jobs for Chrome in CI

Created a cron job for this task and added changes to yml and ini files.

Comment 6

[Marian Raiciof [:marauder]]

This cron job will generate the following target_tasks.json:

[

"test-windows10-64-shippable/opt-raptor-youtube-playback-chromium-e10s",
"test-macosx1014-64-shippable/opt-raptor-youtube-playback-chromium-e10s",
"test-linux64-shippable/opt-raptor-youtube-playback-chromium-e10s",
"test-windows7-32-shippable/opt-raptor-youtube-playback-chromium-e10s"

]

Comment 7

[Marian Raiciof [:marauder]]

Push to try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ede6e786526e38e1939f1fba5ebb880460ca7048

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Can you please provide an update here :marauder?

Comment 9

[Marian Raiciof [:marauder]]

Hi Dave,

I did several tests locally using a local server and the repository from https://github.com/mozilla/perf-youtube-playback
I tested on both branches master and raptor.

The tests are running ok on Chrome but is a perma fail on Chromium.

A few more details for Chrome:
https://phabricator.services.mozilla.com/D41553#1385409

More details for Chromium:
https://phabricator.services.mozilla.com/D41553#1396207

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Let's switch this to Chrome then as that's our preferred target anyway. We'll just need to depend on Greg's work in bug 1563708.

Comment 11

[Marian Raiciof [:marauder]]

Finished the transition from Chromium to Chrome.
Push to try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cb87698cdbfe3b1e93e5b23dcab2046b9ab317b9

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Marian: Can you provide an update on this?

Comment 13

[Marian Raiciof [:marauder]]

Hi Dave,

I did a git bisection locally for the youtube playback repo: https://phabricator.services.mozilla.com/D41553#1499340
to find what is causing the permanent failures on Chrome, but i didn't find any issues with the tests.
And it seems that what we have running on the production server (https://yttest.prod.mozaws.net) has some issues.

https://phabricator.services.mozilla.com/D41553#1523519
Henrik asked Tarek to bring up the server that ran a while back at
http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test
to be able to do that bisection and run on the server not on localhost.

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

From experience, mentions in Phabricator rarely get the attention of the person. It's best to use needinfo in bugzilla. Tarek: could you take a look over the comment from Henrik in https://phabricator.services.mozilla.com/D41553#1523519

Comment 15

[Tarek Ziadé (:tarek)]

Thanks Dave, sorry for not answering in phab. I have restarted the youtube dev instance I had (luckily I did not erase it!)

http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test is back online

@marauder let me know if I can be of further assistance - I can give you access to that dev box if you need.

If you can't reach the URL (dns just changed) replace the domain by ec2-34-244-207-63.eu-west-1.compute.amazonaws.com

Comment 16

[Marian Raiciof [:marauder]]

Hei Tarek,

I can access that URL but i will also need that dev box credentials to jump on raptor branch and check each commit to see which one caused the permanent failures on Chrome.

Thanks!

Comment 17

[Tarek Ziadé (:tarek)]

creds sent by email

Comment 18

[Marian Raiciof [:marauder]]

I rebased the patch and did a push to try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8fb25c17dfbcf8f75c1078413b701a851ddcec32

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bfd693dd19ef
Run raptor-youtube-playback jobs for Chrome in CI r=perftest-reviewers,sparky

Comment 20

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/bfd693dd19ef

Comment 21

[Marian Raiciof [:marauder]]

The cron job is running, the results for Dec 11 will be available here:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedJob=280658921&tier=1%2C2%2C3&fromchange=f83f2771414cf5938a46f44e1b11cdcd5181ea0f&searchStr=ytp

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Results are totally worse for Chrome on Mac:

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=mozilla-central,2189198,1,10&series=mozilla-central,2054686,1,10&timerange=86400

I assume that this might be the same problem with the Mac Minis and Firefox. I don't have that other bug handy. Do you have? Might be good to add this info over there.

How do the other platforms compare with Firefox?

Comment 23

[Marian Raiciof [:marauder]]

Hey Henrik,

Here are the results for various platforms:
https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=mozilla-central,2054686,1,10&series=mozilla-central,2044136,1,10&series=mozilla-central,2044425,1,10&series=mozilla-central,2045003,1,10&series=mozilla-central,2043847,1,10&series=mozilla-central,2044714,1,10&series=mozilla-central,2043558,1,10&timerange=2592000

Mac OS 10 has the highest values (if we exclude the results for Windows 10 Aarch64).

Can you give more details about Mac Minis ?

Thanks!

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I'm not working on this project anymore for a couple of months. So please work with stakeholders of your team or developers to continue. Also why don't you follow-up on my comment from bug 1604207?