Closed Bug 1554967 Opened 2 years ago Closed 1 year ago

Run raptor-youtube-playback jobs for Chrome in CI

Categories

(Testing :: Raptor, task, P2)

task

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: whimboo, Assigned: marauder)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

It would be nice to have the playback performance tests running against Chromium to get comparison numbers with Firefox on desktop and mobile.

Given that also Chrome/Chromium has blocking auto-play turned on, we will have to run those tests muted.

We could already get the tests enabled by specifying the same test_url as for android, or we wait until bug 1554966 has been done, so we know if we have to run with audio enabled.

Bugbug thinks this bug is a enhancement, but please change it back in case of error.

Type: defect → enhancement
Type: enhancement → task
Priority: -- → P2

:jmaher do you have any concerns over turning on additional tests for Chromium? We could consider running these once/week.

Flags: needinfo?(jmaher)

a few jobs once/week isn't going to kill things- especially on a weekend :)

Flags: needinfo?(jmaher)
Duplicate of this bug: 1563736
Assignee: nobody → marian.raiciof
Status: NEW → ASSIGNED

Created a cron job for this task and added changes to yml and ini files.

This cron job will generate the following target_tasks.json:

[

"test-windows10-64-shippable/opt-raptor-youtube-playback-chromium-e10s",
"test-macosx1014-64-shippable/opt-raptor-youtube-playback-chromium-e10s",
"test-linux64-shippable/opt-raptor-youtube-playback-chromium-e10s",
"test-windows7-32-shippable/opt-raptor-youtube-playback-chromium-e10s"

]

Can you please provide an update here :marauder?

Flags: needinfo?(marian.raiciof)

Hi Dave,

I did several tests locally using a local server and the repository from https://github.com/mozilla/perf-youtube-playback
I tested on both branches master and raptor.

The tests are running ok on Chrome but is a perma fail on Chromium.

A few more details for Chrome:
https://phabricator.services.mozilla.com/D41553#1385409

More details for Chromium:
https://phabricator.services.mozilla.com/D41553#1396207

Flags: needinfo?(marian.raiciof)

Let's switch this to Chrome then as that's our preferred target anyway. We'll just need to depend on Greg's work in bug 1563708.

Depends on: 1563708
Summary: Run raptor-youtube-playback jobs for Chromium in CI → Run raptor-youtube-playback jobs for Chrome in CI
Attachment #9084697 - Attachment description: Bug 1554967 - Run raptor-youtube-playback jobs for Chromium in CI. → Bug 1554967 - Run raptor-youtube-playback jobs for Chrome in CI

Marian: Can you provide an update on this?

Blocks: 1582756
Flags: needinfo?(marian.raiciof)

Hi Dave,

I did a git bisection locally for the youtube playback repo: https://phabricator.services.mozilla.com/D41553#1499340
to find what is causing the permanent failures on Chrome, but i didn't find any issues with the tests.
And it seems that what we have running on the production server (https://yttest.prod.mozaws.net) has some issues.

https://phabricator.services.mozilla.com/D41553#1523519
Henrik asked Tarek to bring up the server that ran a while back at
http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test
to be able to do that bisection and run on the server not on localhost.

Flags: needinfo?(marian.raiciof)

From experience, mentions in Phabricator rarely get the attention of the person. It's best to use needinfo in bugzilla. Tarek: could you take a look over the comment from Henrik in https://phabricator.services.mozilla.com/D41553#1523519

Flags: needinfo?(tarek)

Thanks Dave, sorry for not answering in phab. I have restarted the youtube dev instance I had (luckily I did not erase it!)

http://yttest.dev.mozaws.net/2019/main.html?test_type=playbackperf-test is back online

@marauder let me know if I can be of further assistance - I can give you access to that dev box if you need.

If you can't reach the URL (dns just changed) replace the domain by ec2-34-244-207-63.eu-west-1.compute.amazonaws.com

Flags: needinfo?(tarek) → needinfo?(marian.raiciof)

Hei Tarek,

I can access that URL but i will also need that dev box credentials to jump on raptor branch and check each commit to see which one caused the permanent failures on Chrome.

Thanks!

Flags: needinfo?(marian.raiciof) → needinfo?(tarek)

creds sent by email

Flags: needinfo?(tarek)
See Also: → 1597207
Depends on: 1597207
See Also: 1597207
Pushed by mraiciof@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bfd693dd19ef
Run raptor-youtube-playback jobs for Chrome in CI r=perftest-reviewers,sparky
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73

Results are totally worse for Chrome on Mac:

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=mozilla-central,2189198,1,10&series=mozilla-central,2054686,1,10&timerange=86400

I assume that this might be the same problem with the Mac Minis and Firefox. I don't have that other bug handy. Do you have? Might be good to add this info over there.

How do the other platforms compare with Firefox?

Flags: needinfo?(marian.raiciof)
Flags: needinfo?(marian.raiciof)

I'm not working on this project anymore for a couple of months. So please work with stakeholders of your team or developers to continue. Also why don't you follow-up on my comment from bug 1604207?

See Also: → 1576107
You need to log in before you can comment on or make changes to this bug.