1554966 - Investigate running raptor-youtube-playback tests always muted on desktop

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Currently we run the tests muted on Android but not on desktop. It should be investigated if we can run those tests muted everywhere. Therefore the tests have to be run in CI with the Gecko profiler being enabled.

As mentioned by Paul Adenot it shouldn't make a difference. If numbers validate that then run the tests muted across all platforms.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:marauder could you trigger some try jobs to compare running these tests on desktop muted vs audible?

Comment 2

[Marian Raiciof [:marauder]]

Hi Dave,

I enabled the Gecko profiler and changed the url to cover both scenarios: muted vs audible

Here are the links for try:

• Muted = true:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=5e6c79cd95aeb519b023478b1d2db6f30879a9dc

• Muted = false
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d52418adba9068be3d3f4c99e4e26458758c9ed

• Compare link:
  https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=5e6c79cd95aeb519b023478b1d2db6f30879a9dc&newProject=try&newRevision=9d52418adba9068be3d3f4c99e4e26458758c9ed&framework=10

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Paul: are you able to compare the profiles to confirm if there is any difference running muted?
Marian: could you also trigger try runs without the gecko profiler with a few retriggers so we can compare results via the compare view?

Comment 4

[Marian Raiciof [:marauder]]

Hi Dave,
I did 5 retriggers for each job in the previous try runs.

I triggered new jobs without the gecko profiler (removed from raptor-youtube-playback-firefox):

• Muted = false :
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=562e65a1114452ad34dc0a6b0c5943c34f284cbd

• Muted = true :
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=a209d7cbe08314f7a4482a4ea1ac54418db59575
  I'll do the retriggers when the try runs will finish.

• Compare link:
  https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=562e65a1114452ad34dc0a6b0c5943c34f284cbd&newProject=try&newRevision=a209d7cbe08314f7a4482a4ea1ac54418db59575&framework=10

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The comparison view is at https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=562e65a1114452ad34dc0a6b0c5943c34f284cbd&newProject=try&newRevision=a209d7cbe08314f7a4482a4ea1ac54418db59575&framework=10

It looks like muting might have a positive impact on the performance for linux, and a slightly negative impact on other platforms. In all cases the confidence is low, so I think we can proceed with running the tests muted so long as Paul has no concerns.

Comment 6

[Paul Adenot (:padenot)]

This is fine, as long as you compare the same things. Why do we want to mute the videos again?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Paul Adenot (:padenot) from comment #6)

This is fine, as long as you compare the same things. Why do we want to mute the videos again?

Because with GeckoView there is no way yet to get around the problem that autoplay is blocked. As such using muted playback everywhere the numbers across platforms should be closer.

Comment 8

[Marian Raiciof [:marauder]]

Attachment: Bug 1554966 - Investigate running raptor-youtube-playback tests always muted on desktop

Based on the conclusion from the bug mentioned above, we will run youtube playback always muted on desktop.

Comment 9

[Marian Raiciof [:marauder]]

Push to try :
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1561cfe7c062dde757eeda2bf5367eb5d371cf4b

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d87604ba6a87
Investigate running raptor-youtube-playback tests always muted on desktop r=perftest-reviewers,sparky

Comment 11

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/d87604ba6a87