1597207 - [YTP] Youtube Playback Tests are permanent failures on Chrome, while running on production server

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, defect)

Description

[Marian Raiciof [:marauder]]

Attachment: YTP_permanent_failure_on_Chrome_prod_server.png

We have some problems with youtube playback tests on Chrome, these are permanent failures.
https://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true#1573735176237

I did several tests locally using an apache server and the tests are passing.
I talked to Henrik Skupin and Tarek and i also tested on a development server using an up to date repository and the tests are still passing on Chrome.
http://ec2-34-244-207-63.eu-west-1.compute.amazonaws.com/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true#1573735196019

I think that there might be a problem on the production server.
Tested on Chrome version 72.0.3626.119 (Oficial Build) (64-bit)

---

Wei Zhou's opinion on this issue:

The failure might be due to the following error, which can be found in the browser console,

Refused to load media from 'blob:https://yttest.prod.mozaws.net/a844229d-cfe9-4e96-8397-9d6eaa858a79' because it violates the following Content Security Policy directive: "default-src 'none'". Note that 'media-src' was not explicitly set, so 'default-src' is used as a fallback.

Comment 1

[:wezhou]

Attachment: chrome console errors

I took a closer look at this issue, and here is what I've found,

Given the link `https://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true#1573735176237`,

1. In Chrome version 78.0.3904.97 (Official Build) (64-bit), all test videos failed to play, and the errors are similar. Take test 3 as an example, the error is in https://bugzilla.mozilla.org/attachment.cgi?id=9110152.
2. In Firefox version 70.0.1 (64-bit), the same error isn't reproducible and most videos can play.

Because the first error says,

```
/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true#1574223651753:1 Refused to load media from 'blob:https://yttest.prod.mozaws.net/006bfaa7-c0fc-42a8-aa6a-01c55228cbae' because it violates the following Content Security Policy directive: "default-src 'none'". Note that 'media-src' was not explicitly set, so 'default-src' is used as a fallback.
```

I suspect maybe Chrome and Firefox treats "media-src" differently in certain cases. We can set "media-src self;" and see it fixes the issue.

Comment 2

[:wezhou]

I filed https://github.com/mozilla/perf-youtube-playback/pull/7 for it.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thank you :wezhou. I just merged it to the raptor branch and as such it should be present soon on the production instance.

Marian, can you please try in about 1h if the problem still persists or if it is fixed now?

Comment 4

[Marian Raiciof [:marauder]]

Henrik, Wei Zhou,

I tried a few minutes ago and the result is the same: permanent failures for all tests.

URL:
https://yttest.prod.mozaws.net/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true

Error messages in console:

Refused to load media from 'blob:<URL>' because it violates the following Content Security Policy directive: "media-src 'self'".

96 errors like these below (for each test that ran):
/2019/main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&muted=true#1574339347308:1 Refused to load media from 'blob:https://yttest.prod.mozaws.net/ee772ea7-6255-429d-9edf-bf2c01ab1362' because it violates the following Content Security Policy directive: "media-src 'self'".

Uncaught Test failure: The video could not be loaded, either because the server or network failed or because the format is not supported.
Uncaught (in promise) DOMException

I refreshed the site a few times, I cleared all browser data (cookies, cache, etc) - same result.
I can try later in case the changes take longer to apply.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/media-src:

'self'
    Refers to the origin from which the protected document is being served, including the same URL scheme and port number. You must include the single quotes. Some browsers specifically exclude blob and filesystem from source directives. Sites needing to allow these content types can specify them using the Data attribute.

Looks like Chrome is part of this browsers list.

Comment 6

[:wezhou]

Filed https://github.com/mozilla/perf-youtube-playback/pull/8 to add blob: to media-src.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Marian, did you had a chance to test again? The fix went in already 3 days ago.

Comment 8

[Marian Raiciof [:marauder]]

Hi Henrik,

I saw your comment about the revert and i thought Wei Zhou will create another PR.

I tested a few minutes ago and the tests are still failing :

Refused to load media from 'blob:<URL>' because it violates the following Content Security Policy directive: "media-src 'self'".

Refused to load media from 'blob:https://yttest.prod.mozaws.net/16c70933-1390-4dc5-acc5-8ecb43a1b15a' because it violates the following Content Security Policy directive: "media-src 'self'".

Refused to load media from 'blob:https://yttest.prod.mozaws.net/585f7b15-581f-43b1-97dd-045cd091a703' because it violates the following Content Security Policy directive: "media-src 'self'".
....

Uncaught Test failure: The video could not be loaded, either because the server or network failed or because the format is not supported.
Uncaught (in promise) DOMException

I checked different tabs and only the tests that don't use videos nor images are passing, like:
DOM CSS Tests
DOM Document Tests
etc.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh you are right! I totally missed that. Wezhou can you please create a new PR against the Raptor branch please?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No need to recreate the issue. I cherry-picked the changes and fixed the merge issues: https://github.com/mozilla/perf-youtube-playback/commit/e7c1648157fccdfcbbd530a6fb150974292ce480

Changes will be live on production soon. Please re-test.

Comment 11

[Marian Raiciof [:marauder]]

Good news!
The tests are working now on Chrome.

And when they fail we have errors like:

• for red rectangular:
  TestExecutor: Test 71:PlaybackPerf.VP9.1440p30@2X threw an error: Assert failed: Total dropped frames is (89) which should be less than or equal to (1)
  Uncaught Assert failed: Total dropped frames is (89) which should be less than or equal to (1)

• for white rectangular with red border:
  TestExecutor: Test 80:PlaybackPerf.H264.2160p30@2X threw an error: Assert failed: Total dropped frames is (258) which should be less than or equal to (1)
  Uncaught Assert failed: Total dropped frames is (258) which should be less than or equal to (1)

Image attached.

Comment 12

[Marian Raiciof [:marauder]]

Attachment: YTP_working_on_Chrome.png

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Great to hear. So that looks fine then, and there is no more work to be done on this bug! As such I'm closing it as fixed.