1387997 - Check all web platform tests to make sure none of them are disabled because of the old data: URI inheritance model

Status: RESOLVED WORKSFORME

(Core :: DOM: Security, enhancement, P2)

Description

[Christoph Kerschbaumer [:ckerschb]]

We should check:
* for any kind of *.ini files to make sure we have enabled the maximum number of tests possible for the new data: URI inheritance model.

Comment 1

[Christoph Kerschbaumer [:ckerschb]]

Hey Henry,

thanks for loooking into this. I think this will be some semi-automated-manual task. E.g. within testing/web-platform we can do various greps, like e.g:
  grep -r "FAIL" . | grep "data"
  grep -r "NOTRUN" . | grep "data"

What we are looking for are all tests that somehow might include some data: URI that are masked as:
expected: NOTRUN
expected: FAIL

E.g. one of them that I just encountered is a disabled data test for XMLHttpRequest [1]. Not sure if that is relevant to this project or not, but probably worth taking a closer look. What I have been doing for some wpt tests, I explicitly flip the pref before running the test when I encountered tests that we failing/disabled (see [2]). Hope that helps to get you started. Feel free to ping me on IRC/slack.

[1] https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/XMLHttpRequest/data-uri.htm.ini#3
[2] https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/html/browsers/origin/origin-of-data-document.html.ini#3

Comment 2

[Henry Chang [:hchang]]

Thanks for your suggestions, Christoph!

I think I can start from flipping the pref and get all "unexpected success"
test cases.

For those "disabled" test cases... I hope there will be few :p

Thanks!

(In reply to Christoph Kerschbaumer [:ckerschb] from comment #1)
> Hey Henry,
> 
> thanks for loooking into this. I think this will be some
> semi-automated-manual task. E.g. within testing/web-platform we can do
> various greps, like e.g:
>   grep -r "FAIL" . | grep "data"
>   grep -r "NOTRUN" . | grep "data"
> 
> What we are looking for are all tests that somehow might include some data:
> URI that are masked as:
> expected: NOTRUN
> expected: FAIL
> 
> E.g. one of them that I just encountered is a disabled data test for
> XMLHttpRequest [1]. Not sure if that is relevant to this project or not, but
> probably worth taking a closer look. What I have been doing for some wpt
> tests, I explicitly flip the pref before running the test when I encountered
> tests that we failing/disabled (see [2]). Hope that helps to get you
> started. Feel free to ping me on IRC/slack.
> 
> [1]
> https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/
> XMLHttpRequest/data-uri.htm.ini#3
> [2]
> https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/meta/
> html/browsers/origin/origin-of-data-document.html.ini#3

Comment 3

[Christoph Kerschbaumer [:ckerschb]]

James, we are about to flip the pref |security.data_uri.unique_opaque_origin| which means that data: URIs loaded inside and iframe or object will get their own unique, opaque origin. For this effort we updated several wpt tests (See dependencies of Bug 1377244). I would imagine that we encountered all the NOTRUNS and EXPECTED_FAIL cases just by flipping the pref and pushing to TRY, right? Additionaly I did some manual verification (e.g. trying to inspect as many .ini fails that are related as possible), but that's a very cumbersome process.

I think my question is: are there cases for wpt tests that wouldn't show up as a failure on TRY with the pref flipped? I just want to make sure we get the best test coverage we can possibly get and not missing something - thanks!

Comment 4

[James Graham [:jgraham]]

The try run would show any tests that change behaviour due to this change, excluding any that are currently disabled. It seems unlikely that tests were disabled for our behaviour here (usually tests are only disabled because of intermittency), so hopefully there isn't anything else, but [1] is more or less a list of tests that are disabled if you want to investigate further.

[1] http://searchfox.org/mozilla-central/search?q=disabled%3A&case=true&path=testing%2Fweb-platform%2Fmeta

Comment 5

[Christoph Kerschbaumer [:ckerschb]]

Thanks James, I tried to manually verify that we have the best test coverage for wpt tests. I think we can mark this bug as closed.