Bugzilla

(In reply to Daniel Veditz [:dveditz] from comment #15)

> (In reply to Andrea Marchesini [:baku] from comment #4)
> > This is a 'known' issue. We have web-platform tests, (disabled of course) here:
> > https://searchfox.org/mozilla-central/source/testing/web-platform/meta/html/semantics/embedded-content/the-canvas-element
> 
> Did we file a security bug when we turned these off?

No. There are several problems with requiring/expecting that:

* I (and others on my team) who are responsible for maintining the sync with upstream don't have the correct expertise or appropriate resources to identify which wpt failures may represent security issues. Even in "obvious" cases (like if the test is evidently intended to check SOP correctness) there are many reasons for failure other than a security issue.

* Most tests aren't disabled if they fail. Typically disabling is only used in cases where the test is too unstable to run in CI. In the case that the test is marked as expected:FAIL there is generally no review whatsoever, and these are just as likely to represent security issues as other tests.

The way that we need to deal with this is for platform teams to actively monitor the wpt tests that fail (or otherwise aren't passing) in their components. https://github.io/jgraham/wptdash is intended to help with this, and the intent is that we will be able to add annotations to the test results linking them with bugs, so teams are able to triage the Firefox failures (or at least the Firefox-only failures) and so keep the number of untriaged issues to zero, just like we do for incoming bug reports. As a side effect that should allow us to identify potential security issues faster.

So I fully agree that we should be more actively looking at the results of these tests, not only for security bugs, and there are efforts ongoing to make that more workable for platform teams.

(In reply to Daniel Veditz [:dveditz] from comment #15)

> (In reply to Andrea Marchesini [:baku] from comment #4)
> > This is a 'known' issue. We have web-platform tests, (disabled of course) here:
> > https://searchfox.org/mozilla-central/source/testing/web-platform/meta/html/semantics/embedded-content/the-canvas-element
> 
> Did we file a security bug when we turned these off?

No. There are several problems with requiring/expecting that:

* I (and others on my team) who are responsible for maintining the sync with upstream don't have the correct expertise or appropriate resources to identify which wpt failures may represent security issues. Even in "obvious" cases (like if the test is evidently intended to check SOP correctness) there are many reasons for failure other than a security issue.

* Most tests aren't disabled if they fail. Typically disabling is only used in cases where the test is too unstable to run in CI. In the case that the test is marked as expected:FAIL there is generally no review whatsoever, and these are just as likely to represent security issues as other tests.

The way that we need to deal with this is for platform teams to actively monitor the wpt tests that fail (or otherwise aren't passing) in their components. https://jgraham.github.io/wptdash is intended to help with this, and the intent is that we will be able to add annotations to the test results linking them with bugs, so teams are able to triage the Firefox failures (or at least the Firefox-only failures) and so keep the number of untriaged issues to zero, just like we do for incoming bug reports. As a side effect that should allow us to identify potential security issues faster.

So I fully agree that we should be more actively looking at the results of these tests, not only for security bugs, and there are efforts ongoing to make that more workable for platform teams.