Open Bug 937025 Opened 11 years ago Updated 2 years ago

Improve W3C CSP test suite coverage

Categories

(Core :: DOM: Security, defect, P3)

Product:
Core
Component:
DOM: Security
Type:
defect

Tracking

()

People

(Reporter: freddy, Unassigned)

References

(Blocks 1 open bug,
URL
)

Details

(Whiteboard: [domsecurity-backlog3])

Attachments

(1 file)

9vs16.png
174.82 KB, image/png
Details
Attached image 9vs16.png —
Nightly is at 100%, but it sees only 9 tests, whereas chromium sees 16. (I have attached a screenshot to highlight this) We should investigate and improve this. NB: There are some bugs in the test runner, which might cause some additional trouble to track this. We can work around some missing tests due to issue 17 <https://github.com/hillbrad/CSP/issues/17> by running tests individually (in tabs).
Rerunning this in not-nightly on my main profile gives me 16 of 16 tests working. Are we finding a regression here?
(In reply to Frederik Braun [:freddyb] from comment #1) > Are we finding a regression here? Maybe. Are you getting assertions or errors in the error console or command line via debug build? (Enable csp debugging in about:config for more data.)
So there are two things: First, there must be a bug in the test runner, given that we get HTTP 500 for some URLs (and http://www.webappsec-test.info/web-platform-tests/CSP/support/checkReportJs.php seems like an important one ;)) Secondly, I am getting more tests now. Why, I don't know; but it's 22 (aurora) or 21 (nightly) now. On both versions, three tests are failing, which have also been discussed at the working group. They involve bypassing eval/inline restrictions by using blob URIs. Worker created from inline text and loaded via blob URI should not run with policy "default-src *": http://webappsec-test.info/web-platform-tests/CSP/script-src/CSP_1_9.php Script created from inline text and loaded via blob URI should not run with policy "default-src *": http://webappsec-test.info/web-platform-tests/CSP/script-src/CSP_1_11.php Eval created via blob should not run with policy: "script-src blob: www.webappsec-test.info". http://webappsec-test.info/web-platform-tests/CSP/script-src/CSP_1_11_3.php I did not see any interesting errors (despite the HTTP 500s and the obvious CSP violation messages) or assertion failures in the console with csp.debug enabled :(
Steve: can you figure out whether this is issues in gecko or in the test suite?
Flags: needinfo?(sworkman)
Steve, any chance you've made progress on this?
Summary: Improve CSP test suite coverage → Improve W3C CSP test suite coverage
Looks like it is working pretty well to me on a recent Nightly. http://w3c-test.org/tools/runner/index.html?path=/content-security-policy I know there was one outstanding issue with FF on media-src that I need to resubmit a new PR on. (https://github.com/w3c/web-platform-tests/commit/d63e743981749d005ee6de246be26dbc1a3c5ee4) Also, Blink has contributed a lot of tests: https://github.com/w3c/web-platform-tests/pull/1635 They rely on nonstandard console events, but a quick eyeball makes me think they might be amenable to mass-scale automated conversion to testharness.js style, I just haven't had the time to dig into that yet.
(In reply to Sid Stamm [:geekboy or :sstamm] from comment #5) > Steve, any chance you've made progress on this? I did not :/ and I'm going to punt this to Kamil :) Kamil, can you do a bit of investigation here, please? Can you try the test suite at http://w3c-test.org/tools/runner/index.html?path=/content-security-policy with Nightly, Aurora, Beta and Release and report back on the total number of tests run etc. Would also be good to note which tests fail.
Flags: needinfo?(sworkman) → needinfo?(kjozwiak)
Ah, I also noticed that the path has changed. The one that Brad gave in comment 6 (same as comment 7) looks like it's where the latest test suite lives. Or at least it says something like that here - http://webappsec-test.info/
Builds Used: * http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-05-21-03-02-04-mozilla-central/ * http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-05-21-00-40-03-mozilla-aurora/ * http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/38.0.5b3/ * http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/38.0.1/ OS Used: - Win 8.1 x64 VM - OSX 10.10.3 x64 Results: m-c: Passed: 59; Failed: 3; Timeouts: 0; Errors: 0 m-a: Passed: 59; Failed: 3; Timeouts: 0; Errors 0 m-b: Passed: 58; Failed: 4; Timeouts: 0; Errors 1 m-r: Passed: 58; Failed: 4; Timeouts: 0; Errors 1 The following timeouts happen on both Win/OSX and on all the fx channels. If you set the timeout multiplier to something larger than the initial value, the timeouts will disappear (used 5 as the value): * /content-security-policy/media-src/media-src-7_1.html * /content-security-policy/media-src/media-src-7_2.html * /content-security-policy/media-src/media-src-7_3.html The following failures occur on all the channels: * /content-security-policy/script-src/script-src-1_4.html ** assert_throws: function "function () { eval('evalRan = true;') }" threw object "Error: call to eval() blocked by CSP" ("Error") expected object "EvalError" ("EvalError") * /content-security-policy/style-src/style-src-3_4.html ** assert_equals: No report sent. expected "" but got "false" * /content-security-policy/media-src/media-src-7_3_2.html ** assert_unreached: Disllowed track source loaded. Reached unreachable code The following failure/error only happens on m-b & m-r: * /content-security-policy/script-src/script-src-1_9.html ** FAIL: Worker created from inline text and loaded via blob URI should not run with policy default-src * ** Error: assert_unreached: script ran Reached unreachable code I enabled security.csp.debug under about:config but didn't see anything in the browser console other than the generic CSP errors. I went through the test suite on Chrome and Canary and only get the following failure: * /content-security-policy/script-src/script-src-1_9.html Let me know if there's anything else I can do! Perhaps investigate our failures further?
Flags: needinfo?(kjozwiak)
There are known issues with the test suite. I have Pull Requests submitted to fix most of these failures in Firefox (some do represent genuine bugs) but they are awaiting review. If someone would like to do the reviews in the Opera Critic tool so I can land them, that would be awesome. (if you need help navigating the review tool, just let me know) Pull Requests: https://github.com/w3c/web-platform-tests/pull/1830 https://github.com/w3c/web-platform-tests/pull/1831 Reviews: https://critic.hoppipolla.co.uk/r/5005 https://critic.hoppipolla.co.uk/r/5008
Component: Security → DOM: Security
Kamil: did those pull requests fix it?
Flags: needinfo?(kjozwiak)
Builds Used: * http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-06-29-03-02-10-mozilla-central/ * http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015-06-29-00-40-08-mozilla-aurora/ * http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/39.0b7/ * http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/38.0.5/ Win 8.1 x64 & OSX 10.10.3 Results: ================================== Passed: 71, Failed: 4, Timeouts: 1, Errors: 0 [m-c, m-a] Passed: 70, Failed: 5, Timeouts: 1, Errors: 1 [m-b, m-r] > * /content-security-policy/script-src/script-src-1_4.html > ** assert_throws: function "function () { eval('evalRan = true;') }" threw > object "Error: call to eval() blocked by CSP" ("Error") expected object > "EvalError" ("EvalError") Still failing on all the channels (nothing has changed) > * /content-security-policy/media-src/media-src-7_3_2.html > ** assert_unreached: Disllowed track source loaded. Reached unreachable code This one is now timing out rather than failing. Even with a high timeout multiplier, the test timeouts every time on all the channels. > * /content-security-policy/style-src/style-src-3_4.html > ** assert_equals: No report sent. expected "" but got "false" Still failing on all the channels (nothing has changed) > * /content-security-policy/script-src/script-src-1_9.html > ** FAIL: Worker created from inline text and loaded via blob URI should not > run with policy default-src * > ** Error: assert_unreached: script ran Reached unreachable code This one is still only failing on m-b and m-r but with a different error than before: * assert_equals: No report sent. expected "" but got "false" * /content-security-policy/script-src/script-src-1_4_2.html [FAIL] ** assert_throws: function "function () { var funq = new Function..." threw object "Error: call to Function() blocked by CSP" ("Error") expected object "EvalError" ("EvalError") This failure is new and didn't happen in previous testing as per comment # 9. It occurring on all the channels. Ubuntu 14.04.2 Results: ======================= Passed: 66, Failed: 8, Timeouts: 2, Erorrs: 0 [m-c, m-a] Passed: 65, Failed: 9, Timeouts: 2, Erorrs: 1 [m-b, m-r] * /content-security-policy/media-src/media-src-7_1.html [FAIL] ** assert_unreached: Media error handler should be triggered for non-allowed domain. Reached unreachable code * /content-security-policy/media-src/media-src-7_2.html [FAIL] ** assert_unreached: Media error handler should be triggered for non-allowed domain. Reached unreachable code * /content-security-policy/media-src/media-src-7_3.html [TIMEOUT] ** changing the timeout multiplier from one to six didn't help resolving the timeout * /content-security-policy/media-src/media-src-7_3_2.html [TIMEOUT] ** changing the timeout multiplier from one to six didn't help resolving the timeout * /content-security-policy/script-src/script-src-1_4.html ** assert_throws: function "function () { eval('evalRan = true;') }" threw object "Error: call to eval() blocked by CSP" ("Error") expected object "EvalError" ("EvalError") * /content-security-policy/script-src/script-src-1_4_2.html ** assert_throws: function "function () { var funq = new Function..." threw object "Error: call to Function() blocked by CSP" ("Error") expected object "EvalError" ("EvalError") * /content-security-policy/style-src/style-src-3_4.html ** assert_equals: No report sent. expected "" but got "false" * /content-security-policy/script-src/script-src-1_9.html ** assert_equals: No report sent. expected "" but got "false" Sid, looks like the fixes from comment #10 didn't really help. We're still getting the same failures as before. I went through Ubuntu this time around and it seems like there's a few more errors on Ubuntu than on Win/OSX (not surprising). Some of them are timeouts that won't pass even when you set the timeout multiplier to something really high.
Flags: needinfo?(kjozwiak)
I can take another pass through some of these - I remember that one of the timeouts was due to the test harness expecting one of either an onerror or onload handler to always fire, but on FF the CSP blocked successfully but onerror never fired for some media subelements like tracks and subtitles.
That would be helpful, Brad! Chris: do you think this is related to the failed CSP tests that were recently disabled?
Flags: needinfo?(mozilla)
(In reply to Sid Stamm [:geekboy or :sstamm] from comment #14) > Chris: do you think this is related to the failed CSP tests that were > recently disabled? Not all of them, but most likely: * /content-security-policy/style-src/style-src-3_4.html * /content-security-policy/script-src/script-src-1_9.html Same error: 'No report sent' See also: Bug 1090203, Bug 1094323, Bug 1172411 for the failing CSP platforms tests We have to look into that. My first guess would be that ::SetRequestContext() [1] is called without a channel, so that mCallingChannelLoadGroup ends up being null which would cause sending the report to fail. But we should investigate, but that's where I would start. [1] http://mxr.mozilla.org/mozilla-central/source/dom/security/nsCSPContext.cpp#572
Flags: needinfo?(mozilla)
I just landed a PR today after investigating all of the current Web Platform Tests failures in Firefox Nightly for CSP Level 2. https://github.com/w3c/web-platform-tests/pull/2126 I believe that all remaining failures are actually non-compliance by the Firefox implementation. A summary of the test coverage so far is available at: https://w3c.github.io/webappsec/implementation_reports/CSP2_implementation_report.html And tests can be run from: http://web-platform.test:8000/tools/runner/index.html?path=/content-security-policy
Kamil, you already did an assessmet (see comment 12), but that was in June, any chance you can re-run those tests? If they are still failing we should get them fixed.
Flags: needinfo?(kjozwiak)
Whiteboard: [domsecurity-backlog]
It looks like the CSP test suite has added a substantial amount of new tests that we're unfortunately failing on. I went through both OSX and Win 10 using fx48.0a1, fx47.0a2, fx46.0b1 and fx45.0. Chris, let me know if I can do anything else to help move this forward.. I've added the results below: OSX 10.11.13 x64 & Windows 10 x64 VM ==================================== * fx48.0a1 Results: (ran x3 different times) ** Passed: 294 Failed: 29 Timeouts: 7 Errors: 1 Not Run: 13 (timeout multiplier set to 5) * fx47.0a2 Results: (ran x3 different times) ** Passed: 294 Failed: 29 Timeouts: 7 Errors: 1 Not Run: 13 (timeout multiplier set to 5) * fx46.0b1 Results: (ran x3 different times) ** Passed: 294 Failed: 29 Timeouts: 7 Errors: 1 Not Run: 13 (timeout multiplier set to 5) * fx45.0 Results: (ran x3 different times) ** Passed: 294 Failed: 29 Timeouts: 7 Errors: 1 Not Run: 13 (timeout multiplier set to 5) Tests from comment # 12 that are still failing or timing out: > /content-security-policy/script-src/script-src-1_4.html [FAIL] > /content-security-policy/media-src/media-src-7_3_2.html [TIMEOUT] > /content-security-policy/script-src/script-src-1_4_2.html [FAIL] New Failures/Errors & Timeouts: > /content-security-policy/blink-contrib-2/form-action-src-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib-2/form-action-src-get-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib-2/form-action-src-redirect-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib-2/plugintypes-notype-data.sub.html [TIMEOUT] > /content-security-policy/blink-contrib-2/plugintypes-notype-url.sub.html [FAIL] > /content-security-policy/blink-contrib-2/plugintypes-nourl-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib-2/scripthash-default-src.sub.html [FAIL] > /content-security-policy/blink-contrib-2/scripthash-unicode-normalization.sub.html [TIMEOUT] > /content-security-policy/blink-contrib-2/scriptnonce-basic-blocked.sub.html [TIMEOUT] > /content-security-policy/blink-contrib-2/stylehash-allowed.sub.html [FAIL] > /content-security-policy/blink-contrib-2/stylehash-basic-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/frame-src-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/frame-src-cross-origin-load.sub.html [FAIL] > /content-security-policy/blink-contrib/media-src-track-block.sub.html [TIMEOUT] > /content-security-policy/blink-contrib/object-src-no-url-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/shared-worker-connect-src-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/worker-connect-src-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/worker-eval-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/worker-function-function-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/worker-importscripts-blocked.sub.html [FAIL] > /content-security-policy/blink-contrib/worker-set-timeout-blocked.sub.html [FAIL] > /content-security-policy/child-src/child-src-blocked.sub.html [FAIL] > /content-security-policy/child-src/child-src-conflicting-frame-src.sub.html [FAIL] > /content-security-policy/child-src/child-src-cross-origin-load.sub.html [FAIL] > /content-security-policy/child-src/child-src-worker-blocked.sub.html [ERROR] > /content-security-policy/media-src/media-src-redir-bug.sub.html [TIMEOUT]
Flags: needinfo?(kjozwiak) → needinfo?(mozilla)
Thanks Kamil for the detailed list, I'll find an owner for that bug that investigates further what's going on.
Flags: needinfo?(mozilla)
Priority: -- → P3
Whiteboard: [domsecurity-backlog] → [domsecurity-backlog3]
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: