Closed Bug 1222277 Opened 9 years ago Closed 9 years ago

Intermittent opt-in-blocks.https.html | opt_in_method: http-csp

Categories

(Core :: DOM: Security, defect)

Product:
Core
Component:
DOM: Security
Version:
45 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox45 --- affected

People

(Reporter: KWierso, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [domsecurity-intermittent])

Attachments

(1 file)

bug_1222277_intermittent_mcb.patch
10.14 KB, patch
Details | Diff | Splinter Review
if we have one instance in 50 days, is this intermittent or a random glitch in the system?
I believe this is related to a bug in the SSL implementation or in the ServerSocket abstraction in the Python standard library. I have managed to consistently reproduce this when the that particular test is allowed to pass the CSP check in cases where it should be blocked.
Joel, do you know if something in our python config change recently? This is spiking on Mac OS 10.10 today.
Flags: needinfo?(jmaher)
(In reply to Ben Kelly [:bkelly] from comment #7) > Joel, do you know if something in our python config change recently? This > is spiking on Mac OS 10.10 today. We landed Bug 1122236 yesterday - which is very likely causing this problem. Let me investigate.
thanks for being on top of this bkelly. I am not aware of anything, but :ckerschb looks to have a good suspect. Please ping me if there is anything I can do to help on this!
Flags: needinfo?(jmaher)
Christoph, I'm assigning to you based on comment 8.
Assignee: nobody → mozilla
Status: NEW → ASSIGNED
Christoph, this is still a common problem and if we do not have time to work on this, can we turn these tests off?
Flags: needinfo?(mozilla)
(In reply to Andreas Tolfsen ‹:ato› from comment #3) > I believe this is related to a bug in the SSL implementation or in the > ServerSocket abstraction in the Python standard library. I have managed to > consistently reproduce this when the that particular test is allowed to pass > the CSP check in cases where it should be blocked. Ato, how did you reproduce this? The link in comment 0 is not accessible anymore. Did you just run > ./mach web-platform-tests testing/web-platform/tests/mixed-content/blockable/http-csp/cross-origin-http/fetch-request/top-level/keep-scheme-redirect/opt-in-blocks.https.html I have tried that several times but couldn't reproduce on Linux. Is it OS specific? (In reply to Joel Maher (:jmaher) from comment #16) > Christoph, this is still a common problem and if we do not have time to work > on this, can we turn these tests off? I'll give it a try today to see what we can come up with. If we don't have anything by the end of the day we can disable the test for now. We have written 3 of our own mochitests, so we still would have test coverage for the feature. Most likely it is as Ato said, and the problem is not related to the CSP blocking but rather to some SSL error.
Flags: needinfo?(mozilla) → needinfo?(ato)
this looks to be osx 10.10 debug specific, that makes it more difficult. I would be curious why we don't have other python SSL issues in our automation, could it just be this test is the problem, or maybe the wpt harness has some different dependencies than our other automation? if we look at orange factor: https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-14&endday=2016-03-20&tree=all we can see logs, here is an example: https://treeherder.mozilla.org/logviewer.html#?repo=try&job_id=18330912
I agree that the SSLError in the logs is troublesome, but I don't think it has anything to do with this bug; there are multiple tests between the last SSLError and this intermittent, and the SSLError predates the recent increase in intermittency here. Yes, wpt uses different infrastructure here.
Joel, James, I am confused. Can you help me sort something out. Have a look at [1]. In fact all of these tests should pass and there shouldn't be any *.ini file (besides potentially for link-prefetch where our mixed content implementation does not precisely follow the spec). Within Bug 1122236 we landed support for 'block-all-mixed-content'. So all of these test should be enabled and should pass. Is there a simple switch to enable them and push to TRY or do we have to manually delete all those *.ini files? [1] http://mxr.mozilla.org/mozilla-central/search?string=[opt-in-blocks.https.html]
Flags: needinfo?(jmaher)
Flags: needinfo?(james)
For example I was running the following test locally (on mac): > ./mach web-platform-tests testing/web-platform/tests/mixed-content/allowed/http-csp/same-host-https/picture-tag/top-level/keep-scheme-redirect/allowed.https.html The test itself passes but then I get: > Expected FAIL, got PASS So I suppose we can just enable them, the reason I am so confused is because how come those tests just fail intermittently?
So, bearing in mind that I don't know CSP or this particular set of tests very well… The ini files describe various things, including the expected outcome of the tests and whether they are disabled or not. Note that these things are orthogonal; we still run tests that we expect to fail, and signal an issue if they do not in fact fail. AFAICT, only one mixed-content test is actually disabled: /mixed-content/blockable/http-csp/same-host-http/picture-tag/top-level/no-redirect/opt-in-blocks.https.html. If there are other tests that still have ini files then they are presumably not passing. This seems like something you need to investigate (I can help with questions about the testing framework + etc.). If you have a branch where tests are passing then yes, removing the ini files might be the right thing to do (but a Try push will tell you that because there will be lots of TEST-UNEXPECTED-PASS). Understanding why the disabled test is intermittent is very much the point of this bug :)
Flags: needinfo?(james)
Whiteboard: [domsecurity-intermittent]
:jgraham, how can this test be iterated on in try server? for wpt tests the .ini files to toggle the states are confusing and it is hard to know how to hack it so we can just run a single directory of tests and play with the expected states. Can you comment on how we determined this test should be expected fail? Also from looking at the log, this looks to be one specific case of opt-in-blocks.https.html: https://dxr.mozilla.org/mozilla-central/search?q=mixed-content%2Fblockable%2Fhttp-csp%2Fsame-host-http%2Fpicture-tag%2Ftop-level%2Fno-redirect%2Fopt-in-blocks.https.html&redirect=false&case=false I don't understand why we have 10+ different instances of the same test in the tree, or at least .ini files for it.
Flags: needinfo?(jmaher) → needinfo?(james)
If you want to run just a single directory of web-platform-tests you can't just edit the ini files; they aren't manifests like the manifestparser .ini files. The supported way of running one directory of tests on try is |mach try <trychooser syntax> path/to/tests| The determination of the expected behaviour of tests is done by running them on import and adjusting the ini files so that there are no unexpected results. There aren't multiple copies of the same test in the tree. There are multiple tests that have the same filename, but different paths, which have some features in common and some differences (these tests are all generated from a script with each component of the test path corresponding to one option in the input data).
Flags: needinfo?(james)
Thanks for the info :jgraham. The problem here is this is a osx 10.10 issue, so this needs to be run on try- any tips for running a single directory on try?
(In reply to Joel Maher (:jmaher) from comment #26) > Thanks for the info :jgraham. The problem here is this is a osx 10.10 > issue, so this needs to be run on try- any tips for running a single > directory on try? I think you misread my comment above :) (James Graham [:jgraham] from comment #25) > The supported way of running one directory of tests on try is |mach try > <trychooser syntax> path/to/tests|
ah, cool! thanks for pointing out my obvious oversight
James, when applying the attached patch and running tests locally I get the following errors (see underneath). Before continuing to debug I would like to understand why I get 'NOTRUN' on lower two test directories. Is there anything I am missing or I need to enable to make those tests run? > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/form-tag 0:26.50 TEST_END: Thread-TestrunnerManager-1 ERROR, expected OK 'NoneType' object is not iterable Traceback (most recent call last): File "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base.py", line 133, in run_test result = self.do_test(test) File "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/executormarionette.py", line 316, in do_test return self.convert_result(test, data) File "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base.py", line 58, in __call__ result_url, status, message, stack, subtest_results = result TypeError: 'NoneType' object is not iterable ================================================== > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/link-css-tag /link-css-tag/top-level/no-redirect/opt-in-blocks.https.html",2,null,null,[["opt_in_method: http-csp\n origin: cross-origin-http\n source_scheme: https\n context_nesting: top-level\n redirection: no-redirect\n subresource: link-css-tag\n expectation: blocked",3,null,null]]]}]" 0:54.32 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK. Subtests passed 0/1. Unexpected 2 opt_in_method: http-csp origin: cross-origin-http source_scheme: https context_nesting: top-level redirection: no-redirect subresource: link-css-tag expectation: blocked ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Expected PASS, got NOTRUN [Parent] -------- Expected OK, got TIMEOUT ================================================== > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/object-tag 0:54.46 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK. Subtests passed 0/1. Unexpected 2 opt_in_method: http-csp origin: cross-origin-http source_scheme: https context_nesting: top-level redirection: keep-scheme-redirect subresource: object-tag expectation: blocked -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Expected PASS, got NOTRUN [Parent] -------- Expected OK, got TIMEOUT ====================================================
Flags: needinfo?(james)
Flags: needinfo?(ato)
(In reply to Christoph Kerschbaumer [:ckerschb, back on April 11th] from comment #29) > Created attachment 8733551 [details] [diff] [review] > bug_1222277_intermittent_mcb.patch > > James, when applying the attached patch and running tests locally I get the > following errors (see underneath). Before continuing to debug I would like > to understand why I get 'NOTRUN' on lower two test directories. Is there > anything I am missing or I need to enable to make those tests run? NOTRUN is a status that means that a test was created with a call to async_test or similar, but no step was ever run for that test. This typically indicates that either other code (including tests) took so long to run that the timeout was reached before any of the steps of that test were run, or that other code threw an exception before any of the steps of that test were run. > > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/form-tag > > 0:26.50 TEST_END: Thread-TestrunnerManager-1 ERROR, expected OK OK, so it looks like something threw an exception here. > 'NoneType' object is not iterable > Traceback (most recent call last): > File > "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base. > py", line 133, in run_test > result = self.do_test(test) > File > "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/ > executormarionette.py", line 316, in do_test > return self.convert_result(test, data) > File > "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base. > py", line 58, in __call__ > result_url, status, message, stack, subtest_results = result > TypeError: 'NoneType' object is not iterable This looks like a problem with the test harness. > > ================================================== > > > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/link-css-tag > > /link-css-tag/top-level/no-redirect/opt-in-blocks.https.html",2,null,null, > [["opt_in_method: http-csp\n origin: > cross-origin-http\n source_scheme: https\n > context_nesting: top-level\n redirection: > no-redirect\n subresource: link-css-tag\n > expectation: blocked",3,null,null]]]}]" > 0:54.32 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK. > Subtests passed 0/1. Unexpected 2 > opt_in_method: http-csp > origin: cross-origin-http > source_scheme: https > context_nesting: top-level > redirection: no-redirect > subresource: link-css-tag > expectation: blocked > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ----------------------------------------------------------- > Expected PASS, got NOTRUN > > [Parent] > -------- > Expected OK, got TIMEOUT This case is a timeout before any steps of the subtest were run. > ================================================== > > > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/object-tag > > 0:54.46 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK. > Subtests passed 0/1. Unexpected 2 > opt_in_method: http-csp > origin: cross-origin-http > source_scheme: https > context_nesting: top-level > redirection: keep-scheme-redirect > subresource: object-tag > expectation: blocked > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ------------------------------------------------------------------ > Expected PASS, got NOTRUN > > [Parent] > -------- > Expected OK, got TIMEOUT This case looks similar to the one above.
Flags: needinfo?(james)
Should be fixed by what just landed over in bug 1205654.
Assignee: mozilla → ryanvm
Depends on: 1205654
nice, this seems to have stopped on April 1st, not sure if that is an April fools' joke !
Assignee: ryanvm → nobody
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: