Intermittent opt-in-blocks.https.html | opt_in_method: http-csp

RESOLVED WORKSFORME

Status

()

Core
DOM: Security
RESOLVED WORKSFORME
2 years ago
2 years ago

People

(Reporter: KWierso, Unassigned)

Tracking

({intermittent-failure})

45 Branch
intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox45 affected)

Details

(Whiteboard: [domsecurity-intermittent])

Attachments

(1 attachment)

if we have one instance in 50 days, is this intermittent or a random glitch in the system?

Comment 2

2 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* mozilla-release: 1
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* linux32: 4
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-01-04&endday=2016-01-10&tree=all
I believe this is related to a bug in the SSL implementation or in the ServerSocket abstraction in the Python standard library.  I have managed to consistently reproduce this when the that particular test is allowed to pass the CSP check in cases where it should be blocked.

Comment 5

2 years ago
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 3
* try: 1
* mozilla-inbound: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux32: 4
* linux64: 2
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-02-08&endday=2016-02-14&tree=all

Comment 6

2 years ago
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 3
* mozilla-inbound: 2
* mozilla-beta: 1
* mozilla-aurora: 1

Platform breakdown:
* linux32: 3
* linux64: 2
* windows8-64: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-02-22&endday=2016-02-28&tree=all

Comment 7

2 years ago
Joel, do you know if something in our python config change recently?  This is spiking on Mac OS 10.10 today.
Flags: needinfo?(jmaher)
(In reply to Ben Kelly [:bkelly] from comment #7)
> Joel, do you know if something in our python config change recently?  This
> is spiking on Mac OS 10.10 today.

We landed Bug 1122236 yesterday - which is very likely causing this problem. Let me investigate.
thanks for being on top of this bkelly.  I am not aware of anything, but :ckerschb looks to have a good suspect.  Please ping me if there is anything I can do to help on this!
Flags: needinfo?(jmaher)

Comment 10

2 years ago
25 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 7
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 24
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-16&endday=2016-03-16&tree=all

Comment 11

2 years ago
30 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 15
* fx-team: 9
* mozilla-central: 5
* try: 1

Platform breakdown:
* osx-10-10: 30

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-17&endday=2016-03-17&tree=all

Comment 12

2 years ago
Christoph, I'm assigning to you based on comment 8.
Assignee: nobody → mozilla
Status: NEW → ASSIGNED

Comment 13

2 years ago
31 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 20
* fx-team: 8
* try: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 31

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-18&endday=2016-03-18&tree=all

Comment 14

2 years ago
19 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* mozilla-central: 4
* fx-team: 4

Platform breakdown:
* osx-10-10: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-19&endday=2016-03-19&tree=all

Comment 15

2 years ago
127 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 73
* fx-team: 34
* mozilla-central: 13
* try: 7

Platform breakdown:
* osx-10-10: 125
* windowsxp: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-14&endday=2016-03-20&tree=all
Christoph, this is still a common problem and if we do not have time to work on this, can we turn these tests off?
Flags: needinfo?(mozilla)
(In reply to Andreas Tolfsen ‹:ato› from comment #3)
> I believe this is related to a bug in the SSL implementation or in the
> ServerSocket abstraction in the Python standard library.  I have managed to
> consistently reproduce this when the that particular test is allowed to pass
> the CSP check in cases where it should be blocked.

Ato, how did you reproduce this? The link in comment 0 is not accessible anymore. Did you just run

> ./mach web-platform-tests testing/web-platform/tests/mixed-content/blockable/http-csp/cross-origin-http/fetch-request/top-level/keep-scheme-redirect/opt-in-blocks.https.html

I have tried that several times but couldn't reproduce on Linux. Is it OS specific?

(In reply to Joel Maher (:jmaher) from comment #16)
> Christoph, this is still a common problem and if we do not have time to work
> on this, can we turn these tests off?

I'll give it a try today to see what we can come up with. If we don't have anything by the end of the day we can disable the test for now. We have written 3 of our own mochitests, so we still would have test coverage for the feature. Most likely it is as Ato said, and the problem is not related to the CSP blocking but rather to some SSL error.
Flags: needinfo?(mozilla) → needinfo?(ato)
this looks to be osx 10.10 debug specific, that makes it more difficult.

I would be curious why we don't have other python SSL issues in our automation, could it just be this test is the problem, or maybe the wpt harness has some different dependencies than our other automation?

if we look at orange factor:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-14&endday=2016-03-20&tree=all

we can see logs, here is an example:
https://treeherder.mozilla.org/logviewer.html#?repo=try&job_id=18330912
I agree that the SSLError in the logs is troublesome, but I don't think it has anything to do with this bug; there are multiple tests between the last SSLError and this intermittent, and the SSLError predates the recent increase in intermittency here.

Yes, wpt uses different infrastructure here.
Joel, James, I am confused. Can you help me sort something out. Have a look at [1]. In fact all of these tests should pass and there shouldn't be any *.ini file (besides potentially for link-prefetch where our mixed content implementation does not precisely follow the spec).

Within Bug 1122236 we landed support for 'block-all-mixed-content'. So all of these test should be enabled and should pass. Is there a simple switch to enable them and push to TRY or do we have to manually delete all those *.ini files?

[1] http://mxr.mozilla.org/mozilla-central/search?string=[opt-in-blocks.https.html]
Flags: needinfo?(jmaher)
Flags: needinfo?(james)
For example I was running the following test locally (on mac):
> ./mach web-platform-tests testing/web-platform/tests/mixed-content/allowed/http-csp/same-host-https/picture-tag/top-level/keep-scheme-redirect/allowed.https.html

The test itself passes but then I get:
> Expected FAIL, got PASS

So I suppose we can just enable them, the reason I am so confused is because how come those tests just fail intermittently?
So, bearing in mind that I don't know CSP or this particular set of tests very well…

The ini files describe various things, including the expected outcome of the tests and whether they are disabled or not. Note that these things are orthogonal; we still run tests that we expect to fail, and signal an issue if they do not in fact fail.

AFAICT, only one mixed-content test is actually disabled: /mixed-content/blockable/http-csp/same-host-http/picture-tag/top-level/no-redirect/opt-in-blocks.https.html. 


If there are other tests that still have ini files then they are presumably not passing. This seems like something you need to investigate (I can help with questions about the testing framework + etc.). If you have a branch where tests are passing then yes, removing the ini files might be the right thing to do (but a Try push will tell you that because there will be lots of TEST-UNEXPECTED-PASS).

Understanding why the disabled test is intermittent is very much the point of this bug :)
Flags: needinfo?(james)
Whiteboard: [domsecurity-intermittent]

Comment 23

2 years ago
25 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 15
* fx-team: 7
* mozilla-central: 2
* try: 1

Platform breakdown:
* osx-10-10: 25

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-21&endday=2016-03-21&tree=all
:jgraham, how can this test be iterated on in try server?  for wpt tests the .ini files to toggle the states are confusing and it is hard to know how to hack it so we can just run a single directory of tests and play with the expected states.  Can you comment on how we determined this test should be expected fail?

Also from looking at the log, this looks to be one specific case of opt-in-blocks.https.html:
https://dxr.mozilla.org/mozilla-central/search?q=mixed-content%2Fblockable%2Fhttp-csp%2Fsame-host-http%2Fpicture-tag%2Ftop-level%2Fno-redirect%2Fopt-in-blocks.https.html&redirect=false&case=false

I don't understand why we have 10+ different instances of the same test in the tree, or at least .ini files for it.
Flags: needinfo?(jmaher) → needinfo?(james)
If you want to run just a single directory of web-platform-tests you can't just edit the ini files; they aren't manifests like the manifestparser .ini files. 

The supported way of running one directory of tests on try is |mach try <trychooser syntax> path/to/tests|

The determination of the expected behaviour of tests is done by running them on import and adjusting the ini files so that there are no unexpected results.

There aren't multiple copies of the same test in the tree. There are multiple tests that have the same filename, but different paths, which have some features in common and some differences (these tests are all generated from a script with each component of the test path corresponding to one option in the input data).
Flags: needinfo?(james)
Thanks for the info :jgraham.  The problem here is this is a osx 10.10 issue, so this needs to be run on try- any tips for running a single directory on try?
(In reply to Joel Maher (:jmaher) from comment #26)
> Thanks for the info :jgraham.  The problem here is this is a osx 10.10
> issue, so this needs to be run on try- any tips for running a single
> directory on try?

I think you misread my comment above :)

(James Graham [:jgraham] from comment #25)
> The supported way of running one directory of tests on try is |mach try
> <trychooser syntax> path/to/tests|
ah, cool!  thanks for pointing out my obvious oversight
Created attachment 8733551 [details] [diff] [review]
bug_1222277_intermittent_mcb.patch

James, when applying the attached patch and running tests locally I get the following errors (see underneath). Before continuing to debug I would like to understand why I get 'NOTRUN' on lower two test directories. Is there anything I am missing or I need to enable to make those tests run?

> /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/form-tag

 0:26.50 TEST_END: Thread-TestrunnerManager-1 ERROR, expected OK
'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base.py", line 133, in run_test
    result = self.do_test(test)
  File "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/executormarionette.py", line 316, in do_test
    return self.convert_result(test, data)
  File "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base.py", line 58, in __call__
    result_url, status, message, stack, subtest_results = result
TypeError: 'NoneType' object is not iterable

==================================================

> /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/link-css-tag

/link-css-tag/top-level/no-redirect/opt-in-blocks.https.html",2,null,null,[["opt_in_method: http-csp\n                                 origin: cross-origin-http\n                                 source_scheme: https\n                                 context_nesting: top-level\n                                 redirection: no-redirect\n                                 subresource: link-css-tag\n                                 expectation: blocked",3,null,null]]]}]"
 0:54.32 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK. Subtests passed 0/1. Unexpected 2
opt_in_method: http-csp
                                 origin: cross-origin-http
                                 source_scheme: https
                                 context_nesting: top-level
                                 redirection: no-redirect
                                 subresource: link-css-tag
                                 expectation: blocked
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Expected PASS, got NOTRUN

[Parent]
--------
Expected OK, got TIMEOUT

==================================================

> /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/object-tag

 0:54.46 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK. Subtests passed 0/1. Unexpected 2
opt_in_method: http-csp
                                 origin: cross-origin-http
                                 source_scheme: https
                                 context_nesting: top-level
                                 redirection: keep-scheme-redirect
                                 subresource: object-tag
                                 expectation: blocked
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Expected PASS, got NOTRUN

[Parent]
--------
Expected OK, got TIMEOUT

====================================================
Flags: needinfo?(james)

Comment 30

2 years ago
29 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 13
* fx-team: 10
* try: 4
* mozilla-central: 2

Platform breakdown:
* osx-10-10: 29

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-22&endday=2016-03-22&tree=all
Flags: needinfo?(ato)

Comment 31

2 years ago
31 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 9
* mozilla-central: 4
* try: 1

Platform breakdown:
* osx-10-10: 30
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-23&endday=2016-03-23&tree=all

Comment 32

2 years ago
26 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 18
* fx-team: 6
* try: 1
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 26

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-24&endday=2016-03-24&tree=all

Comment 33

2 years ago
24 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 8
* try: 3
* mozilla-central: 3

Platform breakdown:
* osx-10-10: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-25&endday=2016-03-25&tree=all

Comment 34

2 years ago
15 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 5
* try: 4
* fx-team: 4
* mozilla-central: 2

Platform breakdown:
* osx-10-10: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-26&endday=2016-03-26&tree=all

Comment 35

2 years ago
188 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 96
* fx-team: 52
* try: 24
* mozilla-central: 16

Platform breakdown:
* osx-10-10: 187
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-21&endday=2016-03-27&tree=all

Comment 36

2 years ago
26 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* try: 8
* fx-team: 5
* mozilla-central: 2

Platform breakdown:
* osx-10-10: 26

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-28&endday=2016-03-28&tree=all

Comment 37

2 years ago
24 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 16
* fx-team: 6
* try: 1
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-29&endday=2016-03-29&tree=all

Comment 38

2 years ago
30 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 15
* try: 7
* fx-team: 7
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 30

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-30&endday=2016-03-30&tree=all
(In reply to Christoph Kerschbaumer [:ckerschb, back on April 11th] from comment #29)
> Created attachment 8733551 [details] [diff] [review]
> bug_1222277_intermittent_mcb.patch
> 
> James, when applying the attached patch and running tests locally I get the
> following errors (see underneath). Before continuing to debug I would like
> to understand why I get 'NOTRUN' on lower two test directories. Is there
> anything I am missing or I need to enable to make those tests run?

NOTRUN is a status that means that a test was created with a call to async_test or similar, but no step was ever run for that test. This typically indicates that either other code (including tests) took so long to run that the timeout was reached before any of the steps of that test were run, or that other code threw an exception before any of the steps of that test were run.

> > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/form-tag
> 
>  0:26.50 TEST_END: Thread-TestrunnerManager-1 ERROR, expected OK

OK, so it looks like something threw an exception here.

> 'NoneType' object is not iterable
> Traceback (most recent call last):
>   File
> "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base.
> py", line 133, in run_test
>     result = self.do_test(test)
>   File
> "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/
> executormarionette.py", line 316, in do_test
>     return self.convert_result(test, data)
>   File
> "/home/ckerschb/moz/mc/testing/web-platform/harness/wptrunner/executors/base.
> py", line 58, in __call__
>     result_url, status, message, stack, subtest_results = result
> TypeError: 'NoneType' object is not iterable

This looks like a problem with the test harness.

> 
> ==================================================
> 
> > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/link-css-tag
> 
> /link-css-tag/top-level/no-redirect/opt-in-blocks.https.html",2,null,null,
> [["opt_in_method: http-csp\n                                 origin:
> cross-origin-http\n                                 source_scheme: https\n  
> context_nesting: top-level\n                                 redirection:
> no-redirect\n                                 subresource: link-css-tag\n   
> expectation: blocked",3,null,null]]]}]"
>  0:54.32 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK.
> Subtests passed 0/1. Unexpected 2
> opt_in_method: http-csp
>                                  origin: cross-origin-http
>                                  source_scheme: https
>                                  context_nesting: top-level
>                                  redirection: no-redirect
>                                  subresource: link-css-tag
>                                  expectation: blocked
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> -----------------------------------------------------------
> Expected PASS, got NOTRUN
> 
> [Parent]
> --------
> Expected OK, got TIMEOUT

This case is a timeout before any steps of the subtest were run.

> ==================================================
> 
> > /testing/web-platform/meta/mixed-content/blockable/http-csp/cross-origin-http/object-tag
> 
>  0:54.46 TEST_END: Thread-TestrunnerManager-1 Harness TIMEOUT, expected OK.
> Subtests passed 0/1. Unexpected 2
> opt_in_method: http-csp
>                                  origin: cross-origin-http
>                                  source_scheme: https
>                                  context_nesting: top-level
>                                  redirection: keep-scheme-redirect
>                                  subresource: object-tag
>                                  expectation: blocked
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> ------------------------------------------------------------------
> Expected PASS, got NOTRUN
> 
> [Parent]
> --------
> Expected OK, got TIMEOUT

This case looks similar to the one above.
Flags: needinfo?(james)
Should be fixed by what just landed over in bug 1205654.
Assignee: mozilla → ryanvm
Depends on: 1205654

Comment 41

2 years ago
28 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 12
* fx-team: 10
* try: 5
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-04-01&endday=2016-04-01&tree=all

Comment 42

2 years ago
122 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 58
* fx-team: 32
* try: 26
* mozilla-central: 6

Platform breakdown:
* osx-10-10: 122

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1222277&startday=2016-03-28&endday=2016-04-03&tree=all
nice, this seems to have stopped on April 1st, not sure if that is an April fools' joke !
Assignee: ryanvm → nobody
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.