1152332 - nsIProtocolProxyService::asyncResolve calls proxy filters synchronously

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[Jesper Hansen]

I was testing a bug in Foxyproxy and Thunderbird chat and found that nsIProtocolProxyService::asyncResolve didn't do what I had expected from its name or documentation if an addon is handling this call.

Thunderbird's chat makes this call:
(snip from http://mxr.mozilla.org/comm-central/source/chat/modules/socket.jsm#174)
let proxyService = Cc["@mozilla.org/network/protocol-proxy-service;1"]
                     .getService(Ci.nsIProtocolProxyService);
this._proxyCancel = proxyService.asyncResolve(uri, this.proxyFlags, this);

This calls the asyncResolve function in the Addon straight away without becoming glorious asyncness. This is proven by the fact that in onProxyAvailable we check that this._proxyCancel has been set.
(snip from http://mxr.mozilla.org/comm-central/source/chat/modules/socket.jsm#350)
   onProxyAvailable: function(aRequest, aURI, aProxyInfo, aStatus) {
     if (!("_proxyCancel" in this)) {
       this.LOG("onProxyAvailable called, but disconnect() was called before.");
       return;
     }

So in order of code execution this happens:
1. asyncResolve called from thunderbird
2. asyncResolve in addon runs
3. onProxyAvailable called from addon
4. onProxyAvailable in Thunderbird runs
5. _proxyCancel isn't set, return 
6. asyncResolve in addon returns
7. _proxyCancel is now set.


In the addon I made this change to make asyncResolve to return, to make it sort of async:
  asyncResolve : function(aURI, aFlags, aCallback) {
    var timer = Components.classes["@mozilla.org/timer;1"].createInstance(Components.interfaces.nsITimer);
    var self = this;
    timer.initWithCallback(function() {self.aasyncResolve(aURI, aFlags, aCallback)},0, Components.interfaces.nsITimer.TYPE_ONE_SHOT);
  },
  aasyncResolve : function(aURI, aFlags, aCallback) {
    ... 
  },

We should document to make it very clear that this async method isn't guaranteed to be async

Comment 1

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 2

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 4

[Shane Caraveo (:mixedpuppy)]

We may need this for bug 1378459, but not entirely certain yet.  Otherwise we definitely need this to fix the synchronous proxy calls in webextensions (bug 1409878).  While the caller of asyncResolve is handled (sort of) async, asyncResolve calls the proxy filters synchronously.

Comment 5

[Nicklas Boman [:smurfd]]

Hi. i could give this a shot after chat in #webextensions

Comment 6

[Nicklas Boman [:smurfd]]

hm do i understand things correctly?

this is called : https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#1400
which calls this : https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#1316

and its in nsProtocolProxyService::AsyncResolveInternal where calls are not made Async? so its basicly to replace the non-async functions to its ascync brother..

and also are the two bugs mentioned above including this one revolving around the same issue? 
(i have tried to read through all of them a couple of times but was struggling to get heads and tails was so i thought id rather ask)

Comment 7

[Shane Caraveo (:mixedpuppy)]

A bit rusty on this so took a quick look back.  

I believe the problem is nsProtocolProxyService::ApplyFilters (which is called from AsyncResolveInternal):

https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#2111

That loops through any registered filters and calls them synchronously.

Comment 8

[Nicklas Boman [:smurfd]]

Thanks for the look :)

As is see it, there are 4 functions called in nsProtocolProxyService::ApplyFilters 
PruneProxyInfo, GetProxyURI getter_AddRefs and ApplyFilters itself...

The only one i found that has a Async similarity is AsyncGetProxyForURI:
https://searchfox.org/mozilla-central/source/netwerk/base/nsPACMan.cpp#385
(but i guess they are not doing the exact same, since GetProxyURI is also used in AsyncResolveInternal
https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#1316 )

PruneProxyInfo goes through the list of proxies and removes the disabled and disallowed ones.
The ones it does find usable, it will enable...


hmmm ... I guess there has to be something like 
https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#244
.. done in AsyncResolveInternal when the filters are applied?
(or am i completely lost?)

Comment 9

[Nicklas Boman [:smurfd]]

i have realized some things ... i was not Way off but was off.

Im guessing there needs to be done a class Like nsAsyncResolveRequest for applying filters to Do and Handle the callbacks 
https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#103

then todo the whole dance like https://searchfox.org/mozilla-central/source/netwerk/base/nsProtocolProxyService.cpp#1330

Comment 10

[Jason Duell]

Needed soon for Webextensions, so -> P1.

Comment 11

[Nicklas Boman [:smurfd]]

Attachment: 1152332.patch

im not there... but hopefully im on the right way? let me know...

Comment 12

[Gary Chen [:xeonchen]]

Can anyone confirm that removing [1] will "fix" the issue?
Of course this is not a real fix if it works, but I'd like to make it clearer where the problem is.

[1] https://searchfox.org/mozilla-central/rev/03877052c151a8f062eea177f684a2743cd7b1d5/netwerk/base/nsProtocolProxyService.cpp#1371-1374

Comment 13

[Shane Caraveo (:mixedpuppy)]

(In reply to Gary Chen [:xeonchen] (needinfo plz) from comment #12)
> Can anyone confirm that removing [1] will "fix" the issue?
> Of course this is not a real fix if it works, but I'd like to make it
> clearer where the problem is.

nsIProtocolProxyFilter::applyFilter needs to be callable asynchronously.  nsProtocolProxyService::ApplyFilters calls each filter->applyFilter.  It's a synch operation.  

I'm not certain if the patch in comment 11 is going in the right direction, but I think it is attempting to address the right problem.

Comment 14

[Nicklas Boman [:smurfd]]

is it ok to contact you via IRC mixedpuppy or is anyone else more suited?

Comment 15

[Martin Giger [:freaktechnik]]

Comment on attachment 8939277 [details] [diff] [review]
1152332.patch

Hey Nicklas, to get people to respond to questions on Bugzilla, it is usually a good idea to needinfo? them. If you have submitted a patch assign a reviewer that seems to fit and they will redirect if needed (there are guides on finding a reviewer out there, but usually the person helping you with a patch will also be your reviewer). With that in mind I've added mixedpuppy for feedback on the patch, so he has it in his queue for things to look at.

Comment 16

[Nicklas Boman [:smurfd]]

Hey Martin.
Ahh thanks for the feedback, yes i forgot to needinfo. 
One problem i had was i wasnt sure Who to needinfo. 

I have had a short mail discussion with Shane about this, he was in turn going to check with a Gary and Jason...

Comment 17

[Shane Caraveo (:mixedpuppy)]

Some recent changes further delayed some responses, but Jason will be attempting to find a mentor or assignee for this bug in the very near future.

Comment 18

[Jason Duell]

Honza will probably make quick work of this.  Sorry about delay.

Comment 19

[Honza Bambas (:mayhemer)]

Shane, I first need to make few things clear before I start coding this.

1. regarding bug 1409878: how the web-extension API is about to behave (be designed) 

2. I need to know what is the plan/design for bug 1378459 more in detail.  According what I've read so far, are we going to block (for sake of this bug) the proxy resolution calling all filters until all extensions load?  Is that so?  If so, then I need a notification for that to wait with calling filters.


P1 -> P2 since this doesn't block any P1 bugs.  Added block for bug 1378459 based on comment 4.

Comment 20

[Shane Caraveo (:mixedpuppy)]

(In reply to Honza Bambas (:mayhemer) from comment #19)
> Shane, I first need to make few things clear before I start coding this.
> 
> 1. regarding bug 1409878: how the web-extension API is about to behave (be
> designed) 

There is an older document linked to in that bug that outlines roughly what we would do, but I'll simplify a bit.

Any API that we allow to be called in webextensions is expected to work asynchronously.  To support synchronous calls from the proxy service, we had to introduce special support for a pseudo-PAC script, and workarounds using message passing for the extension to talk with that.  The ideal situation is to support a proxy filter in the background script of the extension, but that needs to work asynchronously.

- Use nsIProtocolProxyService.registerChannelFilter to register a filter for the extension
- Implement nsIProtocolProxyChannelFilter, however applyFilter needs to support returning the result asynchronously.

Given that, we can support a filter call in the background script

> 2. I need to know what is the plan/design for bug 1378459 more in detail. 
> According what I've read so far, are we going to block (for sake of this
> bug) the proxy resolution calling all filters until all extensions load?  Is
> that so?  If so, then I need a notification for that to wait with calling
> filters.

The idea there is to suspend all channels until extensions that have a subset of functionality have started up.  That subset is proxy, blocking webrequests and possibly content scripts (questionable I think).  We do not need to wait for extensions that do not use these features.  I beleive aswan was going to take on the startup issue so ni? in case he has more input.

> P1 -> P2 since this doesn't block any P1 bugs.  Added block for bug 1378459
> based on comment 4.

Right now proxy extensions are severely limited, this issue blocks any improvements to address that.  bug 1409878 is not higher priority simply because it cannot be handled without this change.  Once this is fixed, that bug will be addressed right away.

Comment 21

[Honza Bambas (:mayhemer)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #20)
> - Implement nsIProtocolProxyChannelFilter, however applyFilter needs to
> support returning the result asynchronously.

So the whole point is to turn 

nsIProxyInfo applyFilter(...);

to 

void applyFilter(..., in nsIProxyInfoResult aResultCallback);
where nsIProxyInfoResult is an interface with onProxyInfo(in nsIProxyInfo aInfo); method?

the contract may be following:
- when applyFilter returns OK, the callback MUST be called to finalize the roundtrip (can be a footgun by means of the callback never being called by mistake..)
- when applyFilter throws, fail the whole proxy resolution and thus kill the channel?  (for sec reasons)

> The idea there is to suspend all channels until extensions that have a
> subset of functionality have started up.  

good, so no other logic is needed!


> > P1 -> P2 since this doesn't block any P1 bugs.  Added block for bug 1378459
> > based on comment 4.
> 
> Right now proxy extensions are severely limited, this issue blocks any
> improvements to address that.  bug 1409878 is not higher priority simply
> because it cannot be handled without this change.  Once this is fixed, that
> bug will be addressed right away.

then set your now P2 bugs to P1?

this doesn't mean I'm not going to work on this right now, actually, quite the contrary.  after you confirm the above questions I will start writing this.

Comment 22

[Honza Bambas (:mayhemer)]

Also, I assume the filters has to be called serially, not in parallel, right?  Let the order be the order as added?

Comment 23

[Honza Bambas (:mayhemer)]

ah.. there is a position for each filter, didn't look that carefully.

Comment 24

[Honza Bambas (:mayhemer)]

and one more question: is it expected for nsIProtocolProxyService methods accessing filters being called on non-main threads?

Comment 25

[Shane Caraveo (:mixedpuppy)]

(In reply to Honza Bambas (:mayhemer) from comment #21)
> (In reply to Shane Caraveo (:mixedpuppy) from comment #20)
> > - Implement nsIProtocolProxyChannelFilter, however applyFilter needs to
> > support returning the result asynchronously.
> 
> So the whole point is to turn 
> 
> nsIProxyInfo applyFilter(...);
> 
> to 
> 
> void applyFilter(..., in nsIProxyInfoResult aResultCallback);
> where nsIProxyInfoResult is an interface with onProxyInfo(in nsIProxyInfo
> aInfo); method?
> 
> the contract may be following:
> - when applyFilter returns OK, the callback MUST be called to finalize the
> roundtrip (can be a footgun by means of the callback never being called by
> mistake..)
> - when applyFilter throws, fail the whole proxy resolution and thus kill the
> channel?  (for sec reasons)

That seems reasonable.  I'm slightly concerned about the roundtrip since we cannot promise that the extension will do so.  We could probably have some kind of timeout in the extension api itself (ie. the applyFilter call) and throw if it times out.

I'm not certain how important the filter position stuff is at this point.  There is no promise of startup order for extensions, and no promise of order for registering the proxy filter.  We always insert with position 0 [1].  Given that, filter position seems arbitrary anyway.

As far as threading goes, I'd prefer to assume that we will run in the extension process.  The new implementation would be very close to what we already have [2].

[1] https://searchfox.org/mozilla-central/source/toolkit/components/extensions/ProxyScriptContext.jsm#236
[2] https://searchfox.org/mozilla-central/source/toolkit/components/extensions/ProxyScriptContext.jsm#297

Comment 26

[Honza Bambas (:mayhemer)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #25)
> (In reply to Honza Bambas (:mayhemer) from comment #21)
> > (In reply to Shane Caraveo (:mixedpuppy) from comment #20)
> > > - Implement nsIProtocolProxyChannelFilter, however applyFilter needs to
> > > support returning the result asynchronously.
> > 
> > So the whole point is to turn 
> > 
> > nsIProxyInfo applyFilter(...);
> > 
> > to 
> > 
> > void applyFilter(..., in nsIProxyInfoResult aResultCallback);
> > where nsIProxyInfoResult is an interface with onProxyInfo(in nsIProxyInfo
> > aInfo); method?
> > 
> > the contract may be following:
> > - when applyFilter returns OK, the callback MUST be called to finalize the
> > roundtrip (can be a footgun by means of the callback never being called by
> > mistake..)
> > - when applyFilter throws, fail the whole proxy resolution and thus kill the
> > channel?  (for sec reasons)
> 
> That seems reasonable.  I'm slightly concerned about the roundtrip since we
> cannot promise that the extension will do so.  We could probably have some
> kind of timeout in the extension api itself (ie. the applyFilter call) and
> throw if it times out.

In a followup.  I have the same contract for http redirect checking APIs and so far this never broke.

> 
> I'm not certain how important the filter position stuff is at this point. 
> There is no promise of startup order for extensions, and no promise of order
> for registering the proxy filter.  We always insert with position 0 [1]. 
> Given that, filter position seems arbitrary anyway.

maybe I will remove the positioning at all.  that could be there for legacy exts only, if no use in gecko -> remove.

> 
> As far as threading goes, I'd prefer to assume that we will run in the
> extension process.  

I asked about threads, not processes.  I don't care which process this is about to be used as far as the base functionality of the service doesn't break, I need to know which threads will access this.  if more than one, I need a mutex and think a bit.

also, the proxy service needs access to prefs, just FYI.

> The new implementation would be very close to what we
> already have [2].
> 
> [1]
> https://searchfox.org/mozilla-central/source/toolkit/components/extensions/
> ProxyScriptContext.jsm#236
> [2]
> https://searchfox.org/mozilla-central/source/toolkit/components/extensions/
> ProxyScriptContext.jsm#297

will look.

thanks!

Comment 27

[Andrew Swan [:aswan]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #20)
> The idea there is to suspend all channels until extensions that have a
> subset of functionality have started up.  That subset is proxy, blocking
> webrequests and possibly content scripts (questionable I think).  We do not
> need to wait for extensions that do not use these features.  I beleive aswan
> was going to take on the startup issue so ni? in case he has more input.

I don't have anything to add, that is an accurate summary of the plan...

Comment 28

[Honza Bambas (:mayhemer)]

Attachment: v0.1

This is a first version, not that simple as I thought first...

Passes tests adjusted to the new API (next patch).

No idea who should review this these days.. :(  Nick, can you look over the patch or bounce/forward?  Thanks.

TODO:
- indention nits
- update remaining consumers (webext)
- check linux build
- try -a/local testing

Comment 29

[Honza Bambas (:mayhemer)]

Attachment: tests v0.1

only xpcshell tests updates so far (have to check for any remaining tests to update)

Comment 30

[Honza Bambas (:mayhemer)]

Nick, see comment 21 for details what this is supposed to do.  I miss some comments on IDLs in the patch...

Comment 31

[Honza Bambas (:mayhemer)]

Attachment: web-ext update v1

Shane, this seems like the only change to the web extensions back end according how the new API works.

applyFilter takes another argument, a callback implementing nsIProxyProtocolFilterResult with a single method onProxyFilterResult(nsIProxyInfo aProxy) that must be called on by the applyFilter implementation or anytime later asynchronously if applyFilter didn't throw.

Note that when applyFilter throws before calling the callback, it works as before - the chain is not broken and proxy info is left unchanged.

Comment 32

[Honza Bambas (:mayhemer)]

Attachment: tests v1

all occurrences of the API in tests I could find in m-c.

Comment 33

[Honza Bambas (:mayhemer)]

Attachment: v1

Please see comment 21 for what this patch should do.  Everything else should be commented inside the code.

Comment 34

[Honza Bambas (:mayhemer)]

Attachment: web-ext update v1.1

- one return slipped!
- eslint warning fixed

Comment 35

[Shane Caraveo (:mixedpuppy)]

Comment on attachment 8947431 [details] [diff] [review]
web-ext update v1.1

Looks fine, I believe that the tests[1] should cover this change as well.

[1] toolkit/components/extensions/test/xpcshell/test_ext_proxy_*

Comment 36

[Honza Bambas (:mayhemer)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #35)
> Comment on attachment 8947431 [details] [diff] [review]
> web-ext update v1.1
> 
> Looks fine, I believe that the tests[1] should cover this change as well.
> 
> [1] toolkit/components/extensions/test/xpcshell/test_ext_proxy_*

Actually, the patch I've submitted had one major flaw.  Will update.

Thanks for the test hint!

Comment 37

[Honza Bambas (:mayhemer)]

Attachment: webext v1.2

the callback was called twice...

Comment 38

[Honza Bambas (:mayhemer)]

Attachment: core v1.1

- assert/bypass when callback is called twice+
- processing thread assertions added, while the refcnt is t-safe
- updates to IDL comments (more readable)

Comment 39

[Michal Novotny [:michal]]

Comment on attachment 8947555 [details] [diff] [review]
core v1.1

Review of attachment 8947555 [details] [diff] [review]:
-----------------------------------------------------------------

::: netwerk/base/nsIProtocolProxyFilter.idl
@@ +54,5 @@
>     *        the given URI.  This may be null.
>     *
> +   * @param aCallback
> +   *        An object that the implementer is obligated to call on with
> +   *        the result (from within appplyFilter() or asynchronously) when

applyFilter

@@ +56,5 @@
> +   * @param aCallback
> +   *        An object that the implementer is obligated to call on with
> +   *        the result (from within appplyFilter() or asynchronously) when
> +   *        applyFilter didn't throw.  The argument passed to onProxyFilterResult
> +   *        it the proxy (or list of proxies) that should be used in place of

_is_ the proxy

@@ +88,5 @@
>     * @param aProxy
>     *        The proxy (or list of proxies) that would be used by default for
>     *        the given channel. This may be null.
>     *
> +   * @param aCallback

The same typos as above.

::: netwerk/base/nsProtocolProxyService.cpp
@@ +207,5 @@
> +      // false otherwise.
> +      bool mProcessingInLoop;
> +      // true after a filter called back to us with a result, dropped to false
> +      // just before we call a filter.
> +      bool mFilterCalled;

mFilterCalled is IMO a bad name. It sounds like the filter was called and not that it has called back. Something like mFilterCalledBack or mApplyFilterFinished would be better.

Comment 40

[Michal Novotny [:michal]]

Comment on attachment 8947396 [details] [diff] [review]
tests v1

Review of attachment 8947396 [details] [diff] [review]:
-----------------------------------------------------------------

::: netwerk/test/unit/test_protocolproxyservice-async-filters.js
@@ +8,3 @@
>  
>  // These are the major sub tests:
> +// TODO

Looks like you forgot to convert existing tests.

Comment 41

[Honza Bambas (:mayhemer)]

Attachment: tests v1.1

No, just the comment was not updated.  Note that test_protocolproxyservice-async-filters.js is a hg copy of test_protocolproxyservice.js.

Comment 42

[Honza Bambas (:mayhemer)]

Attachment: core v1.2

- typos
- mFilterCalledBack (can't think of a better name)

thanks!

Comment 43

[Honza Bambas (:mayhemer)]

One last issue now remains to be fixed: perma leak of the browser_cache.js test, bug 1330046.

Comment 44

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #43)
> One last issue now remains to be fixed: perma leak of the browser_cache.js
> test, bug 1330046.

OK, the problem is elsewhere.  There is no leak, this is just a taskcluster or try issue.  Even if I do just an arbitrary change to the test code (actually just "touch" the file) and then push to try, it starts perma failing.

Hence, I will land this now.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5818c42acc07d080c885da9a82fcd853e47ddd7&filter-searchStr=tv

The TV oranges are the issue described above.

Comment 45

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/647de96f4c7c
Let connection proxy filters return the result asynchronously - update webextensions back-end, r=mixedpuppy
https://hg.mozilla.org/integration/mozilla-inbound/rev/ce34d962b0df
Let connection proxy filters return the result asynchronously - update/introduce tests, r=michal
https://hg.mozilla.org/integration/mozilla-inbound/rev/77c14093bc69
Let connection proxy filters return the result asynchronously - core changes, r=michal

Comment 46

[Natalia Csoregi [:nataliaCs]]

@mayhemer Can you please take a look at these test-verify failures, they seem to be permafailing.
Push https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=d9da3f3bf43203a0ad3de87adfd8298c5e5e85a2&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=161299145 

Log https://treeherder.mozilla.org/logviewer.html#?job_id=161299145&repo=mozilla-inbound&lineNumber=4906


03:33:42     INFO - TEST-UNEXPECTED-FAIL | browser/components/originattributes/test/browser/browser_cache.js | Cache entries expected for iframe.html: 1, and found 2 - 
03:33:42     INFO - Stack trace:
03:33:42     INFO - chrome://mochitests/content/browser/browser/components/originattributes/test/browser/browser_cache.js:doCheck:248
03:33:42     INFO - Not taking screenshot here: see the one that was previously logged
03:33:42     INFO - TEST-UNEXPECTED-FAIL | browser/components/originattributes/test/browser/browser_cache.js | Cache entries expected for link.css: 1, and found 2 - 
03:33:42     INFO - Stack trace:
03:33:42     INFO - chrome://mochitests/content/browser/browser/components/originattributes/test/browser/browser_cache.js:doCheck:248
03:33:42     INFO - TEST-PASS | browser/components/originattributes/test/browser/browser_cache.js | Cache entries expected for script.js: 1, and found 1 - 
03:33:42     INFO - Not taking screenshot here: see the one that was previously logged
03:33:42     INFO - TEST-UNEXPECTED-FAIL | browser/components/originattributes/test/browser/browser_cache.js | Cache entries expected for img.png: 1, and found 2 - 
03:33:42     INFO - Stack trace:
03:33:42     INFO - chrome://mochitests/content/browser/browser/components/originattributes/test/browser/browser_cache.js:doCheck:248
03:33:42     INFO - Not taking screenshot here: see the one that was previously logged
03:33:42     INFO - TEST-UNEXPECTED-FAIL | browser/components/originattributes/test/browser/browser_cache.js | Cache entries expected for object.png: 1, and found 2 - 
03:33:42     INFO - Stack trace:
03:33:42     INFO - chrome://mochitests/content/browser/browser/components/originattributes/test/browser/browser_cache.js:doCheck:248
03:33:42     INFO - Not taking screenshot here: see the one that was previously logged
03:33:42     INFO - TEST-UNEXPECTED-FAIL | browser/components/originattributes/test/browser/browser_cache.js | Cache entries expected for embed.png: 1, and found 2 -

Comment 47

[Honza Bambas (:mayhemer)]

please see https://bugzilla.mozilla.org/show_bug.cgi?id=1436810#c2 and following.  this is just a stress-tested known intermittent failure which has rate high enough to make in at least one of 32 runs and thus appear permanent.

Comment 48

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/647de96f4c7c
https://hg.mozilla.org/mozilla-central/rev/ce34d962b0df
https://hg.mozilla.org/mozilla-central/rev/77c14093bc69