Closed Bug 1241906 Opened 8 years ago Closed 8 years ago

Firefox stops connecting to the internet until restart spdy suspended channel

Categories

(Core :: Networking, defect)

43 Branch
x86_64
Windows 10
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: mrabinovsky, Assigned: mcmanus)

References

Details

(Whiteboard: [necko-active][spdy])

Attachments

(2 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Build ID: 20160105164030

Steps to reproduce:

Use firefox regularly, tried to access any website on the internet (ie google.com, or microsoft.com, or youtube.com)


Actual results:

Firefox stops loading all web pages, IE11 and Edge and every other internet enabled application on my computer work fine. 

This has happened to me on several computers and for a long period of time now. Restarting firefox fixes the problem until the next time it happens.


Expected results:

Firefox should load the webpage I was trying to access.
OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64
Component: Untriaged → Networking
Product: Firefox → Core
Could be another instance of hang in a socket function on the socket thread?

Michael, if you can reproduce regularly, can you please follow directions at [1] and provide the log?  Also include in a comment what URL fails to load from you to search for it in the log.

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging
Flags: needinfo?(mrabinovsky)
It happens usually once or twice a day, and usually after I leave the computer for a few minutes, though it has happened before while in active use. I'll start the logging and post back here once it has happened again.
Flags: needinfo?(mrabinovsky)
I think this bug is very relevant to investigation in bug 1242755.
(In reply to Michael Rabinovsky from comment #2)
> It happens usually once or twice a day, and usually after I leave the
> computer for a few minutes, though it has happened before while in active
> use. I'll start the logging and post back here once it has happened again.

Hmm.. if that is so sporadic, the log will be huge (GBs!).  

Then maybe better would be to attach a debugger when the behavior manifests and see where we are hanging.  It's a bit more complicated, tho.  I can give you as detailed instructions as possible to do it.
(In reply to Honza Bambas (:mayhemer) from comment #4)

> Hmm.. if that is so sporadic, the log will be huge (GBs!).  
> 
> Then maybe better would be to attach a debugger when the behavior manifests
> and see where we are hanging.  It's a bit more complicated, tho.  I can give
> you as detailed instructions as possible to do it.

I wouldn't mind cutting out irreverent info from the log (after all you will only need a little before it happens up to the point that I restart firefox, and I hope there are time stamps for things.) But if you want to give me instructions to attach a debugger I could do that to, which ever makes it easier for you.
(In reply to Michael Rabinovsky from comment #5)
> (In reply to Honza Bambas (:mayhemer) from comment #4)
> 
> > Hmm.. if that is so sporadic, the log will be huge (GBs!).  
> > 
> > Then maybe better would be to attach a debugger when the behavior manifests
> > and see where we are hanging.  It's a bit more complicated, tho.  I can give
> > you as detailed instructions as possible to do it.
> 
> I wouldn't mind cutting out irreverent info from the log (after all you will
> only need a little before it happens up to the point that I restart firefox,
> and I hope there are time stamps for things.) But if you want to give me
> instructions to attach a debugger I could do that to, which ever makes it
> easier for you.

That's the thing.  We don't know exactly how many lines of the log we may need, but doing it incrementally may work.  Anyway, it's simpler than handling with a debugger, symbols, threads and stacks. 

Thanks!  Let's try the log way!
If you manage to get a log can you write:
NSPR_LOG_MODULES=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,io:5

"io:5" is added at the end of that line.

This will make log even bigger, sorry for that, but I think we need that info too.
Oi. I was restarting firefox so often (for many reasons) recently that it wasn't happening much. Then last night just before I went to bed it happened and I got a log file 7GB large :P

I haven't posted it yet because I haven't had time to go through it. I'll go a head and start logging again with io:5. Do you still want the other log, or should I just scrap it?
(In reply to Michael Rabinovsky from comment #8)
> Oi. I was restarting firefox so often (for many reasons) recently that it

"Many reasons" are they bugs in firefox?

> wasn't happening much. Then last night just before I went to bed it happened
> and I got a log file 7GB large :P
> 
> I haven't posted it yet because I haven't had time to go through it. I'll go
> a head and start logging again with io:5. Do you still want the other log,
> or should I just scrap it?

Please do not delete it. I would like to take a look at that one too.
dd, you should consider omitting nsStreamPump:5,nsHostResolver:5.  I personally don't think it will give you much info and definitely produces huge amount of lines.
(In reply to Honza Bambas (:mayhemer) from comment #10)
> dd, you should consider omitting nsStreamPump:5,nsHostResolver:5.  I
> personally don't think it will give you much info and definitely produces
> huge amount of lines.

That is true.

So 
NSPR_LOG_MODULES=timestamp,nsHttp:5,nsSocketTransport:5,io:5

This should be enough.
Whiteboard: [necko-active]
(In reply to Dragana Damjanovic [:dragana] from comment #9)

> "Many reasons" are they bugs in firefox?

Lol unless updating firefox and addons (this problem happens without addons too, just to be clear)are considered bugs, no. I got a bunch of BSODs from PhotoShop CC and and updated my drivers which require/force computer restarts (thus restarting firefox too).
 
> Please do not delete it. I would like to take a look at that one too.

If you want to take a look at the log as is I can upload it to my web server, otherwise I probably wont get a the time to go through it and cut out junk for a little while longer.


I'll restart the log with NSPR_LOG_MODULES=timestamp,nsHttp:5,nsSocketTransport:5,io:5
(In reply to Michael Rabinovsky from comment #12)
> (In reply to Dragana Damjanovic [:dragana] from comment #9)
> 
> > "Many reasons" are they bugs in firefox?
> 
> Lol unless updating firefox and addons (this problem happens without addons
> too, just to be clear)are considered bugs, no. I got a bunch of BSODs from
> PhotoShop CC and and updated my drivers which require/force computer
> restarts (thus restarting firefox too).

:) 

>  
> > Please do not delete it. I would like to take a look at that one too.
> 
> If you want to take a look at the log as is I can upload it to my web
> server, otherwise I probably wont get a the time to go through it and cut
> out junk for a little while longer.
> 

You can upload it, I will take a look.
I almost forgot the information that goes with it. I restarted firefox at 11:16pm my time (EST) which is 4:16am UTC, however the log goes on for about two minutes after that... I don't know what's up with that.

The sites I was trying to navigate at the time were facebook and adl.org (http://archive.adl.org/braun/dim_13_2_ford.html) I also tried testing gmail.com and google.com to see if they would load.
It appears that for whatever reason my comment prior to that last one with the actual log file was not published. Here is is:

http://mrabinovsky.com/firefox%20logs.rar
this is not a case of ::Poll() not returning afaict.

however there is a giant run of a spdy session spinning - oddly, getting called repeatedly on the read handler and repeatedly having it return would_block without making progress.

cc nick

2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdySession31 1cc77000 data frame read failure 80470007
2016-01-28 04:15:07.885000 UTC - 9112[14117d0]: SpdyStream31::WriteSegments 10c44680 count=32768 state=4
Whiteboard: [necko-active] → [necko-active][spdy]
Michale thanks for the log!

things go off the rails at 04:12:48.037.. that's when you get the tight spdy writesegments loop above. Also interesting to note that you don't get pr_poll in the middle of it.. it appears to be looping in nshttpconnection::onsocketreadable somehow calling transaction::writesegmnets->session::writesegments->stream::writesegments again and again and again, without ever managing to call into socketinputstream (where it would actually try and read from the network)

the actual spdy data looks fine. its in the middle of reading stream 0xF which is a byte range response for a facebook video.. its about 2MB into a 4MB chunk.. all the headers look fine. The last read off the wire is a spdy header for 0x10000 bytes more of stream F.. which is what its trying desparately to read here without actually calling pr_read.
2016-01-27 19:53:06.744000 UTC - 9112[14117d0]: nsHttpConnection::EnsureNPNComplete 1758e980 [.S....www.google.com:443] negotiated to 'spdy/3.1'

so that's odd.. we expect google.com to negotiate h2 instead of spdy. Do you have any networking prefs altered from their defaults? (they will be highlighted in about:config if you type network into the search box.) Thanks.
Flags: needinfo?(mrabinovsky)
I've got a handle on a problem.. I'm not sure if its your only real problem, but it puts so much gunk in the logs, its hard to know. 

just before 4:12 you actually a LOT of data processed in between poll() calls. its a few streams associated with a facebook video. This may be processing slowly beacuse the video processor is chewing CPU and you're doing all this logging for us :).. eventually you see "data frame read failure 80470007" while trying to complete a read of an inprogress video frame.. but you don't see a corresponding log to the input socket stream.

here's what I think happens

1] spdy session reads the frame header from the network - this calls through nsHttpConnection to get the input stream and makes mSocketInCondition be NS_OK as it reads 8 bytes just fine. These bytes are however removed by the spdysession layer as framing overhead.. if it returns to nshttpconnection::onsocketreadable, that function just calls session::writesegments again because the first iteration returned OK and socketincondition is ok

2] session::writesegments calls stream::writesegments which in turn calls transaction::writesegments.. this calls pipeoput::writesegments. normally this invokes the chain of OnWriteSegment handlers which end up going to the network (and setting mSocketInCondition) but I think the pipe to the channel is full (things seem to be running slowly and we've got megs of video floating around - perhaps paused video) so the pipe throws WOULD_BLOCK in nsHttpTransaction (line 832 handles that by calling asyncwait on the pipe.. when it gets that notification (ON THE SOCKET THREAD) it normally tells the network that it wants to read again and then it returns WOULD_BLOCK to stream which returns it to the session

3] the session helpfully says
    // maybe just blocked reading from network
      if (rv == NS_BASE_STREAM_WOULD_BLOCK)
        rv = NS_OK;
and returns OK to the connection

4] The connection sees an ok from writesegments, and an ok from msocketcondition (the last time it tried to read from the network it found 8 bytes there) and realizes that means it needs to loop.. so it spins until the channel pipe unblocks.. but that notification is delivered on the socket thread which is spinning.. so it won't happen. deadlock.

h2 appears to have the same logic as spdy - i.e. the channel pipe blocked vs network blocked events are disambiguated from mSocketInCondition.. which might not actually be set (or might be stale).

there are several code paths that do this mapping of would_block to ok, but the only one of interest is probably in data frame processing as that's the only way to write to the channel

I would like to think that nshttpconnection doesn't need that loop - but I'm sure there is something that keeps a non socket buffer that needs to be looped on as long as it is returning OK and will stall if we don't do that.

need to think a little bit about how to test/resolve this. Michael thanks again for the log.
(In reply to Patrick McManus [:mcmanus] from comment #18)
> 2016-01-27 19:53:06.744000 UTC - 9112[14117d0]:
> nsHttpConnection::EnsureNPNComplete 1758e980 [.S....www.google.com:443]
> negotiated to 'spdy/3.1'
> 
> so that's odd.. we expect google.com to negotiate h2 instead of spdy. Do you
> have any networking prefs altered from their defaults? (they will be
> highlighted in about:config if you type network into the search box.) Thanks.

When I think of highlighted I think of having a different background color (usually yellow) so I'm not sure if this is what you mean, but, a few settings are bold. I, personally, never changed them.

extensions.dta.network.http.max-connections;0

network.cookie.prefsMigrated;true

network.http.max-persistent-connections-per-server;4

network.predictor.cleaned-up;true

I'm glad you got some useful insight from the 7gb log... because as of right now my 11 hour log which includes io:5 is 14.5 gb ;)
Flags: needinfo?(mrabinovsky)
thanks. I meant bold in the about:config. Not sure why you don't get h2 with google - its probably not impt to this bug, but it is a curiosity.

facebook is also running h2, as of today anyhow, and yet you see spdy with them as well in the log. Your prefs don't disable h2 so this is curious.

Can you confirm you're running firefox 43 or 44? It seems a silly question, but this is odd behavior.

Meanwhile, assuming my hypothesis in comment 19 is correct this is only a spdy prbolem (and not a h2 problem) because of the feature in 1204614 (implemented only for h2) that landed in firefox 44.

That's a pretty big feature.. and spdy will be deprecated at some point (which is why it wasn't also done for spdy), so some smaller way forward would be good.
Flags: needinfo?(mrabinovsky)
the log confirms you're on 44

2016-01-28 04:12:45.262000 UTC - 0[1411140]:   User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Flags: needinfo?(mrabinovsky)
(In reply to Patrick McManus [:mcmanus] from comment #22)
> the log confirms you're on 44
> 
> 2016-01-28 04:12:45.262000 UTC - 0[1411140]:   User-Agent: Mozilla/5.0
> (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0

I am on 44, I keep firefox updated always. This has been happening for many versions though.
Michael, thanks again. Debugging remotely is a pita - I know.

I'm going to make a try build with a fix for the deadlock I describe above. Would you feel confident enough testing it to be able to make a guess at whether or not the fix is verified? I would want it verified before requesting backport. The try build would be based on nightly (firefox 47).

Separately, I'm extremely curious how you are negotiating spdy with facebook and google right now given that your prefs indicate you should be using h2. It might just be regional servers that you are connecting to, but its still quite odd.

do you know how to use wireshark to make a packet capture? If so, I'd love to see the handshake between you and facebook and google. It would be simple

1] shutdown firefox
2] startup wireshark and start a packet capture
3] start firefox and goto https://www.google.com/favicon.ico and https://www.facebook.com/favicon.ico
4] stop the packet capture

that would be enough to see if we were sending what we expected and why h2 wasn't used. Thanks!
Flags: needinfo?(mrabinovsky)
I too have a similar problem: Firefox 44 doesn't want to connect to websites, while other Internet applications will. Strangely enough, when I try using HTTP logging, Firefox connects; when I don't use logging, the connection problems return.

I've scanned for viruses and malware, but nothing was found. I've tried running in safe mode, to no avail. I've messed about with the firewall settings and even disabled the Windows firewall, but that did nothing for this problem. Restarting was the only thing that affected this problem and it was very brief.

Here's some information about my environment:
Firefox 44 EME-free 1.0 (Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0)
Window 7 64-bit (Microsoft Windows [Version 6.1.7601])
(In reply to Patrick McManus [:mcmanus] from comment #24)
> Michael, thanks again. Debugging remotely is a pita - I know.
> 
> I'm going to make a try build with a fix for the deadlock I describe above.
> Would you feel confident enough testing it to be able to make a guess at
> whether or not the fix is verified? I would want it verified before
> requesting backport. The try build would be based on nightly (firefox 47).

Sure.

> Separately, I'm extremely curious how you are negotiating spdy with facebook
> and google right now given that your prefs indicate you should be using h2.
> It might just be regional servers that you are connecting to, but its still
> quite odd.

I was using a VPN for the majority of these tests (I generally use VPN) and at some point it may have been in Russia. I switch between severs in the US and Russia depending on my p2p activities. So if spdy was related to what server I was connecting to outside the US, then maybe. However this also happens when I do not use a VPN.

> do you know how to use wireshark to make a packet capture? If so, I'd love
> to see the handshake between you and facebook and google. It would be simple
> 
> 1] shutdown firefox
> 2] startup wireshark and start a packet capture
> 3] start firefox and goto https://www.google.com/favicon.ico and
> https://www.facebook.com/favicon.ico
> 4] stop the packet capture
> 
> that would be enough to see if we were sending what we expected and why h2
> wasn't used. Thanks!

I am familiar with wireshark. I'll try and do it after I get this latest log or after firefox/my computer crash/restart and before I start a new log- which ever comes first.
Flags: needinfo?(mrabinovsky)
(In reply to Michael Rabinovsky from comment #27)
> (In reply to Patrick McManus [:mcmanus] from comment #24)
> > Michael, thanks again. Debugging remotely is a pita - I know.
> > 
> > I'm going to make a try build with a fix for the deadlock I describe above.
> > Would you feel confident enough testing it to be able to make a guess at
> > whether or not the fix is verified? I would want it verified before
> > requesting backport. The try build would be based on nightly (firefox 47).
> 
> Sure.

https://archive.mozilla.org/pub/firefox/try-builds/mcmanus@ducksong.com-d64595522eb77e498b2929683bdf33fe30d745b4/try-win32/
Attachment #8713720 - Flags: review?(hurley)
Assignee: nobody → mcmanus
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
> 
> 1] shutdown firefox
> 2] startup wireshark and start a packet capture
> 3] start firefox and goto https://www.google.com/favicon.ico and
> https://www.facebook.com/favicon.ico
> 4] stop the packet capture
> 
> that would be enough to see if we were sending what we expected and why h2
> wasn't used. Thanks!

So I went a head and captured some packets for you on FF44. I did 6 runs in total. 3 with my default profile and 3 with a new profile. For each profile I did one without a VPN, one with a US VPN, and one with a Russian VPN. I'm not sure how useful the VPN ones will be, since the VPN is supposed to encrypt all the traffic, but maybe since it's being captured on my computer it will still work. I'm no wireshark expert but a quick filter for spdy returned nothing, while http/2 resulted in a few hits.

Take a look yourself: http://mrabinovsky.com/Firefox%20packet%20capture.rar
(In reply to Patrick McManus [:mcmanus] from comment #28)

> https://archive.mozilla.org/pub/firefox/try-builds/mcmanus@ducksong.com-
> d64595522eb77e498b2929683bdf33fe30d745b4/try-win32/

I downloaded and installed your nightly build, I'll go a head and start using it with logging enabled on my normal profile.
so the important issue here is a spdy bug, and hopefully the try build you are using will resolve that. Thanks for trying it out.

Meanwhile, the mystery of why you happen to using spdy against facebook (where the error occurs) and google continues.. both of those sites offer both h2 and spdy but prefer h2 - the same can be said of firefox.

Thanks for the wireshark logs. They are exhaustive - I only needed to look at the first two:
[1] no_vpn_new_profile.pcapng
[2] no_vpn_regular_profile.pcapng

to see a difference.. so vpn isn't an issue - its something in your profile.

In log 1, h2 is used just fine. Streams 19 and 10 are exemplars - h2 is offered in the alpn in the client hello {h2, spdy/3.1, http/1} and h2 is selected by the server in the server hello.

In log 2, h2 is not offered.. the client hello is {spdy/3.1, http/1}.. and spdy/3.1 is selected by each server via alpn - streams 10 and 23 are exemplars. So that's all working fine from a protocol perspective - the question is why h2 is not offered.

So please use the regular profile (corresponding to trace 2) to try and verify the bug - as it is believed to be a spdy specific bug.
mystery solved.. in the new profile you are using tls 1.2 handshakes, and in the "regular" profile you are using tls 1.0 handshakes. H2, by specification, requires at least 1.2, so if you've got that disabled it does not offer h2 - but it will offer spdy which doesn't have such a requirement.

I bet if you look at security.tls in about:config you will see a nondefault, bolded, answer.. probably security.tls.version.max.. which should be 3 but I bet is 0 for you.
(In reply to Patrick McManus [:mcmanus] from comment #33)
> mystery solved.. in the new profile you are using tls 1.2 handshakes, and in
> the "regular" profile you are using tls 1.0 handshakes. H2, by
> specification, requires at least 1.2, so if you've got that disabled it does
> not offer h2 - but it will offer spdy which doesn't have such a requirement.
> 
> I bet if you look at security.tls in about:config you will see a nondefault,
> bolded, answer.. probably security.tls.version.max.. which should be 3 but I
> bet is 0 for you.

It is actually 1. Any way to find out what caused that?
(In reply to Michael Rabinovsky from comment #34)
> It is actually 1. Any way to find out what caused that?

I can only guess - it must be a fairly old profile.

There was a time (a few years ago I think) when firefox had trouble with 'tls intolerant' servers.. i.e. we would handshake with 1.1 or 1.2 and the buggy server, which was a 1.0 server, should have replied with a 1.0 downgrade according the 1.0 spec but instead just hung up.. and you'd get a useless website. Firefox has since been updated to basically silently 'call back' with a lower value and try again when this happens so its not an issue - but for a time there were a lot of workarounds published online that had you change the "version.max" config preference to lock your copy of firefox into only using 1.0 and therefore not triggering the buggy server. Sometimes that came as instructions on how to modify the pref, sometimes it was an addon that just did it for you. All easily forgotten later..
Yes, this is a fairly old profile. The last time I really started fresh was in 2014, or so. For the sake of the trial I am going to leave it at 1. I will remember to change it later though. I'm having a lot of issues with this nightly build, but not the disconnection issue so, so far so good.
Comment on attachment 8713720 [details] [diff] [review]
Spdy deadlock on suspended channel

Review of attachment 8713720 [details] [diff] [review]:
-----------------------------------------------------------------

This all looks reasonable to me.
Attachment #8713720 - Flags: review?(hurley) → review+
Attachment #8715949 - Flags: review?(hurley)
Comment on attachment 8715949 [details] [diff] [review]
test for suspended spdy3.1 channel

Review of attachment 8715949 [details] [diff] [review]:
-----------------------------------------------------------------

::: netwerk/test/unit/test_spdy.js
@@ +160,5 @@
>    do_check_true(this.isSpdyConnection);
>  
>    // Don't want to flood output, so don't use do_check_eq
> +  if (request.originalURI.spec == "https://localhost:" + serverPort + "/big") {
> +    // We know the server should send us the same data as our big post will be,  

nit: trailing whitespace

::: testing/xpcshell/moz-spdy/moz-spdy.js
@@ +155,5 @@
>      hash.update(content);
>      var md5 = hash.digest('hex');
>      res.setHeader("X-Expected-MD5", md5);
> +  } else if (u.pathname == "/huge") {
> +    content = getHugeContent(800 * 1024);

Huh, wonder why I made the name here "Huge" for the original test, but "Big" in the xpcshell file? Go inconsistency!
Attachment #8715949 - Flags: review?(hurley) → review+
See Also: → 1245948
See Also: 1245948
Summary: Firefox stops connecting to the internet until restart → Firefox stops connecting to the internet until restart spdy suspended channel
So I've been using this release of nightly for a few days now with the same profile. Aside from a few completely incompatible add-ons that I could not help but disable, it's pretty much the same. Made sure security.tls.version.max is still set to 1.

Haven't had this particular issue since I started using this release. It would seem that your patch fixed it, as this has been the longest I've gone in years without this happening to me.
thanks michael
https://hg.mozilla.org/mozilla-central/rev/e56a38a1a4ff
https://hg.mozilla.org/mozilla-central/rev/5ca719a45f5d
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Could you please take a look to Bug #1267000

As far as I can see, the latest changes here may be the cause of problem described there.
Flags: needinfo?(mcmanus)
ill have a look. thanks.
Flags: needinfo?(mcmanus)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: