User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:18.104.22.168) Gecko/2008072610 Firefox/3.0.1
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; de; rv:22.214.171.124) Gecko/2008072610 Firefox/3.0.1
When using Firefox on a slow / unreliable internet connection (e.g. GPRS), it often fails to process SNI correctly and the user is left with the main certificate which would be delivered without SNI.
This should be considered serious as it'll leave the user with a certificate issued for the wrong domain and will give him a warning looking like someone was trying to attack him.
I think SNI happens at the NSS library level, changing component.
SNI is a feature of TLS. NSS only does SNI when TLS is enabled.
NSS does not do SNI when TLS is disabled.
PSM has a feature, known by various names such as "transparent fallback"
and "TLS intolerant server recovery", that disables TLS after a server has
connected but then subsequently failed to negotiate a TLS/SSL handshake
when the client was using TLS-only features. PSM finds that the first TLS
handshake attempt fails, and "falls back" to SSL 3.0. Doing so disables
the TLS-only features such as SNI and ECC, both of which require TLS client
So, the complaint here is that PSM falls back in situations where inability
to complete a handshake is due to very faulty network, not to faulty server.
I might suggest that there be a preference to allow users to disable the TLS
intolerant server recovery.
This does not really require a faulty network connection, but is easily reproducable when simply producing "load" by holding F5 when loading a page that requires SNI to work as demonstrated at http://lists.outoforder.cc/pipermail/modules/2008-September/000185.html (I could reproduce this locally with my server which I know to work properly)
That behaviour is kinda annoying because once FF falls back to not using SNI it is not reenabled until FF is restarted.
Email me if you want an account for testing.
What does "holding F5" do (on Linux)?
holding F5 on Windows is for repeatedly reloading the page. On Linux that's like holding Ctrl and repeatedly pressing R (if I remember the shortcuts correctly). But you also might use the toolbar buttons to gain the same effect.
I have observed exactly this behaviour numerous times whilst connecting to an HTTPS server on my LAN.
Just now I experienced this bug by doing the following:
* Connected to https://example.com/, which denotes a domain on my LAN. The correct SSL certificate was used by the server, which proves that TLS was used and that SNI was sent by the browser.
* The page at that URL contains an IMG tag with src= https://subdomain.example.com/. The image did not load. Upon right-clicking the image placeholder and clicking 'View image', I could see the 'invalid security certificate' error message. The server was using a certificate other than that for subdomain.example.com or example.com; it was using the configured 'default' certificate for clients that do not provide SNI.
* After starting up Wireshark to logg HTTPS packets, I refreshed the image URL, and noticed that only SSLv3 negotiation was attempted for subdomain.example.com -- not TLS, and hence no SNI sent.
Upon restarting Firefox and retrying the above steps, the first page, and the image from the subdomain, both loaded correctly. This means that TLS and SNI were successfully used for both connections.
I will note that both subdomain.example.com is a CNAME for example.com; both resolve to the same IP address, of a single webserver, using mod_gnutls for its SSL capabilities.
I estimate that at least 95% of the time, I experience the correct behaviour. I'm not sure what triggers the fallback from TLS to SSLv3, but I doubt this was due to packet loss as this was on an uncongested LAN.
In reply to comment 7, doesn't comment 2 explain this?
Yes, my experience in comment 7 was that of 'transparent fallback' from TLS to SSLv3 as described in comment 2. But I don't see a reason why that should have happened in my situation.
*** Bug 540547 has been marked as a duplicate of this bug. ***
Is this still happening with more recent Firefox versions?
A while ago Honza added some smart code that will cause Firefox to remember a site can do TLS, once we have a good TLS connection.
After that, Firefox should never fall back to SSLv3 (for the remainder of the session).
Yes, I'm still seeing this issue on 3.5.7, even though a good TLS connection to that host/IP was made only a few seconds previously.
I'd say this a dup of bug 498311. The reload button actually works like stop+reload. Stop portion of the reload operation calls PR_Close on the secure socket what marks the server as TLS intolerant (cause of that bug).
Kai, there is a patch in that bug waiting for your review.
Please be aware that we're talking about two different issues. The first (and originally reported) is that on a weak internet connection, it falls back to not using SNI and issuing a wrong cert to the user. I'd still consider this a rather serious issue, especially as SNI will probably get more widespread now that apache supports it by default.
The reload issue seems to be a different one.
Hanno, I agree. The problem is that we are using timeout to detect the server is incapable of TLS. The timeout interval is always hard to define. Too short means false negatives and too long interval cause people to consider the page as not working or taking too long to load. There were a lot of discussions on how the timeout should take and the current value satisfies most of the audience, at least AFAIK.
Mass change owner of unconfirmed "Core:Security UI/PSM/SMime" bugs to nobody.
Search for kaie-20100607-unconfirmed-nobody
This needs a bump. This is a continuing problem that seems to happen even today in the latest nightly builds, and affects even internal Mozilla properties. It happens enough that people don't shrug it off as a fluke.
It would be really, really good for Operations folks if Firefox tried a little bit harder to use TLSv1+ before falling back to SSLv3. It's almost 3 years since the last comment here, and 5 years since the bug was opened... TLSv1.0 is far more established now than it was then, and it seems reasonable to me that we could be a bit more persistent in trying to use it. A longer timeout (as described in comment 15), or even retry the failed handshake once (as described in comment 2) before falling back to SSLv3...
(In reply to Jake Maul [:jakem] from comment #17)
> It would be really, really good for Operations folks if Firefox tried a
> little bit harder to use TLSv1+ before falling back to SSLv3. It's almost 3
> years since the last comment here, and 5 years since the bug was opened...
> TLSv1.0 is far more established now than it was then, and it seems
> reasonable to me that we could be a bit more persistent in trying to use it.
> A longer timeout (as described in comment 15), or even retry the failed
> handshake once (as described in comment 2) before falling back to SSLv3...
Is there a particular service in particular that you'd like us to avoid falling back to SSLv3 on? AUS, the blocklist, etc.?
I guess most Mozilla properties still have to support IE6 which doesn't support SNI anyway.
But, if there are Firefox-specific services, we could change the client code to avoid doing the fallback, e.g. by introducing a new channel load flag that the Firefox side would use to indicate that there should be no fallback. Alternatively, we could consider using HSTS and then change the Firefox code to disable the fallback for HSTS sites.
There are many options. It would be great to know more about the specific issues Operations is facing so we can choose which ones to implement.
% of Firefox Sync requests that use SSLv3, out of total (SSLv3+TLS) requests. '*' marks useragents where I had at least 500 SSLv3 request samples in the dataset. Source data is some fraction of 1 hour's zlb data for Sync, and we've seen this many months ago as well, so I imagine it's been prevalent for a long time.
Firefox/10 5.84% *
Firefox/14 2.86% *
Firefox/15 2.68% *
Firefox/16 2.49% *
Most of our larger sites/services don't have a "problem" per-se... we've long since given up on using SNI support for them (for one reason or another, not just this bug), and give them dedicated IPs. They might still be falling back unnecessarily, but at least they don't get cert errors from it.
The problem is the smaller sites. There are dozens of these, so it's rather harder to justify the overhead of setting up separate IPs for all of them (whereas it's easy in the case of, say, www.mozilla.org). I would not consider these "Firefox-specific services"... they're just websites, and generally work in almost any browser. That said, the user bases do tend to be heavily Firefox-centric, for obvious reasons.
Examples would be sites like these (a selection from our "generic" cluster, which has 32 sites set up for SNI-chosen certs):
The user bases are often more focused and up-to-date than the general internet population... for example the hits to "treestatus.mozilla.org" for the most recent hour is 92% Firefox, and the rest Chrome/Safari/WebKit or bots... not a single "MSIE" hit (insofar as User-Agent logging is trustworthy, anyway). Only 3/8ths is even Windows, and of that only 13% is WinXP. None of it is older than WinXP. Because of this we find it's generally fairly realistic to use TLS SNI for these sites... we don't really have to worry much about browsers not supporting it. Except when it breaks, and browsers that *should* support it, don't. :)
When any of these fail and degrade to SSLv3, the user will get the "default" cert, which happens to be for tbpl.mozilla.org.
Hopefully that adequately explains what we're running into. Let me know if there's anything more I can help with.
Can you (or have you) enabled Strict-Transport-Security for these sites? If so, we can probably use Strict-Transport-Security to disable the fallback, and add these sites to our preloaded list of HSTS sites. The requirement to be on the list is that you have to send the Strict-Transport-Security header with a max-age of 18+ weeks.
In Firefox, Strict-Transport-Security doesn't disable the fallback yet, but it would be relatively simple to make it do so.
Hmm... a few of them perhaps use HSTS already, but definitely not all of them.
I'm not too keen on trying to maintain a list of sites in the preloaded list... they would seem to me to be a little too small and "fluid" to make that easy. I'd feel like we'd be adding and removing sites from it pretty frequently... feels like the wrong approach.
However, I'd definitely be willing to entertain making HSTS a lot more of a default for our sites, and I'd be willing to accept that the first hit might fall back unnecessarily. So we could still do this without using the preloaded list.
That said, this seems like a bit of an indirect solution to the problem... it actually sounds like a future bug report: "Firefox doesn't fall back to SSLv3 when HSTS is present". HSTS doesn't mandate the use of TLSv1+ over SSLv3, so it would seem like a bug (or at least a side effect) that Firefox would behave this way.
One thing that occurs to me is: if we do this (disable fallback when HSTS is present), what happens when you have TLSv1+ disabled in Firefox altogether? To my mind it should allow SSLv3 in that scenario... that is, *fallback to SSL* would be disabled rather than SSLv3 outright. The distinction is maybe not much right now, but it might be more significant if/when Firefox gets TLSv1.1 and/or 1.2 support.
Selfishly however, I think we could make this work in the majority of our particular cases. I'm not sure it's ideal, but I'd take it over the current situation. :)
Should we disable SSLv3 support on TNI-requiring sites, since it's not compatible with their assigned SSL certificate(s)?
(In reply to Jake Maul [:jakem] from comment #22)
> That said, this seems like a bit of an indirect solution to the problem...
> it actually sounds like a future bug report: "Firefox doesn't fall back to
> SSLv3 when HSTS is present". HSTS doesn't mandate the use of TLSv1+ over
> SSLv3, so it would seem like a bug (or at least a side effect) that Firefox
> would behave this way.
There are two ways to fall back from TLS 1.0 to SSL 3.0: the insecure way and the secure way. The insecure way is the thing that is causing you trouble. That is why HSTS should help.
> One thing that occurs to me is: if we do this (disable fallback when HSTS is
> present), what happens when you have TLSv1+ disabled in Firefox altogether?
The secure way (the server explicitly choosing SSLv3) would still work.
> To my mind it should allow SSLv3 in that scenario... that is, *fallback to
> SSL* would be disabled rather than SSLv3 outright. The distinction is maybe
> not much right now, but it might be more significant if/when Firefox gets
> TLSv1.1 and/or 1.2 support.
Yes. This is why I suggested implementing HSTS and putting the sites in the preload list. I worry it might be difficult to make the fallback behavior stricter when we're adding TLS 1.1 and 1.2 support soon.
(In reply to Richard Soderberg [:atoll] from comment #23)
> Should we disable SSLv3 support on TNI-requiring sites, since it's not
> compatible with their assigned SSL certificate(s)?
I think that in theory, an SSLv3-only client could still send the SNI extension. I don't remember if any browsers actually do, however.
Another option is to get SSL certificates that has subjectAltNames for all the hostnames that share IPs. I think many CAs will do this.
(In reply to Brian Smith (:bsmith) from comment #24)
> There are two ways to fall back from TLS 1.0 to SSL 3.0: the insecure way
> and the secure way. The insecure way is the thing that is causing you
> trouble. That is why HSTS should help.
I don't pretend to understand why HSTS helps with this (AIUI, HSTS sets no requirements around protocol or cipher suites... only that "must be HTTPS"). But, in the interest of speed and brevity I'll take your word for it. :)
> Yes. This is why I suggested implementing HSTS and putting the sites in the
> preload list. I worry it might be difficult to make the fallback behavior
> stricter when we're adding TLS 1.1 and 1.2 support soon.
If it's pretty easy to get things onto and off of the list, that might be okay... otherwise I'm happy to skip that step, at least for now. I'm definitely willing to set up some very long-lived HSTS headers where we can- we already do this in some places, it probably won't be a big deal to do it more commonly. We may have a few sites that don't want to force SSL for one reason or another, but in general I think that's a possibility.
My concern is only that we might be making Firefox behave in a way that is unexpected. Perhaps I'm worrying over nothing, however...
> > Should we disable SSLv3 support on TNI-requiring sites, since it's not
> > compatible with their assigned SSL certificate(s)?
In this scenario, what is likely to happen? My understanding is that Firefox is the thing initiating the fallback to SSLv3 by giving up waiting for the server (comment 15), which means the problem would still occur... but instead of an invalid cert, the user would get some other sort of error indicating no session could be established. I suppose this might be an improvement, but it still seems non-ideal. I'd rather it was more persistent in trying to get that TLS handshake completed before giving up and trying SSLv3.
> Another option is to get SSL certificates that has subjectAltNames for all
> the hostnames that share IPs. I think many CAs will do this.
Our CA can do this, and in some ways it's definitely simpler. However, as I understand it our Security Assurance folks prefer not to see unrelated domain names on the same cert. It's also slightly less flexible since certs are no longer "atomic"... content/site migrations become somewhat more painful. I'm not against this outright, but it's not quite the rainbows and unicorns I was hoping for. :)
It's been a long time since I've hit this issue, not sure if it's actually fixed or I just no longer see it due to using faster machines.
In case it's still a problem, how about marking a domain or IP as TLS-capable once the first such connection succeeds, disabling SSL fallback for the current session?
This discussion goes into the completely wrong direction. None of the proposed fixes is an option in many cases. Adding all certs to subjectaltname is pretty much not doable on a shared hosting environment. HSTS is all nice and well for ssl-only setups, but for http/https-mixed setups it doesn't work.
My suggestion would be: disable this fallback. SSLv3 is old and dying, probably mozilla should discuss about disabling it completely at some point in the future. This behavior is made to "fix" broken server-setups, while it breaks perfectly valid server setups. I'd prefer to break some old setups while fixing the valid ones. It makes SNI, a very useful feature, pretty much unusable in many situations. I'm running a shared host and we provide our users with individual certificates - which is nice - but this bug prevents it from being a full replacement for their own IPs.
(In reply to Brian Smith (:bsmith) from comment #18)
> (In reply to Jake Maul [:jakem] from comment #17)
> I guess most Mozilla properties still have to support IE6 which doesn't
> support SNI anyway.
I should clarify this: No version of IE on Windows XP, including the latest (IE8), or any other software that uses Windows XP's SChannel, would be able to use these sites, if they depend on SNI. Accordingly, my opinion is that depending on SNI is inappropriate for many/most public-facing websites. It is especially bad to require SNI on these sites when visitors that run into this problem will get an "untrusted certificate" warning in their browser, because this trains them to ignore these errors and/or to doubt our security practices.
Also, Firefox is not the only browser that does the insecure fallback. However, we may be the only ones doing it based on timeouts; I am not sure if other browsers are doing that and I am also not sure it is worth doing, regardless. See bug 754356 which is about improving/removing the timeout logic.
I think that the patch in bug 754356 may resolve this bug satisfactorily for most people because the timeout logic is the most likely (only?) thing to affect a server that is doing nothing wrong.
(In reply to Brian Smith (:bsmith) from comment #28)
> I should clarify this: No version of IE on Windows XP, including the latest
> (IE8), or any other software that uses Windows XP's SChannel, would be able
> to use these sites, if they depend on SNI.
Yep. As far as I know, this is mainly IE and Safari-on-Windows... Firefox, Chrome, and Opera don't use Schannel, and all support SNI even on WinXP. Our users are heavily Firefox/Chrome, minimal IE, and also minimal WinXP in general. The count of visitors that don't support SNI for these sites actually approaches zero. Where it is a problem (blog.mozilla.org is a great example, since it's an official PR channel) we set up a dedicated IP and eschew SNI altogether.
> Accordingly, my opinion is that
> depending on SNI is inappropriate for many/most public-facing websites.
That's fair... doesn't make sense to require something that your users don't have. However, even if every single user/browser was SNI-capable, this bug still stands in the way of actually relying on SNI for Firefox users. Right now we get more reported problems with SNI-capable Firefox users not being able to use these sites than from SNI-incapable users on any browser.
> I think that the patch in bug 754356 may resolve this bug satisfactorily for
> most people because the timeout logic is the most likely (only?) thing to
> affect a server that is doing nothing wrong.
That definitely seems like a step in the right direction. Thanks! :)
(In reply to Jake Maul [:jakem] from comment #29)
> Yep. As far as I know, this is mainly IE and Safari-on-Windows... Firefox,
> Chrome, and Opera don't use Schannel, and all support SNI even on WinXP. Our
> users are heavily Firefox/Chrome, minimal IE, and also minimal WinXP in
> general. The count of visitors that don't support SNI for these sites
> actually approaches zero. Where it is a problem (blog.mozilla.org is a great
> example, since it's an official PR channel) we set up a dedicated IP and
> eschew SNI altogether.
In this case, for the sites that you are going to require SNI for, you should disable SSL 3.0. (And SSL 2.0 should already be disabled.) This will avoid any users getting a misleading "bad certificate" error.
That's perhaps better, but it doesn't really solve the underlying issue... Firefox would still attempt to fall back to SSLv3 in the "timeout" scenario, wouldn't it? That would just result in a different error. Perhaps it'd be a better error to get, but still far from ideal. :)
I am hopeful that a resolution to bug 754356 will help us out a lot.
(In reply to Jake Maul [:jakem] from comment #31)
> That's perhaps better, but it doesn't really solve the underlying issue...
> Firefox would still attempt to fall back to SSLv3 in the "timeout" scenario,
> wouldn't it? That would just result in a different error. Perhaps it'd be a
> better error to get, but still far from ideal. :)
> I am hopeful that a resolution to bug 754356 will help us out a lot.
Yes, but you should still disable SSL 3.0 independently of bug 754356. The "SSL version not supported" error is much better than untrusted certificate errors, as far as what we're teaching the user.
(In reply to Brian Smith (:bsmith) from comment #32)
> Yes, but you should still disable SSL 3.0 independently of bug 754356. The
> "SSL version not supported" error is much better than untrusted certificate
> errors, as far as what we're teaching the user.
That's fair. I do think that error is more likely to generate a refresh-and-forget scenario, whereas the bad cert has other (worse) outcomes.
On the client side, I'm personally content to wait for bug 754356 to be resolved in some way and see how much impact that has. There are other reasonable-sounding ideas here too, if that doesn't pan out. For example, comment 26 seems like a suitable way to minimize the potential for problems. Comment 27 (disable SSLv3 fallback) seems a little more drastic, and I'm not sure how feasible it would be... but it sounds brutally effective. :)
(In reply to Jake Maul [:jakem] from comment #33)
> That's fair. I do think that error is more likely to generate a
> refresh-and-forget scenario, whereas the bad cert has other (worse) outcomes.
Does refresh-and-forget work in this scenario? I saw above that the client caches the "TLS expired, resort to SSLv3" decision until a browser restart; as long as that caching only occurs upon a successful SSLv3 connect, this should work.
If it's still the same bug I saw years ago, reloading doesn't help. Only a restart does, it seems to cache the SSLv3 fallback somewhere.
"<devd> bsmith: re bug 450280. Don't know how many people connect to those domains via mobile; but a large chunk of iOS and Android systems don't support SNI"
(In reply to Richard Soderberg [:atoll] from comment #34)
> Does refresh-and-forget work in this scenario? I saw above that the client
> caches the "TLS expired, resort to SSLv3" decision until a browser restart;
> as long as that caching only occurs upon a successful SSLv3 connect, this
> should work.
Ouch, I missed that... hopefully that caching happens after an SSLv3 handshake completes successfully, but naively I can see how it might not. This sounds like a separate issue we should investigate and handle in a separate bug.
I wonder if a hard refresh (cmd-shift-r on OSX, I think ctrl-shift-r on Windows/Linux) overrides this behavior and causes it to retry TLSv1.0.
(In reply to Jake Maul [:jakem] from comment #37)
> I wonder if a hard refresh (cmd-shift-r on OSX, I think ctrl-shift-r on
> Windows/Linux) overrides this behavior and causes it to retry TLSv1.0.
Now that we're finally seeing TLS 1.2 used, the fact that it's trivial for an active MITM to downgrade a connection is becoming a serious liability. Features such as GCM and EC crypto might not be available with earlier protocols. Further, some people will have prioritized RC4 only under SSL 3.0 and TLS 1.0, but don't expect that it will be used by modern browsers.
I think the idea to have HSTS disable the downgrade is great. But, HSTS should probably specify the best supported version. Using this information, browsers will know not to downgrade when under attack.
However, this should probably be discussed on the HSTS mailing list.
The fix for bug 754356 was accidentally landed with this bug's bug number. I backed out that commit and then relanded it with the correct bug number:
I believe I just had an intermittent case of this problem happen when using a slow (3G) mifi connection.
Steps to reproduce:
* Make sure FF is not running
* Connect wifi to a Virgin mobile 3G mifi connection
* Launch FF
* Go to https://indiewebcamp.com/irc/today
* Get error warning about certificate, details: "The certificate is only valid for the following names: *.pin13.net , pin13.net"
* Close tab
* Try loading https://indiewebcamp.com/irc/today and get same error.
* Quit FF
* Disconnect wifi from Virgin mobile 3G mifi
* Connect wifi to a router connected to a local/DSL network
* Launch FF
* Go to https://indiewebcamp.com/irc/today
* Note that it loads just fine - no cert error.
SSL 3 and non-secure fallback to SSL 3 were disabled in bug 1076983 in Firefox 34+.