Closed Bug 1072382 Opened 5 years ago Closed 5 years ago

Unable to load GMail, "Error code: ssl_error_inappropriate_fallback_alert" displayed

Categories

(Core :: Security: PSM, defect, major)

x86_64
Windows 8.1
defect
Not set
major

Tracking

()

RESOLVED FIXED
mozilla35
Tracking Status
firefox35 --- fixed
firefox-esr31 35+ fixed

People

(Reporter: bent.mozilla, Assigned: mt, NeedInfo)

References

Details

Attachments

(3 files)

This just started happening yesterday or the day before I think. Loading GMail fails with a network error page with the message "Error code: ssl_error_inappropriate_fallback_alert". Clicking the "Try Again" button doesn't work, the same page is re-displayed. Restarting seemed to fix it the last time I saw this yesterday, so I guess I'll try to restart again now.
Yep, restarting fixed it.
Got the same with GDocs today. Restarting also fixed it.
Get the same with google search, and the HTTPS everywhere extension.
Severity: normal → major
patrick, is this what we were seeing today w/ apln?
(In reply to Doug Turner (:dougt) from comment #4)
> patrick, is this what we were seeing today w/ apln?

unlikely. what we were talking about earlier today was a google.com problem with *DAV methods over http2 (i.e. it isn't a tls handshake problem). That issue doesn't seem related to this report.

this seems more like a problem with scsv - 1036737
Blocks: 1036737
Component: Networking → Security: PSM
I though I'd add a bit more about my setup, using Windows7, with 64-bit Nightly builds.
I have wireshark captures of the TLS negotiations with Google, both good and bad.

I also repeated the failure in gmail, with https everywhere disabled.

As with the others, after a re-start problem went away.

Of interest is that in the client, the bad negotiation specified a TLS 1.1 connection and first cipher suite listed was 0x5600 (Unknown to wireshark)
0x5600 is the downgrade protection cipher suite.  This is only supposed to be sent if we are forced to downgrade from TLS 1.2 to TLS 1.1 or lower.

The error seems to indicate that we're stuck at a lower TLS version somehow, which only happens if a TLS 1.2 handshake failed.  We persist knowledge of failures until restart, so that's why the restart fixes it.

What would be interesting (from Rob North or others), would be a capture that includes the first attempt to connect to gmail after a restart.  The interesting handshake is the one that goes out with TLS 1.2.  There's a chance that version fallback triggers too aggressively on a transient network error: http://dxr.mozilla.org/mozilla-central/source/security/manager/ssl/src/nsNSSIOLayer.cpp#1023

If we can't work this out, I can try modifying the intolerance logic to capture this error code and reset the version intolerance once.  This would have to flag the origin so that it didn't set up an infinite loop.  However, if we're falling back too easily, this will only reduce the odds of this problem, not increase it.
OK, so I'm going to propose a small change that should move this error from a permanent fail (at least until restart) and allow transient errors to recover.  If, at some point, we are able to learn how fallback is occurring for these sites, then we might (might!) do something more.  The 

The change is: If we detect an inappropriate fallback error, we will terminate the fallback attempts and ALSO clear the version intolerance records for that origin.  The alert is a 

Chrome remembers the original reason for failure and switches that in when it encounters the inappropriate_fallback alert.  That might be worth doing at some later point, see bug 1075167.
As described above, this should make the fallback less sticky if an inappropriate_fallback alert is received.  Refreshing the page should work if the fallback was triggered by a transient error of some sort.

Tested in the same way that bug 1036737 was.  The amount of coupling going on in this code makes it very difficult to build properly targetted unit tests.
Assignee: nobody → martin.thomson
Status: NEW → ASSIGNED
Attachment #8497817 - Flags: review?(dkeeler)
Comment on attachment 8497817 [details] [diff] [review]
0001-Bug-1072382-Remove-version-intolerance-marker-on-ina.patch

Review of attachment 8497817 [details] [diff] [review]:
-----------------------------------------------------------------

Ok - this makes sense. In terms of testing, we should put it on the list of things to test when we have the infrastructure for it. Also, next time I'd appreciate a patch with a bit more context - I think 8 lines is common (this looks like 3).
Attachment #8497817 - Flags: review?(dkeeler) → review+
Yeah, my apologies on the context.  `git config --global diff.context 10` applied.

Here's the try run: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=fcc145ea0ad5
Blocks: 1075991
Blocks: POODLE
https://hg.mozilla.org/mozilla-central/rev/4c05532b6342
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
Depends on: 1088950
[Tracking Requested - why for this release]:
Comment on attachment 8497817 [details] [diff] [review]
0001-Bug-1072382-Remove-version-intolerance-marker-on-ina.patch

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
An issue was introduced with bug 1036737, which was uplifted to ESR.  This affects Google properties with a fair degree of regularity.

User impact if declined: Some users will be unable to connect to Google properties.  This can only be fixed by restarting the browser.
Fix Landed on Version: 35
Risk to taking this patch (and alternatives if risky): Little; this has been on Nightly for a long time; Google implement the same mechanism (this corrects an incorrect implementation)
String or UUID changes made by this patch: None
Attachment #8497817 - Flags: approval-mozilla-esr31?
Attachment #8497817 - Flags: approval-mozilla-esr31? → approval-mozilla-esr31+
Martin,

Thanks for getting a patch in for this so quickly.
Duplicate of this bug: 1107578
Since you fixed this one - do you want to add the patch from bug 1088950, too?
Flags: needinfo?(martin.thomson)
:kaie, well, that's a lot more code.  And the only risk is:

When visiting a TLS 1.2 site with the fallback SCSV enabled, if a fallback is induced twice, then you get a fatal error loading the page and you have to hit refresh.  Given the low probability of one erroneous fallback, a double fallback is highly unlikely.  Oh, and it only occurs the first time you visit the site after loading the browser, so you aren't going to get broken images and that sort of thing.

With this patch, the error doesn't require a restart, reloading the page fixes the problem.  

It's not ideal, but I think that I'll recommend we don't take bug 1088950, unless someone feels strongly about it.
Flags: needinfo?(martin.thomson)
So I've been trying to reproduce the original issue for about 2-3 hours and only managed to get the "Error code: ssl_error_inappropriate_fallback_alert" once when loading Google Play. I tried using both Ubuntu and Windows 7 with and without the HTTPS Everywhere extension as it seems Rob North can reproduce it pretty consistently while he has it installed.

Martin, is there an easier way to make this happen? It would make verifying this a lot easier. Rob, would you mind checking to see if the fix is working for you? From comment #6, it seems like you can consistently reproduce the error.
Flags: needinfo?(martin.thomson)
Flags: needinfo?(6jju4k002)
Kamil, this is a tricky one to replicate without some extra help.  You can't just sit there and wait for the network to do its magic :)

See this: https://bugzilla.mozilla.org/show_bug.cgi?id=1036737#c8

Now the script there only performs fallback from TLS 1.2 always.  For this particular issue, you probably only want it to trigger once:

Instead of:
>     if (d[9] === 3 && d[10] === 3) {
I think that this might work (if I have my indices correct):
>    if (d[9] === 3 && d[10] === 3 && !fallen) {
>      fallen = true;
>      // make sure to declare var fallen = false in an outer scope though

That falls back just once.  Restart the script to go again.

Adjust the probability, or create a more deterministic trigger condition if you like.
Flags: needinfo?(martin.thomson)
Martin,

* used the code from Bug # 1036737 comment # 8 with the changes added from comment # 22
* edited /etc/hosts to point www.google.com at 127.0.0.1

Once I had everything setup, I used the "reproduced" build listed below and received the following error every time:
* (Error code: ssl_error_inappropriate_fallback_alert)

After reproducing, I downloaded the latest 31.4.0 esr RC build (listed below) and couldn't reproduce the fallback error anymore. However, I did receive the following error:
* (The connection was interrupted)

Looking the wireshark data, I noticed that I received an "Encryption Alert" entry when I reproduced the issue and didn't receive the alert when using the newer build with the fix. I've also attached the wireshark logs just in case you would like to take quick look.

So it looks like it looks fixed, but I'm not sure about the "(The connection was interrupted)" error message. Is this expected?

Builds:

* Reproduced with: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/12/2014-12-01-00-05-01-mozilla-esr31/
* Couldn't reproduce with: http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.4.0esr-candidates/build1/linux-x86_64/en-US/
Flags: needinfo?(martin.thomson)
(In reply to Kamil Jozwiak [:kjozwiak] from comment #24)
> Created attachment 8546055 [details]

That only includes TCP ACK segments, no SYN or PSH.  I can't make head nor tail of it without any data.

So I've tried the latest ESR and I think I understand what is going on.

Prior to this fix landing, the error was persistent.  If you did a fallback, then Firefox would try to connect with TLS 1.1 and the fallback SCSV constantly.  That produces the ssl_error_inappropriate_fallback_alert as you saw, every time, even when you don't reset the connection.

With this fix, Firefox sees the error and resets its state.  What happens (at least when I tested), is that *after a clean startup* with the script running, you get a *single* alert.  Then, if you retry, you should connect successfully.  If you restart the script, but not the browser, you will see a connection interrupted message instead because we are remembering that www.google.com:443 is available on TLS 1.2.  At that point, Firefox does not fall back and so doesn't trigger the alert.  Firefox remembers that until you restart the browser.

(The problem this bug fixes is that we were incorrectly remembering that www.google.com:443 was intolerant to TLS 1.2, even though we had a signal from it that it probably wasn't.)
Flags: needinfo?(martin.thomson)
I still face this issue on Firefox 35.
http://ww1.sinaimg.cn/large/474ac5b3gw1eoab3ydmy9j20nr0c6tag.jpg
Maybe because Great Firewall of China.
(In reply to Liu Xing from comment #26)
> I still face this issue on Firefox 35.
> http://ww1.sinaimg.cn/large/474ac5b3gw1eoab3ydmy9j20nr0c6tag.jpg
> Maybe because Great Firewall of China.

Maybe.  If the GFW is performing a downgrade attack for the purposes of a MitM, you are unlikely to see an inappropriate_fallback alert from github.com.  If a largely passive network observer is resetting the TLS connection so that it can monitor (and presumably decrypt) a version of TLS that it supports, then that might be the case.  If that error message appears every time, then it could be the latter.  If it only occurs occasionally, it might just be that you are encountering network reliability issues.

The good/bad news is that this is going away in Firefox 36 (I think that it's 36), because it looks like we'll remove the TLS version fallback altogether.
Une erreur est survenue pendant une connexion à personal&private.net. Le serveur a rejeté le message d'établissement de liaison car le client utilise désormais une version de TLS inférieure à la version minimale prise en charge par le serveur. (Code d'erreur : ssl_error_inappropriate_fallback_alert)
Juste after updating firefox.... I'm on 7 x64 bit.
Everything fine I've evenbunistall miscrosot security essential but nope....
No fireware.....

It appear everytimes and occasionaly it ceasse for an hour......
Flags: needinfo?(martin.thomson)
Every update of Ff does the same.... Why? Why??
(In reply to b.cellier962 from comment #29)
> Every update of Ff does the same.... Why? Why??

What version of firefox are you using??
This error code indicates that there is a man-in-the-middle attack on your connection.  If you have something on your network (a proxy, for instance) that is forcing a TLS downgrade, you can get this message.

As for the "Why? Why??", well, now you know about the downgrade attack.

What server causes this message?
Flags: needinfo?(martin.thomson)
(In reply to Martin Thomson [:mt] from comment #31)
> This error code indicates that there is a man-in-the-middle attack on your
> connection.  If you have something on your network (a proxy, for instance)
> that is forcing a TLS downgrade, you can get this message.
> 
> As for the "Why? Why??", well, now you know about the downgrade attack.
> 
> What server causes this message?

I've solve the problem for 1h by kicking my A.Vir But it stopped woking....

35.0.1 - With the Enhanced Steam extension, Firefox could crash (1123732)
Fixed

35.0.1 - Fix a potential startup crash (1122367)
Fixed

35.0.1 - Kerberos authentication did not work with alias (1108971)
Fixed

35.0.1 - SVG / CSS animation had a regression causing rendering issues on websites like openstreemap.org (1083079)
Fixed

35.0.1 - On Godaddy webmail, Firefox could crash (1113121)
Fixed

35.0.1 - document.baseURI did not get updated to document.location after base tag was removed from DOM for site with a CSP (1121857)
Fixed

35.0.1 - With a Right-to-left (RTL) version of Firefox, the text selection could be broken (1104036)
Fixed

35.0.1 - CSP had a change in behavior with regard to case sensitivity resources loading (1122445)

Préférences modifiées importantes
Nom 	Valeur accessibility.typeaheadfind.flashBar	0
browser.cache.disk.capacity	358400
browser.cache.disk.smart_size.first_run	false
browser.cache.frecency_experiment	4
browser.places.smartBookmarksVersion	7
browser.sessionstore.upgradeBackup.latestBuildID	20150122214805
browser.startup.homepage_override.buildID	20150122214805
browser.startup.homepage_override.mstone	35.0.1
dom.mozApps.used	true
extensions.lastAppVersion	35.0.1
gfx.direct3d.last_used_feature_level_idx	0
media.gmp-gmpopenh264.lastUpdate	1421587728
media.gmp-gmpopenh264.version	1.1
media.gmp-manager.lastCheck	1422365226
network.cookie.prefsMigrated	true
places.database.lastMaintenance	1422399625
places.history.expiration.transient_current_max_pages	104858
plugin.disable_full_page_plugin_for_types	application/pdf
plugin.importedState	true
plugin.state.npgoogleupdate	0
plugin.state.npmcafeemss	0
privacy.sanitize.migrateFx3Prefs	true
storage.vacuum.last.index	1
storage.vacuum.last.places.sqlite	1421588856
P.S : I think that might came from a cookie....
I have destroy the evil buging and laging cookie... But still than related websites have the Same SSL problem....
You need to log in before you can comment on or make changes to this bug.