Closed Bug 1468892 Opened 3 years ago Closed 3 years ago
_ERROR _RX _RECORD _TOO _LONG with Avast intercepting TLS 1 .3 connections
254.00 KB, application/octet-stream
5.56 KB, application/octet-stream
341.14 KB, application/octet-stream
112.22 KB, application/octet-stream
70.05 KB, application/x-zip-compressed
40.23 KB, image/png
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0 Build ID: 20180614135649 Steps to reproduce: Attempt to access any one of these sites with FF 61.0b14 (64-bit) 20180614135649 : https://www.windowscentral.com/ https://thehackernews.com/ https://www.androidcentral.com/ https://www.geekgyaan.com/ https://www.onmsft.com/ https://www.bleepingcomputer.com/ Similar to Bug 1467684, but this needs to be classified as MAJOR and affects FF 61. Actual results: Error: Secure Connection Failed ... SSL_ERROR_RX_RECORD_TOO_LONG Repeating sometimes gets: The connection to www.bleepingcomputer.com was interrupted while the page was loading. Expected results: No error... View web page as expected. Works OK with FF 60.0.2 and other browsers.
OK... Maybe back-burner this bug report? It appears that Avast 18.4.2338 (build 18.4.3895.327) may be interfering with something. Access to these sites is OK with FF 61 if Avast is disabled, but I don't know why FF 60, IE, and PaleMoon are NOT affected with Avast ENABLED.
1. Does the problem occur with the latest Nightly in a brand new profile? https://www.mozilla.org/firefox/nightly/all/ https://support.mozilla.com/kb/profile-manager-create-and-remove-firefox-profiles 2. Is Avast's certificate installed in Firefox 60 but not the other versions? 3. (In reply to [:keeler] (use needinfo) from bug 1384302, comment 1) > Can you get a packet trace of the TLS handshake (use e.g. wireshark) and > attach it? Thanks.
Component: Untriaged → Security: PSM
Product: Firefox → Core
Summary: Secure Connection Failed error on some HTTPS websites, no workaround → SSL_ERROR_RX_RECORD_TOO_LONG with Avast | Secure Connection Failed error on some HTTPS websites, no workaround
Same problem with firefox-62.0a1.en-US.win64_20180615100050 with or without a new profile. However, sometimes it works! It also sometimes works correctly if I type in another URL after the failure (like weather.gov), then once loaded use the back arrow upon which the site that had an error loads correctly. A Wireshark trace will take a little while... You can add https://www.wireshark.org/ to the site that fail.
Avast "Web Shield" on for all but the last file. 005_FF60_filtered.pcapng FF 60.0.2 (works!) 006_FF62_filtered.pcapng firefox-62.0a1.en-US.win64_20180615100050 (broken) 007_FF62_filtered.pcapng " 008_FF62_noAvast_filtered.pcapng FF 62.0a1 with Avast off (works) Webshark Filter: not arp and (ip.addr == 172.17.83.18 and (ip.addr == 22.214.171.124 or ip.addr == 126.96.36.199 or dns)) Name: www.bleepingcomputer.com Addresses: 188.8.131.52 184.108.40.206
Thanks! The main difference I see between 60 and 62 is negotiating different TLS 1.3 draft versions. We might be able to tell more if you capture the traces again with the environment variable SSLKEYLOGFILE set to some temporary path (note: you probably want to do that with a new profile or in private browsing mode so you don't leak any cookies (and also don't log in to anything when you do)).
I can't get Firefox to generate the ssl log file. After some investigation it appears that Avast, even when "disabled", modifies the contents of SSLKEYLOGFILE sent to the Firefox environment from a valid path\filename to \\.\aswMonFltProxy\FFFFFA800D450270. If anyone has ideas on how to deal with this, let me know. (No, removing Avast isn't an option since it's needed to debug the issue in this bug report!) I've opened a support ticket with Avast. We'll see how responsive they are. ;-)
Disabling HTTPS scanning works for me: https://support.avast.com/en-ww/article/189/
Good suggestion. It does allow access to "bleepingcomputer.com" and the others, but it doesn't change what Avast does to the SSLKEYLOGFILE variable. It's stuck at the \\.\aswMonFltProxy\FFFFFA80... value (the last 32-bits of the hex number change every time Firefox is started. I suspect that's their hook into HTTPS scanning, but they don't get rid of it when it's "disabled". So... The bug is still present, either with Avast 18 or Firefox 61+, and getting the clear TLS exchange for debug is going to be difficult. What's needed is a "tee" that directs what Avast gets to both the Avast HTTPS scanner and a file.
Not much progress with Avast on this... "do you have the lastest version?", "all browsers have had problems with upgrades", "we haven't heard of this before", etc. I think I finally got the supervisor to record the Bugzilla number and she said it would be forwarded to development, but "no promises". I have an Avast incident ID. Send private email to me if you need it.
I just verified the "bug" is still present with the FF 61.0 build1 release candidate ID 20180618212916. Sometimes I also see the error SSL_ERROR_BAD_MAC_READ, if that helps, but I've seen that with 62 also.
After doing a binary search (fun?) of working vs. non-working versions of FF61 yielded a start of the failure: firefox-61.0a1.en-US.win64_20180419100148.zip OK firefox-61.0a1.en-US.win64_20180419224145.zip Bad <--- Surprise! It looks like there was an "upgrade" to NSS between these builds. See bug 1445731. The big question is what's REALLY broken, Avast or FF? I have my guess, but I'll refrain from pointing fingers.:-) Regardless, I've hit a number of additional sites that get "Secure Connection Failed" errors with FF61+Avast. IMHO I suspect that HTTPS scanners other than Avast's could generate similar problems given what I see in Google search results. I'd hate to see FF 61 hit the streets and generate a raft of complaints -- regardless of what's really to blame.
I also see this issue with Firefox 62 and Avast. Turning off HTTPS scanning seems to get rid of the issue. I will point out up until a week or so ago I had not encountered this issue. This leads me to think a change in Firefox is the cause, but at the same time if this is happening on different versions of Firefox well then maybe Avast is to blame. Either way there does appear to be an issue around Avast's HTTPS scanning and Firefox validating the secure connection. Firefox Version: 62.0a1 Build ID: 20180619220118 Update Channel: nightly User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0 OS: Windows_NT 6.1 Avast Version: 18.5.2342 Build: 18.5.3931.0
What happens if you set "security.tls.version.max" to 3 in about:config?
(Also if this is a result of something Firefox is doing, this bug should be in NSS.)
Assignee: nobody → nobody
Component: Security: PSM → Libraries
Product: Core → NSS
Version: 61 Branch → other
Steps: Enabled HTTPS scanning in Avast Set 'security.tls.version.max' to 3 (default is currently 4) Closed Firefox and restarted computer Launched Firefox and opened 20 tabs each with a site using HTTPS Refreshed all tabs 10 times --Issue NOT observed Close tabs Set 'security.tls.version.max' to 4 Closed and opened Firefox Open same sites as last time Refresh all tabs 10 times --Observed sites failing to load with 'SSL_ERROR_RX_RECORD_TOO_LONG' error. Note the same sites did not fail all 10 refreshes. Hope this helps.
Summary: SSL_ERROR_RX_RECORD_TOO_LONG with Avast | Secure Connection Failed error on some HTTPS websites, no workaround → SSL_ERROR_RX_RECORD_TOO_LONG with Avast intercepting TLS 1.3 connections
It sounds like you can easily repro. Can you please get us a PCAP of a failing connection
Also can you test it with a release version of Chrome?
I don't have Chrome Stable installed so I hope the latest BETA will do. I also added a capture from Firefox with Avast's HTTPS scan turned off.
MT, can you take a look. It looks to me like it is negotiating 1.3 in each case, so it's not obvious what's going on here....
HTTP logs from Firefox while reproducing the issue by navigating to https://www.hardocp.com/ while Avast's HTTPS scan in on. Not sure if this information will help or not.
Sorry for the delay. Yes, setting security.tls.version.max to 3 appears to "fix" the problem. Earlier today I received email from Avast support that had me totally uninstall Avast using avastclear.exe then re-install version 18.5.3931.0 that was just released today. Problem sites still fail, but I did have a couple of sites that previously failed consistently load once. It does appear Avast is monitoring this bug on Bugzilla.
Sadly, those traces aren't really that helpful, thanks partly to better encryption in TLS 1.3. From the outside, that record is fine. It's big (0x105e), but not too big (0x4001). And it certainly seems possible that the last message is an alert from our end being sent out. It's possible that there is a bug in our stack, but I can't tell without more information. I'll have to test with Avast enabled, a prospect I don't look forward to. More forthcoming.
I finally got this reproducing reliably and got traces. I confirmed that Avast installs a trust anchor, but I don't understand how it is intercepting sockets, so I can't test this outside the browser, so my ability to capture traces are extremely limited. It's definitely a problem. It seems like it is transitory, the first connection to the site is quite slow, and it fairly reliably hits this problem. Reload and the site is fine. From the trace, this is distressing, but probably not relevant (this is why you run this sort of thing in a VM): SSL: logging SSL/TLS secrets to \\.\aswMonFltProxy\AB7B5428 Wading through the noise shows this: 5512: TLS13: client installed key for epoch=2 (handshake data) dir=read 5512: TLS13: client state change from wait_server_hello->wait_encrypted_extensions in tls13_HandleServerHelloPart2 (c:/code/gecko/security/nss/lib/ssl/tls13con.c:2597) 5512: SSL3: gather state 1 (need 5 more) 5512: SSL: raw gather data: [Len: 5] 14 03 03 00 01 ..... 5512: SSL3: gather state 2 (need 1 more) 5512: SSL: raw gather data: [Len: 1] 01 . This right here is clearly the ChangeCipherSpec, which we ignore... 5512: SSL: got record of 1 bytes 5512: TLS13: spec=343100416 epoch=2 (handshake data) unprotect 0x0 len=1 5512: TLS13: record too short to contain valid AEAD data 5512: SSL3: gather state 1 (need 5 more) ... some other sockets here 5512: SSL: ssl_Poll flags 6 -> 5 5512: SSL3: ssl3_GatherCompleteHandshake 5512: SSL3: gather state 1 (need 5 more) 5512: SSL: raw gather data: [Len: 5] 3b 9b fb 91 f0 ;.... 5512: SSL3: send alert record, level=2 desc=22 That is garbage. It should be a record header. The only reason we're getting the TOO_LONG error is that this is the first check; 0x91f0 isn't a valid length by any definition. The same sort of garbage doesn't show up in Wireshark traces. Every trace I've managed to capture is fine. I'm going to say that this is a problem with Avast with high probability. I have no idea what it's doing that might cause this to happen. I've run out of time on this for now. Given that the workaround is easy (either disable TLS 1.3 or disable HTTPS scanning in Avast), it seems like we don't have to rush to find a solution.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P3
we have multiple avast/avg users reporting this issue after the update to firefox 61 where tls 1.3 is generally rolled out. given that avast is the most common av provider among firefox users, could we bump up the priority of this bug?
could you take a look into this issue from your end as well?
Hi, thanks to all for the report. We (Avast) already have a fix for that and are just testing it. We'll release in a couple of days. Lukas.
As mentioned this is also having an affect on AVG Users. Is there any timeline/estimate as to when this will be fixed? Are AVG Aware of a potential bug?
avast and avg are the same company by now, so hopefully the fix will be provided in tandem
Yes, we are fixing both avast and avg. What we are missing in the product is a tls/1.3 detection code and our tls/1.2 implementation is failing on the new ciphers.
I don't think this needs to block rollout of Fx61 tomorrow, but I do think we should add a relnote for it and link people to the directions for how to disable HTTPS scanning in Avast until a fixed version is released.
Do we have telemetry data on how frequent SSL_ERROR_RX_RECORD_TOO_LONG is and can we monitor that during 61 rollout?
If my reading of https://searchfox.org/mozilla-central/rev/14cb8f1c238735ba1abe18ad44e39808983c2572/security/manager/ssl/nsNSSIOLayer.cpp#1214 is correct, it should be bucket 25 of SSL_HANDSHAKE_RESULT (SSL_ERROR_RX_RECORD_TOO_LONG is SSL_ERROR_BASE + 25). The tricky part is that if avast is intercepting all connections for a user, presumably they wouldn't be able to send us telemetry unless they disable scanning.
Telemetry is supposed to use tls 1.2 (bug 1325501)
Hi, the fix on Avast side should be released shortly, in the next 24 hours. Disabling https scanning has security implications for other non tls/1.3 sites as well and it might be difficult to make the users turn it back again.
What is the usual time frame for an Avast update to reach 50% and say 95% of your user population?
Lukas, when you say the issue is in TLS 1.3 detection code, are you trying to parse the ServerHello, despite not generating the ClientHello? My understanding is Fx60 also had TLS 1.3 support, albeit with draft-23 instead of draft-28. Why did it start breaking in Fx61? Are you trying to parse out the version field and special-casing draft versions one-by-one? That is non-compliant behavior and will, of course, break every time we add a new version of TLS or a random new server extension. Please see the Protocol Invariants section of the TLS 1.3 draft. https://tlswg.github.io/tls13-spec/draft-ietf-tls-tls13.html#rfc.section.9.3 It is not valid to process ServerHellos in response to ClientHellos containing parameters you didn't select. These parameters may be newer than your product and may change the ServerHello and all subsequent messages in arbitrary ways.
The added release note advising of workarounds is now live. Bug 1471672 was also filed to track possible workarounds we can deploy on our end for users with Avast/AVG installed.
See Also: → 1471672
Disappointed with the release notes. The workaround should also contain the suggestion of setting security.tls.version.max to 3 in about:config.
Lukas, can you please respond to comment 41? What does the typical rollout look like from your end? It would help us better-understand what options we need to consider going forward. Note that we're still exploring the option of disabling TLS 1.3 for Avast/AVG users on our side if it proves practical.
(In reply to Lukas Rypacek from comment #44) > Disappointed with the release notes. The workaround should also contain the > suggestion of setting security.tls.version.max to 3 in about:config. https://www.mozilla.org/en-US/firefox/61.0/releasenotes/ links to this support article: https://support.mozilla.org/en-US/kb/secure-connection-failed-error-message#w_avast-and-avg-security-products That article was just updated to also include the about:config workaround of setting security.tls.version.max to 3.
Ryan, I'll give you some estimates first thing in the morning. We will not use a regular update for this, but rather a quick one, reserved for urgent cases only. It's already past midnight here in Prague and I need to reach to the team for the data about the emergency update distribution speeds. The updated binary should be delivered to connected clients very quickly (4-8 hours after its release)- not sure how fast we reach 50% penetration due to various PC use patterns. We also need to discuss if any further actions will be required - such as a reboot or just a Web Shield component restart. Alice, thanks for the updated support page.
Alice, I'd really prefer that we not tell people to frob about:config. This kind of thing is really hard for us to change later once this is fixed, and then people are just stuck with 1.3 off. I'd far rather disable TLS 1.3 for people who we think have Avast. RyanVM, thoughts on that?
(In reply to Eric Rescorla (:ekr) from comment #48) > Alice, I'd really prefer that we not tell people to frob about:config. This > kind of thing is really hard for us to change later once this is fixed, and > then people are just stuck with 1.3 off. > > I'd far rather disable TLS 1.3 for people who we think have Avast. RyanVM, > thoughts on that? The about:config workaround of manually setting security.tls.version.max to 3 was originally added to the support article in this revision: https://support.mozilla.org/en-US/kb/secure-connection-failed-error-message/revision/162656 ... which was based on this discussion thread post by jscher2000 : https://support.mozilla.org/en-US/forums/contributors/713055 "SSL_ERROR_RX_RECORD_TOO_LONG" with Avast and TLS 1.3 enabled Related discussion: https://support.mozilla.org/en-US/kb/secure-connection-failed-error-message/discuss/7582 The about:config workaround was removed following a series of article updates. I added it back to the article since I thought that users who are concerned about turning off HTTPS scanning in their security software should have an alternative method. In fact, the Avast support article (linked from our KB article) https://support.avast.com/en-en/article/189/ includes this warning: "Important: If you disable HTTPS scanning, any malware delivered by HTTPS traffic is hidden by TLS/SSL encryption and your computer is more vulnerable to threats." Also, Mozilla is considering setting security.tls.version.max to 3 for Avast/AVG users in bug 1471672. I'll NeedInfo Joni Savage the SUMO Content Manager so that she can review the article and make any needed changes.
Flags: needinfo?(alice.wyman) → needinfo?(jsavage)
(In reply to Eric Rescorla (:ekr) from comment #48) > I'd far rather disable TLS 1.3 for people who we think have Avast. RyanVM, > thoughts on that? Yeah, I think we're good to proceed with that. Will follow-up in bug 1471672. I also agree that we should be avoiding sending people to about:config for this when we have better mechanisms in our control for doing it instead (that will greatly reduce the chances of someone changing it and then forgetting).
(In reply to Ryan VanderMeulen [:RyanVM] from comment #50) > (In reply to Eric Rescorla (:ekr) from comment #48) > > I'd far rather disable TLS 1.3 for people who we think have Avast. RyanVM, > > thoughts on that? > > Yeah, I think we're good to proceed with that. Will follow-up in bug 1471672. > I also agree that we should be avoiding sending people to > about:config for this when we have better mechanisms in our control for > doing it instead (that will greatly reduce the chances of someone changing > it and then forgetting). Joni Savage did approve a revision that removed the about:config workaround setting security.tls.version.max to 3 so I should have waited for more feedback before adding it back. I made another revision to https://support.mozilla.org/en-US/kb/secure-connection-failed-error-message#w_avast-and-avg-security-products to again remove the about:config workaround. (Sorry!)
correct if i am wrong, :ekr and :joni but it's my belief that engineering doesn't recommend that we in SUMO document about:config preferences for "normal users" (i.e. the 90% of the population who can't figure out how to backup and restore and how to use the Firefox profile manager) because they can change at any time and they are not supported by engineering and have unpredictable and brittle interactions with other parts of the Firefox code.
Updating the severity
Severity: normal → blocker
Priority: P3 → P1
@rolandtanglao, right, we only recommend about:config for troubleshooting serious issues and always with a disclaimer.
Lukas: I don't work on Firefox, but I work on Chrome which is also shipping TLS 1.3. Can you please provide a response to the questions in comment #42? It's quite encouraging to see you all addressing the immediate issue quickly. Thank you for that. At the same time, we are also interested in ensuring this does not repeat with future versions (a future 1.4 and, more imminently, a switch from draft to final TLS 1.3) and extensions. While we do often need to account for defects in middleware such as AVs, we take their vendors' response into account when deciding how and whether to ship workarounds in Chrome. We expect vendors to be responsive to questions and follow the TLS specification, including and especially the Protocol Invariants section linked above. Thanks!
Hi, we have released the fix via VPS update 180628-08, we are also pushing out a more elaborate update. To answer your concerns, this is just a bug - not an intention to rely on some protocol version specific features. We are parsing out client hello and server hello, but we don't modify it here. In this case, by an error, part of the data was consumed/blocked by the parser. The next chunk was then handed to the browser directly and the browser looked at a random data at the beginning of the next chunk. Don't worry, our TLS/1.3 inspection code follows the specs and we'll release it soon, especially now, when Firefox is pushing it so hard.
TLS 1.3 draft 23 has been shipping in Firefox and Chrome for a while now. I'm not sure of Firefox's timeline, but it's been in stable Chrome since March. So either something changed more recently which triggered this bug, or people are only now noticing it for some reason. Do you know what it was? Firefox 61 includes draft 28, but, if that's the trigger, it's concerning that Avast is sensitive to it. (We actually have been getting reports of issues with Avast and Chrome, though we've never been able to reproduce it, and users reported reinstalling Avast solved it.) Note that parsing the ServerHello is explicitly disallowed by the specification. If you did not produce the ClientHello, there is no guarantee that you can usefully parse the ServerHello. Nor is there a guarantee that values you parse out of the ServerHello have the same meaning. Indeed earlier drafts of TLS 1.3 changed the format of the ServerHello altogether. Other versions used the same cipher suite code points as TLS 1.2, but changed the encryption around them. Cipher suites have tweaked meaning without changing code points before too, in each of TLS 1.1, TLS 1.2, extended-master-secret, and encrypt-then-MAC.
Draft 23 shipped enabled by default in Firefox 60 back in early May, FWIW.
As of now Avast does appear to have patched the problem. Running with Avast update 180628-08 both FF61.0 and FF62.0b3 are not experiencing any problems. Both are making TLS 1.3 connections with bleepingcomputer.com, www.theregister.co.uk, and tls13.crypto.mozilla.org. Sorry to cause you folks so much trouble... 1/2 ;-)
(In reply to Lukas Rypacek from comment #57) > Hi, we have released the fix via VPS update 180628-08, we are also pushing > out a more elaborate update. Lukas, do you have a way of monitoring patch uptake on your end so we have a sense of when we can safely consider rolling back our workarounds?
So with the latest nightly release I am encountering this issue again. The earlier posted workaround steps resolve the issue. So it seems like Avast is the cause again Firefox Version: 63.0a1 Build ID: 20180704100142 Update Channel: Nightly User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0 OS: Windows_NT 6.1 Avast Program Version: 18.5.2342 Build: 18.5.3931.0 Virus Definitions Version: 180704-2
Hmmm actually looking at Avast it seems like there was an update today, so it maybe a change on their side that is causing the issue to happen again.
Lukas, to repeat the question, can you please elaborate on exactly what Avast is doing here? The answers so far have not adequately explained the situation, given comments #58 and #59. Comment #62 is also odd as 180704-2 is larger than 180628-08. Or is that a different branch? I find it quite concerning that Avast is only breaking some TLS 1.3 deployments and not others. Even with the immediate issues fixed, that suggests Avast may break repeatedly in the future as we evolve TLS. Again, we expect vendors to be responsive to questions about issues with their products. We expect vendors whose products use non-complaint implementation strategies to actively work towards redesigning their products to match the specification. Note the Protocol Invariants section in particular. TLS 1.3 is a significant security and performance improvement for the entire Internet. There will be more such improvements to come. If non-compliance in some product holds back these improvements, this must be fixed. It's not acceptable for a small fraction of defective implementations hold back the entire Internet.
(In reply to Meinert from comment #62) > So with the latest nightly release I am encountering this issue again. The > earlier posted workaround steps resolve the issue. So it seems like Avast is > the cause again > > Firefox > Version: 63.0a1 > Build ID: 20180704100142 > Update Channel: Nightly > User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 > Firefox/63.0 > OS: Windows_NT 6.1 > > Avast > Program Version: 18.5.2342 > Build: 18.5.3931.0 > Virus Definitions Version: 180704-2 Hi Meinert, can you please verify the version of the filtering DLL: aswstreamfilter.dll, fixed version is: 18.5.3931.330. BTW, you guys have disabled TLS 1.3 for avast clients on FF 61 release version? To paraphrase David Benjamin, that's odd since 61 > 60.
Lukas: we only see problems with Fx61 and not Fx 60, both in reports and in Telemetry. Our working theory is that you have some problem with the way you are handling -28 but not -23.
I can confirm that aswStreamFilter.dll has a version number of 18.5.3931.330. I also noticed that the version definitions has received a couple updates today. Now on 180704-6.
Thanks. We will investigate. Do you have any specific domain where you can reproduce issues? As to the virus definitions - they are updated usually at least once a day, sometimes several times a day.
Hi Lukas, After updating to the latest definition, 180704-6, it seems as though the issue has been resolved once again. Using both 180704-2 and 180704-4 I had the issue. So I am guessing something got messed up in those pushes. If anything changes I will provide an update. As far as domains, when the issue was occurring visiting https://www.hardocp.com/ was a pretty reliable way for be to reproduce.
Hello, We have released a new version of Web Shield (aswStreamFilter.dll) yesterday (July 24, 2018). This version contains all the fixes and once again enables HTTPS scanning in Firefox (for people that have got the fixed version only). AV version 18.5.3931 aswStreamFilter.dll version: 18.5.3931.434 Filip Avast Team
Correction: aswStreamFilter.dll version: 18.5.3931.343
closing this out, I think we fixed this on our end with normandy recipes and now we have a fix out from avast.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.