Closed Bug 1750647 Opened 4 years ago Closed 2 years ago

Secure Connection Failed An error occurred during a connection to developer.mozilla.org. PR_END_OF_FILE_ERROR

Categories

(Core :: Networking, defect, P3)

Product:
Core
Component:
Networking
Version:
Firefox 96
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: harb, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [necko-triaged])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62

Steps to reproduce:

I clicked on one of your links while trying to get the browser to connect to various URLs such as your own (developer.mozilla.org). Also happens on bing.com. I also "refreshed" Firefox without any different result and also uninstalled Firefox, rebooted the computer and then downloaded and installed it again all with the same result. However, it does seem to go to duckduckgo.com with no problem.

Actual results:

Secure Connection Failed

An error occurred during a connection to developer.mozilla.org. PR_END_OF_FILE_ERROR

Expected results:

Firefox should have displayed the content on the URL site.

The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Component: Untriaged → Networking
Product: Firefox → Core

Hi, can you make a http log:
https://firefox-source-docs.mozilla.org/networking/http/logging.html

Note: the log may contain cookies, so please try to create a log while you are not logged into any site. Also, log contains addresses of sites you are visiting while making the log.

Do you use any firewall?

Thank you.

Flags: needinfo?(harb)

You may also send the log to my email address or to necko@mozilla.com.

When I attempted to go to that logging link here is the result:

"Secure Connection Failed

An error occurred during a connection to firefox-source-docs.mozilla.org. PR_END_OF_FILE_ERROR

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.

Learn more…"

Flags: needinfo?(harb)

I just sent the log file to your email address.

We are getting the error from the TLS layer(NSS).

The log does not have enough information to figure out why. This has most likely to do with something in the network.

Is this starting to happen recently? Was this happening before January 11th? (There was a firefox update on January 11th.).

Has something changed in the network, e.g. a new firewall, new network provider, a new router?

Can you make a wireshark trace? That we can examine what is happening on the network.

Flags: needinfo?(harb)

Dennis, do you have any other ideas?

what I see from the log is that we start TLS handshake and get PR_END_OF_FILE_ERROR before the handshake is done. 0RTT is not used.
Maybe wireshark trace will give us more information.

Flags: needinfo?(djackson)
See Also: → 1710202

I have just downloaded and installed wireshark. Now what do I do with it to get you more of the information you need?

Flags: needinfo?(harb)

Hi Larry,

Thanks for your help so far!

This error code is typically caused by the network connection closing unexpectedly. This could be because we’re connecting to the wrong network address and the server we reach doesn’t want to talk to us, or it could be because a device on the network is interfering with the connection.

It would be great if you could use Wireshark to record the network traffic between your browser and the server so that we can see what is going on. Please be aware that the Wireshark trace will record the websites your browser is connecting to, so be sure to close any unrelated tabs.

To capture a network trace of your TLS sessions, you can start Wireshark and where it says “Enter a capture filter”, type in “tcp port 443”, then double click your network interface. This might be “Wi-Fi: en0” or something similar. You should see a busy looking network activity graph next to the right interface. Once selected, you should see packets starting to scroll by.

If you could now open Firefox and navigate to the websites you mentioned in your report, both the ones that work and the ones that don’t. If you have Chrome or another browser installed, please open that as well and navigate to the same sites so we can see any differences.

Once you’re done, click the red square in Wireshark to stop the capture and use File->Save to save the resulting pcap. You can either attach the pcap to this bug report or use the email address Dragana provided above.

Flags: needinfo?(djackson) → needinfo?(harb)
Assignee: nobody → djackson

I just sent the requested log file via email to Dragana.

From your email, it sounds that this is a new regression in Firefox 96. Can you try to run a tool that could detect the change that cause the regression:

https://mozilla.github.io/mozregression/

I hope the link below contains the information you are asking for.

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=08eb1047d841ce4f1fcee1595dc4ec7ed8cba8f5&tochange=1ef6c9bb62b9fd1339d080ea1dccce66c80eb8c8

The mozregression to version 95 does not cause the issue I have in version 96.

Flags: needinfo?(harb)

Summary notes so far:

  • Version 95 is reported not impacted, Version 96 is impacted. I don't see any commits in the mozregression range that touch the TLS or Necko layers.
  • The issue affects some websites (developer.mozilla.org, bing.com) but not others (duckduckgo.com works)
  • There's no obvious signals in telemetry indicating a widespread issue, although if telemetry does require TLS to be working in order to report.
  • Looking at the provided packet captures:
    • The connection from Firefox to bing.com is using TLSv1.2 with Session Resumption via a ticket. It appears to stall almost immediately but not drop.
    • Chrome and Firefox both use the same IP address, so unlikely to be a DNS issue but we have since this issue before when Firefox is using DoH.
    • The connections from Chrome to bing.com offer TLSv1.3 but negotiate TLSv1.2 and are issued with session tickets.
    • In my own testing, bing.com doesn't support TLSv1.3 and I couldn't get it to issue me a session ticket with TLSv1.2.
    • No evidence of any middlebox interference, the correct server certs are provided.

@Dragana - Do you know if we've changed anything in how session tickets are handled or partitioned recently? Is it possible we're getting stuck with bad TLS state which we should be gracefully recovering from?

@Larry - To help us narrow this down, could you see if either option below fixes the problem:

  • Disabling DNS over HTTPS by setting network.trr.mode to 5 in about:config
  • Disabling the TLS Session Ticket cache by setting security.enable_tls_session_tickets to false. (You may need to create this config value)

For the change in preference to take effect, Firefox will need to be restarted.

Flags: needinfo?(harb)
Flags: needinfo?(dd.mozilla)

Hi Larry, thank you for your help.
I am puzzled by the change log that introduced the regression. There is nothing in it that could have caused it.

I have also added an IP address mapping for www.bing.com to resolve to the same IP address as in your pcap. I get session tickets and they are used. For me the site loads as expected.

I am trying to figure out what is happening. Larry can you create an HTTP log and pcap at the same time to capture the same request. I will try to match the pcap packets to the HTTP log. This may help me understand what is happening.

If it is not big trouble can you do the above for Firefox 95 that works as well?

Thank you.

Flags: needinfo?(dd.mozilla)

Dragana Damjanovic,
I very much appreciate all the assistance I have received on this issue from Mozilla Bugzilla. It has consumed much of my precious time. For that reason I have fallen behind in my work to a severe extent and need to spend all my available time getting caught up on my normal work. For that reason I really cannot spend any more time on this at this time. If by some chance I do have an opportunity I will indeed do so but right now I do not see that I will be able to do so. Again, I thank you very much and please accept my sincere apology for not being able to fulfill your request at this time.

Flags: needinfo?(harb)

Hi Larry,

That's no problem at all, we're grateful for all the help you've already provided.

Hi Larry, thank you for your help. It is very appreciated.

The severity field is not set for this bug.
:kershaw, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(kershaw)

I suspect you may already know this but it has come to mind that I may not have told you that Cloudflare was active under the Firefox Network connection settings and when I unchecked Enable DNS over HTTPS I could still see "Use Provider Cloudflare Default", but it is now grayed out.
I am thinking that if there happens to be some other provider than Cloudflare this issue may not have occurred.

Let me thank you yet again for helping me resolve the issue. It is still working just fine. I hope this helps.

(In reply to Larry H from comment #20)

Let me thank you yet again for helping me resolve the issue. It is still working just fine. I hope this helps.

Hi Larry,

Did you fix this issue by disabling DoH (DNS over HTTPS)? If so, could you try to capture another http log without DoH? Maybe we can find the difference from the log.
Feel free to do this when you have time. Thanks again for your help.

Flags: needinfo?(kershaw) → needinfo?(harb)

I followed Dennis Jackson's instruction in comment 14 which was:
@Larry - To help us narrow this down, could you see if either option below fixes the problem:

Disabling DNS over HTTPS by setting network.trr.mode to 5 in about:config

This caused Firefox to work correctly as soon as it was restarted.
AND:
In my comment 20 above I also provided the following additional information:
"I suspect you may already know this but it has come to mind that I may not have told you that Cloudflare was active under the Firefox Network connection settings and when I unchecked Enable DNS over HTTPS I could still see "Use Provider Cloudflare Default", but it is now grayed out.
I am thinking that if there happens to be some other provider than Cloudflare this issue may not have occurred.

Let me thank you yet again for helping me resolve the issue. It is still working just fine. I hope this helps."

Flags: needinfo?(harb)

The severity field is not set for this bug.
:kershaw, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(kershaw)

Set this to P3 for now, since we don't have enough information to debug this.

Blocks: doh
Severity: -- → S4
Flags: needinfo?(kershaw)
Priority: -- → P3
Whiteboard: [necko-triaged]
Assignee: djackson → nobody
Status: UNCONFIRMED → NEW
Ever confirmed: true

(In reply to Kershaw Chang [:kershaw] from comment #24)

Set this to P3 for now, since we don't have enough information to debug this.

Kershaw Chang,
Would you please explain to me what P3 is and what it means? I am just a user of Firefox and not a developer so I am not familiar with all the terms and acronyms.

Larry

(In reply to Larry H from comment #26)

(In reply to Kershaw Chang [:kershaw] from comment #24)

Set this to P3 for now, since we don't have enough information to debug this.

Kershaw Chang,
Would you please explain to me what P3 is and what it means? I am just a user of Firefox and not a developer so I am not familiar with all the terms and acronyms.

Larry

P3 means this bug will be put in our backlog and will be looked at some point.
The reason I marked this as P3 is that we don't have enough information to figure out why disabling DNS over HTTPS is help to fix this issue.
Once you provide us more information, we will be happy to increase the priority and try to fix this as soon as possible.
Thanks.

Sometimes I can reproduce. Right now (in private window) Ctrl+Shift+R helped, without disabling DoH. Other times clearing cache in about:networking#dns helped IIRC. Seems to be related to server IP?
Will another wireshark capture help?

It's a widespread issue. I keep getting this error while accessing different websites. I am at a point where I am thinking of switching to Chrome.

Could you try to capture a http log that shows this issue?
Thanks.

Flags: needinfo?(remtanmajitenshi)
Flags: needinfo?(achansoriya)

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit BugBot documentation.

Flags: needinfo?(achansoriya)

Close as INCOMPLETE.
Feel free to open another bug if this happens again. Thanks.

Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(remtanmajitenshi)
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.