1711738 - Crash in [@ mozilla::net::DnsAndConnectSocket::TransportSetup::SetupStreams]

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

13 crashes with 9 installations, all OS

Crash report: https://crash-stats.mozilla.org/report/index/a7a05ac8-4146-40d1-9790-0da350210518

MOZ_CRASH Reason: MOZ_DIAGNOSTIC_ASSERT(ent)

Top 10 frames of crashing thread:

0 xul.dll mozilla::net::DnsAndConnectSocket::TransportSetup::SetupStreams netwerk/protocol/http/DnsAndConnectSocket.cpp:1176
1 xul.dll mozilla::net::DnsAndConnectSocket::TransportSetup::OnLookupComplete netwerk/protocol/http/DnsAndConnectSocket.cpp:1249
2 xul.dll mozilla::net::DnsAndConnectSocket::OnLookupComplete netwerk/protocol/http/DnsAndConnectSocket.cpp:419
3 xul.dll mozilla::detail::RunnableFunction<`lambda at /builds/worker/checkouts/gecko/netwerk/dns/DNSListenerProxy.cpp:28:30'>::Run xpcom/threads/nsThreadUtils.h:534
4 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1153
5 xul.dll mozilla::net::nsSocketTransportService::Run netwerk/base/nsSocketTransportService2.cpp:1200
6 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1153
7 xul.dll mozilla::ipc::MessagePumpForNonMainThreads::Run ipc/glue/MessagePump.cpp:300
8 xul.dll MessageLoop::RunHandler ipc/chromium/src/base/message_loop.cc:328
9 xul.dll MessageLoop::Run ipc/chromium/src/base/message_loop.cc:310

Comment 1

[Dragana Damjanovic [:dragana]]

Attachment: Add diagnostic assertions to better understand bug 1711738 and 1711038

Comment 2

[Dragana Damjanovic [:dragana]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b994369bc9e79107b78f2f664aca1849806acf69

Comment 3

[Dragana Damjanovic [:dragana]]

Bug 1705065 was backed out, so 90 is no affected any more.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9099f9d14ed6
Add diagnostic assertions to better understand bug 1711738 and 1711038 r=necko-reviewers,valentin

Comment 5

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/9099f9d14ed6

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

So far 5 assertion failures hitting this in the latest Nightly, e.g. bp-5223a070-9a20-45cc-a79c-d9fd40210610

Comment 7

[Dragana Damjanovic [:dragana]]

Thanks to the new asserts, We have more information what is happening. I think I can fix this issue. I will work on patch today.

Comment 8

[Julien Cristau [:jcristau]]

Can we at least disable the diagnostic asserts asap?

Comment 9

[Dragana Damjanovic [:dragana]]

Attachment: When retrying to resolve a host always set the state to TransportSetupState::RETRY_RESOLVING. When a connection fails the status was set to TransportSetupState::RESOLVING which have a bad influence on the state of DnsAndConnectSocket.

Comment 10

[Dragana Damjanovic [:dragana]]

Attachment: Add more diagnostic assertion to DnsAndConnectSocket.

Comment 11

[Dragana Damjanovic [:dragana]]

(In reply to Julien Cristau [:jcristau] from comment #8)

Can we at least disable the diagnostic asserts asap?

I will try to resolve the issues as soon as possible.

There are 2 kind of crashes:

1. most common one should be resolved by D117706
2. the other kind of crash is:
   a988968b-91c0-4601-a3ad-9acff0210614
   I am not sure how it is getting to that state. I will add more assertions to figure it out.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/af3e019e67c8
When retrying to resolve a host always set the state to TransportSetupState::RETRY_RESOLVING. When a connection fails the status was set to TransportSetupState::RESOLVING which have a bad influence on the state of DnsAndConnectSocket. r=necko-reviewers,kershaw

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eeb43a0ccb88
Add more diagnostic assertion to DnsAndConnectSocket. r=necko-reviewers,kershaw

Comment 14

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/af3e019e67c8
https://hg.mozilla.org/mozilla-central/rev/eeb43a0ccb88

Comment 15

[Dragana Damjanovic [:dragana]]

Comment on attachment 9226936 [details]
When retrying to resolve a host always set the state to TransportSetupState::RETRY_RESOLVING. When a connection fails the status was set to TransportSetupState::RESOLVING which have a bad influence on the state of DnsAndConnectSocket.

Beta/Release Uplift Approval Request

• User impact if declined: Diagnostic asserts added in bug 1705065 showed that DnsAndConnectSocket is sometimes in a wrong state. This patch has fix that. The effect on users is unclear. This may help with crash in 1667102, but I do not have a real good explanation how it would help.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Fixes DnsAndConnectSocket state machine. And fix is verified in Nightly with a lot of diagnostic asserts that make sure that DnsAndConnectSocket is in a proper state.
• String changes made/needed:

Comment 16

[Julien Cristau [:jcristau]]

Given the low (known) impact, and given we're in the last week for beta 90, I'd prefer to let this ride the trains.

Comment 17

[Julien Cristau [:jcristau]]

Should this be marked as fixed now, or is there more work to do? (the leave-open keyword is still around from the first diagnostic patch)

Comment 18

[Dragana Damjanovic [:dragana]]

I was waiting a bit to be sure there is no other issue.

This can be closed now.