1595779 - Crash [@ neqo_http3::connection::Http3Connection<T>::close<T> ] with HTTP3 enabled

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P1)

Description

[Ryan Jones-Ward [:sciguyryan]]

Hi all.

After updating to today's nightly (12/11/2019) I've hit quite a few crashes with the signature [@ neqo_http3::connection::Http3Connection<T>::close<T> ].

This is (obviously) with HTTP3 enabled. Upon disabling it the crash is no longer reproducible.

11c6529d-4342-4f0f-a44e-663970191112 and db7f5454-dccf-4e59-bfba-fa9f00191112 are two examples that I have found but there are more listed here:
https://crash-stats.mozilla.org/signature/?product=Firefox&signature=neqo_http3%3A%3Aconnection%3A%3AHttp3Connection<T>%3A%3Aclose<T>&date=>%3D2019-11-05T15%3A04%3A00.000Z&date=<2019-11-12T15%3A04%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1

Comment 1

[Mike Conca [:mconca]]

I am hitting this crash, but with HTTP3 disabled. I have toggled the network.http.http3.enabled several times over the past few days, but I just confirmed that it was set to FALSE for the last two crashes.

Comment 2

[Lily Kim]

I've just encountered this bug twice in a row while simply having Twitch and Twitter open!

Comment 3

[Ryan Jones-Ward [:sciguyryan]]

(In reply to Mike Conca [:mconca] from comment #1)

I am hitting this crash, but with HTTP3 disabled. I have toggled the network.http.http3.enabled several times over the past few days, but I just confirmed that it was set to FALSE for the last two crashes.

That's interesting. I've yet to hit this without the preference enabled but I'll have to keep an eye out.

Comment 4

[Lily Kim]

I disabled the preference and the constant crashes stopped on my end.

Comment 5

[Marcia Knous [:marcia]]

Adding the macOS specific signature.

Comment 6

[Chris Peterson [:cpeterson]]

I disabled my network.http.http3.enabled pref, but I'm still hitting this crash signature (like a dozen times in a row) when loading this Bugzilla bug dashboard:

https://cpeterso.github.io/burndown/?since=2019-06-01&f1=cf_fission_milestone&o1=anyexact&v1=M4%2CM4.1%2CM5%2CM6%2CMVP

Comment 7

[Julien Cristau [:jcristau] (back April 22)]

"assertion failed: self.state_active()"

AFAICT this assertion was added in bug 1576051.

Comment 8

[Dragana Damjanovic [:dragana]]

The bug was fixed in neqo. I need to pull new version of neqo into mozilla-central.
I will make a quick fix for bug 1595337 (the bug needsto be fixed in neqo) and I will pull a new version of neqo.

Comment 9

[Dragana Damjanovic [:dragana]]

(In reply to Chris Peterson [:cpeterson] from comment #6)

I disabled my network.http.http3.enabled pref, but I'm still hitting this crash signature (like a dozen times in a row) when loading this Bugzilla bug dashboard:

https://cpeterso.github.io/burndown/?since=2019-06-01&f1=cf_fission_milestone&o1=anyexact&v1=M4%2CM4.1%2CM5%2CM6%2CMVP

I need to understand this one:
if your pref is disable you should not be hitting exactly this signature. If you do we have a different bug and I would like a http log to figure out what is happening. You may hit this bug if you had the pref on and you disable it, after disabling it there may be still some http3 connections running and when they are closed we hit this issue.

Comment 10

[Shravan Narayan]

AFAICT this assertion was added in bug 1576051.

So 1576051 was basically a mach vendor rust on the Firefox repo when working on an unrelated feature. Unfortunately, as i understand, mach vendor rust is a bit of a blunt tool, and may choose to upgrade unrelated packages which can be upgraded. From a cursory examination of 1576051, I don't believe any modules added there depends on neqo (will follow-up if I discover anything on further examination). It looks like this break is due to something in upstream neqo?

Comment 11

[Dragana Damjanovic [:dragana]]

(In reply to Shravan Narayan from comment #10)

AFAICT this assertion was added in bug 1576051.

So 1576051 was basically a mach vendor rust on the Firefox repo when working on an unrelated feature. Unfortunately, as i understand, mach vendor rust is a bit of a blunt tool, and may choose to upgrade unrelated packages which can be upgraded. From a cursory examination of 1576051, I don't believe any modules added there depends on neqo (will follow-up if I discover anything on further examination). It looks like this break is due to something in upstream neqo?

When I was doing mach vendor rust for neqo, I was told to do it based on autoland not mozilla-central. Doing it on autoland does not update any other crates except one needed.

Comment 12

[Dragana Damjanovic [:dragana]]

(In reply to Dragana Damjanovic [:dragana] from comment #11)

(In reply to Shravan Narayan from comment #10)

AFAICT this assertion was added in bug 1576051.

So 1576051 was basically a mach vendor rust on the Firefox repo when working on an unrelated feature. Unfortunately, as i understand, mach vendor rust is a bit of a blunt tool, and may choose to upgrade unrelated packages which can be upgraded. From a cursory examination of 1576051, I don't believe any modules added there depends on neqo (will follow-up if I discover anything on further examination). It looks like this break is due to something in upstream neqo?

When I was doing mach vendor rust for neqo, I was told to do it based on autoland not mozilla-central. Doing it on autoland does not update any other crates except one needed.

I just realize there is another fix for that neqo crates problem. I have filed a bug.

Comment 13

[Chris Peterson [:cpeterson]]

(In reply to Dragana Damjanovic [:dragana] from comment #9)

(In reply to Chris Peterson [:cpeterson] from comment #6)

I disabled my network.http.http3.enabled pref, but I'm still hitting this crash signature (like a dozen times in a row) when loading this Bugzilla bug dashboard:

https://cpeterso.github.io/burndown/?since=2019-06-01&f1=cf_fission_milestone&o1=anyexact&v1=M4%2CM4.1%2CM5%2CM6%2CMVP

I need to understand this one:
if your pref is disable you should not be hitting exactly this signature. If you do we have a different bug and I would like a http log to figure out what is happening. You may hit this bug if you had the pref on and you disable it, after disabling it there may be still some http3 connections running and when they are closed we hit this issue.

I had disabled the pref, but then crashing before cleanly quitting Firefox. Perhaps restarting Firefox from the crash reporter dialog was not launching Firefox in a way that would pick up the new pref value? I'm not able to reproduce the crash (with the pref disabled) today.

Comment 14

[Julien Cristau [:jcristau] (back April 22)]

This by far the worst crash in nightly ATM. Dragana can you get this neqo update in today?

Comment 15

[Dragana Damjanovic [:dragana]]

(In reply to Julien Cristau [:jcristau] from comment #14)

This by far the worst crash in nightly ATM. Dragana can you get this neqo update in today?

Just to be clear http3 is disabled by default. I just did a new neqo release and I will make m.-c. patch.

Comment 16

[Dragana Damjanovic [:dragana]]

(In reply to Dragana Damjanovic [:dragana] from comment #15)

(In reply to Julien Cristau [:jcristau] from comment #14)

This by far the worst crash in nightly ATM. Dragana can you get this neqo update in today?

Just to be clear http3 is disabled by default. I just did a new neqo release and I will make m.-c. patch.

I found another bug in neqo that needs to be fix before I can pull the new version. I hope to get a review for the neqo bug today.

Comment 17

[Dragana Damjanovic [:dragana]]

Fixed by bug 1596069.