1330025 - Crash in mozilla::net::Http2Session::ReadSegmentsAgain

Status: RESOLVED INCOMPLETE

(Core :: Networking, defect, P3)

Description

[Randell Jesup [:jesup] (needinfo me)]

+++ This bug was initially created as a clone of Bug #1283823 +++

Still happening at low frequency in 50.1 and later: 
https://crash-stats.mozilla.com/report/index/b7931223-1f1e-48f5-ae12-8f5e92170107
and 51: https://crash-stats.mozilla.com/report/index/cf4acfed-95ef-4af4-8dfe-e21d92161203

(though perhaps not the same bug)

Search to avoid 48 and before:

https://crash-stats.mozilla.com/signature/?version=49.0.1&version=49.0.2&version=50.0&version=50.1.0&version=50.0.2&version=50.0.1&version=53.0a1&version=52.0a2&version=51.0b&signature=mozilla%3A%3Anet%3A%3AHttp2Session%3A%3AReadSegmentsAgain&date=%3E%3D2016-07-10T16%3A57%3A34.000Z&date=%3C2017-01-10T16%3A57%3A34.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_sort=-date&page=1

Not linked on purpose as this is a sec issue, though perhaps not a UAF - some of the crashes have worrying signatures/addresses (EXEC, illegal instructions, etc).

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Patrick - new/different bug?  lots of weird crash causes here

Comment 2

[Patrick McManus [:mcmanus]]

different bug for sure - the other one was h2 proxying which isn't in evidence here..

not much to go on - weird that both of the crashes in comment 0 are in LOG

Comment 3

[Daniel Veditz [:dveditz]]

(In reply to Patrick McManus [:mcmanus] from comment #2)
> not much to go on - weird that both of the crashes in comment 0 are in LOG

I found a couple that weren't on LOG() lines (access violation read rather than write or exec), but yeah, weird.

Comment 4

[Daniel Veditz [:dveditz]]

The most recent version crashing with this signature is Firefox 51.0b99  (odd that it's the release candidate, and not the shipping 51.0.1); no crashes in 52 (Beta) or newer. The last version with a UAF crash I see is Firefox 49. The crashes I see for Firefox 50.1 are mostly exploitable-looking write or exec violations, but the 51.0b99 crashes look fairly benign (read at 0xffffffffffffff8e).

This crash may have morphed elsewhere, but I'm not sure we'll get there from this bug. Call it "worksforme" in current releases?

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

froyd/erahm - any ideas?  As dveditz says, this has almost disappeared (but it's at such a low frequency we mostly see it in beta and release, so lack of 52/53/54 crashes isn't definitive).  And it hides in a raft of failures in older builds (different fixed problem).

However, the remaining crashes seem to (almost) exclusively be in LOG() or LOGN() (LOG3, etc) calls.  This is especially odd.

I think we need to leave this open until 52 gets near end-of-beta or gets to release to declare victory - and I suspect it's still there.

Updated search: https://crash-stats.mozilla.com/signature/?version=50.0&version=50.1.0&version=50.0.2&version=50.0.1&version=53.0a1&version=52.0a2&version=51.0b&version=54.0a1&version=53.0a2&version=52.0b&version=52.0b3&version=52.0b2&version=52.0b1&version=51.0.1&version=51.0&version=51.0b99&signature=mozilla%3A%3Anet%3A%3AHttp2Session%3A%3AReadSegmentsAgain&date=%3E%3D2017-01-03T05%3A33%3A05.000Z&date=%3C2017-02-03T05%3A33%3A05.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_sort=-date&page=1#reports

Example crash in 51.0b99:  https://crash-stats.mozilla.com/report/index/950b8f3c-5d90-4875-bf97-e25d32170124

Comment 6

[Nathan Froyd [:froydnj]]

(In reply to Randell Jesup [:jesup] from comment #5)
> Example crash in 51.0b99: 
> https://crash-stats.mozilla.com/report/index/950b8f3c-5d90-4875-bf97-
> e25d32170124

I took a look at the crash dump in windbg, and the disassembly is pretty gnarly:

xul!mozilla::net::Http2Session::ReadSegmentsAgain+0x88 [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\netwerk\protocol\http\http2session.cpp @ 2311]:
 2311 6492b3eb 57              push    edi
 2311 6492b3ec 68683eec65      push    offset xul!`string' (65ec3e68)
 2311 6492b3f1 56              push    esi
 2311 6492b3f2 8bcb            mov     ecx,ebx
 2311 6492b3f4 e88df474ff      call    xul!mozilla::LazyLogModule::operator mozilla::LogModule * (6407a886)
 2311 6492b3f9 50              push    eax
 2311 6492b3fa e8f579ecff      call    xul!mozilla::detail::log_print (647f2df4)
 2311 6492b3ff 83c410          add     esp,10h

xul!mozilla::net::Http2Session::ReadSegmentsAgain+0x9f [c:\builds\moz2_slave\m-rel-w32-00000000000000000000\build\src\netwerk\protocol\http\http2session.cpp @ 2313]:
 2313 6492b402 8d4f78          lea     ecx,[edi+78h]
 2313 6492b405 e89a16a9ff      call    xul!nsDeque::PopFront (643bcaa4)
 2315 6492b40a 56              push    esi
 2315 6492b40b 8b6e8b          mov     ebp,dword ptr [esi-75h] // XXX crash
 2315 6492b40e e8e872f474      call    d98726fb
 2315 6492b413 ff50e8          call    dword ptr [eax-18h]
 2315 6492b416 54              push    esp
 2315 6492b417 f4              hlt
 2315 6492b418 74ff            je      xul!mozilla::net::Http2Session::ReadSegmentsAgain+0xb6 (6492b419)

So that would explain the 0xffffff8e addresses or so.  The assembly here has three peculiar features:

* The |call d98726fb| instruction: windbg doesn't know of any code at that address.
* The |hlt| instruction at 0x6492b417: what compiler generates a hlt instruction at a location like that?
* The |je| at 0x6492b418 with no apparent test/cmp/flag setting prior to it.  It's certainly possible that one of the previous |call|s set the appropriate flags (I think MSVC is smart enough to do this?), but that's preeetty unusual.

For comparison's sake, the assembly in the relevant block with the xul.dll from:

https://ftp.mozilla.org/pub/firefox/candidates/51.0b9-candidates/build1/win32/en-US/

looks like (from llvm-objdump, I'm not motivated enough to figure out how to get this into windbg with symbols and whatnot):

10a92288:	8d 4f 78 	leal	120(%edi), %ecx
10a9228b:	e8 ca 63 ac ff 	calll	-5479478
10a92290:	56 	pushl	%esi
10a92291:	8b cb 	movl	%ebx, %ecx
10a92293:	8b e8 	movl	%eax, %ebp
10a92295:	e8 cc 4d 6f ff 	calll	-9482804
10a9229a:	50 	pushl	%eax
10a9229b:	e8 ae 4d 6f ff 	calll	-9482834
10a922a0:	59 	popl	%ecx
10a922a1:	59 	popl	%ecx
10a922a2:	85 ed 	testl	%ebp, %ebp
10a922a4:	75 33 	jne	51 

Notice that things look *much* different after the |pushl %esi|; I'm inclined to say somebody was trying to hotpatch our code.  The space between |pushl %esi| and |jne| doesn't quite line up with the assembly we're seeing from the crash, but other code in the same function looks like it's been shuffled around a little bit anyway (e.g. the crash disassembly has instructions that no compiler would ever generate for Firefox code), so that might not be significant?  Or I might be looking at the wrong xul.dll.

All the crash reports with this 0xffffff8e address are submitted very shortly after startup (within seconds) and they all have the same URL.  I think whoever is hitting this particular crash has a virus and/or malware on their computer.

I haven't investigated the other crashes with seemingly random crash addresses.

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

THanks nathan - that really helps.  The code not matching what we shipped is a real red flag.  Are there any unusual DLLs/correlations?

Comment 8

[Frederik Braun [:freddy]]

(In reply to Nathan Froyd [:froydnj] from comment #6)
> Notice that things look *much* different after the |pushl %esi|; I'm
> inclined to say somebody was trying to hotpatch our code.  The space between
> |pushl %esi| and |jne| doesn't quite line up with the assembly we're seeing
> from the crash, but other code in the same function looks like it's been
> shuffled around a little bit anyway (e.g. the crash disassembly has
> instructions that no compiler would ever generate for Firefox code), so that
> might not be significant?  Or I might be looking at the wrong xul.dll.
> 
> All the crash reports with this 0xffffff8e address are submitted very
> shortly after startup (within seconds) and they all have the same URL.  I
> think whoever is hitting this particular crash has a virus and/or malware on
> their computer.
> 
> I haven't investigated the other crashes with seemingly random crash
> addresses.

Could also be antivirus. Do we have crash reports for this that include an email address?
Could be worth a shot, but I honestly don't know how this works - I mostly sit at the submitting end of crash reports :)

Comment 9

[Frederik Braun [:freddy]]

Richard, could we fingerprint the crash report TLS session on our end to identify if (and which) A/V is used? :>

Comment 10

[Andrew McCreight [:mccr8]]

I'm going to mark this sec-other, as I'm not sure how much it is our issue if another program is injecting code into our process...

Comment 11

[David Lawrence [:dkl]]

Removing employee no longer with company from CC list of private bugs.

Comment 12

[edgul]

Dan, can we revisit the sec nature of this one?

Comment 13

[Valentin Gosu [:valentin] (he/him)]

Some stack traces for future reference

https://crash-stats.mozilla.org/report/index/37cadcf0-b7e6-43ce-beba-7e1b00230711

0	xul.dll	mozilla::RefPtrTraits<mozilla::net::nsAHttpSegmentReader>::AddRef(mozilla::net::nsAHttpSegmentReader*)	mfbt/RefPtr.h:53	inlined
0	xul.dll	RefPtr<mozilla::net::nsAHttpSegmentReader>::ConstRemovingRefPtrTraits<mozilla::net::nsAHttpSegmentReader>::AddRef(mozilla::net::nsAHttpSegmentReader*)	mfbt/RefPtr.h:419	inlined
0	xul.dll	RefPtr<mozilla::net::nsAHttpSegmentReader>::assign_with_AddRef(mozilla::net::nsAHttpSegmentReader*)	mfbt/RefPtr.h:64	inlined
0	xul.dll	RefPtr<mozilla::net::nsAHttpSegmentReader>::operator=(mozilla::net::nsAHttpSegmentReader*)	mfbt/RefPtr.h:210	inlined
0	xul.dll	mozilla::net::Http2Session::ReadSegmentsAgain(mozilla::net::nsAHttpSegmentReader*, unsigned int, unsigned int*, bool*)	netwerk/protocol/http/Http2Session.cpp:2794	context
1	xul.dll	PLDHashTable::ComputeKeyHash(void const*) const	xpcom/ds/PLDHashTable.cpp:480	inlined
1	xul.dll	PLDHashTable::Search(void const*) const	xpcom/ds/PLDHashTable.cpp:500	cfi
2	xul.dll	nsTHashtable<nsBaseHashtableET<nsRefPtrHashKey<mozilla::dom::DOMIntersectionObserver>, int> >::GetEntry(mozilla::dom::DOMIntersectionObserver*) const	xpcom/ds/nsTHashtable.h:288	inlined
2	xul.dll	nsBaseHashtable<nsRefPtrHashKey<mozilla::dom::DOMIntersectionObserver>, int, int, nsDefaultConverter<int, int> >::Lookup(mozilla::dom::DOMIntersectionObserver*)	xpcom/ds/nsBaseHashtable.h:637	inlined
2	xul.dll	mozilla::dom::Element::UpdateIntersectionObservation(mozilla::dom::DOMIntersectionObserver*, int)	dom/base/Element.cpp:4216	cfi
3	xul.dll	mozilla::dom::DOMIntersectionObserver::Update(mozilla::dom::Document&, double)	dom/base/DOMIntersectionObserver.cpp:783	cfi

https://crash-stats.mozilla.org/report/index/eb5767fe-96ae-472c-8a6d-fddaa0230808

0	xul.dll	mozilla::RefPtrTraits<mozilla::net::Http2StreamBase>::Release(mozilla::net::Http2StreamBase*)	mfbt/RefPtr.h:54	inlined
0	xul.dll	RefPtr<mozilla::net::Http2StreamBase>::ConstRemovingRefPtrTraits<mozilla::net::Http2StreamBase>::Release(mozilla::net::Http2StreamBase*)	mfbt/RefPtr.h:420	inlined
0	xul.dll	RefPtr<mozilla::net::Http2StreamBase>::~RefPtr()	mfbt/RefPtr.h:85	inlined
0	xul.dll	mozilla::net::Http2Session::ReadSegmentsAgain(mozilla::net::nsAHttpSegmentReader*, unsigned int, unsigned int*, bool*)	netwerk/protocol/http/Http2Session.cpp:2937	context
1	xul.dll	mozilla::net::nsHttpConnection::OnSocketWritable()	netwerk/protocol/http/nsHttpConnection.cpp:1635	inlined
1	xul.dll	mozilla::net::nsHttpConnection::OnOutputStreamReady(nsIAsyncOutputStream*)	netwerk/protocol/http/nsHttpConnection.cpp:2163	cfi
2	xul.dll	mozilla::net::nsSocketOutputStream::OnSocketReady(nsresult)	netwerk/base/nsSocketTransport2.cpp:524	cfi
3	xul.dll	mozilla::net::nsSocketTransport::OnSocketReady(PRFileDesc*, short)	netwerk/base/nsSocketTransport2.cpp:2111	cfi
4	xul.dll	mozilla::net::nsSocketTransportService::DoPollIteration(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>*)	netwerk/base/nsSocketTransportService2.cpp:1391	inlined
4	xul.dll	mozilla::net::nsSocketTransportService::Run()	netwerk/base/nsSocketTransportService2.cpp:1158	cfi

Comment 14

[Daniel Veditz [:dveditz]]

(In reply to Ed Guloien [:edgul] from comment #12)

Dan, can we revisit the sec nature of this one?

It's already marked "sec-other"; what were you thinking? The signs of UAF associated with this signature went away long long ago. It's a very rare crash, and the remaining symptoms are so varied it's hard to imagine they're all from the same cause. These might represent machines with flaky memory, or it's memory corruption from far away causing random crashes and this is just one isolated slice and not very useful for discovering where the real bug is.