Closed Bug 1036779 Opened 10 years ago Closed 6 years ago

crash execing bad address [@ EnterION] (Aurora/32)

Categories

(Core :: JavaScript Engine: JIT, defect)

Product:
Core
Component:
JavaScript Engine: JIT
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox-esr45 --- wontfix
firefox50 --- wontfix
firefox51 --- wontfix
firefox52 --- wontfix
firefox-esr52 --- wontfix
firefox53 --- wontfix
firefox54 --- wontfix

People

(Reporter: jesup, Unassigned)

References

Details

(Keywords: crash, csectype-uaf, sec-other, Whiteboard: [#jsapi:crashes-retriage]) 

https://crash-stats.mozilla.com/report/index/aa44fc81-a80e-42e6-98f1-bd0112140710

More Reports shows ~4000 crashes reported with this signature in the last week; all Windows, 65% Win7.  Pattern goes back at least a month.

In all cases I looked at, EnterIon called a bad/not-executable address (ACCESS_VIOLATION_EXEC)

Marking sec because of the jump to non-nullptr addresses.  (random?)

mjrosenb - please CC anyone appropriate
(In reply to Randell Jesup [:jesup] from comment #0)
> In all cases I looked at, EnterIon called a bad/not-executable address
> (ACCESS_VIOLATION_EXEC)

Luke, could that be an interruption callback issue?
Do we mprotect the trampolines?
Flags: needinfo?(luke)
We don't mprotect the trampolines. Also it's unlikely to be in the EnterIon trampoline; EnterIon or EnterBaseline are on the stack for many JIT crashes.

I looked at the dump for the report in comment 0 and we're crashing at 4d7f01d0:

4d7f01b2:	00 00                	add    %al,(%eax)
4d7f01b4:	dc 05 00 00 00 00    	faddl  0x0
4d7f01ba:	00 00                	add    %al,(%eax)
4d7f01bc:	00 00                	add    %al,(%eax)
4d7f01be:	00 00                	add    %al,(%eax)
4d7f01c0:	74 6a                	je     0x4d7f022c
4d7f01c2:	4f                   	dec    %edi
4d7f01c3:	52                   	push   %edx
4d7f01c4:	80 81 d2 1c 01 5a 5a 	addb   $0x5a,0x5a011cd2(%ecx)
4d7f01cb:	5a                   	pop    %edx
4d7f01cc:	5a                   	pop    %edx
4d7f01cd:	5a                   	pop    %edx
4d7f01ce:	5a                   	pop    %edx
4d7f01cf:	5a                   	pop    %edx
4d7f01d0:	40                   	inc    %eax          <---
4d7f01d1:	06                   	push   %es
4d7f01d2:	80 08 60             	orb    $0x60,(%eax)
4d7f01d5:	06                   	push   %es
4d7f01d6:	80 08 c0             	orb    $0xc0,(%eax)
4d7f01d9:	06                   	push   %es
4d7f01da:	80 08 5a             	orb    $0x5a,(%eax)
4d7f01dd:	5a                   	pop    %edx
4d7f01de:	5a                   	pop    %edx
4d7f01df:	5a                   	pop    %edx
4d7f01e0:	70 06                	jo     0x4d7f01e8
4d7f01e2:	80 08 90             	orb    $0x90,(%eax)

Notice the 0x5a everywhere; this looks like freed memory :(
(In reply to Jan de Mooij [:jandem] from comment #3)
> We don't mprotect the trampolines. Also it's unlikely to be in the EnterIon
> trampoline; EnterIon or EnterBaseline are on the stack for many JIT crashes.
> 
> I looked at the dump for the report in comment 0 and we're crashing at
> 4d7f01d0:
> 
> 4d7f01b2:	00 00                	add    %al,(%eax)
> 4d7f01b4:	dc 05 00 00 00 00    	faddl  0x0
> 4d7f01ba:	00 00                	add    %al,(%eax)
> 4d7f01bc:	00 00                	add    %al,(%eax)
> 4d7f01be:	00 00                	add    %al,(%eax)
> 4d7f01c0:	74 6a                	je     0x4d7f022c
> 4d7f01c2:	4f                   	dec    %edi
> 4d7f01c3:	52                   	push   %edx
> 4d7f01c4:	80 81 d2 1c 01 5a 5a 	addb   $0x5a,0x5a011cd2(%ecx)
> 4d7f01cb:	5a                   	pop    %edx
> 4d7f01cc:	5a                   	pop    %edx
> 4d7f01cd:	5a                   	pop    %edx
> 4d7f01ce:	5a                   	pop    %edx
> 4d7f01cf:	5a                   	pop    %edx
> 4d7f01d0:	40                   	inc    %eax          <---
> 4d7f01d1:	06                   	push   %es
> 4d7f01d2:	80 08 60             	orb    $0x60,(%eax)
> 4d7f01d5:	06                   	push   %es
> 4d7f01d6:	80 08 c0             	orb    $0xc0,(%eax)
> 4d7f01d9:	06                   	push   %es
> 4d7f01da:	80 08 5a             	orb    $0x5a,(%eax)
> 4d7f01dd:	5a                   	pop    %edx
> 4d7f01de:	5a                   	pop    %edx
> 4d7f01df:	5a                   	pop    %edx
> 4d7f01e0:	70 06                	jo     0x4d7f01e8
> 4d7f01e2:	80 08 90             	orb    $0x90,(%eax)
> 
> Notice the 0x5a everywhere; this looks like freed memory :(

Indeed, this looks like a use-after-free of the code, with not much info to dig into.

Also, we should change jemalloc constant such as we do not produce nop-slides with the freed junk memory pattern.
Flags: needinfo?(luke)
Looks bad, though I'm not sure what can be done with this.

> Also, we should change jemalloc constant such as we do not produce nop-slides with the freed junk memory pattern.

That deserves a separate bug filed in the jemalloc component.
Keywords: csectype-uaf, sec-critical
(In reply to Andrew McCreight [:mccr8] from comment #5)
> > Also, we should change jemalloc constant such as we do not produce nop-slides with the freed junk memory pattern.

I filed bug 1044077 for this.
Given UAF, possibly exploitable (guess) and a large number of hits in the field -> critical
Severity: normal → critical
Jandem, do you have any ideas on how we can make progress here?
Component: JavaScript Engine → JavaScript Engine: JIT
Flags: needinfo?(jdemooij)
(In reply to Andrew McCreight [:mccr8] from comment #8)
> Jandem, do you have any ideas on how we can make progress here?

Unfortunately without STR there's not much we can do :( I looked at some more JIT crashes and almost all of them are like the one in this bug, memory corruption.

But are there really more JIT crashes on 32 than on other branches though? On crash-stats, EnterBaselineMethod is at #11 and I don't even see EnterIon.
Flags: needinfo?(jdemooij)
> Unfortunately without STR there's not much we can do :( 

Do the URLs and comments provide any clues?

84 	http://www.welt.de/spiele/online-spiele/article1336655/Sudoku.html
78 	https://www.facebook.com/
65 	http://www.pinterest.com/
32 	http://www.wetter.com/wetter_aktuell/niederschlagsradar/deutschland/
21 	http://24hoursofhappy.com/
19 	http://www.pinterest.com/logout/

"Reading e-mail in Gmail and listening to Pharell's "Happy" with that site open (These are both pinned tabs)."
"very basic window change in Pinterest..."
"Immer beim Spielen von Sudoku"
"trying to log out of Pinterest"
(In reply to David Major [:dmajor] from comment #10)
> Do the URLs and comments provide any clues?

Hm we should test that Sudoku website.

It might be worth testing the Sudoku website with Firefox 32/33/34 (32-bit), and ideally also with different settings for parallel compilation and use counts etc. Hannes can you look into this? I'd do it myself, but I don't really have time until next week...
Flags: needinfo?(hv1989)
I noticed some strange pattern on the table of crash reports:
https://crash-stats.mozilla.com/report/list?product=Firefox&range_unit=days&range_value=28&signature=EnterIon#tab-table

On most builds we have 1-6 crashes. But on certain builds we have much higher volume.
BuildID: # crashes
2014060517: 10037 crashes
2014071618: 2936 crashes

Trying to find out which builds these are, I noticed these buildIds are not on builds in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/07/

I could find one potential source of builds: tete009 unofficial builds:
89% of all these crashes are on a buildId of a tete009 build. (BTW a PGO build, could be related)

So I still have 11% unexplained crashes.
BuildID: # crashes
2014073119: 141
2014072812: 105 (could be 2014-07-29-mozilla-b2g32_v2_0-debug/ or false positive)
2014072219: 200
2014071014: 275
2014070716: 271
2014070315: 62

So except 2014072812 all these buildIds are no official builds (I could not find the build id in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/07/). I have tried a lot of third-party firefox releases, but I couldn't pinpoint them to the actual build.


I think we can lower the severity. Since this isn't on our builds. Also do we need to contact the owner of tete009? And what about the other builds we cannot find the real source?
Flags: needinfo?(hv1989)
Whiteboard: [crashes mostly in unofficial builds]
The original report in comment 0 was from me; I believe this was on Aurora/32 on my windows machine;  I doubt it was a local build (I normally build and test nightly/inbound, not Aurora; I run Aurora for general browsing).

So my guess given the non-zero number of hits on our builds is that there is a bug, but it's timing/compiler/compiler-options related.  Perhaps some of the extra-crashey external builds turn on optimizations we don't which change the timing or open the hole wider.
Group: core-security → javascript-core-security
WFM at this point?
status-firefox32: affected → ---
status-firefox33: affected → ---
status-firefox34: affected → ---
Flags: needinfo?(rjesup)
Don't we have other bugs on file for EnterIon and EnterBaseline crashes? I thought that was a pretty generic signature that was basically unactionable without STR.
Perhaps should be duped to the 'master' EnterIon then.  Or closed in this case perhaps.  

We should have a "Resolved" "Not actionable"....
Too late for firefox 52, mass-wontfix.
status-firefox52: affected → wontfix
jonco - can you decide what to do with this?
status-firefox50: affected → wontfix
status-firefox51: affected → wontfix
status-firefox53: affected → wontfix
status-firefox54: --- → ?
status-firefox-esr52: --- → affected
Flags: needinfo?(jcoppeard)
Bug 1329042 is also about access violations accessing executable memory and may be related.

needinfo'ing Jan as this is JIT related.
Flags: needinfo?(jcoppeard) → needinfo?(jdemooij)
See Also: → 1329042
Flags: needinfo?(jdemooij)
Whiteboard: [#jsapi:crashes-retriage]
No crashes currently newer than 57.0.4 - though I have a suspicion that this is just a signature change.
Yeah, that function is gone, but I suspect this just moves the problem
Problem is moved to a different signature. Bug 858032 tracks the up to date signatures for JIT crashes.

I'm in the middle of doing an up-to-date analysis of JIT crashes elsewhere.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
Group: javascript-core-security
You need to log in before you can comment on or make changes to this bug.