738076 - Crash in [@ nsCARenderer::SetupRenderer ]

Status: RESOLVED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Tobias (:Tobbi) Markus]

I was using a fuzzer to create invalid documents. With one stack of documents, I keep getting the following crash:
https://crash-stats.mozilla.com/report/index/bp-f7684380-f477-463c-9edc-5ae522120321

The top frame looks like a random address in memory to me. I'll attach my stack of fuzzer documents to the bug. The crash happened around document 1400 for me (repeatedly). Unfortunately, it crashes only sometimes. I'll try to get the correct crashing document and then put it as a single testcase

Comment 1

[Tobias (:Tobbi) Markus]

For some reason, I cannot reproduce this crash anymore using my Mac machine. However, here's the stack of documents in a ZIP file:
http://mozilla-uk.org/mangleme/mangleme.zip

Comment 2

[Steven Michaud [:smichaud] (Retired)]

The crash stack is at least partly corrupt.  It's not possible for the mCGData pointer to be invalid and non-null at the line where the crash is supposed to take place:

http://hg.mozilla.org/mozilla-central/annotate/4bdae514b9be/gfx/thebes/nsCoreAnimationSupport.mm#l535

That said, there *is* a bug here:

The line "return NS_ERROR_FAILURE;" is missing after the following line:

http://hg.mozilla.org/mozilla-central/annotate/4bdae514b9be/gfx/thebes/nsCoreAnimationSupport.mm#l533

Comment 4

[Daniel Veditz [:dveditz]]

Given the line the crash stack points at in the second frame (nsCARenderer::SetupRenderer) the raw address ought to be memset(). If so, and given it's crashing accessing null this could simply be the bug Steven noted in comment 2. If so this would not be exploitable. That also might explain why it's so hard to reproduce: you'd only hit that case if "mCGData = malloc(aWidth*aHeight*4)" failed which might heavily depend on what else you're running on your system and what you've opened in Nightly up to that point (for example, you mention running 1400+ mangle testcases first).

That might also indicate a leak assuming you're opening these all in the same tab, which would be useful to figure out.

I also noticed Java in the modules list. That might be chewing up memory and might be required to reproduce this bug (in case anyone's trying it without Java, such as myself). And given Java it might be OS version specific since Apple has their own Java (Tobbi is using 10.6.8 according to crash-stats).

Although we have the .zip archive, I added the testcase-wanted keyword to indicate we still don't know which of the 3256 testcases in there triggered the bug.

We should fix the DoS bug in comment 2 and then re-run the tests to verify that was the only issue.

Comment 5

[Daniel Veditz [:dveditz]]

bclary: could we throw these testcases into one of the automation machines and see what we get?

Comment 6

[Tobias (:Tobbi) Markus]

(In reply to Daniel Veditz [:dveditz] from comment #4)
> Given the line the crash stack points at in the second frame
> (nsCARenderer::SetupRenderer) the raw address ought to be memset(). If so,
> and given it's crashing accessing null this could simply be the bug Steven
> noted in comment 2. If so this would not be exploitable. That also might
> explain why it's so hard to reproduce: you'd only hit that case if "mCGData
> = malloc(aWidth*aHeight*4)" failed which might heavily depend on what else
> you're running on your system and what you've opened in Nightly up to that
> point (for example, you mention running 1400+ mangle testcases first).

I have to admit that these testcases run in one tab only. One testcase opens the next one. However, it seems to be always crashing around the 1400st testcase.

Comment 7

[Bob Clary [:bc] (inactive)]

The url is 404? Can you send the zip file to me? If I can serve these up individually from a web server I can test them. I can use my internal web server so these won't be public. Testing all three branches will take a day or more on Linux and Mac due to the limited number of machines available, but I could get results in a day for Windows XP and Windows 7.

Running each "test" as single url would require a little customization since my normal testing is geared to testing individual pages though I could probably do it locally on Windows XP, Windows 7 and Linux.

Comment 9

[Bob Clary [:bc] (inactive)]

I tried the zip file in a nightly Nightly/14 on Fedora 16 and a debug Nightly/14 on OS X 10.5 both loading each page individually using Spider and by starting at the first page and letting it load each subsequent page. I was not able to reproduce a crash.

Comment 10

[Steven Michaud [:smichaud] (Retired)]

Attachment: Fix

Let's at least restore the missing line.

Comment 11

[Benoit Girard (:BenWa)]

Comment on attachment 612682 [details] [diff] [review]
Fix

No brainer r+.

Comment 12

[Steven Michaud [:smichaud] (Retired)]

Comment on attachment 612682 [details] [diff] [review]
Fix

Landed on mozilla-inbound:
http://hg.mozilla.org/integration/mozilla-inbound/rev/6284f89a2069

Comment 13

[Steven Michaud [:smichaud] (Retired)]

http://hg.mozilla.org/mozilla-central/rev/6284f89a2069

Comment 14

[Tobias (:Tobbi) Markus]

(In reply to Bob Clary [:bc:] from comment #9)
> I tried the zip file in a nightly Nightly/14 on Fedora 16 and a debug
> Nightly/14 on OS X 10.5 both loading each page individually using Spider and
> by starting at the first page and letting it load each subsequent page. I
> was not able to reproduce a crash.

Neither was I able to reproduce the crash anymore. I cleared cache and cookies and other stuff and nothing made the crash happen again.

I think should get moved out of the core-security group, doesn't appear to be a security issue anyway.