1124018 - Intermittent test_file_resurrection_delete.html | application crashed [@ js::CurrentThreadCanAccessRuntime(JSRuntime *)]

Status: RESOLVED FIXED

(Core :: JavaScript: GC, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

0xa5a5a5a5, ruh-roh.

17:45:54 INFO - 1837 INFO TEST-FAIL | dom/indexedDB/test/test_file_resurrection_delete.html | Skipping test in a worker because it's not structured properly
17:45:54 INFO - 1838 INFO Running test in main thread
17:45:54 INFO - 1839 INFO TEST-PASS | dom/indexedDB/test/test_file_resurrection_delete.html | Correct byte length
17:45:54 INFO - 1840 INFO TEST-PASS | dom/indexedDB/test/test_file_resurrection_delete.html | Got correct event type
17:45:54 INFO - 1841 INFO TEST-PASS | dom/indexedDB/test/test_file_resurrection_delete.html | Got correct event type
17:45:54 INFO - 1842 INFO TEST-PASS | dom/indexedDB/test/test_file_resurrection_delete.html | Correct db ref count
17:45:54 INFO - 1843 INFO TEST-PASS | dom/indexedDB/test/test_file_resurrection_delete.html | Correct db ref count
17:45:54 INFO - 1844 ERROR TEST-UNEXPECTED-FAIL | dom/indexedDB/test/test_file_resurrection_delete.html | application terminated with exit code 1
17:45:54 INFO - runtests.py | Application ran for: 0:20:36.348000
17:45:54 INFO - zombiecheck | Reading PID log: c:\users\cltbld\appdata\local\temp\tmpnxki92pidlog
17:45:54 INFO - ==> process 2892 launched child process 1240 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2892.200d1960.850478488 "c:\users\cltbld\appdata\local\temp\tmps4ckpv.mozrunner\plugins\nptest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2892 "\\.\pipe\gecko-crash-server-pipe.2892" plugin)
17:45:54 INFO - ==> process 2892 launched child process 4084 ("C:\slave\test\build\application\firefox\plugin-container.exe" --channel=2892.b51b550.769337565 "c:\users\cltbld\appdata\local\temp\tmps4ckpv.mozrunner\plugins\nptest.dll" -greomni "C:\slave\test\build\application\firefox\omni.ja" -appomni "C:\slave\test\build\application\firefox\browser\omni.ja" -appdir "C:\slave\test\build\application\firefox\browser" - 2892 "\\.\pipe\gecko-crash-server-pipe.2892" plugin)
17:46:03 INFO - mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\3a844c31-8681-49fa-a509-7c50caa3ad88.dmp
17:46:03 INFO - mozcrash Saved app info as C:\slave\test\build\blobber_upload_dir\3a844c31-8681-49fa-a509-7c50caa3ad88.extra
17:46:03 WARNING - PROCESS-CRASH | dom/indexedDB/test/test_file_resurrection_delete.html | application crashed [@ js::CurrentThreadCanAccessRuntime(JSRuntime *)]
17:46:03 INFO - Crash dump filename: c:\users\cltbld\appdata\local\temp\tmps4ckpv.mozrunner\minidumps\3a844c31-8681-49fa-a509-7c50caa3ad88.dmp
17:46:03 INFO - Operating system: Windows NT
17:46:03 INFO - 6.1.7601 Service Pack 1
17:46:03 INFO - CPU: x86
17:46:03 INFO - GenuineIntel family 6 model 30 stepping 5
17:46:03 INFO - 8 CPUs
17:46:03 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_READ
17:46:03 INFO - Crash address: 0xffffffffa5a5a6dd
17:46:03 INFO - Thread 0 (crashed)
17:46:03 INFO - 0 xul.dll!js::CurrentThreadCanAccessRuntime(JSRuntime *) [Runtime.cpp:69cdf42cdaf9 : 806 + 0xb]
17:46:03 INFO - eip = 0x66a4cbee esp = 0x0022f170 ebp = 0x0022f170 ebx = 0x0c64d801
17:46:03 INFO - esi = 0x0022f238 edi = 0xa5a5a5a5 eax = 0x01d11140 ecx = 0xa5a5a5a5
17:46:03 INFO - edx = 0x00000000 efl = 0x00210246
17:46:03 INFO - Found by: given as instruction pointer in context
17:46:03 INFO - 1 xul.dll!js::gc::IsAboutToBeFinalized<JSScript> [Marking.cpp:69cdf42cdaf9 : 451 + 0x69]
17:46:03 INFO - eip = 0x668a00ee esp = 0x0022f178 ebp = 0x0022f184
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 2 xul.dll!js::types::TypeCompartment::sweep(js::FreeOp *) [jsinfer.cpp:69cdf42cdaf9 : 4864 + 0x8]
17:46:03 INFO - eip = 0x669b0194 esp = 0x0022f18c ebp = 0x0022f250
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 3 xul.dll!js::types::TypeZone::beginSweep(js::FreeOp *,bool,js::types::AutoClearTypeInferenceStateOnOOM &) [jsinfer.cpp:69cdf42cdaf9 : 5178 + 0xe]
17:46:03 INFO - eip = 0x66960dce esp = 0x0022f258 ebp = 0x0022f278
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 4 xul.dll!JS::Zone::beginSweepTypes(js::FreeOp *,bool) [Zone.cpp:69cdf42cdaf9 : 119 + 0x17]
17:46:03 INFO - eip = 0x6686efe1 esp = 0x0022f280 ebp = 0x0022f294
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 5 xul.dll!js::gc::GCRuntime::beginSweepingZoneGroup() [jsgc.cpp:69cdf42cdaf9 : 5024 + 0x33]
17:46:03 INFO - eip = 0x667cd399 esp = 0x0022f29c ebp = 0x0022f6dc
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 6 xul.dll!js::gc::GCRuntime::beginSweepPhase(bool) [jsgc.cpp:69cdf42cdaf9 : 5163 + 0x6]
17:46:03 INFO - eip = 0x667cc315 esp = 0x0022f6e4 ebp = 0x0022f75c
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 7 xul.dll!js::gc::GCRuntime::incrementalCollectSlice(js::SliceBudget &,JS::gcreason::Reason) [jsgc.cpp:69cdf42cdaf9 : 5889 + 0x13]
17:46:03 INFO - eip = 0x667f7e0b esp = 0x0022f764 ebp = 0x0022f784
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 8 xul.dll!js::gc::GCRuntime::gcCycle(bool,js::SliceBudget &,JS::gcreason::Reason) [jsgc.cpp:69cdf42cdaf9 : 6084 + 0xa]
17:46:03 INFO - eip = 0x667e9620 esp = 0x0022f78c ebp = 0x0022f7e8
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 9 xul.dll!js::gc::GCRuntime::collect(bool,js::SliceBudget,JS::gcreason::Reason) [jsgc.cpp:69cdf42cdaf9 : 6209 + 0x15]
17:46:03 INFO - eip = 0x667d513a esp = 0x0022f7f0 ebp = 0x0022f880
17:46:03 INFO - Found by: call frame info
17:46:03 INFO - 10 xul.dll!JS::GCForReason(JSRuntime *,JSGCInvocationKind,JS::gcreason::Reason) [jsgc.cpp:69cdf42cdaf9 : 7063 + 0x37]
17:46:03 INFO - eip = 0x667ac0cb esp = 0x0022f888 ebp = 0x0022f8b0
17:46:03 INFO - Found by: call frame info

Comment 1

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=5688650
repository: mozilla-inbound
start_time: 2015-01-20T17:22:32
who: rvandermeulen[at]mozilla[dot]com
machine: t-w732-ix-135
buildname: Windows 7 32-bit mozilla-inbound debug test mochitest-2
revision: 69cdf42cdaf9

1844 ERROR TEST-UNEXPECTED-FAIL | dom/indexedDB/test/test_file_resurrection_delete.html | application terminated with exit code 1
PROCESS-CRASH | dom/indexedDB/test/test_file_resurrection_delete.html | application crashed [@ js::CurrentThreadCanAccessRuntime(JSRuntime *)]
TEST-UNEXPECTED-FAIL | leakcheck | default process: missing output line for total leaks!
Return code: 1
No tests run or test summary not found

Comment 2

[Andrew McCreight [:mccr8]]

Does this look familiar, Jon?  I feel like I've seen some other crash recently about sweeping of type objects.

Comment 3

[Jon Coppeard (:jonco)]

Yes, bug 1091094 looks like the same issue.

Comment 4

[Jon Coppeard (:jonco)]

It's crashing in TypeObject sweeping - Brian, any chance you could take a look?

Comment 5

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=5721588
repository: mozilla-inbound
start_time: 2015-01-21T12:34:56
who: rvandermeulen[at]mozilla[dot]com
machine: t-w732-ix-117
buildname: Windows 7 32-bit mozilla-inbound debug test mochitest-2
revision: 78310b277913

2138 ERROR TEST-UNEXPECTED-FAIL | dom/jsurl/test/test_bug351633-1.html | application terminated with exit code 1
PROCESS-CRASH | dom/jsurl/test/test_bug351633-1.html | application crashed [@ js::CurrentThreadCanAccessRuntime(JSRuntime *)]
TEST-UNEXPECTED-FAIL | leakcheck | default process: missing output line for total leaks!
Return code: 1
No tests run or test summary not found

Comment 6

[Brian Hackett [Laid off!]]

Attachment: patch

I don't know if this is the problem, but if we fail to initialize the allocation site table when we first try to use it during execution, then the table will be deleted but not null'ed out (like we do with other similar tables in jsinfer.cpp), which could lead to a dangling pointer the next time we try to use the table, whether during GC or not.

Comment 7

[Brian Hackett [Laid off!]]

Comment on attachment 8552690 [details] [diff] [review]
patch

[Security approval request comment]
How easily could an exploit be constructed based on the patch?

Difficult, requires an OOM at an allocation that occurs once per compartment.

Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?

No.

Which older supported branches are affected by this flaw?

All, probably.

Do you have backports for the affected branches? If not, how different, hard to create, and risky will they be?

trivial

How likely is this patch to cause regressions; how much testing does it need?

not at all

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Yeah, looks like TypeCompartment::addAllocationSiteTypeObject has the same code all the way back to b2g30.

Comment 9

[Al Billings [:abillings - ex-MoCo]]

Can someone suggest a security rating for this crash. Did we ever figure out a reliable way to trigger it?

Comment 10

[Brian Hackett [Laid off!]]

(In reply to Al Billings [:abillings] from comment #9)
> Did we ever figure out a reliable way to trigger it?

No, I just looked at the code related to where we were crashing at.

Comment 11

[Al Billings [:abillings - ex-MoCo]]

Comment on attachment 8552690 [details] [diff] [review]
patch

sec-approval+. Let's get this into trunk.

Comment 12

[Brian Hackett [Laid off!]]

Comment on attachment 8552690 [details] [diff] [review]
patch

Approval Request Comment
[Feature/regressing bug #]: old
[User impact if declined]: UAF in low memory conditions
[Describe test coverage new/current, TreeHerder]: none
[Risks and why]: none

Comment 13

[Al Billings [:abillings - ex-MoCo]]

Comment on attachment 8552690 [details] [diff] [review]
patch

Approving for beta and aurora.

Normally, we wouldn't take a moderate issue on ESR31 but I kind of made up that rating so you could make a case for it.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d6bdb55ff8bc

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/d6bdb55ff8bc

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8f49a2fc3f0c
https://hg.mozilla.org/releases/mozilla-beta/rev/1d3c24b896a1
https://hg.mozilla.org/releases/mozilla-esr31/rev/4c8e3096563b

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8552690 [details] [diff] [review]
patch

See comment 7 and 12. Normally I wouldn't argue for uplifting a sec-moderate to b2g30/b2g32 in addition to b2g34, but in this case I think it's worth it because:
* It's literally a one-line patch that does the obvious right thing.
* It affects OOM situations, which B2G seems likely to be more prone to.
* The sec-moderate rating was a bit of a guess in the first place.
* The crash was in IndexedDB code, which B2G uses heavily.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/8f49a2fc3f0c

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/logviewer.html#?job_id=1821845&repo=fx-team looks like the same crash?

Comment 20

[Brian Hackett [Laid off!]]

Well, there isn't anything else I can see from the code, so without some STR I don't see a way to move forward.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

*sigh* I'll at least land the safe patch for now even if it's a full solution.

https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/a4739878b511
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/4048cc627fdf
https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/befcfeb9e8b1

Comment 22

[Ed Morley [:emorley]]

This bug came up in the worksforme list as part of bug 1180138, however since patches have landed, I'll mark fixed.