Closed Bug 1153809 Opened 11 years ago Closed 11 years ago

[e10s] crash in _os_lock_corruption_abort | AppleVDADecoder::InitializeSession() in the content process on mac

Categories

(Core :: Audio/Video, defect)

Product:
Core
Component:
Audio/Video
Platform:
All
macOS
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla40
Tracking Flags:
Tracking Status
e10s m8+ ---
firefox40 + fixed

People

(Reporter: jimm, Assigned: smichaud)

References

Details

(Keywords: crash, topcrash-mac)

Crash Data

Attachments

(1 file, 1 obsolete file)

Fix rev1 -- what I will land
1.40 KB, patch
smichaud
: review+
Details | Diff | Splinter Review
report: bp-7599ebf9-0c5d-4a33-ba4a-240ee2150406 Modules like this should not get initialized in the sandbox process. Currently #11 content top crash in Nightly.
I fail to see how that's any different to the initialization done for all the other MediaDataDecoder on all other platforms..
Someone please give STR to make this bug happen. I'll test them on OS X. Others can test them on other platforms, to see what happens.
> Someone please give STR to make this bug happen. I haven't been able to find STR, and that may be quite difficult. But I *have* been able to find one URL that exercises the code added by the patch for bug 1052654: http://clappr.io I don't know how long this URL is going to be available on the web, though. So I'd appreciate a list of URLs that we "officially" use to test the H.264 decoder code added by the patch for bug 1062654. Jya, please provide that if you can. I'm routinely able to exercise this code (and see calls to VTDecompressionSessionCreate()) *without* crashing in today's mozilla-central nightly, with a clean profile (i.e. with e10s mode on), on OS X 10.10.2. If the content process sandbox were involved here, we should crash every time this code runs. Since it doesn't, I doubt the sandbox has anything to do with these crashes. Of course this leaves aside the question of whether or not we *should* call AppleVTDecoder::InitializeSession() from the content process. I have no idea how to answer this, but it does seem to be irrelevant to this bug.
Flags: needinfo?(jyavenard)
Usually when I see this stuff like this it's due to code that expects to be running on the chrome process barfing when it gets initialized in the content process under the sandbox. I'm not sure what this module does or whether its apis will be blocked.. cc'ing bobowen for that.
This same crash has also been reported at bug 1148445. That bug has STR ... but it doesn't work for me (I don't crash).
I just tested http://clappr.io/ again on OS X 10.9.5 with today's m-c nightly. I still don't crash.
Crash Signature: [@ libsystem_platform.dylib@0x2e34] → [@ libsystem_platform.dylib@0x2e34] [@ _os_lock_corruption_abort | _OSSpinLockLockSlow ]
This is easily the #1 Mac topcrasher on the 40 branch.
status-firefox40: affected → ?
Keywords: topcrash-mac
[Tracking Requested - why for this release]: This is easily the #1 Mac topcrasher on the 40 branch, though we can't yet reproduce the crashes.
status-firefox40: ? → affected
tracking-firefox40: --- → ?
These crashes also happen off the trunk (not in e10s mode), in the main process. But they're much less common there. For example: bp-427f493a-335b-4121-9223-b97892150407
> libsystem_platform.dylib@0x2e34 On OS X 10.9.5 atos translates this to: _os_lock_corruption_abort (libsystem_platform.dylib) + 0
Summary: [e10s] crash in AppleVDADecoder::InitializeSession() in the content process on mac → [e10s] crash in _os_lock_corruption_abort | AppleVDADecoder::InitializeSession() in the content process on mac
For what it's worth, we're not the only ones who see this kind of crash in the VideoToolbox: http://www.insanelymac.com/forum/topic/294038-strange-error/ This might turn out to be an Apple bug. I haven't been able to find any similar reports of VideoToolbox crashes on OS X 10.8 or 10.7 -- only on 10.9 and up. In any case, we're likely stuck until we can figure out how to reproduce these crashes.
Moreover all these crashes take place on a secondary thread (whether in the content process or the main process), and seem very likely to be caused either by accessing an uninitialized variable or one that's already been deleted.
There was a crash report in 10.10.0 which got fixed in 10.10.2 (more accurately the reporter couldn't reproduced it once upgrading to 10.10.3. I have reported that particular backtrace to apple: no news yet) The call to VT/VDA::Init will always be done on one of the Media task queue thread pool. This is done before anything else happen, and before anything else is initialized seeing that this is the call to initialize things :) Playing any YouTube videos with MSE enabled (media.mediasource.enabled;true, with webm disabled media.mediasource.webm.enabled;false) would be using the decoder. You can force the destruction/construction of the VT decoder simply by manually changing the video resolution. Here is a test page that will play all type of h264 levels (from 1.0 to 5.2): http://people.mozilla.org/~ajones/h264-level-test/ Here is a plain mp4 one: http://pearce.org.nz/video/h264.html
Flags: needinfo?(jyavenard)
(In reply to Steven Michaud from comment #12) > For what it's worth, we're not the only ones who see this kind of crash in > the VideoToolbox: > > http://www.insanelymac.com/forum/topic/294038-strange-error/ > > This might turn out to be an Apple bug. I haven't been able to find any > similar reports of VideoToolbox crashes on OS X 10.8 or 10.7 -- only on 10.9 > and up. On 10.6-10.8 we use VDA first before attempting to use VT as VT didn't support hardware acceleration until 10.9 You can force the use of VT by disabling hardware acceleration: by setting media.hardware-video-decoding.enabled to false Now this could be HW related in which case this will hide the problem
Thanks for the info, Jya. > You can force the destruction/construction of the VT decoder simply > by manually changing the video resolution. You mean in the Displays system pref panel? Let's hope that gfxCardStatus (https://gfx.io/) can also do this. I don't like to mess with the Displays panel. > Now this could be HW related in which case this will hide the problem I looked at the "Adapter Vendor ID" and "Adapter Device ID" in the crash reports, and didn't see any obvious pattern. But yes, this *could* be hardware related.
(In reply to Steven Michaud from comment #16) > Thanks for the info, Jya. > > > You can force the destruction/construction of the VT decoder simply > > by manually changing the video resolution. > > You mean in the Displays system pref panel? Let's hope that > gfxCardStatus (https://gfx.io/) can also do this. I don't like to > mess with the Displays panel. no no.. within YouTube. Click on the settings icon in the bottom right corner of the video. You can change the resolution there. Each time you do so, a new VT decoder will be created (and the old one deleted first) > I looked at the "Adapter Vendor ID" and "Adapter Device ID" in the > crash reports, and didn't see any obvious pattern. But yes, this > *could* be hardware related. let's hope not... Sounds more to me like an OS bug, and knowing Apple we'll never see a fix until you go to the latest OS
I've given up trying to reproduce these crashes on OS X 10.9.5 (where most of them happen, even though we have far more users on OS X 10.10.2). I played around with the examples from comment #14, plus http://clappr.io/ and a few YouTube videos. I tested with one and two monitors, with dragging a running video from one monitor to another, and with plugging in or unplugging the external monitor while a video was running. I used gfxCardStatus to change the preferred video hardware (from "internal" to "discrete" and vice versa) while a video was running. I tested with today's mozilla-central nightly and a new ASan build I just created, following the instructions at http://people.mozilla.org/~stmichaud/bmo/firefox-asan-howto.txt. I tried testing with libgmalloc with a build in which jemalloc had been disabled, but this hung on startup. (libgmalloc has worked fine for me in the past, but I've also occasionally seen this behavior with it before.) The _os_lock_corruption_abort error shows that these crashes involve trying to use a lock that's been freed or was never initialized. But knowing that isn't enough to help here ... at least not by itself :-(
Forgot to mention that I also played with changing the resolution of running YouTube videos -- but saw no crashes.
> The _os_lock_corruption_abort error shows that these crashes involve > trying to use a lock that's been freed or was never initialized. Actually this isn't quite true. Corruption in the video data we're processing could presumably also lead to the same error -- by tricking the code into thinking something is a lock when it isn't. Jya, have you tried deliberately corrupting the data being processed, to see what would happen?
(Following up comment #20) Actually, data corruption doesn't make much sense as a source of these crashes, either -- since they happen as a "session" is getting initialized. To make any progress here, we may need to log extra information to crash reports when these crashes happen. So we'll need to figure out what that extra information should be.
(In reply to Steven Michaud from comment #20) > > The _os_lock_corruption_abort error shows that these crashes involve > > trying to use a lock that's been freed or was never initialized. > > Actually this isn't quite true. Corruption in the video data we're > processing could presumably also lead to the same error -- by tricking > the code into thinking something is a lock when it isn't. > > Jya, have you tried deliberately corrupting the data being processed, > to see what would happen? no I haven't... The only data we pass is the "extradata" which contains the SPS and PPS H264 NAL. It's a very small data structure. I would expect the VT initialization to simply fail there as designed. We do parse the H264 extradata to extract some informations such as video resolutions, we could also ensure that the data is valid. Having said that, seeing that this is crashing with youtube and it's unlikely youtube would provide crappy data. And lets keep in mind that Apple's own applications (iMovie and Final Cut Pro) appear to suffer the same fate! If it's a race within Apple's own VideoToolbox framework that could occur if say it attempts to initialise multiple instances of the decoder at once ; we could maybe have a static monitor so that only one VT initialisation occurs at a single time across the entire lifetime of firefox.
Could it be it's yet again the stack size that is too small? If you remember that also caused random crashes in VT when hardware acceleration was used.
> Could it be it's yet again the stack size that is too small? Almost certainly not. If nothing else, stack exhaustion tends to have many different "signatures" (at least the top few lines tend to be different). But that's not the case here.
(Following up comment #24) Also, the code where the crashes take place don't have any of the telltale signs described in bug 1062596 comment #3.
> And lets keep in mind that Apple's own applications (iMovie and > Final Cut Pro) appear to suffer the same fate! Yes, this is probably an Apple bug. But it's also a Mac topcrasher. And Apple won't do anything about it unless "their" products are seriously effected, which doesn't seem to be the case. Especially if we can't give them STR or a decent bug report. I'll dig some more through VideoToolbox machine code (in Hopper Disassembler), looking for possible race conditions. I'll also try to think of extra information we can add to this bug's crash logs, which might help us diagnose them. This would work something like https://hg.mozilla.org/mozilla-central/annotate/de27ac2ab94f/dom/plugins/base/nsPluginsDirDarwin.cpp#l490. Jya, can you think of any extra information that might be useful?
The Apple PlatformDecoderModule contains a static counter that is incremented each time we use the framework and decremented after. Once the counter reaches 0, the framework is unlinked. Getting the value of that counter would be of interest: it would tell us if the crash occurs when you have multiple decoders active at the same time. The reason you may not be able to reproduce it is because you're only using one video at a time.
Is the crash in VDA or in VT?? It is very surprising that VDA would be use in 10.9 and later. VT is preferred there. We should never use VDA on 10.9 and later. This wasn't the case until a few days ago where we would be forcing VDA for HD. Do you still see this in Nightly? If not, may be worth back porting this change.
Ah no. I see the crash occurring in both system.
By VDA do you mean VideoDecodeAcceleration framework under /System/Library/Frameworks? I just looked again at the crash reports, and noticed that we sometimes call VTDecompressionSessionCreate() directly (from AppleVTDecoder::InitializeSession()), and sometimes indirectly (from VDADecoderCreate() in AppleVDADecoder::InitializeSession()). But otherwise the crash stacks are exactly the same. Example of direct call: bp-98a320bd-9b5e-4ea4-9382-614d82150410 Example of indirect call: bp-8cefc3c7-f4c7-4ec5-9bff-d85592150408
For me on MacOSX 10.9.5 the steps to reproduce is as simple as "open youtube", it's been a while since I could watch any video on Firefox Nightly (on mac) without bringing all open tabs down, disabling E10S in the preferences panel is a workaround that fix it: - type: /Applications/FirefoxNightly.app/Contents/MacOS/firefox --ProfileManager - create a clean profile - open any youtube channel, https://www.youtube.com/user/portadosfundos for instance Here is the link of a crash report for the reproduce steps above: https://crash-stats.mozilla.com/report/index/3e482ddc-9b85-4a1d-94ce-667632150415 The output on my terminal was: ~ /Applications/FirefoxNightly.app/Contents/MacOS/firefox --ProfileManager ➜ ~ Shumway is registered E/MPEG4Extractor(13787): No width or height, assuming worst case 1080p [Parent 13782] WARNING: pipe error: Broken pipe: file /builds/slave/m-cen-m64-ntly-000000000000000/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 760 ###!!! [Parent][MessageChannel] Error: Channel error: cannot send/recv ###!!! [Parent][MessageChannel] Error: Channel error: cannot send/recv E/MPEG4Extractor(13788): No width or height, assuming worst case 1080p [Parent 13782] WARNING: pipe error: Broken pipe: file /builds/slave/m-cen-m64-ntly-000000000000000/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 760 ###!!! [Parent][MessageChannel] Error: Channel error: cannot send/recv The hardware I am testing on is a MacBookPro5,2 17-inch Mid 2009 http://www.everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-2-duo-3.06-aluminum-17-mid-2009-unibody-specs.html
Fabricio, can you reproduce these crashes using a clean profile? Probably the best way to create a clean profile is to use the Profile Manager (https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles). If you can't, let us know and attach the results of about:support from your current ("unclean") profile. > MacBookPro5,2 I don't have this particular machine to test with. But I do have some older machines, including a MacBookPro5,1. I'll test on them and see what I find.
(Following up comment #32) Oops, I didn't notice that you're already testing with a clean profile.
I can't reproduce these crashes on a 2011 MacBookPro8,2 (testing in OS X 10.9.5), even using gfxCardStatus to force use of either the "discrete" or "integrated" graphics. Later I'll try on my 2008 MacBookPro5,1. But that machine is really old and doesn't have much hard drive space, so getting OS X 10.9.5 onto it might be tricky.
Fabricio, how much RAM does your machine have? It's possible that's the key factor, and not the graphics hardware.
Flags: needinfo?(fabricio)
> Later I'll try on my 2008 MacBookPro5,1. I did manage to get OS X 10.9.5 onto this ancient machine, and I crash there loading http://clappr.io/ in today's mozilla-central nightly. Hooray!! This means I'll be able to find out a lot more about this bug. Interestingly, I crash with either of its two kinds of graphics hardware -- Nvidia GeForce 9400M or Nvidia GeForce 9600M GT. (I used gfxCardStatus to force the use of one or the other. I also crash if I don't use gfxCardStatus, and the OS chooses which to use.) The machine has 4GB of RAM. In contrast, my MacBookPro8,2 has 8GB and my current laptop (a Retina MacBookPro10,1) has 16GB. So I still don't know whether it's the (relatively) small amount of RAM or the graphics hardware that's the trigger. Fabricio, please let us know how much RAM your machine has, and what graphics hardware.
(In reply to Steven Michaud from comment #35) > Fabricio, how much RAM does your machine have? > > It's possible that's the key factor, and not the graphics hardware. 4GB. ECC: Disabled Upgradeable Memory: Yes BANK 0/DIMM0: Size: 2 GB Type: DDR3 Speed: 1067 MHz Status: OK Manufacturer: 0x80AD Part Number: 0x484D54313235533642465238432D47372020 Serial Number: 0x2B90AB37 BANK 1/DIMM0: Size: 2 GB Type: DDR3 Speed: 1067 MHz Status: OK Manufacturer: 0x80AD Part Number: 0x484D54313235533642465238432D47372020 Serial Number: 0x2AB0ABBA
Flags: needinfo?(fabricio)
Graphics: NVIDIA GeForce 9400M: Chipset Model: NVIDIA GeForce 9400M Type: GPU Bus: PCI VRAM (Total): 256 MB Vendor: NVIDIA (0x10de) Device ID: 0x0863 Revision ID: 0x00b1 ROM Revision: 3436 gMux Version: 1.7.10 And NVIDIA GeForce 9600M GT: Chipset Model: NVIDIA GeForce 9600M GT Type: GPU Bus: PCIe PCIe Lane Width: x16 VRAM (Total): 512 MB Vendor: NVIDIA (0x10de) Device ID: 0x0647 Revision ID: 0x00a1 ROM Revision: 3436 gMux Version: 1.7.10 I am using the first one at the moment.
(In reply to Steven Michaud from comment #30) > By VDA do you mean VideoDecodeAcceleration framework under > /System/Library/Frameworks? Yes. VDA was introduced in 10.6.3, and only mac with nvidia 9400M and later are supported. It's a 100% hardware accelerated framework. Not all machines support it. VideoToolBox became official in 10.8, it is likely using VDA internally. VideoToolBox is available in 10.7 in PrivateFrameworks and in 10.6 (names were different at the time, but we have a hack allowing to use the framework on 10.6 too). HW acceleration only became official in 10.9, so for all machines < 10.9 we prefer the official VDA. Starting with 10.9, we will only use VideoToolBoox (VT) > > I just looked again at the crash reports, and noticed that we sometimes call > VTDecompressionSessionCreate() directly (from > AppleVTDecoder::InitializeSession()), and sometimes indirectly (from > VDADecoderCreate() in AppleVDADecoder::InitializeSession()). But otherwise > the crash stacks are exactly the same. How VT or VDA works internally, I'm not sure... they seem to have common code path. Glad you can reproduce it. The 9400M is the first GPU that supports HW decoding.
tracking-e10s: --- → m8+
This has the following regression range on my MacBookPro5,1 running OS X 10.9.5: firefox-2015-03-17-03-02-06-mozilla-central firefox-2015-03-17-07-33-44-mozilla-central http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=436686833af0&tochange=008b3f65a7e0 André Reinald's patch for bug 1083344 is in that range, and turning off the sandbox (by setting security.sandbox.content.level to '0') makes the crashes go away. So I was wrong, and this *does* have to do with the content process sandbox. I'm going to try to figure out what rules to add to the sandbox's ruleset to make these crashes go away.
Assignee: nobody → smichaud
Status: NEW → ASSIGNED
Weird that it would only affect a particular OS and a particular hardware...
Yeah, this bug's weirdness quotient is pretty high ... and may get higher :-)
I've got a fix for this that works on OS X 10.9.5 (on my MacBookPro5,1) -- a single line added to the content sandbox ruleset. But I also need to test it on OS X 10.10, so I'll need to install that version of OS X on my machine (on a separate partition). We don't need to worry about OS X 10.8.5 and below -- we only use a minimal (allow everything) content process sandbox on those version of OS X. That's why these crashes only happen on OS X 10.9.5 and up.
My fix is NVidia-specific. So I looked again through the "adapter vendor ids" and "adapter device ids" in Socorro's list of crash stacks. Almost all the crashes on OS X 10.9.5 are for some kind of NVidia graphics hardware -- so we're probably safe there. So are many of those on OS X 10.10. But there are also a lot of crashes for Intel HD 3000 graphics (under device id 0x0116 or 0x0226). So we probably need yet another vendor-specific hack. Fortunately my MacBookPro8,2 has this kind of graphics hardware. So I'll install OS X 10.10 there, too, and see what happens. It's amazing how useful old hardware can sometimes be! :-)
This almost done. But I've got a bit more testing I want to do ... which will need to wait for next week.
Attached patch Fix (obsolete) — — Splinter Review
I don't actually crash on my MacBookPro8,2 machine, in either OS X 10.9.5 or 10.10.3. But I do see another hardware-specific error, which this patch also works around. So I already know that this patch "fixes" these crashes on my MacBookPro5,1 (with its NVidia graphics hardware). And I have reason to hope that it also fixes those (on OS X 10.10) related to Intel HD 3000 graphics hardware. I've started a set of tryserver builds: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5f2b67b4b29 By the way, I don't crash in OS X 10.10.3 even on my MacBookPro5,1. But I see an error message (in the system console) indicating that the same hardware specific access denial happens there as on OS X 10.9.5 -- it just doesn't cause a crash on 10.10.3, at least on my machine.
Attachment #8595623 - Flags: review?(areinald)
Comment on attachment 8595623 [details] [diff] [review] Fix Review of attachment 8595623 [details] [diff] [review]: ----------------------------------------------------------------- Like all rules set up in bug 1083344, I guess the 2 you add result from trial/error. To me they are fine. ::: security/sandbox/mac/Sandbox.mm @@ +392,5 @@ > " (appleevent-destination \"com.apple.preview\")\n" > " (appleevent-destination \"com.apple.imagecaptureextension2\"))\n" > + "; bug 1153809\n" > + " (allow iokit-open (iokit-user-client-class \"NVDVDContextTesla\"))\n" > + " (allow iokit-open (iokit-user-client-class \"Gen6DVDContext\"))\n" For style consistency I'd rather write those 2 rules as: (allow iokit-open (iokit-user-client-class "...") (iokit-user-client-class "...")) Feel free to take this comment into account and carry on the r+.
Attachment #8595623 - Flags: review?(areinald) → review+
> Like all rules set up in bug 1083344, I guess the 2 you add result > from trial/error. Yes, I'm afraid so :-( > For style consistency I'd rather write those 2 rules as: > > (allow iokit-open > (iokit-user-client-class "...") > (iokit-user-client-class "...")) Will do.
Carrying forward areinald's r+.
Attachment #8595623 - Attachment is obsolete: true
Attachment #8596042 - Flags: review+
Comment on attachment 8596042 [details] [diff] [review] Fix rev1 -- what I will land And ... I didn't manage to land my patch before mozilla-inbound went down. I'll try again later.
https://hg.mozilla.org/mozilla-central/rev/925df25da5c3
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
status-firefox40: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla40
There haven't been any crashes with either of this bug's two signatures in builds made since I landed my patch. So I'm calling this VERIFIED.
Status: RESOLVED → VERIFIED
See Also: → 1164480
For the record, we're still seeing some of these crashes on the 41 branch. But all are on OS X 10.10.1, and with GeForce GT 740 graphics hardware: https://crash-stats.mozilla.com/signature/?platform=Mac+OS+X&version=41.0a1&signature=_os_lock_corruption_abort+|+_OSSpinLockLockSlow&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=platform_version&_columns=reason&_columns=address&_columns=adapter_vendor_id&_columns=adapter_device_id&page=1 As best I can tell, no machine from Apple uses this kind of graphics hardware. So these crashes are presumably happening on Hackintoshes.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: