Closed
Bug 1445067
Opened 6 years ago
Closed 6 years ago
black flashing and content process crash when reading article on the atlantic (IPDL error [PCompositorBridgeChild]: "constructor for actor failed")
Categories
(Core :: Audio/Video: cubeb, defect, P2)
Tracking
()
RESOLVED
FIXED
mozilla61
Tracking | Status | |
---|---|---|
firefox-esr52 | --- | unaffected |
firefox-esr60 | --- | fixed |
firefox59 | --- | unaffected |
firefox60 | + | fixed |
firefox61 | --- | fixed |
People
(Reporter: dbaron, Assigned: u480271)
References
()
Details
(Keywords: crash, regression, Whiteboard: [gfx-noted])
Attachments
(4 files)
59 bytes,
text/x-review-board-request
|
kinetik
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
kinetik
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
kinetik
:
review+
|
Details |
999 bytes,
patch
|
kinetik
:
review+
jcristau
:
approval-mozilla-release+
jcristau
:
approval-mozilla-esr60+
|
Details | Diff | Splinter Review |
On a Linux nightly (64-bit, running Ubuntu 17.10), I was trying to read this article on the Atlantic: https://www.theatlantic.com/magazine/archive/2018/04/the-last-temptation/554066/?utm_source=atlfb In the time it took to read the article, I hit three or four graphics failures some of which led to content process crashes. After a bit of time (reading, scrolling down via a mix of touchpad and PgDn key) the content area would turn completely white, in some cases I'd see some black rectangles flash in both the chrome and content areas. Switching to a different tab showed similar graphics glitches. If I didn't close the tab with the Atlantic article quickly, I'd get a content process crash such as bp-cee9abe4-1b8a-4c3d-9dae-7253f0180312. I saw something similar a month or so ago as well: bp-fc87bb38-5b88-44f1-9026-b607f0180220. This might be related to bug 1400108 or bug 1416726 (the latter appears to currently have a fix on mozilla-inbound).
Comment 1•6 years ago
|
||
triage note: haven't been able to reproduce this on my Ubuntu 16.04 machine. Ryan, does this look like the same linux FD issue you've been battling?
Flags: needinfo?(rhunt)
Whiteboard: [gfx-noted]
Reporter | ||
Comment 2•6 years ago
|
||
FWIW, I still see this in today's build, on https://hg.mozilla.org/mozilla-central/rev/8f1b2f872f0ea358a0412eb8b8687f08d47f6621, which has bug 1416726. I didn't get a content-process crash today, though, but I: * looked at the article in comment 0 until the entire page went white (no black flashing, no content process crash) * closed the tab * opened the article again, read a bit more * opened a second atlantic article, linked from the bottom, in another tab I then got black flashing, inability to paint most things in content (painting them just produced mostly white with some black rectangles), and had to actually restart the browser (not just close those two Atlantic tabs) to get things working again.
Comment 3•6 years ago
|
||
Just confirming some details since I'm bad at reading crash reports: * You definitely *aren't* using webrender, right? * Before ever visiting the page, is the GPU process properly enabled? What compositor are you getting? (about:support)
Reporter | ||
Comment 4•6 years ago
|
||
* no webrender * basic compositor * no idea how to tell about the gpu process
I can reliably reproduce this issue on 60.0b3 and 61.0a1 (2018-03-15) (Ubuntu 17.10 with clean profile) by opening a large number of tabs of the article and waiting a few minutes. Crash on 61.0a1 bp-33e3b42f-a1d5-4d06-874f-277f10180315 looks like Bug 1416726. 61.0a1 log: > [Child 7404, MediaPlayback #3] WARNING: 7fb2f69dd100 OpenCubeb() failed to init cubeb: file /builds/worker/workspace/build/src/dom/media/AudioStream.cpp, line 388 > [Child 7404, MediaPlayback #3] WARNING: Decoder=7fb2f6252a00 [OnMediaSinkAudioError]: file /builds/worker/workspace/build/src/dom/media/MediaDecoderStateMachine.cpp, line 3746 > [Child 7404, MediaPlayback #1] WARNING: Decoder=7fb2f6252a00 Decode error: NS_ERROR_OUT_OF_MEMORY (0x8007000e) - virtual mozilla::MediaResult mozilla::FFmpegVideoDecoder<57>::DoDecode(mozilla::MediaRawData*, uint8_t*, int, bool*, mozilla::MediaDataDecoder::DecodedData&): image allocation error: file /builds/worker/workspace/build/src/dom/media/MediaDecoderStateMachine.cpp, line 3445
Blocks: 1416726
status-firefox60:
--- → affected
Cannot reproduce on 59.0.1. The video sometimes freezes on some tabs which is similar to what happens on 60+ before things start to go bad but it doesn't produce log warnings (with NSPR_LOG_MODULES=all:4), graphic artifacts or crash. Everything continues to work normally except for the video controls but those can be unstuck by setting the video to fullscreen and playing.
status-firefox59:
--- → unaffected
Comment 7•6 years ago
|
||
This could be, the crash signature in bug 1416726 is general for IPC failures so a lot of things can trigger it. I'll try and reproduce this when I get access to a linux desktop later today. Kestrel, if you can reliably reproduce it and have time using mozregression [1] could help us narrow down what changed from 59 to today to cause this issue. [1] https://mozilla.github.io/mozregression/
Flags: needinfo?(rhunt)
Regression window: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=3fcd9eb7c60afb5064b2b5f88ebc62225698f642&tochange=33873f5e18b985b326cfd61ec605856b59df88f2 Regressed by: Bug 1428952
Component: Graphics: Layers → Audio/Video: cubeb
Keywords: regression
OS: Unspecified → Linux
Version: Trunk → 61 Branch
Updated•6 years ago
|
Flags: needinfo?(dglastonbury)
Updated•6 years ago
|
Rank: 15
Priority: -- → P2
(In reply to Kestrel from comment #5) > I can reliably reproduce this issue on 60.0b3 and 61.0a1 (2018-03-15) > (Ubuntu 17.10 with clean profile) by opening a large number of tabs of the > article and waiting a few minutes. > > Crash on 61.0a1 bp-33e3b42f-a1d5-4d06-874f-277f10180315 looks like Bug > 1416726. > > 61.0a1 log: > > > [Child 7404, MediaPlayback #3] WARNING: 7fb2f69dd100 OpenCubeb() failed to init cubeb: file /builds/worker/workspace/build/src/dom/media/AudioStream.cpp, line 388 > > [Child 7404, MediaPlayback #3] WARNING: Decoder=7fb2f6252a00 [OnMediaSinkAudioError]: file /builds/worker/workspace/build/src/dom/media/MediaDecoderStateMachine.cpp, line 3746 > > [Child 7404, MediaPlayback #1] WARNING: Decoder=7fb2f6252a00 Decode error: NS_ERROR_OUT_OF_MEMORY (0x8007000e) - virtual mozilla::MediaResult mozilla::FFmpegVideoDecoder<57>::DoDecode(mozilla::MediaRawData*, uint8_t*, int, bool*, mozilla::MediaDataDecoder::DecodedData&): image allocation error: file /builds/worker/workspace/build/src/dom/media/MediaDecoderStateMachine.cpp, line 3445 Kestrel, Are there other errors before this log? Does it still reproduce if you set pref "media.cubeb.sandbox" to false?
Flags: needinfo?(dglastonbury) → needinfo?(kestrel)
Assignee | ||
Comment 10•6 years ago
|
||
David, Does it still reproduce if you set pref "media.cubeb.sandbox" to false?
Flags: needinfo?(dbaron)
Comment 11•6 years ago
|
||
(In reply to Dan Glastonbury (:kamidphish) | needinfo me | Jury duty Mar-12 - Mar-23 from comment #9) > Are there other errors before this log? Here's two complete logs for failures that happened just now on latest 61.0a1, the log is empty until the bug occurs. https://pastebin.mozilla.org/9080424 https://pastebin.mozilla.org/9080426 > Does it still reproduce if you set pref "media.cubeb.sandbox" to false? No. The log is also empty.
Flags: needinfo?(kestrel)
Reporter | ||
Comment 12•6 years ago
|
||
Today it doesn't seem to reproduce in the same configuration I was using a few days ago (other than updated nightly). It's possible it was a function of the ads I was getting, or something else about the site.
Flags: needinfo?(dbaron)
Comment 13•6 years ago
|
||
I can consistently reproduce this bug with just the background video from the article with the cubeb sandbox enabled. https://cdn.theatlantic.com/assets/media/video/upload/0418_WEL_Gerson_Lead-optimized_(2).mp4 Setting it to loop causes it to eventually fail with an error and crash: > "Video can't be played because the file is corrupt" Time to failure is predictable and reduces linearly with the number of tabs that have the looping video. 1 tab 50 minutes 2 tabs 25 minutes 5 tabs 10 minutes 10 tabs 5 minutes 15 tabs 3 minutes The video is 3 seconds long and it takes about 50 minutes to fail which equates to roughly 1000 loops. There is no indication of a memory leak with system RAM or VRAM and they are not close to full utilization. Changing media.cubeb.logging_level to "verbose" causes Firefox to crash so I tried "normal" logging but it didn't provide anything useful. I tried MOZ_LOG=cubeb:5 but it crashes on startup.
Comment 14•6 years ago
|
||
Dan, what are next steps? Can you repro using comment 13?
Flags: needinfo?(dglastonbury)
Assignee | ||
Comment 15•6 years ago
|
||
I haven't been able to repro yet. Now that the cubeb-mixer stuff looks to be done, I'll deep dive into this.
Flags: needinfo?(dglastonbury)
Assignee | ||
Comment 16•6 years ago
|
||
I've repro'd tab crash when running 15 tabs of the video reported in comment 13. Looking into why there appears to be a huge memory leak.
[Tracking Requested - why for this release]: We're shipping cubeb remoting in 60 on Linux, perhaps we should track this?
tracking-firefox60:
--- → ?
Updated•6 years ago
|
status-firefox-esr52:
--- → unaffected
Assignee | ||
Comment 19•6 years ago
|
||
(In reply to Julien Cristau [:jcristau] from comment #18) > Any luck since last week, Dan? I've been on PTO. I'm currently looking into what looks like a leak of the shmem regions for each stream. Will update when I discover more.
Flags: needinfo?(dglastonbury)
Assignee | ||
Comment 20•6 years ago
|
||
I've found a few issues in the audioipc code that resulted in leaking a couple of file descriptors and some shmem for each created cubeb stream. Everything was being cleaned up correctly at shutdown, but for long running processes creating and destroying at lot of streams this lead to OOM. (The sample video creates/destroys a cubeb stream every 3 seconds, one on each loop).
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 24•6 years ago
|
||
On a build with the attached patches, I ran for over an hour with 15 videos looping with out running out of memory. Monitoring with htop showed no rampant growth in virtual memory usage.
Comment 25•6 years ago
|
||
mozreview-review |
Comment on attachment 8968763 [details] Bug 1445067 - P1: Update audioipc to commit 219a811. https://reviewboard.mozilla.org/r/237464/#review243226
Attachment #8968763 -
Flags: review?(kinetik) → review+
Comment 26•6 years ago
|
||
mozreview-review |
Comment on attachment 8968764 [details] Bug 1445067 - P2: Update cubeb-pulse-rs to commit f90aecf. https://reviewboard.mozilla.org/r/237466/#review243228
Attachment #8968764 -
Flags: review?(kinetik) → review+
Comment 27•6 years ago
|
||
mozreview-review |
Comment on attachment 8968765 [details] Bug 1445067 - P3: Update rust crates. https://reviewboard.mozilla.org/r/237468/#review243230
Attachment #8968765 -
Flags: review?(kinetik) → review+
Comment 28•6 years ago
|
||
I can confirm the try build fixes the issue with cubeb sandbox enabled. Previously with 15 videos looping every 3 seconds, virtual memory grew by 1GB for each content process and playback failed after a few minutes. Now virtual memory is stable with no failure for over an hour.
Comment 29•6 years ago
|
||
Pushed by dglastonbury@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/884c9dd8ec73 P1: Update audioipc to commit 219a811. r=kinetik https://hg.mozilla.org/integration/autoland/rev/d838f3629523 P2: Update cubeb-pulse-rs to commit f90aecf. r=kinetik https://hg.mozilla.org/integration/autoland/rev/6910c35cb4f7 P3: Update rust crates. r=kinetik
Comment 30•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/884c9dd8ec73 https://hg.mozilla.org/mozilla-central/rev/d838f3629523 https://hg.mozilla.org/mozilla-central/rev/6910c35cb4f7
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Comment 31•6 years ago
|
||
Is there something we can do for 60 here, bearing in mind we're building the first release candidate today?
Flags: needinfo?(dglastonbury)
Updated•6 years ago
|
status-firefox-esr60:
--- → affected
Assignee | ||
Comment 32•6 years ago
|
||
(In reply to Julien Cristau [:jcristau] from comment #31) > Is there something we can do for 60 here, bearing in mind we're building the > first release candidate today? I saw the emails and was confused because I when I pulled mozilla/beta the changes were there. (It looked like central had been merged into beta. Looking at the release calendar, it was way past beta 8.)
Flags: needinfo?(dglastonbury) → needinfo?(jcristau)
Comment 33•6 years ago
|
||
60 is on release already. We built the first RC today.
Flags: needinfo?(jcristau) → needinfo?(dglastonbury)
Assignee | ||
Comment 34•6 years ago
|
||
MozReview-Commit-ID: CMzPhzhdqsh
Attachment #8972213 -
Flags: review?(kinetik)
Assignee | ||
Comment 35•6 years ago
|
||
Comment on attachment 8972213 [details] [diff] [review] Uplift fix for FF60. r?kinetik Approval Request Comment [Feature/Bug causing the regression]: 1428952 [User impact if declined]: Long running process, creating/destroying many audio streams will eventually exhaust file descriptors or memory. [Is this code covered by automated tests?]: No [Has the fix been verified in Nightly?]: Yes [Needs manual test from QE? If yes, steps to reproduce]: No [List of other uplifts needed for the feature/fix]: None [Is the change risky?]: No [Why is the change risky/not risky?]: The patch is very small and easy to code review. The fix was identified to fix the leak and has been on Nightly for weeks with out issue. [String changes made/needed]: None
Attachment #8972213 -
Flags: approval-mozilla-release?
Assignee | ||
Comment 36•6 years ago
|
||
Comment on attachment 8972213 [details] [diff] [review] Uplift fix for FF60. r?kinetik [Approval Request Comment] If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fixes memory leak introduced after Firefox 52 in audio streams on Linux. User impact if declined: Long running process, creating/destroying many audio streams will eventually exhaust file descriptors or memory. Fix Landed on Version: 61 Risk to taking this patch (and alternatives if risky): String or UUID changes made by this patch: None See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.
Attachment #8972213 -
Flags: approval-mozilla-esr60?
Assignee | ||
Comment 37•6 years ago
|
||
Thanks, I made the smallest patch that fixes the problem against Firefox 60. I've tested by running a local smoke test of 15 tabs of the looping video for 2 hours.
Flags: needinfo?(dglastonbury) → needinfo?(ryanvm)
Updated•6 years ago
|
Attachment #8972213 -
Flags: review?(kinetik) → review+
Updated•6 years ago
|
Flags: needinfo?(ryanvm) → needinfo?(jcristau)
Comment 38•6 years ago
|
||
Comment on attachment 8972213 [details] [diff] [review] Uplift fix for FF60. r?kinetik thanks Dan, approved for 60 rc2 and 60esr
Flags: needinfo?(jcristau)
Attachment #8972213 -
Flags: approval-mozilla-release?
Attachment #8972213 -
Flags: approval-mozilla-release+
Attachment #8972213 -
Flags: approval-mozilla-esr60?
Attachment #8972213 -
Flags: approval-mozilla-esr60+
Comment 39•6 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-release/rev/ea99ff7d0404
Comment 40•6 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-esr60/rev/ea99ff7d0404
You need to log in
before you can comment on or make changes to this bug.
Description
•