Closed Bug 1407487 Opened 7 years ago Closed 7 years ago

Enable cubeb remoting on Nightly

Categories

(Core :: Audio/Video: cubeb, defect, P3)

Product:
Core
Component:
Audio/Video: cubeb
Platform:
All
Linux
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla58
Tracking Flags:
Tracking Status
firefox58 --- fixed

People

(Reporter: u480271, Assigned: u480271)

References

(Blocks 1 open bug)

Details

Attachments

(5 files, 1 obsolete file)

Bug 1407487 - P1: Remove --enable-cubeb-remoting.
59 bytes, text/x-review-board-request
rillian
: review+
kinetik
: review+
Details
Bug 1407487 - P2: Enable cubeb remoting on Nightly.
59 bytes, text/x-review-board-request
kinetik
: review+
Details
Bug 1407487 - P4: Disable backtrace in error-log to work around bug 1409276.
59 bytes, text/x-review-board-request
u480271
: review+
Details
Bug 1407487 - P3: Exclude Android when enabling cubeb-remoting.
59 bytes, text/x-review-board-request
u480271
: review+
Details
Bug 1407487 - P5: Support connections from within chrome process in audioipc.
59 bytes, text/x-review-board-request
u480271
: review+
Details 
      No description provided.
Assignee: nobody → dglastonbury
Blocks: 1362220
Depends on: 1407490
See Also: → 1407490
Rank: 25
Priority: -- → P2
I'm going to call this a P3 because this is tied to a new feature, not a regression/fix.
Comment on attachment 8917648 [details]
Bug 1407487 - P1: Remove --enable-cubeb-remoting.

https://reviewboard.mozilla.org/r/188596/#review193844
Attachment #8917648 - Flags: review?(kinetik) → review+
Comment on attachment 8917649 [details]
Bug 1407487 - P2: Enable cubeb remoting on Nightly.

https://reviewboard.mozilla.org/r/188598/#review193846

::: modules/libpref/init/all.js:644
(Diff revision 1)
>  
>  // Log level for cubeb, the audio input/output system. Valid values are
>  // "verbose", "normal" and "" (log disabled).
>  pref("media.cubeb.logging_level", "");
>  
> +#ifdef NIGHTLY_BUILD

It might be nice to make this NIGHTLY_BUILD && XP_LINUX to avoid confusing anyone that sees the pref enabled on other platforms.  Not a big deal though.
Attachment #8917649 - Flags: review?(kinetik) → review+
When I re-enable sandboxing on local machine, I see:

0 INFO SimpleTest START
1 INFO TEST-START | dom/media/mediasource/test/test_AVC3_mp4.html
2 INFO TEST-PASS | dom/media/mediasource/test/test_AVC3_mp4.html | Receive a sourceopen event 
3 INFO TEST-PASS | dom/media/mediasource/test/test_AVC3_mp4.html | fetchWithXHR load uri='avc3/init.mp4' status=200 
4 INFO Loading buffer: [0, 687)
5 INFO SourceBuffer buffered ranges grew from TimeRanges:  to TimeRanges: 
6 INFO TEST-PASS | dom/media/mediasource/test/test_AVC3_mp4.html | fetchWithXHR load uri='avc3/segment1.m4s' status=200 
7 INFO Loading buffer: [0, 696869)
8 INFO SourceBuffer buffered ranges grew from TimeRanges:  to TimeRanges: [0, 3.84)
9 INFO TEST-PASS | dom/media/mediasource/test/test_AVC3_mp4.html | continuous buffered range 
10 INFO TEST-PASS | dom/media/mediasource/test/test_AVC3_mp4.html | got loadeddata 
11 INFO TEST-PASS | dom/media/mediasource/test/test_AVC3_mp4.html | endOfStream completed 
GECKO(32291) | MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
GECKO(32291) | MEMORY STAT | vsize 1902MB | residentFast 167MB | heapAllocated 18MB
12 INFO TEST-OK | dom/media/mediasource/test/test_AVC3_mp4.html | took 6422ms
13 INFO TEST-START | dom/media/mediasource/test/test_AudioChange_mp4.html
14 INFO TEST-PASS | dom/media/mediasource/test/test_AudioChange_mp4.html | Receive a sourceopen event 
15 INFO TEST-PASS | dom/media/mediasource/test/test_AudioChange_mp4.html | readyState is HAVE_NOTHING 
16 INFO TEST-PASS | dom/media/mediasource/test/test_AudioChange_mp4.html | fetchWithXHR load uri='aac20-48000-64000-init.mp4' status=200 
17 INFO Loading buffer: [0, 1246)
18 INFO got durationchange event
19 INFO got loadedmetadata event
20 INFO SourceBuffer buffered ranges grew from TimeRanges:  to TimeRanges: 
21 INFO TEST-PASS | dom/media/mediasource/test/test_AudioChange_mp4.html | got loadedmetadata event 
22 INFO TEST-PASS | dom/media/mediasource/test/test_AudioChange_mp4.html | fetchWithXHR load uri='aac20-48000-64000-1.m4s' status=200 
23 INFO Loading buffer: [0, 24328)
24 INFO SourceBuffer buffered ranges grew from TimeRanges:  to TimeRanges: [0, 3.029333)
25 INFO got loadeddata event
26 INFO got canplay event
27 INFO TEST-PASS | dom/media/mediasource/test/test_AudioChange_mp4.html | got canplay event 
28 INFO got play event
29 INFO got playing event
GECKO(32291) | Sandbox: seccomp sandbox violation: pid 32351, tid 32469, syscall 16, args 33 21537 139883795421916 90 0 139883956404288.  Killing process.
GECKO(32291) | Sandbox: crash reporter is disabled (or failed); trying stack trace:
GECKO(32291) | Sandbox: frame #01: ???[/lib64/libpthread.so.0 +0x123b0]
GECKO(32291) | Sandbox: frame #02: ioctl[/lib64/libc.so.6 +0x1065e7]
GECKO(32291) | Sandbox: frame #03: _ZN3std3sys3imp3ext3net10UnixStream15set_nonblocking17hf01f3ab34222916fE[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x57b7964]
GECKO(32291) | Sandbox: frame #04: _ZN8audioipc10connection10Connection3new17h93c927a8bd24a323E[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x53671f0]
GECKO(32291) | Sandbox: frame #05: _ZN86_$LT$audioipc..connection..Connection$u20$as$u20$std..sys..imp..ext..io..FromRawFd$GT$11from_raw_fd17hce078adbd2c49afeE[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x536761a]
GECKO(32291) | Sandbox: frame #06: _ZN90_$LT$audioipc_client..context..ClientContext$u20$as$u20$cubeb_backend..traits..Context$GT$11stream_init17h2749a4263730a6bfE[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x4fcdc20]
GECKO(32291) | Sandbox: frame #07: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x4fc4e9d]
GECKO(32291) | Sandbox: frame #08: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x3b5675a]
GECKO(32291) | Sandbox: frame #09: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2c6cfb9]
GECKO(32291) | Sandbox: frame #10: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2c6ce18]
GECKO(32291) | Sandbox: frame #11: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2e25520]
GECKO(32291) | Sandbox: frame #12: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2e24c1d]
GECKO(32291) | Sandbox: frame #13: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2e26719]
GECKO(32291) | Sandbox: frame #14: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2e294ba]
GECKO(32291) | Sandbox: frame #15: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2caabc3]
GECKO(32291) | Sandbox: frame #16: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2caaadd]
GECKO(32291) | Sandbox: frame #17: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2cb98e3]
GECKO(32291) | Sandbox: frame #18: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x2cf250c]
GECKO(32291) | Sandbox: frame #19: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x1023745]
GECKO(32291) | Sandbox: frame #20: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x102af6b]
GECKO(32291) | Sandbox: frame #21: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x10319d8]
GECKO(32291) | Sandbox: frame #22: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x1031a9d]
GECKO(32291) | Sandbox: frame #23: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x102fb8a]
GECKO(32291) | Sandbox: frame #24: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x1041687]
GECKO(32291) | Sandbox: frame #25: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x15160e0]
GECKO(32291) | Sandbox: frame #26: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x14b5365]
GECKO(32291) | Sandbox: frame #27: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x102e907]
GECKO(32291) | Sandbox: frame #28: ???[/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libnspr4.so +0x27d2a]
GECKO(32291) | Sandbox: frame #29: ???[/lib64/libpthread.so.0 +0x736d]
GECKO(32291) | Sandbox: frame #30: clone[/lib64/libc.so.6 +0x110bbf]
GECKO(32291) | Sandbox: frame #31: ??? (???:???)
GECKO(32291) | Sandbox: end of stack.
GECKO(32291) | [Parent 32291, Gecko_IOThread] WARNING: pipe error (88): Connection reset by peer: file /home/djg/Mozilla/gecko-unified/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
GECKO(32291) | [Parent 32291, Gecko_IOThread] WARNING: pipe error (50): Connection reset by peer: file /home/djg/Mozilla/gecko-unified/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
GECKO(32291) | [Parent 32291, Gecko_IOThread] WARNING: pipe error (42): Connection reset by peer: file /home/djg/Mozilla/gecko-unified/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
GECKO(32291) | [Parent 32291, Gecko_IOThread] WARNING: pipe error (37): Connection reset by peer: file /home/djg/Mozilla/gecko-unified/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
GECKO(32291) | ###!!! [Parent][MessageChannel] Error: (msgtype=0x150083,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv
Buffered messages finished
TEST-UNEXPECTED-TIMEOUT | dom/media/mediasource/test/test_AudioChange_mp4.html | application timed out after 330 seconds with no output
30 ERROR Force-terminating active process(es).
Determining child pids from psutil
Found child pids: [32415]
Killing process: 32415
Failed to retrieve MOZ_UPLOAD_DIR env var
Can't trigger Breakpad, just killing process
psutil found pid 32415 dead
Killing process: 32291
Not taking screenshot here: see the one that was previously logged
Can't trigger Breakpad, just killing process
TEST-INFO | Main app process: killed by SIGKILL
Buffered messages finished
31 ERROR TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_AudioChange_mp4.html | application terminated with exit code -9
runtests.py | Application ran for: 0:05:42.419610
zombiecheck | Reading PID log: /tmp/tmpT62SaJpidlog
==> process 32291 launched child process 32351
==> process 32291 launched child process 32415
zombiecheck | Checking for orphan process with PID: 32351
zombiecheck | Checking for orphan process with PID: 32415
Stopping web server
Stopping web socket server
Stopping ssltunnel
websocket/process bridge listening on port 8191
Stopping websocket/process bridge
leakcheck | refcount logging is off, so leaks can't be detected!
runtests.py | Running tests: end.
Buffered messages finished
Running manifest: dom/media/test/mochitest.ini
pk12util: PKCS12 IMPORT SUCCESSFUL
MochitestServer : launching [u'/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/xpcshell', '-g', u'/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin', '-v', '170', '-f', u'/home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpkXPAuY.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/home/djg/Mozilla/objdir-desktop-opt.noindex/_tests/testing/mochitest/server.js']
runtests.py | Server pid: 32553
runtests.py | Websocket server pid: 32556
runtests.py | websocket/process bridge pid: 32564
runtests.py | SSL tunnel pid: 32585
runtests.py | Running with e10s: True
runtests.py | Running tests: start.
Comment on attachment 8917648 [details]
Bug 1407487 - P1: Remove --enable-cubeb-remoting.

https://reviewboard.mozilla.org/r/188596/#review194246
Attachment #8917648 - Flags: review?(giles) → review+
(In reply to Dan Glastonbury :kamidphish from comment #7)
> GECKO(32291) | Sandbox: seccomp sandbox violation: pid 32351, tid 32469,
> syscall 16, args 33 21537 139883795421916 90 0 139883956404288.  Killing
> process.
> GECKO(32291) | Sandbox: crash reporter is disabled (or failed); trying stack
> trace:
> GECKO(32291) | Sandbox: frame #01: ???[/lib64/libpthread.so.0 +0x123b0]
> GECKO(32291) | Sandbox: frame #02: ioctl[/lib64/libc.so.6 +0x1065e7]
> GECKO(32291) | Sandbox: frame #03:
> _ZN3std3sys3imp3ext3net10UnixStream15set_nonblocking17hf01f3ab34222916fE[/
> home/djg/Mozilla/objdir-desktop-opt.noindex/dist/bin/libxul.so +0x57b7964]

It looks like we'll end up here: https://github.com/rust-lang/rust/blob/84bbd14e3f392d63524482e2f0c69cd6fcd7f2c5/src/libstd/sys/unix/net.rs#L332 which does:

    libc::ioctl(*self.as_inner(), libc::FIONBIO, &mut nonblocking) })

Bug 1405891 recently changed the handling of ioctl() in the sandbox, but it looks like it's only supposed to affect TTY-related ioctls.
If necessary we can probably work around this by using fcntl to set O_NONBLOCK instead (assuming the sandbox permits that), or opening the socket as non-blocking directly by passing SOCK_NONBLOCK to socket()/accept().
If I'm reading the code correctly, the FIO* block of ioctls is also considered TTY related: https://dxr.mozilla.org/mozilla-central/source/security/sandbox/linux/SandboxFilter.cpp#717-741

I think it'd be stragihtforward to add a whitelist for FIONBIO. Jed, can you confirm I read this correctly?
Flags: needinfo?(jld)
Sorry about the confusion around “tty-related”; I should've described that change better.

We're already allowing FIOCLEX because the rust stdlib uses it rather than the equivalent fcntl[1], so I'm fine with adding FIONBIO.  (Filtering fcntl is planned, in bug 1328896, but my work-in-progress patch already allows setting O_NONBLOCK.)


[1] To support Linux kernels older than 2.6.24, according to the comments; not that they'd be doing seccomp-bpf, but apparently Rust supports them.  Incidentally this also breaks on ESXi, which implements fcntl but not FIOCLEX and probably not FIONBIO either: https://github.com/rust-lang/rust/issues/41347#issuecomment-294937148
Flags: needinfo?(jld)
Depends on: 1408821
Filed bug 1408821 to whitelist FIONBIO.  Try push of Dan's changes with the whitelist patch included: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5fb91da49a6dbf481840441963861f604c630f4e
Pushed by mgregan@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0ff7dc9aebd9
P1: Remove --enable-cubeb-remoting. r=kinetik,rillian
https://hg.mozilla.org/integration/autoland/rev/806976ac38b8
P2: Enable cubeb remoting on Nightly. r=kinetik
This looks to have triggered compiler crashes on pretty much all Android builds: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=806976ac38b8971776a5c078d2a093e347876f06.  Can y'all comment?
Flags: needinfo?(kinetik)
Flags: needinfo?(giles)
That's wierd, unless XP_LINUX is defined for Android builds. Should be linux-only.

Maybe also bug 1409276, since build scripts are involved?
Flags: needinfo?(giles)
(In reply to Ralph Giles (:rillian) | needinfo me from comment #16)
> That's wierd, unless XP_LINUX is defined for Android builds. Should be
> linux-only.
> 
> Maybe also bug 1409276, since build scripts are involved?

It's plausible, except the push immediately before this one has green Android builds:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=0a9f04d67bd0c4bb76933ed13eb55e7828be48a4

And all subsequent Android builds crash with the same error in the compile checks during configure.
(In reply to Ralph Giles (:rillian) | needinfo me from comment #16)
> That's wierd, unless XP_LINUX is defined for Android builds. Should be
> linux-only.

I think XP_LINUX _is_ defined for Android builds, based on stanzas like:

#if defined(XP_LINUX) && !defined(OS_ANDROID)

throughout the tree.  I think this needs to be backed out and relanded :/
Aha. I saw during review that XP_LINUX was set based solely on `target.kernel == 'Linux'` but didn't think about the implications for Android.

Is there a better define which means *-linux-gnu?
(In reply to Ralph Giles (:rillian) | needinfo me from comment #19)
> Aha. I saw during review that XP_LINUX was set based solely on
> `target.kernel == 'Linux'` but didn't think about the implications for
> Android.
> 
> Is there a better define which means *-linux-gnu?

I'm not expert here.  snorp?
Flags: needinfo?(snorp)
Oops, this code is not intended for Android for now.

It looks like we need |defined(XP_LINUX) && !defined(MOZ_WIDGET_ANDROID)| in Gecko code (OS_ANDROID is defined in the chromium/ipc code) and |CONFIG['OS_ARCH'] == 'Linux' and CONFIG['OS_TARGET'] != 'Android'|.  Let's back out and reland.
Flags: needinfo?(kinetik)
We're also being bitten by bug 1409276 for Linux Hazard builds: https://treeherder.mozilla.org/logviewer.html#?job_id=137744314&repo=autoland&lineNumber=13767

I think the simplest fix for that is to opt-out of error-chain's default features, since we can live without backtraces.
P3 could be rolled in to P1, but it doesn't seem possible to revise MozReview changes created by someone else... I couldn't carry the existing reviews forward.
Attachment #8919588 - Attachment is obsolete: true
Attachment #8919588 - Flags: review?(dglastonbury)
(In reply to Matthew Gregan [:kinetik] from comment #30)
> Missed adding a Cargo.lock update:
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=58e8fcee77b6d2d0f5f9e62983b8e39c78b2e94a

I expect this will still have the mda3/webspeech timeouts that philor mentioned in comment 23.  I suspect that's a regression from bug 1407490; I'll investigate tomorrow.
(In reply to Matthew Gregan [:kinetik] from comment #31)
> I expect this will still have the mda3/webspeech timeouts that philor
> mentioned in comment 23.  I suspect that's a regression from bug 1407490;
> I'll investigate tomorrow.

Confirmed.  This broke connections to the audioipc server from within the same process, which worked before bug 1407490 made the path unique.  I extended the temporary fix to handle this case and pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f79376b459c583d10b884802accaff25b3e73d22
Comment on attachment 8919596 [details]
Bug 1407487 - P3: Exclude Android when enabling cubeb-remoting.

https://reviewboard.mozilla.org/r/190466/#review196114
Attachment #8919596 - Flags: review?(dglastonbury) → review+
Comment on attachment 8919589 [details]
Bug 1407487 - P4: Disable backtrace in error-log to work around bug 1409276.

https://reviewboard.mozilla.org/r/190448/#review196116
Attachment #8919589 - Flags: review?(dglastonbury) → review+
Comment on attachment 8919992 [details]
Bug 1407487 - P5: Support connections from within chrome process in audioipc.

https://reviewboard.mozilla.org/r/190938/#review196118
Attachment #8919992 - Flags: review?(dglastonbury) → review+
Pushed by mgregan@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/13e2bc3638bc
P1: Remove --enable-cubeb-remoting. r=kinetik,rillian
https://hg.mozilla.org/integration/mozilla-inbound/rev/317843e4b256
P2: Enable cubeb remoting on Nightly. r=kinetik
https://hg.mozilla.org/integration/mozilla-inbound/rev/61961b3f86be
P3: Exclude Android when enabling cubeb-remoting.  r=kamidphish
https://hg.mozilla.org/integration/mozilla-inbound/rev/2b88b91f35f9
P4: Disable backtrace in error-log to work around bug 1409276.  r=kamidphish
https://hg.mozilla.org/integration/mozilla-inbound/rev/64cecfd341a1
P5: Support connections from within chrome process in audioipc.  r=kamidphish
Depends on: 1410702
Depends on: 1410704
Noticed these improvement:

== Change summary for alert #10101 (as of October 19 2017 06:32 UTC) ==

Improvements:

  3%  cpstartup content-process-startup linux64 opt e10s     268.50 -> 259.50

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=10101
Depends on: 1463001
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: