1551195 - Fennec Beta 68 x86 crash when sharing the microphone

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Audio/Video, defect, P1)

Description

[Stefan Deiac]

Environment:
Device: Lenovo Yoga Tablet 2 (Android 4.4.2) - x86;
Build: Nightly 68.0a1 (2019-05-07) and Beta 68.0b1 ESR builds;

Steps to reproduce:

1. Go to: https://goo.gl/8fOcQf;
2. Share your video/microphone;

Expected result:
A video starts playing from the camera;

Actual result:
The app crash immediately after sharing the video/microphone

Notes:

• Not reproducible on Google Pixel 3XL (Android 9), Samsung Galaxy S9 (Android 8.0), Samsung Galaxy S8 (Android 9).
• https://crash-stats.mozilla.org/report/index/c0863278-14a6-4756-a93a-87ae60190513#tab-details

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Does this only reproduce with the Try builds?

Comment 2

[Stefan Deiac]

I tested with the official builds Beta 67.0b18 and Nightly 68.0a1 (2019-05-13) but I can't reproduce the problem.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I had Stefan try out a 68-as-Beta simulation build from April 30, which predates any of our Fennec ESR patches landing and the issue is still reproducible. So that's pointing to this being a general issue with Beta68. Any thoughts on this, Dan? I've also given Stefan some pointers for trying to narrow down the regression window a bit, but that may be a time-consuming endeavor and I'm not sure how narrow of a window will be possible.

Comment 4

[Dan Minor [:dminor]]

Hi Stefan, I see cubeb on the crash stack which is the library we use to interface with the operating system audio libraries. Can you please retest with with only the microphone shared to see if this reproduces without video? That would help narrow things down. Thank you!

(Visiting this page: https://mozilla.github.io/webrtc-landing/gum_test.html and selecting audio should work)

Comment 5

[Stefan Deiac]

Hello, I tried to reproduce this issue using the link from comment 4 and my findings are:

• When I share both of them (microphone and camera) - the application crash;
• When I share only the microphone - the application crash;
• When I share only the camera - the application doesn't crash;

Additionally, I tried with appr.tc and talky.io and the issue is still reproducible.

Comment 6

[Dan Minor [:dminor]]

Alex, do you mind having a look at this? From the crash stack, it looks like it might be cubeb related.

Comment 7

[Paul Adenot (:padenot)]

I need the line number in the stack frame 32 in the crash report above. I can also do with the exact libxul.so from a build that crashes, maybe, but it'd be easier with a good crash report.

Comment 8

[Alex Chronopoulos [:achronop]]

Stefan, can you try a debug build in case we get a better stack. If it does not reproduce in Beta 67.0b18 can we find which commit broke this?

Comment 9

[Stefan Deiac]

Hello, I tried to debug all versions from simulation builds and those are my findings:

• The crash is reproducible between March 24th to March 28th (the 28th build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8b506be75dd0f9d640bc7501eb953cb198588f89);
• From March 29th to April 16th the crash isn't anymore reproducible (the 29th build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c85f749c44797a1334abb2e70867b9c7836cdf4f);
• From April 17th the crash reappears (https://treeherder.mozilla.org/#/jobs?repo=try&revision=8433039e48485c08d99e6934621089bb7f5c0cde&searchStr=x86);

Comment 10

[Paul Adenot (:padenot)]

Yes, we need the line it crashes, the commit that broke it is my patch set in bug 1531833.

I think this is related to non-standard behaviour of audio methods on some devices.

Comment 11

[Stefan Deiac]

Attachment: Logcatx86crash.txt

Hello, there is the logcat of the crash.

Comment 12

[Paul Adenot (:padenot)]

Stefan, can you repro on the device with this build, which has additional logging, and provide the logcat ? Thanks.

Comment 13

[Stefan Deiac]

Attachment: logcat.txt

Hello, I tried to reproduce the issue using your build from comment 12. Anyway, I'll attach a logcat with my findings. Hope it helps.

Comment 14

[Paul Adenot (:padenot)]

You mean that it works with this build? Indeed this log seem to be of something that works and does not crash.

Comment 15

[Stefan Deiac]

Yeah, the crash doesn't occur anymore with this build.

Comment 16

[Paul Adenot (:padenot)]

That's annoying, it only changes logging. Alex, do you know what could happen here, have you seen this? Probably a subtle race?

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

From a chat with Stefan last week, the issue seemed only to reproduce up to May 7th: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&revision=7845efee95bebe266d76844c198cf70bb14916

Also had hacked together a debug build with which Stefan was able to reproduce:
Can you try with https://treeherder.mozilla.org/#/jobs?repo=try&revision=077c9a9040f257e9e953d5ab5214c6aa0af7b836 ? That's a debug build based on the same code like the last failing opt one.

Comment 18

[Alex Chronopoulos [:achronop]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #17)

Also had hacked together a debug build with which Stefan was able to reproduce:
Can you try with https://treeherder.mozilla.org/#/jobs?repo=try&revision=077c9a9040f257e9e953d5ab5214c6aa0af7b836 ? That's a debug build based on the same code like the last failing opt one.

Can you please open the about:crashes page of that debug build and provide the crash-ids listed there. I hope that a crash report from a debug build will provide some extra information. Right now the report is vague. Not sure where we crash.

Comment 19

[Stefan Deiac]

https://crash-stats.mozilla.org/report/index/361d9509-a608-432c-8472-d360c0190520

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thanks to :willkg for the explanation why the a resymbolification of the crash stack didn't make it more readable. The libmedia.so is from Android and the symbol upload job provides the symbols for Fennec. Are crashes in system files on Android 4.2 x86 in general?

Comment 21

[Paul Adenot (:padenot)]

If it does not repro after May 7th, can we try to find what fixed it? There was quite some changes there, if I have a range I can have a look and find what fixed it.

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Pushlog last bad to first good: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6fd64908d113d7cdf085be1e2858e10dd69643e3&tochange=65a693623cee0837b4ad0d23241c84cd3ea23e3a
Let me know if more builds are needed for bisection.

Stefan also mentioned "it seems that from 03-29 to 04-16 the issue disappears and from 03-24 to 03-28 is it reproducible".

Comment 23

[Alex Chronopoulos [:achronop]]

Most likely Bug 1549699 fixed it.

Comment 24

[Marcia Knous [:marcia]]

If there are no recent crashes, can we close this one out?

Comment 25

[Chris Peterson [:cpeterson]]

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #24)

If there are no recent crashes, can we close this one out?

I see only 8 reports in Fennec 68.0b5 and none from Fennec 69 Nightly. We can probably close this bug as fixed by bug 1549699.