1034147 - [B2G][Flame] libaudio_route missing symbols causing audio failures on v188 (e.g. cannot using camera to record)

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Camera, defect)

Description

[Andrew Osmond [:aosmond] (he/him)]

Attachment: record_audio_failure.log

This tracks an intermittent failure I have experienced in the last two weeks.

Steps to reproduce:
1) Load an build from 2.0 or master (at least).
2) Attempt to start record video
3) See an error dialog containing only a period pop up
4) All subsequent attempts will fail as well.

Initial analysis from log provided below.

--

First failure appears to be for playing the "start recording" sound notification:

07-03 07:11:31.801   280  1144 E AudioPolicyManagerALSA: getDeviceForStrategy() speaker device not found for STRATEGY_SONIFICATION
07-03 07:11:31.801   280  1144 W AudioPolicyManagerALSA: getOutput() could not find output for stream 5, samplingRate 48000,format 1, channels 1, flags 4
07-03 07:11:31.801  1804  1895 E AudioTrack: Could not get audio output for stream type 5
07-03 07:11:31.801  1804  1895 E libOpenSLES: AudioTrack::initCheck status 4294967274
07-03 07:11:31.801  1804  1895 W libOpenSLES: Leaving Object::Realize (SL_RESULT_CONTENT_UNSUPPORTED)

The critical failure happens further below:

07-03 07:11:32.031  1804  1822 I Gecko   : [../../../../b2g_v20/dom/camera/GonkRecorder.cpp:1064]clipAudioBitRate: encoder 3
07-03 07:11:32.031  1804  1822 I Gecko   : [../../../../b2g_v20/dom/camera/GonkRecorder.cpp:1080]Intended audio encoding bit rate (156000) is too large and will be set to (96000)
07-03 07:11:32.031  1804  1822 I Gecko   : [../../../../b2g_v20/dom/camera/GonkRecorder.cpp:1086]clipAudioSampleRate: encoder 3
07-03 07:11:32.031  1804  1822 I Gecko   : [../../../../b2g_v20/dom/camera/GonkRecorder.cpp:1108]clipNumberOfAudioChannels: encoder 3
07-03 07:11:32.031  1804  1822 E AudioRecord: Unsupported configuration: sampleRate 48000, format 1, channelMask 0xc
07-03 07:11:32.031  1804  1822 I Gecko   : [../../../../b2g_v20/dom/camera/GonkRecorder.cpp:737]audio source is not initialized

This appears to happen because the audio hardware device did not initialize successfully as the given parameters are valid. The error bubbles up to apps a little later:

07-03 07:11:32.061  1804  1822 I Gecko   : [../../../../b2g_v20/dom/camera/GonkRecorder.cpp:1462]stop
07-03 07:11:32.061  1804  1822 I Gecko   : [Child 1804] WARNING: Camera control API(6) failed with 0x80004005: file ../../../../b2g_v20/dom/camera/CameraControlImpl.cpp, line 375
07-03 07:11:32.061  1804  1822 I Gecko   : CameraControlImpl::OnUserError : aContext='StartRecording' (6), aError=0x80004005
07-03 07:11:32.061  1804  1804 I Gecko   : DOM OnUserError aContext=6, error='GeneralFailure'

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

This was reproducible on JB occasionally, but never on KK once the audio dependencies were satisfied. Any similar looking bugs reported have had different root causes rather than an audio failure at startup.

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

Reproduced on flame v188 / b2g-inbound debug build.

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

Attachment: bug1034147_flame_v188_record_error.log

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

Attachment: bug1034147_flame_v188_record_error_from_boot.log

Better log capturing the issue from bootup.

Comment 5

[Andrew Osmond [:aosmond] (he/him)]

Must be a new problem. It happens if one does a full flash on top of v188 instead of a shallow gecko/gaia flash. One of our dependencies, libaudio_route, is missing some symbols added in a patch in the codeaurora repo [1] which is missing from the AOSP repo.

mwu: I looked at the B2G manifest files, and it looks like base-kk.xml [2] is what controls which repo we pull platform/system/media from. My reading of this is that it would come from the AOSP project, and the contents seem to match up, but git config shows that it comes from codeaurora:

> $ git config -l
> ...
> remote.caf.url=git://codeaurora.org/platform/system/media
> remote.caf.projectname=platform/system/media
> remote.caf.fetch=+refs/heads/*:refs/remotes/caf/*

The way repo works seem to hide the actual branch from which we pull from in git, although maybe you can provide a suggestion on how to read this and map it back to the manifest file:

> $ git branch -v
> * (no branch) 20c2fb4 Merge "Merge commit 'android-4.4.1_r1' into merge_branch"
> $ git log
> commit 20c2fb4c896aa59f2e8379d755f439dc59a5cf9b
> commit 20c2fb4c896aa59f2e8379d755f439dc59a5cf9b
> Merge: b9e4326 114b4f6
> Author: Linux Build Service Account <lnxbuild@localhost>
> Date:   Sat Dec 7 00:25:25 2013 -0800
> 
>     Merge "Merge commit 'android-4.4.1_r1' into merge_branch"
> 
> commit 114b4f6f3d4dfc99e13f4ec6773e60368d7ad157
> Merge: b9e4326 52a64a5
> Author: Manish Kumar <[e-mail]>
> Date:   Thu Dec 5 20:39:06 2013 -0800
> 
>     Merge commit 'android-4.4.1_r1' into merge_branch
>     
>     * commit 'android-4.4.1_r1':
>       Camera: Make partialResult quirk fields hidden, not system.
>       Camera: Add partialResult quirk metadata fields

Any suggestions on how to read from the proper repo with the patch from August in master?

[1] https://www.codeaurora.org/cgit/quic/la/platform/system/media/commit/audio_route/audio_route.c?h=LNX.LA.2.7.3&id=4d1023f3e1b5fa269485f53d8cd73b28722e54d2
[2] https://github.com/mozilla-b2g/b2g-manifest/blob/master/base-kk.xml

Comment 6

[Andrew Osmond [:aosmond] (he/him)]

To workaround, you need to reload the original libaudioroute onto the device. This is enough to get everything working again.

1) Flash vendor image, v188+.
2) Ensure backup_flame folder is deleted before doing a full flame build.
3) Do a full flash from your flame build.
4) adb remount && adb push ./backup-flame/system/lib/libaudioroute.so /system/lib && adb reboot

Comment 7

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Override platform/system/media revision, v2

Found the time to answer my own questions :). I was blind.

This PR overrides the version to sync for platform/system/media; I was not sure if I should update *everything* or who exactly decides to do that, but at least this PR is very low risk since there is only one change [1] and it should be source/binary backwards compatible with earlier vendor images.

[1] https://www.codeaurora.org/cgit/quic/la/platform/system/media/commit/?id=c1332c21c608f4932a6d7e83450411cde53315ef

Comment 8

[Michael Wu [:mwu]]

How about tracking the branch that that tag came from? I think it's LNX.LF.3.5.2.1_1.

I think a lot of the repos in here should actually be tracking LNX.LF.* now, as https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/commit/?h=release&id=ee59d7f38a7e9356bd45f7b2840834214aeac397 seems to suggest.

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

Comment on attachment 8512161 [details] [review]
Override platform/system/media revision, v2

(In reply to Michael Wu [:mwu] from comment #8)
> How about tracking the branch that that tag came from? I think it's
> LNX.LF.3.5.2.1_1.
> 
> I think a lot of the repos in here should actually be tracking LNX.LF.* now,
> as
> https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/commit/
> ?h=release&id=ee59d7f38a7e9356bd45f7b2840834214aeac397 seems to suggest.

LNX.LF.3.5.2.1_1 works fine. It has the same contents as the tag at the moment, so same level of risk (i.e. nil). Updated the pull request as such.

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Override platform/system/media revision, v1 -- 2.1 branch

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Override platform/system/media revision, v1 -- 2.0 branch

Comment 12

[Andrew Osmond [:aosmond] (he/him)]

[Blocking Requested - why for this release]: Full flash images contain an outdated libaudioutil preventing audio from working at all.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Comment on attachment 8512972 [details] [review]
Override platform/system/media revision, v1 -- 2.1 branch

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Flame v188 image has dependencies on recent changes in libaudioroute.
User impact if declined: Audio will be broken in full flash images, including sound effects, camera recording, and music playback.
Testing completed: Manually built and flashed full image, audio drivers now load on flame. Camera sound effects and recording functional.
Risk to taking this patch (and alternatives if risky): Minimal, only new functions added, libraries remain source/binary compatible.
String or UUID changes made by this patch: None.

Comment 14

[Andrew Osmond [:aosmond] (he/him)]

Comment on attachment 8512973 [details] [review]
Override platform/system/media revision, v1 -- 2.0 branch

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Flame v188 image has dependencies on recent changes in libaudioroute.
User impact if declined: Audio will be broken in full flash images, including sound effects, camera recording, and music playback.
Testing completed: Manually built and flashed full image, audio drivers now load on flame. Camera sound effects and recording functional.
Risk to taking this patch (and alternatives if risky): Minimal, only new functions added, libraries remain source/binary compatible.
String or UUID changes made by this patch: None.

Comment 15

[Carsten Book [:Tomcat]]

https://github.com/mozilla-b2g/b2g-manifest/commit/50a9bf24c439ae435e1c264cf254219cab163b3b

Comment 16

[Pete Moore [:pmoore][:pete]]

This has broken b2g bumper for dolphin builds, since this head does not exist on http://sprdsource.spreadtrum.com:8085/b2g/android/platform/system/media which is the default for dolphin (dolphin is not using code aurora but still inherits from base-kk.xml

Comment 17

[Carsten Book [:Tomcat]]

had to revert this, because of causing problems for releng

Comment 18

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Override platform/system/media revision, v3

Moved where I add the branch to base-kk-caf.xml, this should now only affect flame-kk at present, looking at the dependencies between manifest files.

Note the 2.1 and 2.0 pull requests are unaffected since the flame-kk.xml file itself contained the platform/system/media reference.

Comment 19

[Michael Wu [:mwu]]

Comment on attachment 8513602 [details] [review]
Override platform/system/media revision, v3

Oops - should have caught this the first time. base-kk.xml should generally not refer to specific revisions .

Comment 20

[Andrew Osmond [:aosmond] (he/him)]

Let's try this again. With less destructive force :).

Comment 21

[bhavana bajaj [:bajaj]]

Can QA help verify this atleast on trunk before we uplift on branches?

Comment 22

[Carsten Book [:Tomcat]]

https://github.com/mozilla-b2g/b2g-manifest/commit/8b8379d3b433275de2be9bdb5663e1dbff727b52

Comment 23

[Tony Chung [:tchung]]

(In reply to bhavana bajaj [:bajaj] from comment #21)
> Can QA help verify this atleast on trunk before we uplift on branches?

We think this will fix the dependency bugs in bug 1085759.   I've already asked nthomas to spin me a b2g-i build that includes a flame KK pvtbuild with v188 bits and this patch included.   Also adding qawanted to see if anyone else can flag this.

Comment 24

[Tony Chung [:tchung]]

Verified this against a private Jenkins build that nthomas respun for me.  Audio works again.  This build runs with v188 binaries, and picks up the patch in this bug.   however, not marking this status=verified yet until this is retested on a pvtbuild generated against m-c.  

Gaia-Rev        8ae6598f3ab7b0c34ac42a73083ddb74266affba
Gecko-Rev       https://hg.mozilla.org/users/nthomas_mozilla.com/mozilla-central/rev/d4fab3f7782b
Build-ID        20141030203020
Version         36.0a1
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141030.201724
FW-Date         Thu Oct 30 20:17:37 EDT 2014
Bootloader      L1TC10011880

Comment 25

[Viorela Ioia [:viorela]]

This issue is not reproducible in latest master. 

Device firmware (date)     31 Oct 2014 04:25:03
Device firmware (incremental)     eng.cltbld.20141031.072452
Device firmware (release)     4.4.2
Device identifier     flame
Gaia date     30 Oct 2014 10:56:06
Gaia revision     8ae6598f3ab7
Gecko build     20141031040201
Gecko revision     e0b505a37b1c
Gecko version     36.0a1
Bootloader      L1TC10011880

Marking this as verified, as it works on m-c.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

v2.1: https://github.com/mozilla-b2g/b2g-manifest/commit/473de548fffb1a6ffddaf6d01bfe3debe389becd
v2.0: https://github.com/mozilla-b2g/b2g-manifest/commit/df219894d6241ef0cfb78a4b63f80cfae23e46b7

Comment 28

[Joshua Mitchell (Inactive)]

issue is verified fixed - removing outdated keywords

Comment 29

[Shine(Leave from Mozilla)]

Attachment: video

This issue has been verified successfully on Flame 2.0/2.1/2.2
See attachment: Verify_1034147.MP4
Reproducing rate: 0/3
Reproducing Steps:
1.Full flash the latest build.
2.Launch Camera and switch to Camcorder.
3.Start record video
** It can record video successfully, the error dialog cannot pop up.

Flame 2.0 build:
Gaia-Rev        856863962362030174bae4e03d59c3ebbc182473
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/2d0860bd0225
Build-ID        20141209160202
Version         32.0

Flame 2.1 build:
Gaia-Rev        c226db212db4d824c09617cd6dc407b2d4258d9b
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/cf8bebfa4703
Build-ID        20141209170126
Version         34.0

Flame 2.2 build:
Gaia-Rev        bd4dcc8c4582e2368b47b0e62506d3031fb2fc09
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/f1f48ccb2d4e
Build-ID        20141208160206
Version         37.0a1