Closed Bug 1166758 Opened 9 years ago Closed 6 years ago

[Aries] Listening to music while the screen is locked results in intermittent beeps/buzzing noises

Categories

(Firefox OS Graveyard :: GonkIntegration, defect, P2)

Product:
Firefox OS Graveyard
Component:
GonkIntegration
Platform:
ARM
Gonk (Firefox OS)
Type:
defect

Tracking

(tracking-b2g:+, b2g-master affected)

Status:
RESOLVED WONTFIX
Project Flags:
tracking-b2g +
Tracking Flags:
Tracking Status
b2g-master --- affected

People

(Reporter: cwiiis, Assigned: bwu)

References

Details

(Whiteboard: [spark][POVB])

Attachments

(1 file)

Dump-audio-pcm-in-AudioStream.patch
4.96 KB, patch
Details | Diff | Splinter Review 
[Blocking Requested - why for this release]: Users should be able to play music normally.

STR:

Prerequisite: Aries device running master with mp3 music loaded on it. Reproduced on a Sony Xperia Z3C.

1- Open the music app and play anything
2- Lock the device

Expected:

Music plays normally.

Actual:

Music plays, but intermittently there are sort of buzzing noises, where it sounds like perhaps a buffer is under-running.

This only happens using the speaker and headphone port, using bluetooth audio is unaffected. This also doesn't seem to happen while actively using the device (so perhaps a scheduling/priority issue?)
blocking-b2g: spark? → spark+
Flags: needinfo?(squibblyflabbetydoo)
This sounds like a Gecko/Gonk issue. You're asking the wrong person; I have only a rough understanding of the Gecko layer.
Flags: needinfo?(squibblyflabbetydoo)
Paul, any ideas about what to do here, or who to ask?
Flags: needinfo?(padenot)
No idea. This happens for both mp3 and ogg, so it's not the special offload code for mp3 that is the issue (I'm actually not sure it's enabled). Also I mostly hear this at the beginning of the a track, it's fine after 10s or so.

Listening carefully, glitches can last for multiple hundreds of milliseconds. I've seen stutter (reading the same portion of the buffer multiple times), and proper random buffer reading.

The buffer queue we have, when using opensl (we should check it's used, I think it is, but better be sure since cubeb can fallback automatically), is four elements long, and AudioStream.cpp requests 100ms of latency. This means each buffer we have to fill is roughly 25ms. Our code missing one or more deadlines would not explain the duration of the gliches.

I know that on my android tablet, when listening carefully with headphones, since recent Android versions, glitches occur as well on all apps (spotify, vlc, etc.), which would point to a problem down the line.

There are a number of things we could try to attack this:
- Record glitches over a simple sine wave changing pitch, to try to understand (just plug the device into the mic socket and record with audacity or whatever)
- Get a regression range
- Try with different base image (is this running a different android base image than, say, a flame gecko master/gaia master?)
- Make sure we are making each deadline in the code, maybe by implementing a very simple log of callbacks ? We should measure a time and a duration of each callback, and correlate the time with a recording of glitches to see what's up
- We should look at the thread priorities when turning the screen off. It might be that android lower the threads priorities, or do other energy saving measures, I don't know
Flags: needinfo?(padenot)
Gabriele, as things to point to this possibly being a scheduling issue, is this something you have any comment on?
Flags: needinfo?(gsvelto)
I don't have an aries yet, I should get it this week though, when I do I can try it out and see if I reproduce. From the description in comment 0 though I doubt this might be due to scheduling, I would expect the app to stutter when the phone is under load, not when it's idle.

Two other things come to mind: a power management problem whereas the phone is entering a low-power state where there's not enough CPU time to properly feed the audio output or an issue with the DSP itself (IIRC most audio decoding is not done on the CPU but rather on a dedicated DSP, the former does need to wake from time to time to feed it with data though).
Flags: needinfo?(gsvelto)
(In reply to Gabriele Svelto [:gsvelto] from comment #5)
> Two other things come to mind: a power management problem whereas the phone
> is entering a low-power state where there's not enough CPU time to properly
> feed the audio output or an issue with the DSP itself (IIRC most audio
> decoding is not done on the CPU but rather on a dedicated DSP, the former
> does need to wake from time to time to feed it with data though).

This should be a CPU problem, we don't have code to offload ogg vorbis decoding to a DSP chip, and I've heard it happen with ogg vorbis tracks. We only use DSP offloading for mp3, and when the screen is off.
Good to know, then it might be a frequency scaling problem. Try watching the frequency when locking the homescreen and see how far it drops:

watch 'adb shell cat -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq'
Hrm, so, I can't run adb commands when the screen is off, it does not find the device anymore, how can I keep adb working in this situation ?
Flags: needinfo?(gsvelto)
Since bug 810092 adb is disabled on user builds when the screen is locked. To work around it you either need a non-user build (userdebug and eng both work fine) or you simply need to disable the lockscreen in the prefs to prevent this behavior.
Flags: needinfo?(gsvelto)
Cool, that worked:

> ~  » adb shell cat -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
> 300000$
> ~  » adb shell cat -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
> 652800$

First is screen off, second is screen on.
300MHz is pretty low but should still be plenty enough for decoding audio. Can you check how much CPU time is being used in both cases with:

adb shell top -s cpu -m 5 -d 1

That should give us a hint if we're CPU starved or not.
Playing an mp3 in the music app:

Screen off: 

> User 16%, System 37%, IOW 0%, IRQ 0%
> User 65 + Nice 0 + Sys 151 + Idle 189 + IOW 0 + IRQ 0 + SIRQ 0 = 405
> 
>   PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
> 15612  0  27% R    42 278100K  82016K  fg u0_a1561 /system/b2g/b2g
> 11599  0   5% S    14  72304K  30812K  fg media    /system/bin/mediaserver
> 11859  2   3% R     1   1336K    484K     root     top
> 12743  0   1% R   100 552304K 164576K     root     /system/b2g/b2g
>  1192  0   0% S     6   6228K    496K     root     /system/bin/mpdecision



Screen on:

> User 10%, System 12%, IOW 0%, IRQ 0%
> User 39 + Nice 0 + Sys 47 + Idle 285 + IOW 0 + IRQ 0 + SIRQ 0 = 371
> 
> PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
> 15612  3  15% S    42 278108K  81676K  fg u0_a1561 /system/b2g/b2g
> 11599  0   3% S    14  72304K  30812K  fg media    /system/bin/mediaserver
> 12743  2   3% S   100 560656K 164940K     root     /system/b2g/b2g
> 11859  0   1% R     1   1336K    484K     root     top
>  171  3   0% D     1      0K      0K     root     boost_sync/3
(In reply to Paul Adenot (:padenot) from comment #12)
> >   PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
> > 15612  0  27% R    42 278100K  82016K  fg u0_a1561 /system/b2g/b2g
              ^^^

There's your problem. top is reporting CPU usage as a percentage of total usage so the b2g process consuming 27% means that it's consuming 100% CPU time of a single core and then some more (possibly in other threads). So apparently a single core running at 300MHz is not enough for this job.

While I find this rather surprising we have to find a viable solution. One thing is to verify why the phone is not being pushed to higher frequency automatically. The CPU frequency scaling governor should do that but it's perfectly possible that it's switching between a higher and a lower power state because the CPU load is just a little bit more than what the low power state provides (the measurement with the screen off suggests that 400-450MHz would probably be enough).

Admitted that we find a way to push the phone to a higher frequency in this situation that's not an optimal solution as we'd be hurting battery life pretty badly. The best course of action would be to profile this workload in this low power state and see if we can optimize it to work within this envelope.
Note that this is supposed to be _heavily_ multi-threaded, so it should be distributed on multiple core and it should be fine.

I'm gonna do the measurement again with -t to check if there is a thread that is individually at 25%+ or the like.
Using `adb shell top -s cpu -m 5 -d 1 -t` so we see threads:

Screen off:

> User 21%, System 18%, IOW 0%, IRQ 0%
> User 48 + Nice 0 + Sys 40 + Idle 132 + IOW 0 + IRQ 0 + SIRQ 0 = 220
> 
>   PID   TID PR CPU% S     VSS     RSS PCY UID      Thread          Proc
>   295   694  0   6% S  47516K  15168K  fg media    AudioOut_3      /system/bin/mediaserver
>  4974  4974  3   5% R   1488K    484K     shell    top             top
>  4309  4851  0   3% S 208936K  81120K  fg u0_a4309 gle.mp3.decoder /system/b2g/b2g
>  4309  4846  0   2% S 208936K  81120K  fg u0_a4309 MediaPl~back #1 /system/b2g/b2g
>  4309  4858  0   2% S 208936K  81120K  fg u0_a4309 MediaPl~back #5 /system/b2g/b2g

Screen on:

> User 21%, System 10%, IOW 0%, IRQ 0%
> User 74 + Nice 0 + Sys 38 + Idle 234 + IOW 0 + IRQ 0 + SIRQ 0 = 346
>
>   PID   TID PR CPU% S     VSS     RSS PCY UID      Thread          Proc
>  4974  4974  0   2% R   1488K    484K     shell    top             top
>   295   694  3   2% S  48540K  15168K  fg media    AudioOut_3      /system/bin/mediaserver
>  4309  4309  3   2% S 208944K  81132K  fg u0_a4309 Music           /system/b2g/b2g
>  4309  4846  3   0% S 208944K  81132K  fg u0_a4309 MediaPl~back #1 /system/b2g/b2g
>  4309  4848  3   0% S 208944K  81132K  fg u0_a4309 MediaPl~back #3 /system/b2g/b2g

This is definitely not hitting the CPU too hard. I was thinking that maybe priorities would change, but quickly looking at it does not show anything interesting. I could use another pair of eyes, though, I'm not too familiar with Android, I did my comparison with `ps`, that shows the niceness.
Paul, what are our next steps here? Who should we contact for help?
Flags: needinfo?(padenot)
Steven, does someone on your team has access to an Aries (Sony Z3C) device and has some time to help with this ? I'm thinking maybe it would be easier for someone with mobile and/or android experience to characterize this ?
Flags: needinfo?(padenot) → needinfo?(slee)
(In reply to Paul Adenot (:padenot) from comment #15)
> This is definitely not hitting the CPU too hard.

Indeed, it looks spread out on the different cores. You could try using kernel tracing to verify that the threads are really running on different cores but I doubt the scheduler would lump them all on the same one.

> I was thinking that maybe
> priorities would change, but quickly looking at it does not show anything
> interesting. I could use another pair of eyes, though, I'm not too familiar
> with Android, I did my comparison with `ps`, that shows the niceness.

We've been using control groups since bug 1081871, there should be no change in priority assignments though when turning the screen on or off and indeed the PCY column shows the main process' threads being in the same control group (fg stands for foreground).
Naoki, can you still repro this?
Flags: needinfo?(nhirata.bugzilla)
I can't seem to reproduce the issue on this build : https://tools.taskcluster.net/task-inspector/#IZU4TkDOQ--KQulTLtSldA/0

Chris, can you give this build a try to see if that resolve your issue?
I'm not sure what the change is.
Flags: needinfo?(nhirata.bugzilla) → needinfo?(chrislord.net)
Can anyone still repro this?
Keywords: qawanted
Issue still reproduces on latest Spark 3.0

Reproduced issue on 3 out of 3 mp3's tested, at intervals a weird buzzing or skipping noise can be heard. Easiest to reproduce while screen is locked.

Device: Aries 3.0
Build ID: 20150608104950
Gaia: ea27c4ed5b6083c9e21d233d4804372ac4d5d353
Gecko: 4700d1cdf489
Gonk: 3af1ede0d0956cfbf9c549df7cd9a6807a9efdf2
Version: 41.0a1 (3.0)
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:41.0) Gecko/41.0 Firefox/41.0
QA Whiteboard: [QAnalyst-Triage?]
status-b2g-master: --- → affected
Flags: needinfo?(ktucker)
Keywords: qawanted
Using master built from a check-out of about 2 hours ago (give or take), this still occurs for me. Tested on a .m4a file.
Flags: needinfo?(chrislord.net)
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(ktucker)
I found another way to generate noises. XD 
I am not sure whether the root causes are the same. Here is my STR.
1. play any song from music app and kill mediaserver
2. stop music app
3. play any sound, ex DTMF tone, or play a song from music app.
4. You will always hear noise in the beginning

BTW, CJ will take over this bug.
Flags: needinfo?(sthugo) → needinfo?(cku)
blocking-b2g: spark+ → 2.5+
After thinking about this for some time, I think the issue lies in AudioStream.cpp. I have not properly trouble shot this thought, this is just by ear.
Flags: needinfo?(cku)
Alastor, could you help on this?
Flags: needinfo?(alwu)
Blake, could you help to comment?
Component: AudioChannel → Audio/Video: Playback
Flags: needinfo?(alwu) → needinfo?(bwu)
Product: Firefox OS → Core
Ken, can you help out here and find an assignee please? We already lost a couple of dogfooders because of this bug.
Flags: needinfo?(kchang)
Perhaps, this is because phone uses the lower cpu operation frequency when phone screen is off. We can try if wake lock mechanism could fix this problem.

Blake, please take a look, thanks.
Assignee: nobody → bwu
Flags: needinfo?(kchang)
It would be a good idea to check:
- The number of frames requested by the AudioStream::DataCallback each time, when the screen is on and off
- The state of the CircularBuffer in both cases. It might be under-runing or something.

Both in AudioStream.cpp.
(In reply to Paul Adenot (:padenot) from comment #32)
> It would be a good idea to check:
> - The number of frames requested by the AudioStream::DataCallback each time,
> when the screen is on and off
> - The state of the CircularBuffer in both cases. It might be under-runing or
> something.
> 
> Both in AudioStream.cpp.
Thanks for your feedback! 
Normally, for the music case on B2G audio is playing via AudioOffloadPlayer[1] to save CPU resource.
However, in Z3C, AudioOffloadPlayer fails to get started @[2] and mAudioSink->Open()[3]. 
From what I can tell so far, when creating Android audio track in AudioOutput::Open()@AudioOutput.cpp[4], some error happens in Android AudioTrack.cpp. 
I am going to dig into.  

[1]https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/MediaOmxCommonDecoder.cpp?case=true&from=MediaOmxCommonDecoder.cpp#68
[2]https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/MediaOmxCommonDecoder.cpp?case=true&from=MediaOmxCommonDecoder.cpp#82 
[3]https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/AudioOffloadPlayer.cpp?from=AudioOffloadPlayer.cpp#158
[4]https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/AudioOutput.cpp?case=true&from=AudioOutput.cpp#114
Flags: needinfo?(bwu)
(In reply to Blake Wu [:bwu][:blakewu] from comment #34)
> (In reply to Paul Adenot (:padenot) from comment #32)
> > It would be a good idea to check:
> > - The number of frames requested by the AudioStream::DataCallback each time,
> > when the screen is on and off
> > - The state of the CircularBuffer in both cases. It might be under-runing or
> > something.
> > 
> > Both in AudioStream.cpp.
> Thanks for your feedback! 
> Normally, for the music case on B2G audio is playing via
> AudioOffloadPlayer[1] to save CPU resource.

It depends on the codec. I've had glitches with opus files as well, and those are not playing using AudioOffloadPlayer, afaik.

> However, in Z3C, AudioOffloadPlayer fails to get started @[2] and
> mAudioSink->Open()[3]. 
> From what I can tell so far, when creating Android audio track in
> AudioOutput::Open()@AudioOutput.cpp[4], some error happens in Android
> AudioTrack.cpp. 
> I am going to dig into.

Thanks, let me know if you need more info.
(In reply to Paul Adenot (:padenot) from comment #35)
> (In reply to Blake Wu [:bwu][:blakewu] from comment #34)
> > (In reply to Paul Adenot (:padenot) from comment #32)
> > > It would be a good idea to check:
> > > - The number of frames requested by the AudioStream::DataCallback each time,
> > > when the screen is on and off
> > > - The state of the CircularBuffer in both cases. It might be under-runing or
> > > something.
> > > 
> > > Both in AudioStream.cpp.
> > Thanks for your feedback! 
> > Normally, for the music case on B2G audio is playing via
> > AudioOffloadPlayer[1] to save CPU resource.
> 
> It depends on the codec. I've had glitches with opus files as well, and
> those are not playing using AudioOffloadPlayer, afaik.
Thanks for this info. So there are two problems, one is AudioOffloadPlayer cannot be used due to some failure in creating AudioTrack and the other is normal audio path does not work well either.
Set "qawanted" to know if this bug happens on Flame or not.
Keywords: qawanted
QA Contact: ktucker
I cannot reproduce this on Flame Master or Flame 2.2

No buzzing sounds or distortion was observed when playing a song while the device was locked. 

Environmental Variables:
Device: Flame 2.5 (Full Flash)(KK)(319mb)
Build ID: 20150817061040
Gaia: 60489c1ff8c5d1633fc4837d4f8019623d4e1940
Gecko: a6eeb28458fd2652e12e57334f046b7776d75bb4
Gonk: c4779d6da0f85894b1f78f0351b43f2949e8decd
Version: 43.0a1 (2.5)
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:43.0) Gecko/43.0 Firefox/43.0

Device: Flame 2.2 (Full Flash)(KK)(319mb)
Build ID: 20150817032503
Gaia: 335cd8e79c20f8d8e93a6efc9b97cc0ec17b5a46
Gecko: 82ec88fa015c
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 37.0 (2.2)
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0
QA Whiteboard: [QAnalyst-Triage+]
Flags: needinfo?(jmercado)
Keywords: qawanted
QA Whiteboard: [QAnalyst-Triage+]
Flags: needinfo?(jmercado)
Blocks: 1188643
See Also: → 1200111
I have created the bug 1200111 for the problem of AudioOffloadPlayer mentioned in comment 36. This bug will focus on the normal audio playback.
Priority: -- → P2
Dump PCM data in AudioStream.cpp
I tried to dump PCM data in AudioStream @ AudioStream::DataCallback(). The dumped PCM data sounds no problem, so it looks all data before going to AudioTrack is ok.
Thanks Blake. Do we have access to the modification Sony did to the file AudioTrack.cpp ?
(In reply to Paul Adenot (:padenot) from comment #42)
> Thanks Blake. Do we have access to the modification Sony did to the file
> AudioTrack.cpp ?
I had sent a email to Sony's contact window and he said they cannot share us AudioTrack.cpp.
Can we try on Lollipop ?
Flags: needinfo?(bwu)
(In reply to Paul Adenot (:padenot) from comment #44)
> Can we try on Lollipop ?
yeah. I think so. 
I am trying to get aries-L code and build it now. 
Set qawanted to have QA's help to test Aries-L image.
Flags: needinfo?(bwu)
Keywords: qawanted
One thing I noticed on a flight last week, this buzzing did _not_ occur when headphones were connected via bluetooth.  No physical wire connecting the two.
(In reply to Nick Desaulniers [:\n] from comment #46)
> One thing I noticed on a flight last week, this buzzing did _not_ occur when
> headphones were connected via bluetooth.  No physical wire connecting the
> two.

Indeed, see comment #0 :)
Yes, see the initial comment. I think there is some weirdness with some function signature in libmedia.so. Maybe we should do a quick readlelf to check what's up ?
(In reply to Blake Wu [:bwu][:blakewu] from comment #45)
> (In reply to Paul Adenot (:padenot) from comment #44)
> > Can we try on Lollipop ?
> yeah. I think so. 
> I am trying to get aries-L code and build it now. 
> Set qawanted to have QA's help to test Aries-L image.

Naoki, can you help us on getting an Aries L build? Or does any Aries L build work? Like, do we have a standard Aries L build that we could test for reference?
Flags: needinfo?(nhirata.bugzilla)
I have tried the Aries-L image and the music sounds *no* problem. But the decoding pipeline of Aries-L is a little different, MediaOmxReader(Aries-L) v.s. MediaFormatReader(Aries-KK). I will try to make Aries-kk play with MediaOmxReader tomorrow. If there is still no noise, then this problem is more likely related to gonk integration part, instead of gecko.
QAwanted request fulfilled at comment 50.
Flags: needinfo?(nhirata.bugzilla) → needinfo?(jmercado)
Keywords: qawanted
Flags: needinfo?(jmercado)
(In reply to Blake Wu [:bwu][:blakewu] from comment #50)
> I have tried the Aries-L image and the music sounds *no* problem. But the
> decoding pipeline of Aries-L is a little different, MediaOmxReader(Aries-L)
> v.s. MediaFormatReader(Aries-KK). I will try to make Aries-kk play with
> MediaOmxReader tomorrow. If there is still no noise, then this problem is
> more likely related to gonk integration part, instead of gecko.
It turns out even Aries-kk plays via MediaOmxReader, noise still happens.
Component: Audio/Video: Playback → GonkIntegration
Product: Core → Firefox OS
Blocks: 1168609
Our current theory is that Bug 1168609 will be resolved when this bug (re: GonkIntegration) is fixed.
Hi Blake -- This needs to be a P1 because it is blocking a P1 bug (see Bug 1168609).  Can you or someone on your team look at this this week?  I believe the FxOS drivers want to resolve all the P1 bugs this week.  Thanks!
Flags: needinfo?(bwu)
Priority: P2 → P1
So, this seem to be fixed with the last OTA update, which makes no sense. I'm not closing this bug until we know why and we have non-anecdotal evidences it is fixed (I've been listening to music on my phone all weekend without glitches, but that's only my experience).
(In reply to Paul Adenot (:padenot) from comment #55)
> So, this seem to be fixed with the last OTA update, which makes no sense.
> I'm not closing this bug until we know why and we have non-anecdotal
> evidences it is fixed (I've been listening to music on my phone all weekend
> without glitches, but that's only my experience).

I often find this gets 'fixed' by regressions where we peg the CPU (like the push-notifications regressions, and things like that) - maybe stick the phone in airplane mode to make sure? (though I suppose that's not necessarily a guarantee of working around such regressions)
Thanks for the heightened awareness on this everyone! Yes, as Maire mentions above, this is a p1 blocker that we need confirmed resolved asap. :)
(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #54)
> Hi Blake -- This needs to be a P1 because it is blocking a P1 bug (see Bug
> 1168609).  Can you or someone on your team look at this this week?  I
> believe the FxOS drivers want to resolve all the P1 bugs this week.  Thanks!
Current problem is this bug could be in Sony's library (libmedia.so or libaudioflinger.so) which we don't have source codes to debug. I have asked Sony's contact window (Alin) for source codes, he said it is not possible to share us the codes. So currently it is very difficult to debug this problem.
Flags: needinfo?(bwu)
(In reply to Paul Adenot (:padenot) from comment #55)
> So, this seem to be fixed with the last OTA update, which makes no sense.
> I'm not closing this bug until we know why and we have non-anecdotal
> evidences it is fixed (I've been listening to music on my phone all weekend
> without glitches, but that's only my experience).
After syncing the latest gecko and gaia codes, I still can repro this problem. It usually happens after around 30 seconds counting from the time I turn off the screen.
(In reply to Blake Wu [:bwu][:blakewu] from comment #59)
> (In reply to Paul Adenot (:padenot) from comment #55)
> > So, this seem to be fixed with the last OTA update, which makes no sense.
I didn't let my phone to do OTA update. I just synced the code base, built it and flash the image. Maybe some thing is updated via OTA which fixed this problem. We may check what is updated in OTA.
On my *dev* phone (not the foxfooding), I can reproduce if the lock screen is on. Without the lock screen I don't have any issue. (same music files, same everything)
You need to have the screen off.
Before doing anything, I'd appreciate if someone could try what I suggested in comment 48, to make sure we're calling the right function.

It might well be that we are calling a function with the wrong signature or the like.
I stand corrected with comment 61. It happens without the lock screen, but less often and not as loud.
(In reply to Paul Adenot (:padenot) from comment #63)
> Before doing anything, I'd appreciate if someone could try what I suggested
> in comment 48, to make sure we're calling the right function.
> 
> It might well be that we are calling a function with the wrong signature or
> the like.
I had checked Z3C's AudioTrack.h (Sony sent to me) and confirmed Sony didn't modify Android public interfaces but they added some interfaces and some member variables. I have sent it to you in mail. You may want to double check it. 
So what else should I check via readelf?
Flags: needinfo?(padenot)
BTW, I still can repro this problem on *FoxFooding* phone. Like the same situation I found on my debug phone, it usually happens around 30 seconds after locking screen and usually after that there would be no noise.
So, here is a plan of action:
- We need to know the following:
  - In which .so is libwhilelm, and whether we compile it or we use Sony's.
    - If we use Sony, we need to go higher up, that is, in cubeb, but since it works on all other devices, we can't do much.
    - If we compile our own, we might want to have a look at if we're calling the right functions from AudioTrack.h, since they changed a couple things.
  - The symbols exported by libmedia.so, to make sure we are exactly calling the right function and all. We know there is some weirdness with the memory in the function a function"set" of the AudioTrack class, but we haven't checked which version yet. We need to know that.
- Once we know that, we need to do the following:
  - Fix the memory trashing (or uninitialized read, I can't remember) in `set`, by calling the right function or whatever in opensl. We might want to talk to Julian Seward to have some help with valgrind. I know there is a bug about it, but I can't find it
  - Trace the initialization path for an AudioTrack and make sure every value goes where it should go. This involves reading part of libwilhelm and the android bridge to audio track.

My current theory is that the latency request we make to open the audio channel is not respected and that we're glitching because our callbacks are too small at some point, especially when the screen is being turned off and the CPU is down-scaled.
Flags: needinfo?(padenot)
Paul, 
Thanks for your feedback! Have some short update below and will further investigate. 
(In reply to Paul Adenot (:padenot) from comment #67)
> So, here is a plan of action:
> - We need to know the following:
>   - In which .so is libwhilelm, and whether we compile it or we use Sony's.
>     - If we use Sony, we need to go higher up, that is, in cubeb, but since
> it works on all other devices, we can't do much.
>     - If we compile our own, we might want to have a look at if we're
> calling the right functions from AudioTrack.h, since they changed a couple
> things.
We use our own. An easy way to find out is go to device/sony/shinano-common/extract-files.sh to check if this library is in the extracted list. If yes, that means Sony library will be used. 
>   - The symbols exported by libmedia.so, to make sure we are exactly calling
> the right function and all. We know there is some weirdness with the memory
> in the function a function"set" of the AudioTrack class, but we haven't
> checked which version yet. We need to know that.
> - Once we know that, we need to do the following:
>   - Fix the memory trashing (or uninitialized read, I can't remember) in
> `set`, by calling the right function or whatever in opensl. We might want to
> talk to Julian Seward to have some help with valgrind. I know there is a bug
> about it, but I can't find it
I think you were saying bug Bug 1184066.
See Also: → 1184066
(In reply to Blake Wu [:bwu][:blakewu](OoO from 10/19~10/21) from comment #68)
> >   - Fix the memory trashing (or uninitialized read, I can't remember) in
> > `set`, by calling the right function or whatever in opensl. We might want to
> > talk to Julian Seward to have some help with valgrind. I know there is a bug
> > about it, but I can't find it
> I think you were saying bug Bug 1184066.

To summarise bug 1184066, the impression there is that, in :kinetic's words

 libwilhelm (where the OpenSL ES to AudioTrack wrapper lives) and
 libmedia (where AudioTrack lives) disagree on the size of the AudioTrack class.

and my guess is that the size disagreement is 4 bytes.
Per discussions in bug 1184066 and this bug, we should use both libwilhelm and libmedia pulled from Sony to avoid this library mismatch problem. I tried to use those libraries from Sony Android image and encountered this same problem, music cannot be played, as bug 1184066 comment 13. If I only replace libmedia.so with the one from Android image, device cannot reboot. It seems this libmedia.so doesn't match our Android image in FxOS. So I am going to find the matched/correct libmedia first and then use its corresponding Sony's libwilhelm to see if this noise problem will be gone or not. 
If this noise problem still exists, we may need to contact Sony to check this bug together.
Set P2 per Sony Aries specific issue, also we can unblock this bug in v2.5 (Keep blocking:2.5+ just for tracking). Per comment 70, this bug probably requires Sony's library support for the solution. Blake will follow up this in further release.
Priority: P1 → P2
tracking-b2g: --- → +
After pushing the right version of libmedia and libwilhelm pulled from Sony Android's image, music cannot be played due to OnMediaSinkAudioError[1] which results from dlopen(libOpenSLES.so, RTLD_LAZY) failure[2].
Maybe we should change libOpenSLES.so as well. Continue to check it. 

[1]https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaDecoderStateMachine.cpp#2909
[2]https://dxr.mozilla.org/mozilla-central/source/media/libcubeb/src/cubeb_opensl.c#226
In triage, its decided to unblock for 2.5 and handle as part of backlog.
blocking-b2g: 2.5+ → ---
So basically having a functional Music application on Aries in 2.5 is not a priority. Good to know given the effort we put towards having the NGA Music app ready for 2.5....
If I were a consumer, I would have returned the phone over this issue.
(In reply to Hubert Figuiere [:hub] from comment #74)
> So basically having a functional Music application on Aries in 2.5 is not a
> priority. 

The discussion in triage was that this is dependent on work from Sony, so it does not make sense to block on this if we have no ability to fix it. If that is not the case and we have a fix for it, then obviously we will take it.
IMO, this is a very important bug for quality and I will continue checking this bug to see what else we can do here. AFAIK, sony stops maintaining kk, so it would not be possible to ask them to check this bug with us.
(In reply to Blake Wu [:bwu][:blakewu] from comment #72)
> After pushing the right version of libmedia and libwilhelm pulled from Sony
> Android's image, music cannot be played due to OnMediaSinkAudioError[1]
> which results from dlopen(libOpenSLES.so, RTLD_LAZY) failure[2].
> Maybe we should change libOpenSLES.so as well. Continue to check it. 
The dlopen error is "dlopen failed: could not load library "libwilhelm.so" needed by "libOpenSLES.so"; caused by cannot locate symbol "_ZN7android10DataSource16getCurrentOffsetEPx" referenced by "libwilhelm.so"..." 
_ZN7android10DataSource16getCurrentOffsetEPx(android::DataSource::getCurrentOffset(long long*)) is defined in Sony's libstagefright. It looks like Sony's libwilhelm needs that function but our libstagefright doesn't have it. If we want to use Sony's libwilhelm, we need to use their libstagefright as well...
Whiteboard: [spark] → [spark][POVB]
Whats the status here?
Flags: needinfo?(bwu)
So far, all evidences lead to that the root cause could be in Sony's library. I don't have ideas to move this bug forward. But this problem only happens on Aries KK version, not found on L and Flame, so I think we can wont-fix this problem on KK.
Flags: needinfo?(bwu)
(In reply to Blake Wu [:bwu][:blakewu] from comment #80)
> So far, all evidences lead to that the root cause could be in Sony's
> library. I don't have ideas to move this bug forward. But this problem only
> happens on Aries KK version, not found on L and Flame, so I think we can
> wont-fix this problem on KK.

Are we switching aries to L any time soon?
(In reply to Chris Lord [:cwiiis][PTO till Jan 4th] from comment #81)
> (In reply to Blake Wu [:bwu][:blakewu] from comment #80)
> > So far, all evidences lead to that the root cause could be in Sony's
> > library. I don't have ideas to move this bug forward. But this problem only
> > happens on Aries KK version, not found on L and Flame, so I think we can
> > wont-fix this problem on KK.
> 
> Are we switching aries to L any time soon?
It should happen on *2.6* I think.
(In reply to Blake Wu [:bwu][:blakewu] from comment #80)
> So far, all evidences lead to that the root cause could be in Sony's
> library. I don't have ideas to move this bug forward. But this problem only
> happens on Aries KK version, not found on L and Flame, so I think we can
> wont-fix this problem on KK.

It will not be possible to get L soon: Camera is still not stable, even though an experimental driver has been published by Sony. We need a solution for KK in the mean time. It took me just 10 minutes to push the libs people referrenced in this bug. So far, after pushing libwilhelm.so and libOpenSLES.so, we need libstagefright.so and then libdrmframework.so.

After that, Gecko fails to boot with missing symbol: dlopen failed: cannot locate symbol "_ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_" referenced by "libxul.so"...
That symbol is being defined by libgcastv2_support.so:
> $ objdump -tT backup-aries/system/lib/libgcastv2_support.so|grep --color _ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_
> 00010f7d g    DF .text	00000288 _ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_
Sadly, pushing everything that has this symbol to the system does not help.
So I guess we need to tell Gecko that the prototype is not the same:
> $ objdump -tT out/target/product/aries/system/lib/libstagefright.so|grep --color "_ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_"
> 0008cabd g    DF .text	0000024c _ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_
> $ objdump -tT backup-aries/system/lib/libstagefright.so|grep --color "_ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_"
> 000ca4f1 g    DF .text	00000348 _ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_S5_S5_
(In reply to Alexandre LISSY :gerard-majax from comment #86)
> So I guess we need to tell Gecko that the prototype is not the same:
> > $ objdump -tT out/target/product/aries/system/lib/libstagefright.so|grep --color "_ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_"
> > 0008cabd g    DF .text	0000024c _ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_
> > $ objdump -tT backup-aries/system/lib/libstagefright.so|grep --color "_ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_"
> > 000ca4f1 g    DF .text	00000348 _ZN7android17FindAVCDimensionsERKNS_2spINS_7ABufferEEEPiS5_S5_S5_S5_S5_

Changed and hacked but then I get more and more undefined symbols referenced from libxul that should be within libstagefright: getOutputGraphicBufferFromIndex, canOffloadStream, ...

I really don't know media playback code and I don't feel confident hacking libstagefright that much. We really have no way to hack that another way?

I mean from:
(In reply to Blake Wu [:bwu][:blakewu] from comment #43)
> (In reply to Paul Adenot (:padenot) from comment #42)
> > Thanks Blake. Do we have access to the modification Sony did to the file
> > AudioTrack.cpp ?
> I had sent a email to Sony's contact window and he said they cannot share us
> AudioTrack.cpp.

And:
(In reply to Julian Seward [:jseward] from comment #69)
> (In reply to Blake Wu [:bwu][:blakewu](OoO from 10/19~10/21) from comment
> #68)
> > >   - Fix the memory trashing (or uninitialized read, I can't remember) in
> > > `set`, by calling the right function or whatever in opensl. We might want to
> > > talk to Julian Seward to have some help with valgrind. I know there is a bug
> > > about it, but I can't find it
> > I think you were saying bug Bug 1184066.
> 
> To summarise bug 1184066, the impression there is that, in :kinetic's words
> 
>  libwilhelm (where the OpenSL ES to AudioTrack wrapper lives) and
>  libmedia (where AudioTrack lives) disagree on the size of the AudioTrack
> class.
> 
> and my guess is that the size disagreement is 4 bytes.

Cannot we infer where there is a discrepency ? A size diffrence of 4 bytes should be something we can hack, right?

I really don't know much about media codebase, but don't we have some header for AudioTrack that is used?
Alexandre, 
Thanks for your help try to move this bug forward! 
Recently I am thinking and trying to find an ultimate way to fix this kind of bugs. When porting FxOS to non-nexus devices, we extract and use some of phone-maker's libraries which are changed by phone-maker. Sometimes that would make us very difficult fix some problems, like this bug and camera bug, related to those proprietary libraries. I am thinking instead of hacking their libraries maybe we could replace those phone-maker's libraries with our libraries and if needed, we may need to change the libraries in HAL(on QC platform, we can use the HAL on CodeAurora). In this way, that will give us more control on the codes and whole SW stack. I am going to try it on media part first. 
How do you think?
I don't really care how we fix it, but so far I fear there is too much differences (or at least for me) with libstagefright and how it's used in Gecko. Hence why I suggested that if the problem is the size of the AudioTrack class, maybe we can teach Gecko some trick ....

I used that in the past when hacking a Keon build to run on HTC Desire Z device (ICS base) regarding GPS: there was a difference in some gps related class size that was hitting Gecko (there is a check on this). Hacking members I could find a way.
The condition was there: https://dxr.mozilla.org/mozilla-central/source/dom/system/gonk/GonkGPSGeolocationProvider.cpp#456

The class being used on my HTC Desire Z was bigger than the one defined in Keon's headers. So I hacked by adding some padding or stuff like that :). This resulted in working GPS interface :)
(In reply to Blake Wu [:bwu][:blakewu] from comment #88)
> Alexandre, 
> Thanks for your help try to move this bug forward! 
> Recently I am thinking and trying to find an ultimate way to fix this kind
> of bugs. When porting FxOS to non-nexus devices, we extract and use some of
> phone-maker's libraries which are changed by phone-maker. Sometimes that
> would make us very difficult fix some problems, like this bug and camera
> bug, related to those proprietary libraries. I am thinking instead of
> hacking their libraries maybe we could replace those phone-maker's libraries
> with our libraries and if needed, we may need to change the libraries in
> HAL(on QC platform, we can use the HAL on CodeAurora). In this way, that
> will give us more control on the codes and whole SW stack. I am going to try
> it on media part first. 
> How do you think?

This is what we tried by pushing built libmedia.so and getting no sound at all, right? Well maybe pushing libmedia.so with hacked AudioTrack class can help too?
On a Kitkat Z3c tree:
> $ ls -hal ./frameworks/av/include/media/AudioTrack.h ./gecko/dom/media/AudioTrack.h ./gecko/dom/system/gonk/android_audio/AudioTrack.h
> -rw-rw-r-- 1 alex alex  38K févr. 12  2015 ./frameworks/av/include/media/AudioTrack.h
> -rw-rw-r-- 1 alex alex 1,1K avril 24  2015 ./gecko/dom/media/AudioTrack.h
> -rw-rw-r-- 1 alex alex  19K avril  3  2015 ./gecko/dom/system/gonk/android_audio/AudioTrack.h

Which one is the one we care about? It looks like dom/media is a false positive, but the two others seems quite competitive.
Primivite hacking would suggest we don't really care about dom/system/gonk/android_audio/AudioTrack.h and that only frameworks/av/include/media/AudioTrack.h is really important.
(In reply to Alexandre LISSY :gerard-majax from comment #91)
> 
> This is what we tried by pushing built libmedia.so and getting no sound at
> all, right? Well maybe pushing libmedia.so with hacked AudioTrack class can
Yeah. per comment 72. 
> help too?
Maybe, I have not tried that. Another question is how to hack AudioTrack. 

(In reply to Alexandre LISSY :gerard-majax from comment #92)
> On a Kitkat Z3c tree:
> > $ ls -hal ./frameworks/av/include/media/AudioTrack.h ./gecko/dom/media/AudioTrack.h ./gecko/dom/system/gonk/android_audio/AudioTrack.h
> > -rw-rw-r-- 1 alex alex  38K févr. 12  2015 ./frameworks/av/include/media/AudioTrack.h
> > -rw-rw-r-- 1 alex alex 1,1K avril 24  2015 ./gecko/dom/media/AudioTrack.h
> > -rw-rw-r-- 1 alex alex  19K avril  3  2015 ./gecko/dom/system/gonk/android_audio/AudioTrack.h
> 
> Which one is the one we care about? It looks like dom/media is a false
> positive, but the two others seems quite competitive.
/frameworks/av/include/media/AudioTrack.h is used in libmedia.so.
I think I fixed the offloading problem in bug 1200111, by fixing the offloading path there's no more glitches while playing the audio tracks supporting it, like MP3, but if I play .ogg files (which doesn't support offloading), glitches still show up :(
This is awesome!

How about AAC? (.m4a)

Would it be hard to support offloading for Ogg/Vorbis?
Flags: needinfo?(jgomez)
(In reply to Juan Gomez [:_AtilA_] (CET/CEST) from comment #95)
> I think I fixed the offloading problem in bug 1200111, by fixing the
> offloading path there's no more glitches while playing the audio tracks
> supporting it, like MP3, but if I play .ogg files (which doesn't support
> offloading), glitches still show up :(
That's because ogg is decoded and played by CPU and offloading (audio DSP) doesn't support ogg. 
So once CPU is busy, it may not be able to decode music data in time.
It's not a CPU issue, it's a AudioTrack issue. We've verified the data is decoded in time just fine when it glitches.
Hello everyone,
I was at the Firefox OS hackathon saturday and Juan has applied a patch concerning this bug on my Z3C phone. Maybe you know which one Alexandre?

Now, my phone do absolutely no sound whatever the application is launched. I can't hear when someone call me or the music, alarm clock etc... This seems logical if the patch is for Gecko.  Strangely, I have sound when I use headphones... But I don't know about what the patch did so I'm not sure this is the fault of the patch. I hadn't this problem before, but I haven't anymore the intermittent beeps/buzzing noises! :D

Tonight I was at Paris Mozilla office, but there were no developpers and the contract with Mozilla don't allowed me to flash the phone by myself with the previous build. Do you know what can I do?

Thank you so much for reading this.
Philippe, this code is free so we can send you another .so :)
Flags: needinfo?(philippe.joulot)
Hubert, Audio is not my filed (I didn't even know what offloading was before this) :) so I cannot tell you how hard is it to implement that. Maybe Paul can answer that :)
Flags: needinfo?(jgomez) → needinfo?(ptheriault)
Sorry, not that Paul :)
Flags: needinfo?(ptheriault) → needinfo?(padenot)
(In reply to Paul Adenot (:padenot) from comment #98)
> It's not a CPU issue, it's a AudioTrack issue. We've verified the data is
> decoded in time just fine when it glitches.
Yeah. That's the problem of library mismatch. I were trying to say after that problem is fixed, if we still use CPU to decode and play music data without DSP assistance (offload mode), we may hit another problem described in comment 95 (ogg files) that playback could be not continuously played since CPU is busy in doing other things.
I don't know, and the right approach is to make the CPU decoder work anyways, we can figure out offloading for AAC later.
Flags: needinfo?(padenot)
(In reply to Hubert Figuiere [:hub] from comment #96)
> This is awesome!
> 
> How about AAC? (.m4a)
> 
> Would it be hard to support offloading for Ogg/Vorbis?
From my understanding, offloading requires a dedicated DSP to offload CPU loading. That DSP is usually provided by HW platform. Therefore, supporting Ogg/Vorbis offloading depends on the capability of that DSP.
(In reply to Alexandre LISSY :gerard-majax from comment #100)
> Philippe, this code is free so we can send you another .so :)

I flashed my phone with the last Foxfooding build so my headphones work again (I can call people! :) ) But now the bug of the sound is still here. When I was testing the patch, the phone was frozen twice and I could not launch any apps. The only solution was to shutdown the phone.
Flags: needinfo?(philippe.joulot)
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: