836638 - Alarm can't go off (clock doesn't start up) under "process buster" stress

Status: VERIFIED FIXED

(Core :: General, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

STR
 (1) Open Clock application
 (2) Set an alarm for 10 minutes or so in the future
 (3) Load http://people.mozilla.com/~cjones/membuster.com
 (4) Choose "Bust processes"
 (5) Wait for tabs to stabilize and load example.com.  Usually takes 30s or so.
 (6) Put phone to sleep

The alarm never goes off.  Unlocking the phone shows a "ghost" Clock app entry in the task switcher.  This indicates b2g tried to launch Clock but it crashed while being initialized.

Comment 1

[Justin Lebar (not reading bugmail)]

This is essentially the same as bug 836199.

There are essentially two problems here, and we probably need to fix both.

The one is that the browser app creates ten foreground processes.  I'm trying to fix that, and it's hard.  We may be hitting APZC bugs or something; I'm not sure yet.  The numerous fg processes make it hard for the alarm app to compete for CPU cycles and make it more likely to be killed.

The other is that the clock app launches with regular fg permission.  We cannot guarantee that the LMK won't kill the alarm app unless it's running with higher permission than the other fg apps.  I'm thinking that we need to elevate the permission of the alarm/communications/etc. app when it is sent a "high-priority" system message (potentially right from the start when it starts up) and drop it down only after it has handled said message.

I wish we weren't hitting this stuff at the eleventh hour.  :(

Comment 2

[Gene Lian (I already quit Mozilla)]

(In reply to Justin Lebar [:jlebar] from comment #1)
> The other is that the clock app launches with regular fg permission.  We
> cannot guarantee that the LMK won't kill the alarm app unless it's running
> with higher permission than the other fg apps.  I'm thinking that we need to
> elevate the permission of the alarm/communications/etc. app when it is sent
> a "high-priority" system message (potentially right from the start when it
> starts up) and drop it down only after it has handled said message.
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For the alarm, we used to let the AlarmService.jsm listen to the "SystemMessageManager:HandleMessageDone" IPC message to know when to release the CPU wake lock, which might be the same time point to drop down the priority.

For the phone call, I used to introduce the similar mechanism at [1] but was rejected in the long run, which might be useful here. Another alternative place could be RadioInterfaceLayer.handleCallDisconnected() (when the phone call ends) but it doesn't accurately indicate the time point of finishing handling the system message.

[1] attachment #705699 [details] [diff] [review] and attachment #705726 [details] [diff] [review]

Comment 3

[Justin Lebar (not reading bugmail)]

> we used to let the AlarmService.jsm listen to the "SystemMessageManager:HandleMessageDone" IPC 
> message to know when to release the CPU wake lock, which might be the same time point to drop down 
> the priority.

Yes, that's exactly my thinking.

Comment 4

[Justin Lebar (not reading bugmail)]

Bug 836654 may be all we need here.

Comment 5

[Justin Lebar (not reading bugmail)]

Also, in this situation, I think the alarm service will permanently hold on to the CPU wake lock.

We take the CPU wake lock before we send the alarm system message, and we don't release it until we get SystemMessageManager:HandleMessageDone.  But that message never comes if the alarm process crashes.

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

Andrew, should we + 836654 as well then?

Justin, you're looking at the other bug (836199) this depends on - are you looking at 836654 and this one as well?

Comment 7

[Justin Lebar (not reading bugmail)]

I'm looking at all these bugs.

(OT, but it would be really helpful if you'd say "bug XXX" so that bugs are linkified and so we can hover over the links to get the bug titles.)

Comment 8

[Justin Lebar (not reading bugmail)]

> I'm looking at all these bugs.

... but I could definitely use help with the Gaia stuff.

Comment 9

[Lawrence Mandel [:lmandel] (use needinfo)]

Justin, assigning to you as you're looking at this. We'll try to find some help if you need it.

Clearing needinfo on Andrew as he's +ed bug 836654.

Comment 10

[David Scravaglieri [:scravag]]

Justin, any news on this ?

Comment 11

[Justin Lebar (not reading bugmail)]

Please see the dependent bugs.

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

We'll leave this on the blockers for now, but can qa verify if they have seen this in the last 3 weeks or so?

Comment 13

[Justin Lebar (not reading bugmail)]

I can reproduce this easily.

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Thanks jlebar; removing qawanted tag.

Comment 15

[Justin Lebar (not reading bugmail)]

All of the dependent bugs are fixed here (well, aside from bug 836199, but that doesn't count).  All that remains to be done is to verify that this bug is in fact fixed.

Comment 16

[Angela Hubenya]

The alarms go through, after the "Bust processes" was selected and fully loaded. 
Tested on the nightly build: 20130225070200, as well as the master build: 20130225030609. Changing status to WFM

Comment 17

[Justin Lebar (not reading bugmail)]

Thanks for testing!

(We usually use WFM when we're not sure what fixed a bug.  But here, it's clear where the fixes came from.)

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

Resolving branches per comment 16, 17

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

removing qawanted.

Comment 20

[Nicholas Nethercote [inactive]]

>  (3) Load http://people.mozilla.com/~cjones/membuster.com

FWIW, that should be http://people.mozilla.com/~cjones/membuster.html.

Comment 21

[Justin Lebar (not reading bugmail)]

Attachment: membuster testcase

Here's the membuster testcase.

Comment 22

[Justin Lebar (not reading bugmail)]

And for ease of typing on a phone, I made http://bit.ly/membuster.

Comment 23

[Angela Hubenya]

The alarm went off as it is supposed to, after the "bust process" was ran. 
Issue is Verified as fixed in the following branches-- 

Unagi Build ID: 20130408075816
Kernel Date: Dec 5
Gecko: http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/3b51ced33215
Gaia: 8cf8dbd430af5e72a2ad542824421debe6de29d4


Unagi Build ID: 20130408075751
Kernel Date: Dec 5
Gecko: http://hg.mozilla.org/releases/mozilla-b2g18/rev/24b27125d907
Gaia: b973e66f328d8e09e364654e17ddab9f2c938bc4