1105511 - mozSettings API stuck in a state where all get() and set() calls never returns

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Using master (v2.2), sometimes the keyboard just stops showing up.  This generally happens after using the phone for a few days.  I've seen it a number of times, and so has kgrandon.  I think khuey did as well before he gave up on dogfooding.

In particular, the symptoms are that at some point I focus a textfield, and while I see a caret blinking in that textfield, the keyboard doesn't show up.  This symptom happens with all textfields across all apps, and I need to reboot the phone to get a keyboard again.

My phone is currently in this state if there's anything you want me to debug in the next few hours.

Comment 1

[Kevin Grandon :kgrandon]

"Me too", so adding this to my bug for tracking. Unfortunately I'm not too sure what would help us debug this.

Adding Tim/Rudy to the bug. Are you guys aware of this issue? What would be helpful for debugging this?

Comment 2

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Is the mozSettings API still working? You could verify that by toggle a setting in Settings > Keyboards > Built-in Keyboards, or connect with console of any app.

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Hmmm.  Settings seem somewhat broken.  The date and time settings have "set automatically" unchecked and everything else black and show my setting as 12-hour, even though I'm actually on 24-hour time.  But it is working well enough to toggle homescreen layout from four columns to three columns (and back).

Going to Settings -> Keyboards shows me no options under the "Keyboard Settings" header and only a "Select Keyboards" menu under the "Selected Keyboards" header.  When I open that "Select Keyboards" submenu, it is blank.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(Should this remind me of bug 1081780 / bug 1089619?)

Comment 5

[Tim Guan-tin Chien [:timdream] (please needinfo)]

It's a mozSettings API issue then. So the Keyboard activation/deactivation is actually chained in a promise queue involving setting a settings key. If the DOMRequest never resolve we will be blocked.

This bug is essentially what I sent out on dev-b2g last week. 

I can fix this bug in Keyboard app by moving that set() call out of the queue but it would not be a real fix -- or we could throw this bug to DOM and find out what's wrong with it.

Comment 6

[Rudy Lu [:rudyl] (inactive after 2015/7/3)]

We were hit by a similar issue in bug 1093121 and it turned out caused by bug 1092941, a mozSettings bug.
However, from bug 1092941 comment 30, it seems Julien was hit by the settings issue again recently.

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Other symptoms of the current state of my phone:
 * can't place phone calls
 * can receive phone calls, but it doesn't say who's calling (neither number nor contact)

Comment 8

[:gerard-majax]

[Blocking Requested - why for this release]: reported on v2.1 by bug 1092941 comment 30

Comment 9

[:gerard-majax]

How long does it takes for you to trigger the bug?

Comment 10

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I updated my phone at 16:10 on Tuesday.

I filed this bug at 14:17 on Wednesday -- though I think I first noticed the keyboard wasn't working while I as getting lunch at about 12:30.

So it took about 20-22 hours after booting the phone.

However, I've certainly run much longer than that without seeing it.

I'd been using the GPS and some other things more intensively than usual in the past 24 hours, though.

Comment 11

[Kevin Grandon :kgrandon]

Yeah, reproducing is not on the order of minutes for me. Sometimes it takes a day, sometimes a week.

Comment 12

[:gerard-majax]

Attachment: Nexus S logcat, after 4+ days

Not sure if those are related, but I could see it on my Nexus S that is booted for more than 4 days. I have no symptom of the issue so far.

Comment 13

[:gerard-majax]

Attachment: Improve debug and error reporting in mozSettings

Comment 14

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: logcat of errors that happen a few seconds after starting settings app

Nothing at all in logcat when I tap in a text input, but I do get this when I start the settings app (requested by gerard-majax on IRC).

Comment 15

[:gerard-majax]

Looking at about:memory on the device, I see some settings observer leaking:
> 456 (100.0%) -- settings-observers-suspect
> ├──242 (53.07%) ── referent(topic=lockscreen.lock-message)
> └──214 (46.93%) ── referent(topic=powersave.threshold)

The |lockscreen.lock-message| one leaks on each unlocking of the device.

Comment 16

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

FWIW, while seeing the bug, I currently see:

108 (100.0%) -- settings-observers-suspect
├───68 (62.96%) ── referent(topic=powersave.threshold)
└───40 (37.04%) ── referent(topic=lockscreen.lock-message)

Given that yours is worse, sounds like a leak worth filing, but seems more likely to be unrelated.

Comment 17

[:gerard-majax]

Attachment: Improve debug and error reporting in mozSettings

Comment 18

[:gerard-majax]

Isn't it what you reported yesterday/this week?

Comment 19

[Benjamin Bouvier [:bbouvier] (inactive)]

That happened to me a few times when I was using master. Now that I am on 2.1, I don't experience this issue anymore. Note that I need to reboot my phone a few times a day anyways, because of another issue after which even the adb connection doesn't work and thus I can't get a logcat.

Comment 20

[Benjamin Bouvier [:bbouvier] (inactive)]

Reading more thoroughly, I also get the symptoms as described in comment 7. Moreover, whenever this happens, I can't change any setting (by the status bar nor the settings app), e.g. switching wifi off/on, changing volume, etc..

Comment 21

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Benjamin Bouvier [:bbouvier] from comment #20)
> Reading more thoroughly, I also get the symptoms as described in comment 7.
> Moreover, whenever this happens, I can't change any setting (by the status
> bar nor the settings app), e.g. switching wifi off/on, changing volume, etc..

I've seen this before. Other than the symptoms, I also found the CPU usage is very high when this happens. That was recorded here https://bugzilla.mozilla.org/show_bug.cgi?id=1064695

Comment 22

[:gerard-majax]

Attachment: Improve debug and error reporting in mozSettings

Comment 23

[Nicolas Silva [:nical]]

(In reply to Alexandre LISSY :gerard-majax from comment #18)
> Isn't it what you reported yesterday/this week?

Yes. On an up to date 2.1 flame.

Comment 24

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Kan-Ru Chen [:kanru] from comment #21)
> I've seen this before. Other than the symptoms, I also found the CPU usage
> is very high when this happens. That was recorded here
> https://bugzilla.mozilla.org/show_bug.cgi?id=1064695

I used to see high CPU usage when I saw things like this, but I don't yesterday/today.  (Maybe bug 1089619 fixed that part?)

Comment 25

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So I started poking around in the gc and cc logs generated by get_about_memory.py.  The GC log is more interesting, I think.

In the parent process, there are three SettingsManager objects, each associated with a different window.  One has no entries in _locks, one has 3, and one has 159 (although there are 164 objects that have it as a _settingsManager).  (That's in the first of the about:memory dumps I took.)

There's one object in this process with a settingsLockQueue (from SettingsRequestManager.jsm).  The queue appears not to be being processed:  I took two dumps over time.  The first had 410 elements, the second had 456 elements, of which the initial 410 were the same.

The first item in that settingsLockQueue is an item for a lock with ID 60915c7c-6567-4c73-ad11-3256de5f036a, which seems to correspond to this SettingsLockInfo object:

0xacc75e80 G Object <no private>
> 0xb0097540 G type
> 0xb17aeb38 G shape
> 0xb00b1010 B lockID
> 0xab4ed9c0 G tasks
> 0xaafcec10 G queuedSets
> 0xb17af700 G _transaction
> 0xaf2d8760 B _mm
> 0xb000a980 G getObjectStore

That tasks points to an empty array.

_transaction is:

0xb17af700 G IDBTransaction <no private>
> 0xaf2d7520 G type
> 0xab4df208 G shape

I'm not sure what else might be interesting here.

Comment 26

[Julien Wajsberg [:julienw]]

(In reply to Alexandre LISSY :gerard-majax from comment #18)
> Isn't it what you reported yesterday/this week?

yep.

A very easy way to find if your phone is in this state is launching the Settings app and see if the "airplane mode" icon stays disabled. If it stays disabled, your mozSettings is broken. Another clue is that at the end of the Settings first panel you have the title "Operator Services" but nothing beneath.

Comment 27

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to David Baron [:dbaron] (UTC-8) (needinfo? for questions) from comment #25)
> 0xb17af700 G IDBTransaction <no private>
> > 0xaf2d7520 G type
> > 0xab4df208 G shape

Oh, but if I then jump back to the CC log:

0xb17af700 [gc] JS Object (IDBTransaction)
> 0xb2a485b0 type_proto
> 0xb2a471f0 parent
> 0xa9ce5e00 UnwrapDOMObject(obj)

the last leads to:

0xa9ce5e00 [rc=2] DOMEventTargetHelper 
> 0xb2a471f0 mScriptOwner
> 0xab121520 mListenerManager
> 0xafa66870 mDatabase
> 0xa9ca78c0 mObjectStores[i]

which is also pointed to by:

0xa9ca78c0 [rc=1] IDBObjectStore
> 0xa9ce5e00 mTransaction


So I could probably poke at these objects in gdb if there's something interesting to look at.  (Need to go to Thanksgiving dinner now, though.)

Comment 28

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Updating the summary. If people still need to Keyboard app to unchain the call (as comment 5 states) please clone a bug and needinfo me.

Comment 29

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: some poking at the IDBTransaction and related structures in gdb

Comment 30

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

One possibility is:
[2014-11-27 23:45:05 -0800] <khuey> dbaron: it looks a little suspicious that the "cannot get object store" branch doesn't removeLock
[2014-11-27 23:45:20 -0800] <khuey> although that seems unlikely to be the issue here
[2014-11-27 23:48:29 -0800] <khuey> dbaron: although, IDBTransaction.objectStore("foo") will fail if the transaction is finished

>    let store = lock.getObjectStore(aTask.principal);
>    if (!store) {
>      if (DEBUG) debug("Rejecting Set task on lock " + aTask.data.lockID);
>      return Promise.reject({task: aTask, error: "Cannot get object store"});
>    }

(That seems like one way that a finalize might not lead to the next task being processed.  It's also possible the problem was no finalize at all...)

Comment 31

[Flaburgan]

I have most of the symptoms on a Flame running up-to-date 2.1:
* Can receive calls but without seeing the contact
* Cannot switch on of WiFi, 3G, Airplane mode nor silence mode, either by using the notification panel,  the setting app or the power menu.
* The hour becomes wrong, but maybe not related.

I don't know what to do to reproduce, but I have this problem daily, sometime twice or third per day. I do not do something specific with the phone (no data connection or GPS enabled, it's just next to me at work...)

Comment 32

[:gerard-majax]

So this is indeed affecting a lot of people dogfooding. I know we are focusing on Flame for this purpose, but it's a bit strange that I absolutely never met this issue on my dogfooding Nexus S device.

Considering David's comment 30, I'd like to land the attached patch after adding some way to be able to detect when the queue does not get processed.

Comment 33

[:gerard-majax]

Attachment: Improve debug and error reporting in mozSettings

Comment 34

[:gerard-majax]

Given current feedback about the non processed queue, I've added some memory reporting stuff to have a better view of this. I hope it will be able to confirm what david saw.

Comment 35

[Peter Bylenga [:PBylenga]]

bug 1093932 looks related to this one, should we Resolve duplicate to consolidate?

Comment 36

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I just hit this bug again, and I know what I was doing when it started.

I confirmed in the gc log that the settingsLockQueue is huge, so it is this bug.

What happened was the following:
 * I tapped in the rocketbar and typed "Lincoln second inaugural" (or similar)
 * I hit enter
 * the keyboard went away, but no search results showed up
 * then the keyboard no longer came up (the settings lock queue was stuck)

At the time I hit it, I had the patch to bug 1106324 applied, so that is NOT the problem.

Comment 37

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Ok, looking at the logcats, it looks like some accept/reject callback is throwing an exception and we're not handling it, which means finalize never gets called on something, which means we get a jammed lock queue. The problem is going to be figuring out where/why that's happening. I thought I'd covered wrapping most of what could throw in SettingsRequestManager but that may not be the case. Is there a way to add instrumentation to promises to get more information on where something is throwing?

Comment 38

[:gerard-majax]

We have bug 1106896 that is going to land very soon, this should allow us to have a better feedback and less noisy output on error cases for Settings.

Comment 39

[:gerard-majax]

(In reply to David Baron [:dbaron] (UTC-8) (needinfo? for questions) from comment #36)
> I just hit this bug again, and I know what I was doing when it started.
> 
> I confirmed in the gc log that the settingsLockQueue is huge, so it is this
> bug.
> 
> What happened was the following:
>  * I tapped in the rocketbar and typed "Lincoln second inaugural" (or
> similar)
>  * I hit enter
>  * the keyboard went away, but no search results showed up
>  * then the keyboard no longer came up (the settings lock queue was stuck)
> 
> At the time I hit it, I had the patch to bug 1106324 applied, so that is NOT
> the problem.

So maybe those symptoms are coming from several bugs indeed? Because I clearly tested that when there was an error at getObjectStore() level, without bug 1106324, then the queue would get blocked.

Comment 40

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Alexandre LISSY :gerard-majax from comment #39)
> So maybe those symptoms are coming from several bugs indeed? Because I
> clearly tested that when there was an error at getObjectStore() level,
> without bug 1106324, then the queue would get blocked.

I don't see any reason to think that there are different bugs; I think the queue was just getting stuck in a different way.  (I confirmed from the GC log that the queue was stuck.)

Comment 41

[:gerard-majax]

(In reply to David Baron [:dbaron] (UTC-8) (needinfo? for questions) from comment #40)
> (In reply to Alexandre LISSY :gerard-majax from comment #39)
> > So maybe those symptoms are coming from several bugs indeed? Because I
> > clearly tested that when there was an error at getObjectStore() level,
> > without bug 1106324, then the queue would get blocked.
> 
> I don't see any reason to think that there are different bugs; I think the
> queue was just getting stuck in a different way.  (I confirmed from the GC
> log that the queue was stuck.)

Yes, that was my point, sorry if it was not clear. Yesterday, I saw the issue reproduced twice in 10 mins on gregor's flame.

Comment 42

[:gerard-majax]

So, yesterday, Gregor triggered the issue. We captured logcat, and I could see nothing useful yet. However, I spotted something that may be linked to the "No such process" things we talked about:

> 12-04 21:05:47.874   210   534 I Gecko   : [Parent 210] WARNING: pipe error (157): Connection reset by peer: file ../../../gecko/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 457

Comment 43

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

I'm really convinced something isn't getting cleared after an app dies. It's just a question of what app, when, and why. Will work on repro'ing and getting bug

Comment 44

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Oops hit enter too fast. Getting bug 1107329 started.

Comment 45

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

The repro in bug 1103178 gets me a VERY reliable repro to this. If I smack the "set" button a whole bunch while setting wallpaper, the settings queue jams. Have watched it happen with debug messages on. Diagnosing now.

Comment 46

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Ok, as :julienw said in https://bugzilla.mozilla.org/show_bug.cgi?id=1092941#c34, it looks like bug 1092941 is our culprit. It tries to use principals from a structure that only caches some principals, not all, so we get a silent failure checking a null principal for permissions, which is what causes the lock queue to jam. I'm trying to figure out a good fix for this right now, but we've at least got a repro and a cause.

Comment 47

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Attachment: Patch 1 (v1) - Check validity of principals before doing permissions check

Comment 48

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Attachment: Patch 2 (v1) - Store principals with locks instead of with tasks

I cannot remember why I passed principals with tasks instead of storing them as lock members on lock creation. As far as I can tell, there's no reason that we shouldn't be able to store principals with locks, which means we don't have to worry about pulling from a cache after child process death. The locks can just continue to finalize using information they already have. This seems to fix the issues we were seeing on my phone.

Comment 49

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Attachment: Patch 3 (v1) - Make observer principal cache name reflect content

mmPrincipals was a pretty crappy name for something that didn't actually cache ALL of the principals for all message managers. Renamed it to something more obvious and put in a comment explaining what it does.

Comment 50

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Doing a try run before starting on reviews.

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=6e21f9360154

Comment 51

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

https://tbpl.mozilla.org/?tree=Try&rev=6e21f9360154

Comment 52

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Try seems happy, so submitting for review.

Comment 53

[Julien Wajsberg [:julienw]]

Don't forget to ask for approval on v2.1 once it's resolved.

Comment 54

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Did some research, it looks like the idea to start using principals came from me as part of bug 1061510. Before Bug 1061510, we were caching message managers in locks, which caused issues due to how we were checking permissions against them. I changed it to using principals, but also attached the principals to tasks instead of to locks, most likely because principals are transmitted with messages and in settings each incoming message is considered a task, so I figured I would just store a principal per message/task, which ended up not working here because we now have situations where we need to create new tasks for a lock with no incoming message.

Given this, the change from storing principals in tasks to storing them with their respective locks seems sane.

Comment 55

[Tony Chung [:tchung]]

triage blocking:2.1+

Comment 57

[Tim Guan-tin Chien [:timdream] (please needinfo)]

\o/

Comment 58

[:gerard-majax]

(In reply to Kyle Machulis [:kmachulis] [:qdot] (USE NEEDINFO?) from comment #45)
> The repro in bug 1103178 gets me a VERY reliable repro to this. If I smack
> the "set" button a whole bunch while setting wallpaper, the settings queue
> jams. Have watched it happen with debug messages on. Diagnosing now.

I can confirm the easy STR.

Comment 59

[:gerard-majax]

That does indeed fix the issue reproduced when tapping very fast on the "Set" button, and I have good reasons to believe this is also how gregor was reproducing this. Let's run try to make sure we don't break anything :)

Comment 60

[:gerard-majax]

This try must be green: https://tbpl.mozilla.org/?tree=Try&rev=d9f5d56de138

Specifically, we want to make sure we won't break again Mnw tests, like in bug 1092941.

Comment 61

[:gerard-majax]

The failures on Gij 2 seems to be unrelated and also visible on several Gaia-Try PR.

Comment 62

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

https://hg.mozilla.org/integration/b2g-inbound/rev/56266e35a53f
https://hg.mozilla.org/integration/b2g-inbound/rev/0b948e1e1891
https://hg.mozilla.org/integration/b2g-inbound/rev/d5fde9d729e4

Comment 63

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Comment on attachment 8532696 [details] [diff] [review]
Patch 1 (v1) - Check validity of principals before doing permissions check

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: 
Testing completed: 
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch:

Comment 64

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Sorry, hit enter in the wrong place when doing approval flagging.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Bug 1061510 and Bug 1092941
User impact if declined: Phone will lock up randomly due to race condition
Testing completed: Multiple try runs, manual device testing for repros
Risk to taking this patch (and alternatives if risky): Settings affects everything. So considerably risky, but downsides are worse.
String or UUID changes made by this patch: None

Comment 65

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Comment on attachment 8532699 [details] [diff] [review]
Patch 2 (v1) - Store principals with locks instead of with tasks


[Approval Request Comment]
Bug caused by (feature/regressing bug #): Bug 1061510 and Bug 1092941
User impact if declined: Phone will lock up randomly due to race condition
Testing completed: Multiple try runs, manual device testing for repros
Risk to taking this patch (and alternatives if risky): Settings affects everything. So considerably risky, but downsides are worse.
String or UUID changes made by this patch: None

Comment 66

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

Comment on attachment 8532704 [details] [diff] [review]
Patch 3 (v1) - Make observer principal cache name reflect content

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: 
Testing completed: 
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch:

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Bug 1061510 and Bug 1092941
User impact if declined: Phone will lock up randomly due to race condition
Testing completed: Multiple try runs, manual device testing for repros
Risk to taking this patch (and alternatives if risky): Settings affects everything. So considerably risky, but downsides are worse.
String or UUID changes made by this patch: None

Comment 67

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/56266e35a53f
https://hg.mozilla.org/mozilla-central/rev/0b948e1e1891
https://hg.mozilla.org/mozilla-central/rev/d5fde9d729e4

Comment 68

[bhavana bajaj [:bajaj]]

Ive' requested QA verification on 1103178/1093932 on trunk to ensure those are fixed before uplifting this. Kyle given the risk here, if you can highlight some testcases that may hit the code path you are modifying I can request QA to help with additional targeted testing here.

Comment 69

[:gerard-majax]

Attachment: [v2.1] Patch 3 (v1) - Make observer principal cache name reflect content

Comment 70

[:gerard-majax]

Comment on attachment 8534875 [details] [diff] [review]
[v2.1] Patch 3 (v1) - Make observer principal cache name reflect content

Patch 3 needed a small change (conflict around the end of the removeObserver() method) to make it applied on v2.1 branch.

Comment 71

[:gerard-majax]

Comment on attachment 8532704 [details] [diff] [review]
Patch 3 (v1) - Make observer principal cache name reflect content

This version does not apply to v2.1

Comment 72

[:gerard-majax]

Comment on attachment 8534875 [details] [diff] [review]
[v2.1] Patch 3 (v1) - Make observer principal cache name reflect content

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Bug 1061510 and Bug 1092941
User impact if declined: Phone will lock up randomly due to race condition
Testing completed: Multiple try runs, manual device testing for repros
Risk to taking this patch (and alternatives if risky): Settings affects everything. So considerably risky, but downsides are worse.
String or UUID changes made by this patch: None

Comment 73

[:gerard-majax]

For uplifting, please note we do need to apply:
 - attachment 8532696 [details] [diff] [review]
 - attachment 8532699 [details] [diff] [review]
 - attachment 8534875 [details] [diff] [review]

Comment 75

[Johan Lorenzo [:jlorenzo]]

(In reply to bhavana bajaj [:bajaj] from comment #68)
> Ive' requested QA verification on 1103178/1093932 on trunk to ensure those
> are fixed before uplifting this.

I checked bug 1103178 and bug 1093932, I am not able to fall into that stuck state with the patches given by :gerard-majax in comment 73.

By reading the changes made, I'd be glad to know what kind of action fill the taskQueue in order to do further testing.
I've seen this issues multiple times by sending SMS, making phone calls and killing these apps on my dogfooding device. So, in the meantime, I'll dogfood 2.1 with this patch in order to see if things are going better.

Comment 76

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

(In reply to bhavana bajaj [:bajaj] from comment #68)
> Ive' requested QA verification on 1103178/1093932 on trunk to ensure those
> are fixed before uplifting this. Kyle given the risk here, if you can
> highlight some testcases that may hit the code path you are modifying I can
> request QA to help with additional targeted testing here.

Verifying those two bugs are about the best we can do. There are many of operations on the phone that hit settings, but it's hard to test specifically for this because it requires app shutdown for doing an API test, which isn't something we have a great place to test (it could be done in a gaia integration test but isn't specifically a gaia problem?).

Comment 77

[Johan Lorenzo [:jlorenzo]]

I hit that bug again this morning on my 2.1 dogfood device, even with the 3 patches applied yesterday. I did nothing except receiving some SMS, leaving the device in sleep mode, and trying to forward one of the SMS. I couldn't do it, so I checked settings and I've got the same symptoms as bug 1093932.

Has anybody else seen the issue on master in the last 48 hours?

Comment 78

[:gerard-majax]

Attachment: FindMyDevice/FxIdentity related logs

I've noticed those logs today, each time I do unlock the lockscreen. I'm wondering how much it can be related ...

Comment 79

[Martin Shuman [:Marty]]

I hit this issue yesterday after testing Email for about 3 hours, with a little bit of time spent in Clock and Calendar before that.  The main focus of my testing had been exploring Text Selection (Cut/Copy/Paste) throughout the app.  After these hours of testing, I navigated to Settings, and saw that 'Operator Services' was present at the bottom of the menu, and I couldn't enable or access the Developer menu.

Environmental Variables:
Device: Flame 2.2 (319MB)
BuildID: 20141211040208 (Full Flash)
Gaia: 1a956437210d2b16988d2ddbf40c9a38d8474435
Gecko: d92db71d4e67
Gonk: 263b5f41f7733c5577fb101eb4dc8ac5c11cfa8d
Version: 37.0a1 (2.2) 
Firmware Version: v188-1
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0

Comment 80

[:gerard-majax]

Email app does not use Settings, as far as I can tell ...

> $ egrep -R 'SettingsHelper|SettingsListener|mozSettings' apps/email/js/*
> $

So I doubt it's because of your use of Email.

Comment 81

[:gerard-majax]

Martin, had you any chance to capture a logcat when the issue got either reproduced or noticed ?

Enabling logshake in developper menu can help: if you are loosing adb access when the issue is reproduced, shaking should still be able to get your logs. After rebooting, you can access the logs on the sdcard, in the logs/ directory.

Comment 82

[Martin Shuman [:Marty]]

Attachment: logcat-text-selection.txt

I did manage to get a logcat after identifying the issue.  I couldn't access the Developer menu, but logging and USB debugging had already been enabled, and still seemed to be working properly.

I don't really know at what point I hit this issue, so there might not be much in the log for you to see, but I have attached it here.

Comment 83

[:gerard-majax]

Do you need anything else before evaluating approval ?

Comment 84

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/4025179a636a
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/f811d4d9a002
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/05ab73bfe2d0