815895 - 11/27 update bricks unagi phone (stuck on the mozilla black screen)

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::System, defect, P1)

Description

[[:Cww]]

Attachment: logcat

After applying the update from 11/27:

If it matters, I updated while my phone was plugged in (but not to a computer, just to a power brick). I initiated the update from settings.

1) You end up in a state with black text and mozilla in the corner but sometimes your lock screen would be visible for a second when tapping the power button.

2) Holding the power button does bring up the power menu

3) Restarting from that menu restarts the phone but doesn't fix the problem. Phone looks like it boots normally, screen powers off and then powers on in the mozilla state and eventually (a few seconds) the screen powers off again.

4) Ditto powering it down. (Note, it seems to take 7-8 seconds to power down... I can't remember if that's always been the case).

5) Removing the battery doesn't seem to help.

6) Removing the SIM card doesn't seem to help. 

Attaching a logcat.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

This is preventing us from releasing an update to the B2G Test Drivers who are on a 13 day old build already.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Fwiw, I was able to download and apply this update without issue.  I will attach my logcat for comparison.

Comment 3

[Zandr Milewski [:zandr]]

I'm bricked, and won't be in a Mozilla office for another week, so if someone can point me at docs for recovery, I'm happy to give it a try.

Comment 4

[[:Cww]]

Interestingly, screenshots work from the mozilla screen, you just don't see a notification that one was saved.

Also, flashing the build directly works great (well, all my data is gone but that's flashing for you).

Comment 5

[[:Cww]]

You'll have to flash your phone I think. I don't know if there are docs, if not I'll probably write them for non-local dogfooders.

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

The logcat looks perfectly normal.  All processes are launched as expected.

Looks like a gaia UI bug.

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Also, this sounds really familiar ...

Comment 8

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

We had a similar bug caused in the past by a setting related to the first-run app being wrong.  I wonder if we have a race condition or some other bug in the default value for those settings.

Comment 9

[[:Cww]]

Some things to note: 

I'd never run/tested the first-run experience prior to installing the update so it wouldn't surprise me that a setting was missing.

Flashing the phone to the 11-15 nightly build and installing the 11-27 nightly update doesn't cause this problem (but you have to go through and complete the first-run experience to start the update.)

Flashing the phone to the 11-06 nightly build (which doesn't have first-run) and trying to update gave me a weird problem (it didn't fully download the update but offered me a chance to install anyway).  When I installed, it resulted in the exact problem listed above, mozilla screen and nothing else.

So, first-run is a likely candidate. I can work on a tighter regression range if someone in QA doesn't beat me to it.

Comment 10

[[:Cww]]

Ok, this is fully reproducible: flash to 11/06 nightly, set up wifi, check for updates.  Walk away.

Eventually the phone will reboot and you'll be stuck.

PS, a way to test OTA updates using an alternate channel would be a great idea.

Comment 11

[Armen [:armenzg]]

I'm in the same state.
I put my SIM last night inside the phone and I was offered a download (I did not check for an update).
The update was around 40+MB. IIRC I was on 11-14.
This morning I finally got prompted to install which I did.
After restart I was on the "mozilla screen".
FTR I was able to hear a vibration for a text message.

zandr, I believe these are the instructions for the unagi:
https://intranet.mozilla.org/B2G_Team/Unagi#Install_Boot_to_Gecko

lsblakk also wrote a script to help you with this:
https://github.com/mozilla-b2g/dogfood-setup

I have not tested those steps though.

Comment 13

[Ravi Pina [:ravi]]

I was offered an update last night too and am now in the same bricked state.  I'll try to reimage.

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Ravi Pina [:ravi] from comment #13)
> I was offered an update last night too and am now in the same bricked state.
> I'll try to reimage.

Ravi - re-image from https://releases.mozilla.com/b2g-testdrivers/unagi_stable_2012-11-27.zip and then please run dogfood-setup.sh -- see https://etherpad.mozilla.org/b2g-testdrivers-flashing for complete info

Comment 15

[Damon Sicore (:damons)]

fwiw, my phone bricked, too.  Are we on a clear path here to get past this, or are we just going to instruct everyone to re-flash?  Did we somehow get past bricking a large category of users here?  Also, who owns this?

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Damon Sicore (:damons) from comment #15)
> fwiw, my phone bricked, too.  Are we on a clear path here to get past this,
> or are we just going to instruct everyone to re-flash?  Did we somehow get
> past bricking a large category of users here?  Also, who owns this?

Some people were getting past this update and others weren't however we are about to do a hard stop and reflash everyone who is a B2G Test Driver to the newly-created Beta channel builds that have some kernel updates in them we weren't able to provide in an update anyway, so the update was left up in case some folks could use it even though the majority of people who are still checking for updates will need to be reflashed since they would have to be anyway.  The build is being tested today and will be announced later today or early tomorrow in conjunction with shipping out more phones to testers in remote areas.

Comment 17

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Damon Sicore (:damons) from comment #15)
> fwiw, my phone bricked, too.  Are we on a clear path here to get past this,
> or are we just going to instruct everyone to re-flash?  Did we somehow get
> past bricking a large category of users here?  Also, who owns this?

I'm owning the part about keeping Test Drivers on current builds as best we can.  QA and devs need to own the update path testing and looking closer at this particular update sequence to uncover what happened to Gaia in the process.

Comment 18

[Marshall Culpepper [:marshall_law]]

Taking this

Comment 19

[Alex Keybl [:akeybl]]

Mass Modify: All un-milestoned, unresolved blocking-basecamp+ bugs are being moved into the C3 milestone. Note that the target milestone does not mean that these bugs can't be resolved prior to 12/10, rather C2 bugs should be prioritized ahead of C3 bugs.

Comment 20

[Dietrich Ayala (:dietrich)]

any update Marshall? was the root cause ever found?

Comment 21

[Marshall Culpepper [:marshall_law]]

Hey guys, I was finally able to reproduce this with a slight modification to comment 10. After the 11/27 FOTA update has been applied, you still need to make sure to apply the subsequent OTA update. After that has been applied, and b2g restarts, you will see this mozilla screen. It seems that the 11/06 build as the starting point is key here.

tl;dr: This looks to be caused by a settings IndexedDB change of some kind (possibly schema change?). You can workaround this by using adb, but beware you will lose your settings:

adb shell stop b2g
adb shell mv /data/local/indexedDB/chrome /data/local/indexedDB/_chrome
adb shell start b2g

Here's what I did to narrow this down to the settings DB:
* I first moved /data/local/indexedDB to /data/local/_indexedDB, and restarted b2g to see if this was a database problem. This showed a crash report, and then b2g loaded normally.
* Afterward I wiped the new indexedDB, and restored the "bad" version. I repeated this for the "chrome" directory.
* I then pulled down /data/local/indexedDB/chrome locally, and grabbed the db name from each database:

% find chrome -name '*.sqlite' -exec echo -n \{\} ": " \; -exec sqlite3 \{\} 'select name from database;' \;
chrome/226660312ssm.sqlite : sms
chrome/2588645841ssegtnti.sqlite : settings
chrome/3104902905ascetiitvi.sqlite : activities
chrome/3249156127nsetta_ts.sqlite : net_stats
chrome/4045445992aslmar.sqlite : alarms

* Then I systematically removed and replaced each directory, and restarted b2g until Gecko loaded again. B2G successfully loaded (again, after a crash prompt) for the sqlite with the "settings" database

Anyone knowledgeable about IndexedDB to take this one over? I can send "bad" sqlite database to someone for further debugging, if it helps..

Comment 22

[Marshall Culpepper [:marshall_law]]

After talking some with gwagner today, it sounds like this might have happened because of new default settings appearing in Gaia's settings.json, but they didn't properly make their way into the SettingsDB.

gwagner has opened Bug 821814 w/ a patch that might solve this problem. I will be testing it soon to verify..

Comment 23

[Alex Keybl [:akeybl]]

We'll always catch this bug in final build update testing. There's no way this issue will go unnoticed when it reproduced on a majority of tested phones (comment 10). We haven't seen the issue since in Nightly/Beta updates. blocking-basecamp-

Comment 24

[Alex Keybl [:akeybl]]

(when we have time again, we should investigate)