11/27 update bricks unagi phone (stuck on the mozilla black screen)

RESOLVED WONTFIX

Status

Firefox OS
Gaia::System
P1
blocker
RESOLVED WONTFIX
6 years ago
3 years ago

People

(Reporter: Cww, Assigned: marshall_law)

Tracking

({b2g-testdriver, foxfood})

unspecified
B2G C3 (12dec-1jan)
ARM
Gonk (Firefox OS)
b2g-testdriver, foxfood

Firefox Tracking Flags

(blocking-basecamp:-, b2g18+)

Details

(Whiteboard: [target:12/21])

Attachments

(1 attachment)

103.01 KB, text/plain
Details
(Reporter)

Description

6 years ago
Created attachment 685905 [details]
logcat

After applying the update from 11/27:

If it matters, I updated while my phone was plugged in (but not to a computer, just to a power brick). I initiated the update from settings.

1) You end up in a state with black text and mozilla in the corner but sometimes your lock screen would be visible for a second when tapping the power button.

2) Holding the power button does bring up the power menu

3) Restarting from that menu restarts the phone but doesn't fix the problem. Phone looks like it boots normally, screen powers off and then powers on in the mozilla state and eventually (a few seconds) the screen powers off again.

4) Ditto powering it down. (Note, it seems to take 7-8 seconds to power down... I can't remember if that's always been the case).

5) Removing the battery doesn't seem to help.

6) Removing the SIM card doesn't seem to help. 

Attaching a logcat.
This is preventing us from releasing an update to the B2G Test Drivers who are on a 13 day old build already.
Severity: normal → blocker
Priority: -- → P1
blocking-basecamp: --- → +
Keywords: b2g-testdriver, dogfood
Fwiw, I was able to download and apply this update without issue.  I will attach my logcat for comparison.
I'm bricked, and won't be in a Mozilla office for another week, so if someone can point me at docs for recovery, I'm happy to give it a try.
(Reporter)

Comment 4

6 years ago
Interestingly, screenshots work from the mozilla screen, you just don't see a notification that one was saved.

Also, flashing the build directly works great (well, all my data is gone but that's flashing for you).
(Reporter)

Comment 5

6 years ago
You'll have to flash your phone I think. I don't know if there are docs, if not I'll probably write them for non-local dogfooders.
The logcat looks perfectly normal.  All processes are launched as expected.

Looks like a gaia UI bug.
Component: General → Gaia
Also, this sounds really familiar ...
We had a similar bug caused in the past by a setting related to the first-run app being wrong.  I wonder if we have a race condition or some other bug in the default value for those settings.
(Reporter)

Comment 9

6 years ago
Some things to note: 

I'd never run/tested the first-run experience prior to installing the update so it wouldn't surprise me that a setting was missing.

Flashing the phone to the 11-15 nightly build and installing the 11-27 nightly update doesn't cause this problem (but you have to go through and complete the first-run experience to start the update.)

Flashing the phone to the 11-06 nightly build (which doesn't have first-run) and trying to update gave me a weird problem (it didn't fully download the update but offered me a chance to install anyway).  When I installed, it resulted in the exact problem listed above, mozilla screen and nothing else.

So, first-run is a likely candidate. I can work on a tighter regression range if someone in QA doesn't beat me to it.
(Reporter)

Comment 10

6 years ago
Ok, this is fully reproducible: flash to 11/06 nightly, set up wifi, check for updates.  Walk away.

Eventually the phone will reboot and you'll be stuck.

PS, a way to test OTA updates using an alternate channel would be a great idea.
I'm in the same state.
I put my SIM last night inside the phone and I was offered a download (I did not check for an update).
The update was around 40+MB. IIRC I was on 11-14.
This morning I finally got prompted to install which I did.
After restart I was on the "mozilla screen".
FTR I was able to hear a vibration for a text message.

zandr, I believe these are the instructions for the unagi:
https://intranet.mozilla.org/B2G_Team/Unagi#Install_Boot_to_Gecko

lsblakk also wrote a script to help you with this:
https://github.com/mozilla-b2g/dogfood-setup

I have not tested those steps though.

Comment 13

6 years ago
I was offered an update last night too and am now in the same bricked state.  I'll try to reimage.
(In reply to Ravi Pina [:ravi] from comment #13)
> I was offered an update last night too and am now in the same bricked state.
> I'll try to reimage.

Ravi - re-image from https://releases.mozilla.com/b2g-testdrivers/unagi_stable_2012-11-27.zip and then please run dogfood-setup.sh -- see https://etherpad.mozilla.org/b2g-testdrivers-flashing for complete info

Updated

6 years ago
Component: Gaia → Gaia::System
fwiw, my phone bricked, too.  Are we on a clear path here to get past this, or are we just going to instruct everyone to re-flash?  Did we somehow get past bricking a large category of users here?  Also, who owns this?
OS: Mac OS X → Gonk (Firefox OS)
Hardware: x86 → ARM
(In reply to Damon Sicore (:damons) from comment #15)
> fwiw, my phone bricked, too.  Are we on a clear path here to get past this,
> or are we just going to instruct everyone to re-flash?  Did we somehow get
> past bricking a large category of users here?  Also, who owns this?

Some people were getting past this update and others weren't however we are about to do a hard stop and reflash everyone who is a B2G Test Driver to the newly-created Beta channel builds that have some kernel updates in them we weren't able to provide in an update anyway, so the update was left up in case some folks could use it even though the majority of people who are still checking for updates will need to be reflashed since they would have to be anyway.  The build is being tested today and will be announced later today or early tomorrow in conjunction with shipping out more phones to testers in remote areas.
(In reply to Damon Sicore (:damons) from comment #15)
> fwiw, my phone bricked, too.  Are we on a clear path here to get past this,
> or are we just going to instruct everyone to re-flash?  Did we somehow get
> past bricking a large category of users here?  Also, who owns this?

I'm owning the part about keeping Test Drivers on current builds as best we can.  QA and devs need to own the update path testing and looking closer at this particular update sequence to uncover what happened to Gaia in the process.
(Assignee)

Comment 18

6 years ago
Taking this
Assignee: nobody → marshall
Mass Modify: All un-milestoned, unresolved blocking-basecamp+ bugs are being moved into the C3 milestone. Note that the target milestone does not mean that these bugs can't be resolved prior to 12/10, rather C2 bugs should be prioritized ahead of C3 bugs.
Target Milestone: --- → B2G C3 (12dec-1jan)
Whiteboard: [target:12/21]
any update Marshall? was the root cause ever found?
(Assignee)

Comment 21

6 years ago
Hey guys, I was finally able to reproduce this with a slight modification to comment 10. After the 11/27 FOTA update has been applied, you still need to make sure to apply the subsequent OTA update. After that has been applied, and b2g restarts, you will see this mozilla screen. It seems that the 11/06 build as the starting point is key here.

tl;dr: This looks to be caused by a settings IndexedDB change of some kind (possibly schema change?). You can workaround this by using adb, but beware you will lose your settings:

adb shell stop b2g
adb shell mv /data/local/indexedDB/chrome /data/local/indexedDB/_chrome
adb shell start b2g

Here's what I did to narrow this down to the settings DB:
* I first moved /data/local/indexedDB to /data/local/_indexedDB, and restarted b2g to see if this was a database problem. This showed a crash report, and then b2g loaded normally.
* Afterward I wiped the new indexedDB, and restored the "bad" version. I repeated this for the "chrome" directory.
* I then pulled down /data/local/indexedDB/chrome locally, and grabbed the db name from each database:

% find chrome -name '*.sqlite' -exec echo -n \{\} ": " \; -exec sqlite3 \{\} 'select name from database;' \;
chrome/226660312ssm.sqlite : sms
chrome/2588645841ssegtnti.sqlite : settings
chrome/3104902905ascetiitvi.sqlite : activities
chrome/3249156127nsetta_ts.sqlite : net_stats
chrome/4045445992aslmar.sqlite : alarms

* Then I systematically removed and replaced each directory, and restarted b2g until Gecko loaded again. B2G successfully loaded (again, after a crash prompt) for the sqlite with the "settings" database

Anyone knowledgeable about IndexedDB to take this one over? I can send "bad" sqlite database to someone for further debugging, if it helps..
(Assignee)

Comment 22

6 years ago
After talking some with gwagner today, it sounds like this might have happened because of new default settings appearing in Gaia's settings.json, but they didn't properly make their way into the SettingsDB.

gwagner has opened Bug 821814 w/ a patch that might solve this problem. I will be testing it soon to verify..
We'll always catch this bug in final build update testing. There's no way this issue will go unnoticed when it reproduced on a majority of tested phones (comment 10). We haven't seen the issue since in Nightly/Beta updates. blocking-basecamp-
blocking-basecamp: + → -
(when we have time again, we should investigate)
tracking-b2g18: --- → 19+

Updated

5 years ago
tracking-b2g18: 19+ → +
Keywords: foxfood
Keywords: dogfood
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.