[OTA update] much of my user data blown away in today's update

RESOLVED FIXED

Status

defect
P1
critical
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: dietrich, Assigned: fabrice)

Tracking

({smoketest})

Dependency tree / graph

Firefox Tracking Flags

(blocking-basecamp:+, firefox18 fixed)

Details

(Whiteboard: [dogfooding-blocker])

Attachments

(2 attachments)

Reporter

Description

7 years ago
* homescreen customizations gone
* browser bookmarks and history gone
* all calendar accounts deleted
* all email accounts deleted
* not sure what else

Settings seem to be intact, as well as apps installed from Marketplace.
Reporter

Updated

7 years ago
blocking-basecamp: --- → ?
The updates themselves can't touch user data, but we need to understand what happened.

Dietrich, do you see any errors in logcat?

Were there any changes to the app DB schemas that might have caused this?

Did something change in the identification of apps that would have led to us stranding their old DBs?

Did something in IDB change?  (I sure hope not, on aurora!)
Severity: normal → critical
blocking-basecamp: ? → +
Assignee

Comment 2

7 years ago
Given that all the apps private data has been removed, but not the chrome databases (like the settings) I tend to think that we hit this codepath:
https://mxr.mozilla.org/mozilla-central/source/dom/apps/src/Webapps.jsm#215

But since the apps from the marketplace are still there, we loaded the apps registry from /data/local/webapps/webapps.json properly and we should not remove anything.

unfortunately Dietrich's logcat is not useful thanks to the ONCRPC flood.
Coincidence-wise, SQLite 3.7.14.1 landed last night in Bug 801281, but we were only updating from 3.7.14 and if you look at the diff there were almost no changes.  So we should rule that out.
(In reply to Fabrice Desré [:fabrice] from comment #2)
> Given that all the apps private data has been removed, but not the chrome
> databases (like the settings) I tend to think that we hit this codepath:
> https://mxr.mozilla.org/mozilla-central/source/dom/apps/src/Webapps.jsm#215
> 

Is there any logging we can add there to determine why the apps are blown away?

Guys, this blocks updates on the stable channel.  Sorry Dietrich, but thanks for being a guinea pig :/.
Whiteboard: [dogfooding-blocker]
Priority: -- → P1

Comment 5

7 years ago
:cjones :fabrice :marshall_law
who owns this?  for the time being assigning to :marshall_law unless some other owner pops up.
Assignee: nobody → marshall
Assignee

Comment 6

7 years ago
We first need to reproduce it. Dietrich, were you on the stable channel? (where the last update is from 2012-10-24)
Assignee

Comment 7

7 years ago
I reflashed a stable build from 2012-10-23, installed a couple of apps, played a bit with the browser, added some contacts and changed some settings.

Then I updated to the 2012-10-24, but no data loss occured nor setting changes.
I believe that dietrich was on the nightly channel.  There wasn't a 10-26 stable update.

dietrich, please confirm.
Flags: needinfo?(dietrich)
Assignee

Comment 9

7 years ago
Dietrich confirmed that he was on the nightly channel. I will try to reproduce again.
Flags: needinfo?(dietrich)
Posted file logcat
I can confirm that a poor man's update from 10-25 nightly to 10-26 (re-flash /system with 10-26) blew away my homescreen customizations, but not my settings or contacts.  So I think this is the same problem as described in comment 0.

A *very* suspicious-looking character in logcat is

E/GeckoConsole(  106): [JavaScript Error: "PermissionsTable.jsm: expandPermissions: Unknown Permission: background-sensors" {file: "resource://gre/modules/PermissionsInstaller.jsm" line: 260}]
E/GeckoConsole(  106): [JavaScript Error: "Error: PermissionsTable.jsm: expandPermissions: Unknown Permission: background-sensors" {file: "resource://gre/modules/PermissionsInstaller.jsm" line: 389}]
Doing the same poor man's update from 10-25 --> 10-29 exhibited the same symptom, loss of data.

The permissions error wasn't in logcat, so that's not the culprit.
Assignee

Comment 14

7 years ago
There's also Bug 786295 - That would fit "nicely" with comment #0, where everything backed by indexedDB disappears.
(In reply to Fabrice Desré [:fabrice] from comment #14)
Yeah, before bug 786295 we didn't have the ability to clear indexedDB apps in response to the uninstall notification.
Assignee

Comment 17

7 years ago
But we still should not clear anything. Assigning to me since this is triggered from Webapps.jsm
Assignee: marshall → fabrice
(In reply to Chris Jones [:cjones] [:warhammer] from comment #18)
> Confirmed to be bug 786295.

Wait, to be clear, the bug is that we're calling this clearPrivateData function during an update, not that IndexedDB is listening to that signal.
That's fine, but bug 786295 introduced this regression.  I understand that that cset was an innocent bystander, of sorts.
I put a dump() at all the _clearPrivateData() calls in Webapps.jsm, including _clearPrivateData() itself, and none of them were hit ...  The plot thickens.
Assignee

Comment 22

7 years ago
yeah, just did the same here, forcing runUpdate to be true. We never hit this path, as expected.
Oh! I think this happened because bug 786295 change the naming scheme for indexedDB databases. So technically no data was deleted, we're just looking for it at the "wrong" filename.

So yes, this will lose a bunch of data on update, but it's a one-shot deal. It's not expected to happen again. I.e. this will only happen when updating this once, it won't happen on every update.
Assignee

Comment 24

7 years ago
Posted patch patchSplinter Review
Patch renaming indexedDB folders when running an update.

Before:

fabrice@fabrice-x230:~/b2g-distro$ adb shell ls /data/local/indexedDB
1002+app+++browser.gaiamobile.org+f
1004+app+++costcontrol.gaiamobile.org+f
1006+app+++homescreen.gaiamobile.org+f
1009+app+++system.gaiamobile.org+f
chrome

After:

fabrice@fabrice-x230:~/b2g-distro$ adb shell ls /data/local/indexedDB
1002+f+app+++browser.gaiamobile.org
1004+f+app+++costcontrol.gaiamobile.org
1006+f+app+++homescreen.gaiamobile.org
1009+f+app+++system.gaiamobile.org
chrome
Assignee

Updated

7 years ago
Attachment #676490 - Flags: review?(jonas)
We need some temporary pref to force this upgrade path right?  Won't firstRun be false for deployed dogfood devices?
Assignee

Comment 27

7 years ago
(In reply to Chris Jones [:cjones] [:warhammer] from comment #25)
> We need some temporary pref to force this upgrade path right?  Won't
> firstRun be false for deployed dogfood devices?

No, this test is true for "real" first run and for gecko+gaia updates.
Keywords: smoketest
Backed out along with several others in order to get inbound green again, after a busted landing meant we lost test coverage for 7 pushes, and now have multiple failures:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=a145ded68994

Backout:
https://hg.mozilla.org/integration/mozilla-inbound/rev/553fb59b9ca0
\o/

Do you mind filing a followup to remove this temporary upgrade code?
Assignee

Updated

7 years ago
Component: Gaia → General
Assignee

Comment 33

7 years ago
(In reply to Chris Jones [:cjones] [:warhammer] from comment #32)
> \o/
> 
> Do you mind filing a followup to remove this temporary upgrade code?

I filed bug 807216
https://hg.mozilla.org/mozilla-central/rev/58ddfb88815a
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.