790849 - Don't store information in /data about built-in apps before first startup

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Currently we at build time create various information in the /data directory to support the preinstalled built-in apps. For example we populate the permissions database and the app registry with information for these apps.

That won't work after a factory reset since a factory reset will clear most, if not all, of /data. In particular it will clear any dynamic content since we want to make sure that it's all reset back to default.

The only thing that should live in /data is things about pre-installed 3rd party apps. Ideally those too wouldn't live in /data since it means that they get lost on factory reset, but that's an acceptable solution for v1.

Comment 1

[[:fabrice] Fabrice Desré]

We have everything in place to move the app registry to /system, and bug 787439 will let us get rid of the permission database.

The pieces still in /data are:
- the settings database.
- user.js.
- the prebuilt offline cache if any.

The settings database could be built on first run, but it takes quite a while when we prebuild it.

user.js only has a couple of prefs that can't move to b2g/app/b2g.js, and we could user environment variables to replace them in /system/bin/b2g.sh

For the offline cache, I'm not sure what to do. This stuff will always be painful!

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

How large is the settings database? Could we copy it from /system to /data on first-run?

Actually, speaking of which, how are we going to handle v2 likely needing to add additional preferences? Possibly not something we need to worry about for now. Simply putting code in v2 to handle it might be ok. But it would be nice to have a plan to make sure we don't dig ourselves into a hole.

user.js sounds like we can set on first run?

The offline cache is a problem. We'll want that for the marketplace app I think.

Comment 3

[[:fabrice] Fabrice Desré]

(In reply to Jonas Sicking (:sicking) from comment #2)
> How large is the settings database? Could we copy it from /system to /data
> on first-run?

It's an indexedDB, current size is 520K. Copying on first run is definitely an option.

> Actually, speaking of which, how are we going to handle v2 likely needing to
> add additional preferences? Possibly not something we need to worry about
> for now. Simply putting code in v2 to handle it might be ok. But it would be
> nice to have a plan to make sure we don't dig ourselves into a hole.

I think we could do that:
- store the settings in /system/b2g/settings.json when building the image to flash.
- at first run, iterate over these settings and only set new ones.

> user.js sounds like we can set on first run?

Or we can also simply copy it from /system to /data in b2g.sh (which is before even running gecko).
 
> The offline cache is a problem. We'll want that for the marketplace app I
> think.

We may have to package the assets in a zip and at first run do roughly what the build system does now (it's an xpcshell script).

And of course, we need to take care of not breaking the desktop builds!

Comment 4

[Andrew Overholt [:overholt]]

Marshall, can you take a look at this?

Comment 5

[[:fabrice] Fabrice Desré]

re-assigning to me since Marshall has way more blockers than I do now.

Comment 6

[Jason Smith [:jsmith]]

Did you mean to change the assignee field?

Comment 7

[[:fabrice] Fabrice Desré]

(In reply to Jason Smith [:jsmith] from comment #6)
> Did you mean to change the assignee field?

yes, sorry...

Comment 8

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

ftfy

Comment 9

[[:fabrice] Fabrice Desré]

I made to PR to move user.js out of /data/local when flashing:
https://github.com/mozilla-b2g/gonk-misc/pull/30
https://github.com/mozilla-b2g/gaia/pull/4987

Comment 10

[[:fabrice] Fabrice Desré]

Attachment: Patch to remove the settings db

This is the gecko part to remove the prebuilt indexedDB for settings from /data/local. Instead we just store a json file with the initial values and load it when first creating the settings db.

The file is located in /system/b2g/defaults/ on device, and in profile/ on desktop. PR are coming, but this patch should fail gracefully with the current setup so it can land before.

Comment 11

[Gregor Wagner [:gwagner]]

Comment on attachment 668573 [details] [diff] [review]
Patch to remove the settings db


> /* static functions */
>-let DEBUG = 0;
>-if (DEBUG) {
>-  debug = function (s) { dump("-*- SettingsDB: " + s + "\n"); }
>-} else {
>-  debug = function (s) {}
>+function debug(s) {
>+  dump("-*- SettingsDB: " + s + "\n");
> }
> 

Comment the dump? 
It would make to keep a single "Creating SettingsDB" in the log when we actually create it.


>+    let chan = NetUtil.newChannel(settingsFile);
>+    let stream = chan.open();
>+    // Obtain a converter to read from a UTF-8 encoded input stream.
>+    let converter = Cc["@mozilla.org/intl/scriptableunicodeconverter"]
>+                    .createInstance(Ci.nsIScriptableUnicodeConverter);
>+    converter.charset = "UTF-8";
>+    let rawstr = converter.ConvertToUnicode(NetUtil.readInputStreamToString(
>+                                            stream,
>+                                            stream.available()) || "");
>+    let settings = JSON.parse(rawstr);

Can we catch the error and print an error message if there is a syntax error in the file?

Comment 12

[[:fabrice] Fabrice Desré]

pushed with nits addressed:
https://hg.mozilla.org/integration/mozilla-inbound/rev/50005364bca4

Comment 13

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/50005364bca4

Comment 14

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Does this mean that we don't store any of the information menation in comment 1 in /data before the first run?

Comment 15

[[:fabrice] Fabrice Desré]

(In reply to Jonas Sicking (:sicking) from comment #14)
> Does this mean that we don't store any of the information menation in
> comment 1 in /data before the first run?

bug 787439 will let us remove the prebuilt permissions.sqlite, so the latest piece will be the offline cache for preloaded hosted apps. I'm not sure yet what's the best way to deal with this.

Comment 16

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I checked with Chris Lee a while ago and he said it was ok for third party preinstalled apps to not survive factory reset. So we could let appcached apps livr in /data. Though we'd have to create a packaged shell for the marketplace or some such.

Might be better to find a real solution. At least if we have time.