1474562 - Enable ExtensionStorageIDB backend on Nightly

Status: VERIFIED FIXED

(WebExtensions :: Storage, defect, P1)

Description

[Luca Greco [:rpl] [:luca] [:lgreco]]

This issue goal is enabling the ExtensionStorageIDB backend on Nightly (and it is going to depend from the bugzilla issues that have been identified as blockers for being able to enable this pref on Nightly).

Comment 1

[Luca Greco [:rpl] [:luca] [:lgreco]]

Added Bug 1470213, which will provide a new proposed telemetry event probe to collect some per-addon information related to the data migration.

Besides the telemetry event probe the issue also contains ongoing discussions (and preliminary patches) related to:

- some changes related to the behavior of the data migration itself (e.g. Shane would prefer if we could rename the old JSONFile after a successfull data migration, instead of removing it, which was also my original choice when we were reviewing the data migration as part of Bug 1406181)
- some additional fixes related to exceptions (like QuotaExceeededError) that may be raised when we open the connection to the IndexedDB storage
- the need for some kind of readonly access to the data storage in an existing IndexedDB storage when there are quota issues (event as a chromeOnly feature only available to the Firefox internals)

Some (or all) of these additional issues/discussiongs may be moved into their own bugzilla issue (and added to this issue ad blockers).

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Added Bug 1474557 as an additional blocker.

Bug 1474557 is actually also a regression for the storage.local API even when the Firefox profile is configured to still use the current JSONFile backend, and so it is a fix needed also on Firefox 62 beta (where Bug 1406181 has been initially landed).

Comment 3

[Luca Greco [:rpl] [:luca] [:lgreco]]

Added Bug 944918 as an additional blocker.

It is actually a generic IndexedDB issue, but I verified (currently tested manually) that it may also (unsurprisingly) affect the extensions that have been migrated to the storage.local IndexedDB backend:

creating "any unexpected file in the directory where the storage.local IndexedDB database lives" makes the ExtensionStorageIDB.open to raise an exception and prevent the extension to access the stored data.

Comment 4

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 1474562 - Enable ExtensionStorageIDB on Nightly.

Review commit: https://reviewboard.mozilla.org/r/259758/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/259758/

Comment 5

[Shane Caraveo (:mixedpuppy)]

Comment on attachment 8995278 [details]
Bug 1474562 - Enable ExtensionStorageIDB on Nightly.

https://reviewboard.mozilla.org/r/259758/#review267056

assuming this waits for bug 1477015

Comment 6

[Luca Greco [:rpl] [:luca] [:lgreco]]

Comment on attachment 8995278 [details]
Bug 1474562 - Enable ExtensionStorageIDB on Nightly.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/259758/diff/1-2/

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8236a828c129
Enable ExtensionStorageIDB on Nightly. r=mixedpuppy

Comment 8

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/8236a828c129

Comment 9

[CosminB]

Attachment: Bug1474562.png

This issue is verified as fixed on Firefox 63.0a1 (20180805231147) under Win 7 64-bit and Mac OS X 10.13.3.

The preference extensions.webextensions.ExtensionStorageIDB.enabled is set to true by default.

Please see the attached screenshot.

Comment 10

[juraj.masiar]

Is there a plan to backport this to ESR 60? 
So that users downgrading from 63+ in upcoming year (until next ESR) won't be affected by the add-on storage clear issue?

Comment 11

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to juraj.masiar from comment #10)
> Is there a plan to backport this to ESR 60? 

There are no plans to backport this to ESR 60 (the patch attached to this bug contains only the single line
change needed to turn on the storage.local backend in Nightly on 63, but the actual feature is composed by a bigger amount of changes, too much to be reasonable to think about backporting them on the previous Firefox versions).  

> So that users downgrading from 63+ in upcoming year (until next ESR) won't
> be affected by the add-on storage clear issue?

In general downgrading a Firefox profile from a new to an older Firefox version is not advisable (and not supported),
because the differences between the two versions may potentially (and very likely) trigger in the profile 
a number of unexpected issues (which will be pretty hard to reproduce and/or investigate).

May I ask which is the bugzilla issue number related to the add-on storage clear issue you are mentioning
above (if there is one that you are referring to)?

(I'd like to double-check if it is actually something that is meant to be fixed by the new storage.local backend,
or if it is an additional or an unrelated issue).

Comment 12

[juraj.masiar]

I'm not sure there is a bugzilla page for that, but it's a real thing. I got here through Caitlin's link here:
https://discourse.mozilla.org/t/downgrade-from-firefox-63-clears-local-storage-of-all-add-ons/30753/6
But first time I read it here:
https://www.ghacks.net/2018/08/06/dont-downgrade-firefox-63/

Comment 13

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to juraj.masiar from comment #12)
> I'm not sure there is a bugzilla page for that, but it's a real thing. I got
> here through Caitlin's link here:
> https://discourse.mozilla.org/t/downgrade-from-firefox-63-clears-local-
> storage-of-all-add-ons/30753/6
> But first time I read it here:
> https://www.ghacks.net/2018/08/06/dont-downgrade-firefox-63/

The new storage.local backend has been enabled for the first time in 63, but only on the Nightly channel,
it is not enabled on the released and beta versions of Firefox 63.
And so there is not the storage.local backend used in Firefox 60 ESR and the non-nightly
versions of Firefox 63 is the same one (the one based on the JSON file).

Comment 14

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Luca Greco [:rpl] from comment #13)
> And so there is not the storage.local backend used in Firefox 60 ESR and the
> non-nightly
> versions of Firefox 63 is the same one (the one based on the JSON file).

Ouch, I just noticed that I made a typo in my previous comment, here is what I actually meant:

the release version of Firefox 63 and Firefox 60 ESR both use the same storage.local backend
(the one based on the JSON file).

Comment 15

[juraj.masiar]

I see, so the problem could affect only Nightly users, right? Then the article is a bit misleading...
So when is this gonna activate in Beta / Release channels?

So far I've received only few reports from users of my addon that lost their data - but they were able to restore them from backup feature I've build using IndexedDB API (which persists even after downgrade).

Comment 16

[David Durst [:ddurst]]

(In reply to juraj.masiar from comment #15)
> I see, so the problem could affect only Nightly users, right? Then the
> article is a bit misleading...
> So when is this gonna activate in Beta / Release channels?
> 
> So far I've received only few reports from users of my addon that lost their
> data - but they were able to restore them from backup feature I've build
> using IndexedDB API (which persists even after downgrade).

This problem will only affect Nightly users until we enable it on other channels. Which we would like to do as early as 64 beta, but release will not see it until 65.

But, again, issues that users have from downgrading are not something we can completely support. We are trying to minimize any issues with migrating (and I'm not sure why someone would downgrade from, say, 65 to 63 or to 60ESR), but my only concern here would be if your users are experiencing dataloss with your extension as part of *migrating* to IDB.

Comment 17

[juraj.masiar]

One question though - I was contacted by one user that lost add-on data due to normal Firefox upgrade to 63.0. Is that possible? He said he didn't tried Beta or Nightly.

And he was able to restore his data using my history (backup) feature in my add-on that stores backups in IndexedDB.

Actually the few other users that reported data loss before also claimed they didn't installed Beta or Nightly. (these reports started only several weeks ago). Also note that my add-on is used by 50k users so few users is not much.

Comment 18

[juraj.masiar]

Please disregard my last message, I found a bug on my cloud server that is most probably responsible for the data loss :(