1890585 - Have MiscDataBackupResource stage the ActivityStream "snippets" IndexedDB data stores under storage/permanent/chrome

Status: RESOLVED FIXED

(Firefox :: Profile Backup, task, P3)

Description

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

Spun out from bug 1885944. This bug will be about staging the IndexedDB databases stored under storage/permanent/chrome, presuming bug 1887978 comes to fruition.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Is the intent to backup everything in storage/permanent/chrome, or just specific things? I see in my profile "remote-settings" is 64M and presumably it could easily end up being stale for the purposes of a newly installed Firefox. It seems like remote-settings is also generating the related IDB databases named in https://searchfox.org/mozilla-central/source/__GENERATED__/services/settings/dumps/last_modified.json (and (partially?) sourced from https://searchfox.org/mozilla-central/source/services/settings/dumps it would seem).

Comment 2

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

RemoteSettings is exactly the sort of thing I think we can skip backing up, since we'll just repopulate it not long after launch if it's missing.

I see the following IndexedDB databases under storage/permanent/chrome:

1. normandy-preference-rollout
2. ActivityStream
3. normandy-addon-rollout
4. push-api
5. shield
6. SaveToPocket
7. remote-settings

I need to double-check with the associated teams, but I suspect we can probably skip backing up the following:

1. normandy-preference-rollout
2. normandy-addon-rollout
3. push-api
4. shield
5. remote-settings

We can probably get away with not backing up SaveToPocket and ActivityStream. SaveToPocket appears to contain a cache of recent articles saved to Pocket, which appears to be designed such that it can "expire" and be empty.

The ActivityStream one is trickier. That one contains a thing called snippets, but despite snippets support having been removed, the data it contains is more related to impressions and session data for ASRouter messages. I also think it is the part of our storage that lets Firefox remember what messages a client has seen (and therefore, which ones not to show again). That might be more worthy of backing up.

If a general solution to safe IndexedDB copying isn't in the cards, we might be able to just serialize the contents of the databases that we care about, and re-insert them into the IndexedDB database after a restoration.

Comment 3

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

Hey barret,

Can I presume we're okay skipping backing up the normandy-preference-rollout, normandy-addon-rollout and shield IndexedDB databases?

Comment 4

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

Hey jjalkanen,

I see you as the current triage owner of DOM: Push Subscriptions. Feel free to redirect me if I have the wrong person.

Does it make sense to skip backing up the pushapi and pushHttp2 IndexedDB databases? Presume that these backups might, for example, be restored across multiple machines over time. Also note that we're also skipping backing up service worker registration.

Comment 5

[Jari Jalkanen [:jari]]

Kagami, do you know if it would be good to back these up?

Comment 6

[Beth Rennie [:beth] (she/her)]

(In reply to Mike Conley (:mconley) (:⚙️) from comment #3)

Hey barret,

Can I presume we're okay skipping backing up the normandy-preference-rollout, normandy-addon-rollout and shield IndexedDB databases?

Yep these are only used by normandy, no reason to back them up.

Comment 7

[Kagami Rosylight [:saschanaz] (they/them)]

(In reply to Mike Conley (:mconley) (:⚙️) from comment #4)

Does it make sense to skip backing up the pushapi and pushHttp2 IndexedDB databases? Presume that these backups might, for example, be restored across multiple machines over time. Also note that we're also skipping backing up service worker registration.

We don't use pushHttp2, it should be fully removed, that's part of bug 1874195.

Backing up pushapi depends on whether we plan to actually migrate the push subscription to other machines. I honestly don't know whether that's possible. Maybe that's a jrconlin question (unless you already know the answer).

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #7)

Backing up pushapi depends on whether we plan to actually migrate the push subscription to other machines. I honestly don't know whether that's possible. Maybe that's a jrconlin question (unless you already know the answer).

Given that we wouldn't be migrating any content-installed ServiceWorkers, I guess the question would depend on any system-based push subscriptions (which I can't speak to).

Comment 9

[JR Conlin [:jrconlin,:jconlin]]

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #7)

Backing up pushapi depends on whether we plan to actually migrate the push subscription to other machines. I honestly don't know whether that's possible. Maybe that's a jrconlin question (unless you already know the answer).

It wouldn't be a good idea to have the push endpoints used by more than one User Agent (UA). (I could see the value of having them backed up and restored to the same UA, or a complete UA replacement, but having the endpoints appear on two different UAs means that Push would break for one of those UAs, and it would be very difficult for the user to know which.)

The other complication is that unless you also capture and copy the associated encryption key database, none of the Push messages would decrypt and Push would break in all sorts of weird ways. (e.g. The subscriptions would still be "live" but the user would never get a notification or action from a Push message. Sites where the user had originally created a Subscription would continue to think that there's one, so the user wouldn't be able to automatically reconnect.)

(For folk that are interested: Push works by assigning a UserAgent Identifier (UAID) to a given instance of Firefox. That UAID is how Push knows what notifications to deliver to what device. Internally, to prevent potential abuse, the Push Server only accepts a "live" connection from one UAID at a time. Mobile devices do a daily "check-in" to confirm that their list of active subscriptions match what the server has, but desktop doesn't do that.)

Frankly, I'd say that Push wasn't really created with this sort of thing in mind, so it might be a good idea to just skip any push backup / restore option to avoid a lot of potential "user surprise" headaches. Just note that they will need to re-create any Push Subscriptions they may have had before. As a side note, I'll add a ticket to investigate how Push might be able to handle this better in the future.

Comment 10

[Kagami Rosylight [:saschanaz] (they/them)]

Thank you both for the details! That means: no, or at least not now.

Comment 11

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

Great, thanks for the context, everybody.

Comment 12

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

Hey thecount, do you know if it's okay if we skip backing up the SaveToPocket IndexedDB database for our backup project? It appears to be some kind of "recently saved pages" cache that is periodically expired. Or is there value in having that be backed up, restorable at a later time, and (potentially) transferred to new (maybe multiple) machines?

Similar question to you, pdahiya for the snippets table inside of the ActivityStream IndexedDB database. As far as I can tell, that's using the snippets table in there to store information about impressions and messages that the profile has seen. Is that correct? Whatever it contains - is there value in having that be backed up, restorable at some later time, and (potentially) transferred to new (maybe multiple) machines?

Comment 13

[Punam Dahiya [:pdahiya]]

Similar question to you, pdahiya for the snippets table inside of the ActivityStream IndexedDB database. As far as I can tell, that's using the snippets table in there to store information about impressions and messages that the profile has seen. Is that correct? Whatever it contains - is there value in having that be backed up, restorable at some later time, and (potentially) transferred to new (maybe multiple) machines?

@mconley that's correct messaging system uses snippets table inside ActivityStream IndexedDb to capture impressions, blocked message ids, message group impressions for a profile. If plan here is to offer user an option to restore to a certain timestamp profile state (to help recover from data loss / corruption) than it makes sense to backup this data.

Comment 14

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

I talked to Scott about this separately. He said to me that the SaveToPocket IndexedDB database was safe to skip for backing up, as it just contains the most recently Pocket'd articles, which is only useful when Pocket tries to show recent saves. These expire anyways over time. For simplicity's sake, we'll not back that up for now.

Comment 15

[Mike Conley (:mconley) (:⚙️) (PTO June 8 - 12)]

Attachment: Bug 1890585 - Have MiscDataBackupResource backup the ActivityStreamStorage snippets table. r?#backup-reviewers!

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e2b46d3af38
Have MiscDataBackupResource backup the ActivityStreamStorage snippets table. r=backup-reviewers,kpatenio

Comment 17

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/6e2b46d3af38