1063412 - Restore and backup profile through webide for any given runtime

Status: RESOLVED WONTFIX

(DevTools Graveyard :: WebIDE, defect)

Description

[Frederik Braun [:freddy]]

I want to try porting the backup_restore_profile shell script[1] to the WebIDE.

I guess this could be in the dropdown on the top right, somewhere between "Runtime Info", "Permission Table", "Screenshot", "Disconnect" and so forth.

Optimally, this would open a new sub page just like "Permission Table" and "Runtime Info" do. I want the website to cover the following use cases
* "quick! back up everything, now"
* (de)selecting apps, just doing settings
* provide an overview of previous backups, with a timestamp and possibly the  originating runtime

I might need some support to provide a visually appealing site, but am happy to do the JavaScript bits.

[1] https://github.com/Mozilla-TWQA/B2G-flash-tool/blob/master/backup_restore_profile.sh

Comment 1

[Panos Astithas (he/him) [:past] (please ni?)]

Yes please!

Comment 2

[Frederik Braun [:freddy]]

I have been toying around with the interfaces and noticed that the device object I can get has a shell() interface. According to the adbhelper scripts on Github[1], there should at least also be a push() function? I could not easily figure out why that's missing here.
I also noticed that pull is missing, which is an obvious blocker. Filing a dependent bug right now..

My current approach is very meager. But I'm uploading in case someone bothers to give some early feedback. Just in case I'm on a very wrong track :)

[1] https://github.com/mozilla/adbhelper/blob/master/adb.js

Comment 3

[Frederik Braun [:freddy]]

Attachment: bug-1063412-webide-backup.patch

Comment 4

[Frederik Braun [:freddy]]

I have talked this over with Paul, who suggested creating an actor or use the device actor (see toolkit/devtools/server/actors/device.js). I realized that something which lives in chrome code is not particularly useful, as we need to shut down the b2g process before we perform the backup (to avoid races).

We also do not want to store the backup we create on disk: If we create a backup of everything, the created image would be (at worst, assuming terrible compression) double the size of the existing data. This would mean, backups would fail with a disk space usage of 50% or more.

We want to do this in an app-agnostic way: For now, this means backing up the directories for profile, apps and settings that do not live within the profile (e.g. wifi config) - maybe through something like 'adb pull' (as the shell script mentioned in comment the bug description does). In the future we could use the App Export/Import mechanisms that are being worked on in bug 982874.

Comment 5

[Paul Rouget [:paul]]

Frederik, I encourage you to talk to jryans and ochameau. They might have some smart ideas of how to implement that properly.

Comment 6

[[:fabrice] Fabrice Desré]

fwiw, I plan to spend some time working on the backup/restore api for b2g. Hopefully we should cross roads at some point!

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Fabrice Desré [:fabrice] from comment #6)
> fwiw, I plan to spend some time working on the backup/restore api for b2g.
> Hopefully we should cross roads at some point!

Fabrice, is there a bug already filed for this work?

Comment 8

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

First of all, this is a great idea! :D

Like Paul's suggestion, I think it would be nice to implement this with an actor, assuming it's possible to do.  The main reason is that we'll soon offer WiFi debugging, which doesn't use ADB.  So, if we'd like to offer this feature via WiFi too, then we need it to use a DevTools actor.

However, it does sound challenging to do especially, for the restore case...  If b2g is running while you restore, it seems like it could overwrite what you just restored on restart.

Do we know what parts of the profile are held open while b2g is running?  Can we force them to flush to disk (from chrome code) just before we'd do a backup?

But we may have to go with ADB (and thus USB-only) if we can't find a good way to make it work from within DevTools.

I'll keep thinking about this problem to see if there's a better way.

Comment 9

[[:fabrice] Fabrice Desré]

Depending on exactly what you want to backup, this can be done safely while gecko is running. For the backup part of the api (no bug yet, sorry, only discussions in dev-b2g) the idea is to backup:
For each app:
- indexedDB
- localStorage
- cookies
- app package
Chrome databases:
- settings
- sms
- contacts
- datastore?

We already have support to prevent indexedDB & co. to do any writes when we are low on disk space. We could re-use this mechanism to freeze the DB before doing a backup. I'm not sure yet about the restoration step, this looks a bit more tricky.

A totally different option is to do something like that: when we want to backup/restore, restart gecko to use a different profile, and from this profile run the backup or restore of the main profile. Lots of details here to figure out also...

Comment 10

[Alexandre Poirot [:ochameau]]

Fabrice, Is it this old thread on dev-b2g, or is there anything newer?
https://groups.google.com/forum/#!searchin/mozilla.dev.b2g/restore/mozilla.dev.b2g/VPkqmLWP4vk/TDJamYXHUSUJ

I'm convinced that for a user-facing interface, we would need something like what fabrice is describing. Per app, data centric with stable/standards data formats.

In the meantime, I think what you tried to achieve in the submitted patch should be doable via webIDE and adb-helper, but I'm not sure it will be very stable across fxos versions. Data file format changes and may break over releases. But I would love to see an experiment to prove how far we can get with such approach and measure the value of such feature in WebIDE.
Core developers already have some scripts, like in gaia build system to do some backup/restore.
But that's still a small set of people...

Comment 11

[Alexandre Poirot [:ochameau]]

Also I hope you came across Yuan Xulei post on:
https://groups.google.com/forum/#!searchin/mozilla.dev.b2g/restore/mozilla.dev.b2g/VPkqmLWP4vk/TDJamYXHUSUJ
and have seen its pc-sync-tool project!

It is very interesting pattern where it uses adb to communicate with a magic app using regular API to backup/restore APIs data. It won't save app specific data, but at least help with API that expose a single shared database.

Comment 12

[[:fabrice] Fabrice Desré]

(In reply to Alexandre Poirot [:ochameau] from comment #10)
> Fabrice, Is it this old thread on dev-b2g, or is there anything newer?
> https://groups.google.com/forum/#!searchin/mozilla.dev.b2g/restore/mozilla.
> dev.b2g/VPkqmLWP4vk/TDJamYXHUSUJ

No, there's no other discussion I know of.

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Frederik, the latest version of ADB Helper now has push and pull support.

Feel free to finish porting the script if you'd like, and we can think about how to evolve the future of backups separately.  It would great to at least have something!

Comment 14

[Frederik Braun [:freddy]]

Thanks for the ping, Ryan.
I am going to work on this during Q4, albeit with a lower priority.
The patch I previously attached for feedback still describes the status quo.

Comment 15

[[:fabrice] Fabrice Desré]

Just one thing about the current method used in this patch: you can't use it to do a backup of b2g version N and restore after flashing version N+1 reliably. That's because nothing guarantees that the app ids will be stable between the versions, but some file names include the app id and browser flag (eg storage paths).

Comment 16

[Frederik Braun [:freddy]]

I'm not going to work in this anymore. Feel free to pick this up.

Comment 17

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

FYI Askeing.  What would be cool is to combine your tool into webide.

Comment 18

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

WebIDE is slated for removal (bug 1314811), so there are no plans to add to the current set of features.