821985 - Missing easy way to perform gaia databases backup and restore

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, enhancement)

Description

[:gerard-majax]

Currently, when dogfooding, there is no easy way to:
 - make a backup/restore of the device from the Makefile, it involves playing with adb
 - reset Gaia profile and restore majors parts (SMS, contacts, etc)

I have the need for such enhancements to simplify dogfooding process.

Comment 1

[:gerard-majax]

Attachment: Link to Github https://github.com/mozilla-b2g/gaia/pull/7025

Pull request #7025 on Github.

Comment 2

[Fabien Cazenave [:kaze]]

If it works, I’ll use it for sure!

Comment 3

[James Lal [:lightsofapollo]]

This would be awesome to land to land for internal dog fooding? Can we have this reviewed ASAP ?

Comment 4

[Alexandre Poirot [:ochameau]]

Comment on attachment 692580 [details]
Link to Github https://github.com/mozilla-b2g/gaia/pull/7025

Forwarding the review to vivien as I have to focus on jetpack now.

Comment 5

[:gerard-majax]

Maybe I should also handle /data/local/webapps/ and /data/local/webapps.json, to backup installed webapps ?

Comment 6

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #5)
> Maybe I should also handle /data/local/webapps/ and
> /data/local/webapps.json, to backup installed webapps ?

I've sent a change to the pull request, it now backup and restores also webapps.

Comment 7

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Comment on attachment 692580 [details]
Link to Github https://github.com/mozilla-b2g/gaia/pull/7025

Let's just make sure to handle the production case as well and the patch looks good otherwise.

Comment 8

[Staś Małolepszy :stas]

I tested the patch from attachment 692580 [details] and it works very well.  Thanks so much for this, Alexandre!

The number one request from the localizers testing the phones was exactly this -- to be able to restore contacts after flashing a new version of Gaia, with translation fixes.

I'd like to nominate this for tef.  As I understand it, the risk is very low.  The patch doesn't affect other make targets, so if you continue using "make production", you won't even notice the change (Alexandre, please correct me if this is not the case).

OTOH, the testers can start using "make reinstall" which turns on backing up and restoring.

Comment 9

[Staś Małolepszy :stas]

Comment on attachment 692580 [details]
Link to Github https://github.com/mozilla-b2g/gaia/pull/7025

Vivien, can you take another look at this?  FWIW, I've been successfully using this patch with MAKECMDGOALS=production MOZILLA_OFFICIAL=1.

Comment 10

[Daniel Coloma:dcoloma]

(In reply to Staś Małolepszy :stas from comment #8)
> I tested the patch from attachment 692580 [details] and it works very well. 
> Thanks so much for this, Alexandre!
> 
> The number one request from the localizers testing the phones was exactly
> this -- to be able to restore contacts after flashing a new version of Gaia,
> with translation fixes.
> 
> I'd like to nominate this for tef.  As I understand it, the risk is very
> low.  The patch doesn't affect other make targets, so if you continue using
> "make production", you won't even notice the change (Alexandre, please
> correct me if this is not the case).
> 
> OTOH, the testers can start using "make reinstall" which turns on backing up
> and restoring.

This is clearly not a release blocker. I'd suggest we land this first in master and v1-train, and then decide whether we need to uplift this to v1.0.1. Can we wait for that?

By the way, are dogfooders still on v1.0.1 or are they already in v1-train?

Comment 11

[Daniel Coloma:dcoloma]

> 
> By the way, are dogfooders still on v1.0.1 or are they already in v1-train?

sorry, I read dogfooders when you said localizers.

Comment 12

[Staś Małolepszy :stas]

(In reply to Daniel Coloma:dcoloma from comment #10)
 
> This is clearly not a release blocker. I'd suggest we land this first in
> master and v1-train, and then decide whether we need to uplift this to
> v1.0.1. Can we wait for that?

I'd like to offer a counter-argument:  there's no point in waiting with the uplift to v1.0.1.  I realize this doesn't block the release per-se, and we can release without this.  Just as we *can* release without unit tests.

We've invested a lot of effort over the last few days in extensive localization testing.  20+ localizers now have phones and we're seeing a lot of changes and fixes landing in the localization repositories.  The localizers are similar to dogfooders -- so your mistype is justifiable -- in that they use the phone as their primary phones.  Losing your contacts every time you |make install-gaia| with new translations severely handicaps their .

I fully understand the process and I support landing on master first, then v1-train.  I still want to do this, with lower priority.  But I need this on v1.0.1 too, and the time window is small.  Realistically, if we decide to wait, we probably won't need this on v1.0.1 at all, because we won't have an opportunity to update the localizations a few weeks from now.  This may be the right call from the stability-point of view, which is why I'd like to solicit help in assessing the risk of this patch.  As far as I understand it, the patch is borderline not-part-of-the-build (NPOTB), so the risk is very low.


> By the way, are dogfooders still on v1.0.1 or are they already in v1-train?

Most of them are on v1-train, but 8 high-priority locales are on 1.0.1, with a deadline coming any day now (mid-April).

Thanks, Daniel.

Comment 13

[Staś Małolepszy :stas]

Comment on attachment 692580 [details]
Link to Github https://github.com/mozilla-b2g/gaia/pull/7025

Alexandre, btw, should you also add device-backup to .gitignore?

Comment 14

[Daniel Coloma:dcoloma]

> 
> I fully understand the process and I support landing on master first, then
> v1-train.  I still want to do this, with lower priority.  But I need this on
> v1.0.1 too, and the time window is small.  Realistically, if we decide to
> wait, we probably won't need this on v1.0.1 at all, because we won't have an
> opportunity to update the localizations a few weeks from now.  This may be
> the right call from the stability-point of view, which is why I'd like to
> solicit help in assessing the risk of this patch.  As far as I understand
> it, the patch is borderline not-part-of-the-build (NPOTB), so the risk is
> very low.

I know the risk is low, so please land it asap in master and when some basic testing is done let's uplift then.

Comment 15

[:gerard-majax]

(In reply to Staś Małolepszy :stas from comment #13)
> Comment on attachment 692580 [details]
> Link to Github https://github.com/mozilla-b2g/gaia/pull/7025
> 
> Alexandre, btw, should you also add device-backup to .gitignore?

Done, I also renamed device-backup to backups, as suggested by vivien.

Comment 16

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

The pull request needs some love before merging.

Comment 17

[:gerard-majax]

(In reply to Vivien Nicolas (:vingtetun) (:21) from comment #16)
> The pull request needs some love before merging.

I think I did address all of your comments a while ago, do you have new ones to make ?

Comment 18

[Jack Mitchell]

Now that the Geeksphones are shipping this is becoming a big issue, I use the Geeksphone Keon as a primary device and if I'm running testing versions regularly to help fix and report, I need a way to easily backup and restore personal data.

I tried applying the patch listed here but it fails to apply, a rebase onto master is probably in order...

Comment 19

[Jan Jongboom [:janjongboom] (inactive)]

I strongly +1 this issue. It merges cleanly on v1-train btw. Anyway, tested this on my v1-train GeeksPhone and worked flawlessly.

Comment 20

[Chris Lord [:cwiiis]]

Adding a needinfo? wrt comment #17.

Comment 21

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

(In reply to Alexandre LISSY :gerard-majax from comment #17)
> (In reply to Vivien Nicolas (:vingtetun) (:21) from comment #16)
> > The pull request needs some love before merging.
> 
> I think I did address all of your comments a while ago, do you have new ones
> to make ?

I thought this stuff has merged already. My comment was because the PR was not merging anymore.

Comment 22

[Chris Lord [:cwiiis]]

This work hasn't merged, do we need to add a checkin-needed flag?

Comment 23

[[:fabrice] Fabrice Desré]

I'm not sure this code is right though. This doesn't save the profile and the some useful files in /data/local like the permissions.sqlite which is tightly coupled with the webapps that are saved.
Saving the offline cache is also missing it seems, and that will break some apps.

Comment 24

[:gerard-majax]

(In reply to Fabrice Desré [:fabrice] from comment #23)
> I'm not sure this code is right though. This doesn't save the profile and
> the some useful files in /data/local like the permissions.sqlite which is
> tightly coupled with the webapps that are saved.
> Saving the offline cache is also missing it seems, and that will break some
> apps.

Yep. When I first worked on this, I've asked about those, and I've been told it was too risky to save permissions and/or the profile. For now, it more or less suited my needs, but feel free to improve it if you have solutions :)

Comment 25

[:gerard-majax]

FYI, to avoid the reset-gaia step, I worked on a shell script that ease building and providing MAR package update for Gecko and Gaia, it's available in bug 933789.

Comment 26

[Michael Henretty [:mikehenrty][:mhenretty]]

Just to add information from the B2G mailing list[1], Mozilla China has a demo of such a tool working:

"We create a demo tool for this. It allows user to manage, backup and restore data (such as contacts, sms, pictures, music...) of a Firefox OS 1.3 phone from a host computer. For details and screenshots, see:http://1drv.ms/1fUzBbk

The tool is composed of a host application running in a Firefox desktop tab and a standalone device application.

The host application can establish a connection to the device application by either USB cable or WIFI. Once connected, use can view the data and manage them directly. The host application and the device application keeps data synchronized after user's operations.

I have a idea that we may integrate these functions into the app manager of desktop and extend the app manager to something like a device manager. User can manage or backup their phones' data through Firefox on a computer without installing other tools.

The source code is available on github:
Host application - https://github.com/MozillaOnline/pc-sync-tool
Device application - https://github.com/MozillaOnline/pc-sync-gaia-app"


1.) https://groups.google.com/forum/#!topic/mozilla.dev.b2g/VPkqmLWP4vk

Comment 27

[Alexandre Poirot [:ochameau]]

Yes, there is a quite large overlap with what we are doing around the app manager.
The adb helper addon is very similar to the binary files being shipped into this firefox addon. We would like to land it into Firefox (bug 957164) so that we can use the app manager with a device without any other dependencies other than Firefox (and a driver on windows or udev rule on linux).
Wa also have a work in progress to use Wifi to communicate with devices and also ease connecting to a device! (bug 962308)

Otherwise, the app manager is really developer-oriented, I'm not sure that's the best place to expose and promote user tools, but this is the kind of feature that is also useful for developers... and we clearly should aim to share the communication layer between Firefox and devices!