837229 - Firefox for Android should allow bookmark import by JSON file

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, enhancement)

Description

[Markus Popp]

Since Firefox Sync often has issues (e.g. https://twitter.com/mozservices/status/296732429069201408), I'm proposing an alternative method to import bookmarks.

It should be possible to import a JSON bookmark file, exported by Firefox Desktop, into Firefox Mobile. So if an outage or any other problem occurs a user still has the choice to transfer the JSON file to the mobile phone, and import it into Firefox Mobile right there, just like it works on Firefox Desktop.

Comment 1

[Markus Popp]

The sync server is down again, I lost all bookmarks on mobile and have no way to restore them.

Comment 2

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Markus Popp from comment #1)
> The sync server is down again, I lost all bookmarks on mobile and have no
> way to restore them.

Wow! This is bad news. Yeah, it's bad when the Sync system is down, but we should not drop bookmarks (or any data) when the system is down.

Is there a separate bug for the loss of data?

As for this bug, exporting/importing data sounds like a decent feature - we have been discussing backing up to dropbox, box or other online service recently - but this seems like a time consuming workaround for this particular situation.

I wonder if allowing people to switch to a different Sync instance would be a better idea? We do support private Sync servers IIRC.

Comment 3

[Richard Newman [:rnewman]]

(In reply to Mark Finkle (:mfinkle) from comment #2)
> (In reply to Markus Popp from comment #1)
> > The sync server is down again, I lost all bookmarks on mobile and have no
> > way to restore them.
> 
> Wow! This is bad news. Yeah, it's bad when the Sync system is down, but we
> should not drop bookmarks (or any data) when the system is down.

We don't. The Sync client typically doesn't touch local data at all if the server returns a non-success status code, and it certainly doesn't ever wipe data without being told to do so.

If your bookmarks are being wiped, it's because the client has successfully received a wipe command from another client (i.e., you chose Reset Sync on one of your devices).

Even if the connection gets interrupted part-way through, the most that would happen is some bookmarks get shuffled around.

For stable-state bookmarks (i.e., not a first sync), it essentially doesn't matter if the Sync server is up or not. The Sync client was designed to be fault-tolerant.

The worst case you could be seeing is that you managed to successfully sync the clients collection, processed the wipe command your other client sent, but then couldn't sync bookmarks. In that case, things will fix themselves the next time your device is able to sync.

Comment 4

[Richard Newman [:rnewman]]

Note that if the local database somehow loses data (DB corruption?), Sync typically won't know to restore it unless you issue a Reset Sync on another device.

Comment 5

[Markus Popp]

Here are some comments/ideas/suggestions from me in a related bug report:

https://bugzilla.mozilla.org/show_bug.cgi?id=849294#c7

I intended to file them as separate bug reports/enhancement requests, but maybe we can filter in advance what's doable and reasonable.

In addition to that, some improved reporting process and status checking would be appreciated. The @mozservices Twitter account didn't mention any problems at all, and also https://services.mozilla.com/status/ displayed "The Firefox Sync service is operating normally." the entire time when clearly it didn't.

Comment 6

[Markus Popp]

(In reply to Richard Newman [:rnewman] from comment #3)

> If your bookmarks are being wiped, it's because the client has successfully
> received a wipe command from another client (i.e., you chose Reset Sync on
> one of your devices).

As I see it, the problem is that the user data is wiped before the sync process is started, and when the sync process fails, the user is left with whatever has been synced until the point of failure (potentially nothing at all).

As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=849294#c7 the solution to this problem may be to run updates in a transaction. And only if the entire sync process is successful, commit the transaction to the database, so the user if left with either all updates (including wiping the database) or no updates at all.

Comment 7

[Richard Newman [:rnewman]]

(In reply to Markus Popp from comment #6)

> As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=849294#c7 the
> solution to this problem may be to run updates in a transaction. And only if
> the entire sync process is successful, commit the transaction to the
> database, so the user if left with either all updates (including wiping the
> database) or no updates at all.

That is fundamentally impossible to achieve. 

I am hoping that PICL implements two-phase writes, but true transactional syncing against the local database can't be done, for several reasons. 

And the way Sync does command processing could be slightly reordered, but not feasibly before we EOL the product. 

No better status reporting mechanism is slated for similar reasons. I encourage you to involve yourself with the PICL project if you have requirements you want to share. 

In general, Sync doesn't care if a server goes down for a day. And the services Twitter account is only occasionally updated, and typically only when there's a large-scale outage; individual server failures are a daily occurrence, and the client's fault tolerance should mask them.

Comment 8

[Markus Popp]

What is the PICL project?

I have no idea how Firefox Sync currently works. But having some fairly reasonable database background myself (I work for MySQL and I'm a MySQL certified DBA and DEV) I know that if you want to update a database in an all-or-nothing way, transactions are the way to go. AFAIK, SQLite (which AFAIK is used on the client side) supports transactions, and my educated guess is that you use a database system which supports transactions on the backend too. This makes it hard for me to imagine that it's fundamentally impossible to use transactions here.

Maybe it's difficult to update the current implementation, but designing something like Firefox Sync from scratch should definitely involve transactions. I would even consider this a perfect use case to explain somebody why transactions are necessary.

Comment 9

[Richard Newman [:rnewman]]

(In reply to Markus Popp from comment #8)
> What is the PICL project?

https://wiki.mozilla.org/Identity/AttachedServices

> I have no idea how Firefox Sync currently works. But having some fairly
> reasonable database background myself (I work for MySQL and I'm a MySQL
> certified DBA and DEV) I know that if you want to update a database in an
> all-or-nothing way, transactions are the way to go. AFAIK, SQLite (which
> AFAIK is used on the client side) supports transactions, and my educated
> guess is that you use a database system which supports transactions on the
> backend too. This makes it hard for me to imagine that it's fundamentally
> impossible to use transactions here.

Android content providers don't allow transactions.

Even if we skip through the sanctioned Android interfaces and grab a direct DB handle, and even if we assume that the shitty old version of sqlite that a given Android OEM shipped includes transaction support, a big sync takes *minutes*. We can't hold a transaction open for that long; it'll either be exclusionary (and screw the UI) or it'll collide with UI transactions. And maybe Android will screw up a long, huge transaction anyway.

And syncing involves interleaved reads and writes, where some of those writes take place *on a remote server via HTTP*. We don't have distributed transaction support, so there will always be an opportunity for inconsistency.

Even on desktop, I would have a really hard time imagining a multi-minute Places transaction that doesn't cause enormous problems.

Phrased differently: if we could do it, we'd have done it.


> Maybe it's difficult to update the current implementation, but designing
> something like Firefox Sync from scratch should definitely involve
> transactions. I would even consider this a perfect use case to explain
> somebody why transactions are necessary.

And if we could go back in time and advise Google on the design of Android 2.3, and redesign the entire architecture of the desktop browser to make long-lived transactions feasible, then we would do so.

I anticipate different structures being used to address this.

Comment 10

[Markus Popp]

If fixing Sync to avoid these kinds of problems is hard/impossible to do, perhaps the original request here should be given more priority (to allow bookmark import by JSON file).

Then there would at least be a workaround if Sync fails, so the JSON file can be uploaded to the mobile phone and bookmarks imported from there manually.

Comment 11

[Richard Newman [:rnewman]]

(In reply to Markus Popp from comment #10)

> Then there would at least be a workaround if Sync fails, so the JSON file
> can be uploaded to the mobile phone and bookmarks imported from there
> manually.

There are some tradeoffs here.

1. How many users are realistically going to use this feature? To find/create a bookmark backup, put it somewhere their phone can get it, and do the import? Is it worth spending time to build, test, and maintain, and offering in primary UI? Perhaps an add-on is a better idea.

2. Bookmark backups don't carry GUIDs (Bug 627487). A user who does this, thinking it'll speed up their first sync, is actually going to cause Sync (or PICL) to do a lot more work, which makes it more likely they'll hit a bug.

3. In my personal opinion, bookmarks are less important than history or passwords. Are we really talking about profile import? And if we are, then that really overlaps with Profile In the CLoud, no?

4. Does this mean we should also be thinking about bookmark backups on the phone? And are they going to be the same format as desktop, when the DB schema is so different? (This implies that importing Places bookmarks is actually a fair amount of work: it's essentially the same as running a sync.)

It seems like there's a different set of tradeoffs in other solutions: for example, implement Reset Sync on the device, so you can proactively say "blow away everything local and apply the network contents".

Comment 12

[Markus Popp]

So if you reject my ideas, what are your ideas to fix this problem? I'm all ears.

Comment 13

[Richard Newman [:rnewman]]

(In reply to Markus Popp from comment #12)
> So if you reject my ideas, what are your ideas to fix this problem? I'm all
> ears.

Not a rejection; just things to consider.

I'd review patches that added Reset Sync on Android, although I don't have resources to write the code.

If we get traction on Bug 627487, then some kind of bookmark import functionality might work out, but I would expect that we'd explore something more comprehensive than just tagging in a menu entry somewhere. (There have been some discussions about network backup and restore.)

But with regards to this precise bug: I'd rather figure out why you seem to be losing bookmarks, and address that directly.

Comment 14

[Lu Wei]

I am not using Firefox on desktop, but using Firefox for Android on phone. So I hope Firefox for Android could accept bookmarks from other browsers, whether json file or html file. The format should be the same as Firefox because it is Firefox derived (Serpent 52).

Comment 15

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.