Last Comment Bug 769476 - bookmarks reordered on desktop after syncing with android
: bookmarks reordered on desktop after syncing with android
Status: NEW
[sync:bookmarks]
: qawanted
Product: Android Background Services
Classification: Client Software
Component: Android Sync (show other bugs)
: unspecified
: All Android
: -- major
: ---
Assigned To: Nobody; OK to take it and work on it
:
:
Mentors:
: 765279 1204372 (view as bug list)
Depends on: 814801 bookmarksync 814331
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-28 15:18 PDT by Allison Naaktgeboren :ally
Modified: 2016-09-02 09:52 PDT (History)
17 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Android and Firefox sync logs (107.74 KB, application/octet-stream)
2012-10-13 16:44 PDT, Mihai Todor
no flags Details
Logs from home PC (21.19 KB, application/octet-stream)
2012-10-26 22:02 PDT, dragondrago
no flags Details
Logs from Android (70.59 KB, text/plain)
2012-10-26 22:03 PDT, dragondrago
no flags Details

Description Allison Naaktgeboren :ally 2012-06-28 15:18:23 PDT
There are multiple hypothesis about what the cause of this behavior is. We may have more than one issue in play, but we're guessing in the dark at this point. 

QA, Given keheldar's detailed replies, we'd like to know if you can create reliable reproductions of the jumbling issues. 

Detailed discussion (including comments from 2 sync engineers) & STR (serious props to kheldar) : https://support.mozilla.org/en-US/questions/927574

NB: To (most) users, bookmark reordering == data loss
Comment 1 Richard Newman [:rnewman] 2012-06-28 15:39:05 PDT
We are very much guessing in the dark, but there are almost certainly two issues.

An interrupted first sync -- partial data on the server -- is a very predictable source of reordering, and one that we're (broadly speaking) ignoring.

A robust client will fix positions of records to ensure a contiguous set. That should only happen if a client uploads a folder but is interrupted before it can upload the children, which is an insignificant risk on all but a first sync.

However, I've seen reordering on a mature Sync account, so a partial upload issue cannot be the whole story. I'm much more concerned about "I added an Android device to my existing stable Sync constellation, and now all of my bookmarks moved".

That *could* be caused by a partial *download* (client doesn't receive all records but isn't informed of success). If a user has thousands of bookmarks, that could happen. But ordinarily it should not… hence the guessing in the dark.
Comment 2 Richard Newman [:rnewman] 2012-06-28 16:23:00 PDT
Oh, and a difference in behavior between Android and desktop: when desktop sees partial data, it computes a new order locally, but -- as far as I can tell -- doesn't reupload a reordered folder unless something else changes (add or remove a child locally).

Android does its best to ensure that the local and remote data are identical, so when it modifies a folder (including its child array) it uploads the modified record.

Formerly, then, if you:

* Sync desktop; quit before done
* Sync some other device
* Sync desktop again

you'd see no issue, but if you:

* Sync desktop; quit before done
* Sync some other device
* Modify the partial folder on other device
* Sync desktop again

you'd see reordering. If the other device is Fennec Native, you'll see possible reordering in either case.

Mitigation: have desktop upload records in a strict partial-safe order. Probably the most partial-safe order for:

root
  Toolbar
    A
    B
    C
  Menu
    F1
      D
      E
    F2
      G
      H

is to leave folders until last, walking them leaf-to-trunk, but that imposes the maximal amount of churn on other clients (every record has to be applied then moved).

Another mitigation is to ensure that records are proximal to their folders, so they have more chance of hitting the server together.
Comment 3 Richard Newman [:rnewman] 2012-06-28 23:38:01 PDT
Walking the dog, five further late-night unvetted possibilities came to mind.

This is the one I think is most likely.

Desktop uploads bookmark records in an arbitrary order. The server will also return them in an arbitrary order -- *probably* with folders first, but who knows?

If Android Sync fails to finish storing records -- some exception is thrown, or we're killed with OOM, or whatever -- it will have a sample of records from the other device. More importantly, it either won't reach a repositioning stage, or will have to handle partial data when it does.

If it tries a sync again, it will re-download records, because our timestamp is still 0. But during the reconciling stage it will see that the local record was modified _later_ -- it made it to the database! -- and won't apply the incoming record.

We'll finish the download, and then we'll upload our own computed records, with our own computed positions.

I haven't explored this possibility yet, but it seems promising.



Four more possibilities:

Firstly, is desktop propagating position changes when they occur?

Secondly, is it using the same method to indicate position changes as we are on Android? (There's a "pos" attribute to a record, and the children array on the parent.)

Thirdly, is Sync on desktop uploading records with the correct children array?

Fourthly, is Fennec preserving the positions of records?

Imagine that a user adds a bunch of bookmarks to their toolbar. Each addition causes an instant sync. Then they rearrange the toolbar. No sync occurs. Their phone syncs, and -- for whatever reason -- touches the record on the server again. Desktop will download the toolbar record and undo the user's position changes.

Or imagine if desktop is uploading the wrong children array, or an empty children array. Android will reconstruct one and upload it. Or if Fennec mutates the bookmark order after insertion, bad things will happen.

This log from my Aurora profile, recorded as I moved a couple of toolbar entries around, might illustrate such:

1340950738001	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1340950748082	Sync.Tracker.Bookmarks	TRACE	onItemMoved: 4416
1340950748082	Sync.Tracker.Bookmarks	TRACE	Adding changed ID: toolbar,1340950748
1340950748358	Sync.SyncScheduler	DEBUG	Global Score threshold hit, triggering sync.
1340950749077	Sync.Tracker.Bookmarks	TRACE	Saving json to disk: weave/changes/bookmarks.json
1340950760491	Sync.Tracker.Bookmarks	TRACE	onItemMoved: 4584
1340950760492	Sync.Tracker.Bookmarks	TRACE	Adding changed ID: toolbar,1340950760
1340950760579	Sync.SyncScheduler	DEBUG	Global Score threshold hit, triggering sync.
1340950761493	Sync.Tracker.Bookmarks	TRACE	Saving json to disk: weave/changes/bookmarks.json
1340950762485	Sync.SyncScheduler	DEBUG	Global Score threshold hit, triggering sync.
1340950767464	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1340950767464	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1340950767981	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1340950767981	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1340950768797	Sync.ErrorHandler	DEBUG	Beginning user-triggered sync.
1340950768801	Sync.Service	DEBUG	User-Agent: Firefox/15.0a2 FxSync/1.17.0.20120624042007.
1340950768801	Sync.Service	INFO	Starting sync at 2012-06-28 23:19:28
1340950768802	Sync.SyncScheduler	DEBUG	Clearing sync triggers and the global score.
1340950768803	Sync.Service	INFO	In sync().
1340950768803	Sync.Status	INFO	Resetting Status.
1340950768803	Sync.Status	DEBUG	Status.service: success.status_ok => success.status_ok
1340950769288	Sync.Resource	DEBUG	mesg: GET success 200 https://phx-sync542.services.mozilla.com/1.1/holygoat/info/collections
1340950769288	Sync.Resource	DEBUG	GET success 200 https://phx-sync542.services.mozilla.com/1.1/holygoat/info/collections

versus:

1340951048047	Sync.Tracker.Bookmarks	TRACE	onItemAdded: 4654
1340951048048	Sync.Tracker.Bookmarks	TRACE	Adding changed ID: 0biQzJtu--x_,1340951048
1340951048048	Sync.Tracker.Bookmarks	TRACE	Adding changed ID: menu,1340951048
1340951048050	Sync.Tracker.Bookmarks	TRACE	onItemChanged: 4654, bookmarkProperties/description (anno)
1340951048155	Sync.SyncScheduler	DEBUG	Global Score threshold hit, triggering sync.
1340951048156	Sync.Service	DEBUG	User-Agent: Firefox/15.0a2 FxSync/1.17.0.20120624042007.
1340951048156	Sync.Service	INFO	Starting sync at 2012-06-28 23:24:08
1340951048158	Sync.SyncScheduler	DEBUG	Clearing sync triggers and the global score.
1340951048159	Sync.Service	INFO	In sync().

That is, repositioning a bookmark does not (unlike addition) cause an instant sync to occur.
Comment 4 Tracy Walker [:tracy] 2012-06-29 07:35:57 PDT
I am moderating testday for a couple hours this morning, then I'll attempt to repro rnewman's STR's in comment #2.  I think those are what the user described;  Their data in the cloud was mangled somehow. They *may* have shut down during initial sync. Reset from desktop fixes the problem. It sounds like the user has a significant amount of tool bar bookmarks.
Comment 5 Richard Newman [:rnewman] 2012-06-29 09:59:46 PDT
(In reply to Tracy Walker [:tracy] from comment #4)
> I am moderating testday for a couple hours this morning, then I'll attempt
> to repro rnewman's STR's in comment #2.  I think those are what the user
> described;  Their data in the cloud was mangled somehow. They *may* have
> shut down during initial sync. Reset from desktop fixes the problem. It
> sounds like the user has a significant amount of tool bar bookmarks.

Yeah, you'll need to have more items (much greater than 100) than can be uploaded and applied in a single request (100 items). A bad network connection might help.

I will see if I have time to try to write a test for my main hypothesis.
Comment 6 Richard Newman [:rnewman] 2012-07-06 10:23:22 PDT
Tracy, any news?
Comment 7 Richard Newman [:rnewman] 2012-07-06 10:53:56 PDT
*** Bug 765279 has been marked as a duplicate of this bug. ***
Comment 8 Tracy Walker [:tracy] 2012-07-09 10:23:09 PDT
Sorry, I wasn't able to get to this that Friday. I've been on vacation since.  I'll try to repro after s-c train testing.
Comment 9 Tracy Walker [:tracy] 2012-07-12 09:47:42 PDT
I've been attempting various scenarios for the better part of yesterday afternoon and this morning.  I have yet to trigger desktop BM reordering.
Comment 10 Mihai Todor 2012-10-13 10:05:24 PDT
Hi guys,

This just happened to me again today. I installed Firefox 16.0.1 (the non-beta version) on my HTC One X and I wanted to see if this bug is fixed, so I paired the device to my laptop (which is also running Firefox 16.0.1) and I let the sync do its job on the phone. Just after it finished, the order of my bookmarks on the laptop became an incomprehensible mess. I haven't looked at all of them, but it definitely messed up my bookmarks toolbar (changed folder order, moved subfolders directly into the toolbar, rearranged the bookmarks inside the folders, etc).

Perhaps I should mention that I'm also using the Xmarks plugin as an extra backup (luckily).

This is not cool, guys. I want to be able to use the sync functionality on my phone. Please let me know if I can be of further assistance. I can provide a dump of my bookmarks to let you test this issue. I can also provide whatever logs you want, if needed, just let me know.
Comment 11 Mihai Todor 2012-10-13 10:10:59 PDT
PS: I'm a C++ software engineer, so don't be shy to go into technical details, if needed.
Comment 12 Richard Newman [:rnewman] 2012-10-13 13:17:30 PDT
(In reply to Mihai Todor from comment #10)

> haven't looked at all of them, but it definitely messed up my bookmarks
> toolbar (changed folder order, moved subfolders directly into the toolbar,

…

You mean you had

  Toolbar
    Folder A
      Folder B
        Item 1
        Item 2

and ended up with

  Toolbar
    Folder B
      Item 2
      Item 1
    Folder A

?

If yes, then that's *very* odd.

There's no straightforward way for bookmarks sync on either platform to cause that to occur -- it doesn't fall-back up the hierarchy at all, instead using the Unsorted Bookmarks folder as the default location.

(That is, if somehow Folder A disappeared, Folder B would be found in Unsorted Bookmarks, not in the Toolbar.)

Related questions, if the answer is yes: did you originally create Folder B in the toolbar, then move it into Folder A? How recently did you create that folder hierarchy?


> Perhaps I should mention that I'm also using the Xmarks plugin as an extra
> backup (luckily).

For all we know, that could be the cause.


> This is not cool, guys. I want to be able to use the sync functionality on
> my phone. Please let me know if I can be of further assistance. I can
> provide a dump of my bookmarks to let you test this issue. I can also
> provide whatever logs you want, if needed, just let me know.

If you can reproduce this, Trace logs for bookmark sync on desktop (see http://bit.ly/q6fBL1) and mobile (see http://160.twinql.com/how-to-file-a-good-android-sync-bug) would be excellent. We haven't been able to reproduce any symptoms.

We have extensive tests covering the components of this system, which means that the bug is hiding in some (probably long-term) interaction between components.
Comment 13 Mihai Todor 2012-10-13 16:43:16 PDT
Hi Richard,

Thank you very much for the quick reply. It seems that you are lucky today, and I was able to replicate it (to a certain degree). Here is the setup:

1. Firefox 16.0.1 on my personal laptop (Windows 7 x64). Extensions:
- Adblock Plus 2.1.2
- Cookies Manager+ 1.5.1.1
- Element Hiding Helper for Adblock Plus 1.2.3
- Firebug 1.10.4
- FireFTP 2.0.7
- Flashblock 1.5.15.1
- gTranslate 0.9
- JavaScript Deobfuscator 1.6.3
- Live HTTP headers 0.17
- NoScript 2.5.7rc5
- Romanian spelling dictionary 1.12
- SQLite Manager 0.7.7
- Xmarks 4.1.3

2. HTC One X, Android 4.0.4 and HTC Sense 4.1, connected to the internet using WiFi, via Connectify, that is running on my laptop.

I enabled the services.sync.log.appender.file.logOnSuccess option in Firefox on my laptop.

I completely removed Firefox from my phone and installed Firefox 16.0.1 via Play Store. Afterwards, I started it.

At this point, I connected the phone via USB cable to my laptop and I started the adb logger.

Afterwards, I went to my phone and started setting up sync as instructed. I went on my laptop to introduce the required codes and, on my phone, Firefox instructed me that the sync will work in the background. I went to Settings > Accounts & sync > Firefox Sync and waited until it finished. After this, I stopped the ADB logger.

Now, I waited for sync to finish messing up my bookmarks on the laptop, which happened in about 20 seconds or so.

This time, the damage was smaller and I did not notice any updates to the folder structure anymore, just rearranged bookmarks. Earlier today I was very upset that it messed up my bookmarks again (I was a bit scared that I lost something) and I am not 100% sure if it really messed up the folder structure, but it definitely added some stuff inside the Bookmarks Toolbar (the main Bookmarks Toolbar folder), because I remember seeing it had more items than it was supposed to (I should’ve taken a dump of them for closer inspection and comparison). Perhaps it threw in there some bookmarks from my Bookmarks Menu, but I don’t remember what (my Unsorted Bookmarks folder is empty).

Anyway, on my laptop, Firefox produced 4 sync logs for you to analyze. Please find them and the android log attached. I really hope you’ll be able to get to the bottom of this issue.

If needed, I can also send you a dump of my bookmarks, but I don’t really want to upload them here for anybody to download, so please provide an email address where I should send them.

I’ve been using Firefox for quite some time now, and I really like it, so if you need more feedback or help from me, please let me know and I will be happy to assist.
Comment 14 Mihai Todor 2012-10-13 16:44:33 PDT
Created attachment 671149 [details]
Android and Firefox sync logs
Comment 15 Nick Alexander :nalexander 2012-10-13 16:45:46 PDT
> I’ve been using Firefox for quite some time now, and I really like it, so if
> you need more feedback or help from me, please let me know and I will be
> happy to assist.

Hi Mihai,

Glad you like Firefox enough to help us with this extremely frustrating bug!  I'm going to look at your logs now, and I know rnewman will too.
Comment 16 Nick Alexander :nalexander 2012-10-13 16:49:44 PDT
   6443:10-14 00:44:09.170 I/FxSync  (26905): ServerSyncStage :: Reset timestamps for org.mozilla.gecko.sync.stage.AndroidBrowserBookmarksServerSyncStage@412a0bc8
   6840:10-14 00:44:15.870 I/FxSync  (26905): BrowserRepoSession :: BEGIN: creating record -> GUID map.
   6842:10-14 00:44:15.920 I/FxSync  (26905): BrowserRepoSession :: END: creating record -> GUID map.
   9566:10-14 00:47:11.420 E/FxSync  (26905): BrowserRepoSession :: Finish called but 2949 bookmark(s) have been placed in unsorted bookmarks and not been reparented.
   9567:10-14 00:47:11.440 I/FxSync  (26905): ServerSyncStage :: Received 5000 and sent 657 records in 182.28 seconds.

What this means is that our download limit is being hit, with terrible results :(  We have considered this -- see Bug 730142 for a possible solution -- and your experience suggests we need to make this higher priority.

I'm going to let rnewman think about this before we make any concrete plans, though.
Comment 17 Mihai Todor 2012-10-13 16:57:23 PDT
@Nick: Is everybody at Mozilla working on weekends? :) I really appreciate that you're taking the time to investigate this now, but it is not a big emergency, so please take your time.

It's nice that you managed to locate a possible cause so quickly and I do confirm that I have a large number of bookmarks and a complex folder hierarchy in the Bookmarks Toolbar.

The first time I tried FF Sync on my phone (more than 6 months ago), I lost some bookmarks and it really was frustrating. I haven't reactivated it until today and, sadly, I noticed the same issue. I should be more paranoic when it comes to backups :P
Comment 18 Richard Newman [:rnewman] 2012-10-13 17:23:22 PDT
(In reply to Mihai Todor from comment #17)
> @Nick: Is everybody at Mozilla working on weekends? :)

We sometimes try not to :)

> What this means is that our download limit is being hit, with terrible
> results :(  We have considered this -- see Bug 730142 for a possible
> solution -- and your experience suggests we need to make this higher
> priority.

You're one of the 0.1% of users who have enough bookmarks that it's considered impractical to process them all. That very likely means you are *not* encountering this specific bug; you're encountering a known limitation.

A limit is a necessary evil -- if for no reason other than Android will kill the Sync process if it takes more than five minutes to complete the whole sync operation on the phone. If we get killed like that, we'll keep retrying, so you'll have your Sync client pulling down five megs of bookmarks a few times a day.

Unlike history, a partial download of bookmarks is destructive.

What's happening here is something like this:

* Desktop has uploaded some or all of its bookmarks to the server. We'll assume all for simplicity, but if it didn't there'll be additional corruption.

* Android downloads 5,000 of them, sorted by importance.

* About 3,000 of those bookmarks don't have their parent folder — that folder is outside of the set of 5,000 downloaded records. Those are moved into Unsorted Bookmarks, because there's nowhere else to put them. This could happen because we hit a limit, or it could happen because the server data is inconsistent -- a bad client could easily have uploaded junk, or we could just be seeing the server state in the middle of a sync over a bad network.

* The phone constructs the sanest arrangement it can manage, such that each record has a parent, a consecutive position within that parent, and so on.

* We then upload that structure to the server, so that other clients end up with the same set. (The alternative is that we *don't* upload records, but that means the contents of the phone won't get positioned correctly in the merge case, amongst other things.)  You can think of this as the well-behaved client 'fixing' the corrupt server.

* The desktop downloads the changed records and repositions/relocates accordingly.

There are a few complex things we can do to try to guard against this case, but no perfect solution.

* Remove the limit, and hope that your phone successfully syncs before it consumes your entire mobile data allowance.

* Increase the limit, and orchestrate a sequence of syncs so as to get a whole 5-minute run at each of the big collections.

* Try to track which folders haven't been reconciled locally, and don't upload those which have only been repositioned as a consequence of partial data. Maybe we'll get fixed up in the future.

* Batch downloads (Bug 730142), which will require some kind of similar tracking.

* Introduce some kind of state which will allow us to stretch a multi-part sync over multiple Android syncs.

* Build a local cache of server data against which we do the actual bookmark syncing (two-phase sync). We wouldn't do any syncing against a part of the folder tree until that tree was complete.


The combination of batch downloads and two-phase sync is probably the only way to 'solve' this problem, at the cost of delayed gratification (you wouldn't see any bookmarks in Fennec until we had all of them) and increased memory and CPU cost.

If we simply implement batching _and refuse to upload or reposition until we've downloaded everything_, we get an 80% solution. Flawed, but would work in the general case.


Unfortunately for you, we won't have time to work on this until the new year. The best advice I can give you is to begin by turning off bookmark sync, because it's guaranteed to not work for you. There are a few things you could do:

* Find a way to import your bookmarks to Fennec. If the bookmark structure is initially the same on both devices, no corruption will occur.

* Use a different tool for handling bookmarks — pinboard.in, for example. The Firefox bookmark UI (and data store, really) aren't optimized for users who have thousands or tens of thousands of bookmarks, and even if this whole thing would function at all, the Sync client on each end is doing a huge amount of work to manipulate a model of that bookmark structure each time you sync. Better to just store the data somewhere central.
Comment 19 Mihai Todor 2012-10-13 18:30:43 PDT
(In reply to Richard Newman [:rnewman] from comment #18)

Wow, thanks for the highly detailed reply, Richard. I was not expecting such rich feedback.

While I am not familiar with the Firefox internals, here are my 2 cents, based on my experience with large scale applications:

----
* Build a local cache of server data against which we do the actual bookmark syncing (two-phase sync). We wouldn't do any syncing against a part of the folder tree until that tree was complete.
----

This seems like the sanest solution to me, and, to be honest, I fail to see why would anyone want to complicate things so much, considering the rest of the solutions that you mentioned above.

I just exported all my bookmarks as JSON. The file has ~700KB and it contains at least 3 years of bookmarks (2618 items without the associated folders to be more precise - a surprisingly small number, I would say). If I archive it as zip without any special compression, I get ~160KB.

Now, I think we can all agree that downloading a few megabytes (in the worse case I can imagine) on a mobile phone won't cause major headaches for anybody. Actually, I download & upload pictures much larger than that on a daily basis.

At this point, I can think of only one pain in the ass: what if the user already has some bookmarks on his mobile phone and wants to merge them with the ones received from the server? I would answer this with one simple two choice dialogue: either loose 'em or let the phone churn on bookmark tree merging for a few minutes in a background thread. If this is such a big issue, the phone could upload the partial list to the server and ask it to perform merging (I can imagine some cute DDoS attacks in this scenario), but I doubt this is really needed, even on older phones.

To make things even faster, why not send the entire bookmark store to the client (phone) as a native Firefox store (SQLite database, probably)? If properly archived, I doubt it will take up much more space than the JSON string and the client won't need to parse it.

For the sake of argument, let's say that the user's internet speed is really slow and he has a couple of MB of bookmarks to sync and an older phone. Why not just warn him? "Hey, dude, it's going to take one hour to fetch all your stuff." Also, I can imagine some scalability issues with your servers, but I guess that's not the big problem here.

Still, as an easy-to-implement alternative, I propose to have a "power-user setting" somewhere that enables bookmark freaks like me to get read-only bookmark access on certain devices. I find it pointless most of the times to bookmark stuff from my phone, considering that the mobile version doesn't provide any advanced bookmark management ( https://support.mozilla.org/en-US/questions/828180 ). This could also make sense as a sanity feature, if I sync my bookmarks on a work / public computer, where multiple people have access.

I hope my feedback helps and I'll wait patiently for you to come up with a solution for this, even if it will take until next year :) I don't want to mess around with another bookmark manager, like pinboard.in, since it doesn't have a nice integration in the browser...
Comment 20 Richard Newman [:rnewman] 2012-10-13 19:50:00 PDT
> I just exported all my bookmarks as JSON. The file has ~700KB and it
> contains at least 3 years of bookmarks (2618 items without the associated
> folders to be more precise - a surprisingly small number, I would say). If I
> archive it as zip without any special compression, I get ~160KB.

Bear in mind that Sync's records are structured and encrypted (which also means they don't compress well). My 500 bookmarks occupy about 500KB on the server.

Out of interest, how often do you delete bookmarks, and how long have you had Sync enabled on your desktop? And how many folders do you have?

> Now, I think we can all agree that downloading a few megabytes (in the worse
> case I can imagine) on a mobile phone won't cause major headaches for
> anybody. Actually, I download & upload pictures much larger than that on a
> daily basis.

That's not strictly true; it's a little more complex than that. For example, we support ARMv6 devices on flaky networks, and there are some overheads to consider, but moving on…

> At this point, I can think of only one pain in the ass: what if the user
> already has some bookmarks on his mobile phone and wants to merge them with
> the ones received from the server?

Which is always the case, with the exception of a deliberate Reset Sync. There's always merging to do. (The code paths don't even differ after a reset's initial wipe.)

> I would answer this with one simple two
> choice dialogue: either loose 'em or let the phone churn on bookmark tree
> merging for a few minutes in a background thread.

It's always a background thread, but consider that these are often phones with very limited processor and memory capacities, and Android allows us only 8MB of heap.

> If this is such a big
> issue, the phone could upload the partial list to the server and ask it to
> perform merging (I can imagine some cute DDoS attacks in this scenario), but
> I doubt this is really needed, even on older phones.

Not possible; Sync uses client-side encryption. The server doesn't have the cleartext, and IMO should not.

(Not to mention how much slower upload is from a mobile phone.)

> To make things even faster, why not send the entire bookmark store to the
> client (phone) as a native Firefox store (SQLite database, probably)? If
> properly archived, I doubt it will take up much more space than the JSON
> string and the client won't need to parse it.

Firefox for Android uses a completely different storage backend, and a different version of SQLite. (Ignoring for a moment that this would require clients to maintain snapshots and store them somewhere, and that the protocol would have to be changed to support snapshotting, and that someone would have to pay for the storage….)

> For the sake of argument, let's say that the user's internet speed is really
> slow and he has a couple of MB of bookmarks to sync and an older phone. Why
> not just warn him? "Hey, dude, it's going to take one hour to fetch all your
> stuff." Also, I can imagine some scalability issues with your servers, but I
> guess that's not the big problem here.

Mostly that background sync operations don't have UI to do so, and we don't really know how long it'll take until we're part-way through.

> Still, as an easy-to-implement alternative, I propose to have a "power-user
> setting" somewhere that enables bookmark freaks like me to get read-only
> bookmark access on certain devices.

Honestly, doing this would pretty much entail solving the problem we're discussing -- incremental download of large bookmark collections with safe reconciling. All of the same persistence requirements (that is, if I see a folder record before I've seen all of its children, I need to keep it around to use in later parenting operations) still apply.

There's no way to offload work to the server, because your data is stored as opaque encrypted blobs; we can't lazily query for a list of bookmarks by folder structure, for example.

Beyond that, we tend to eschew adding configuration UI for this kind of thing, because if it's visible and obvious it confuses users, and if it's not visible and obvious it might as well not exist, because people will only discover it when they've already encountered a problem.

> I hope my feedback helps and I'll wait patiently for you to come up with a
> solution for this, even if it will take until next year :) I don't want to
> mess around with another bookmark manager, like pinboard.in, since it
> doesn't have a nice integration in the browser...

Fair enough!

Thanks for taking the time to write, and for your patience.
Comment 21 Mihai Todor 2012-10-13 20:36:08 PDT
(In reply to Richard Newman [:rnewman] from comment #20)

Oh, yeah, now I remember reading that Firefox does encrypt bookmarks. That is one nice privacy feature, I have to say, and I am sure that some people do appreciate it, but, indeed, this complicates many things. Incremental encrypted backups are hard to get right... On the other hand, I find it strange that Mozilla doesn't want to sell at least some aggregated bookmarks data to 3rd parties. I think some people would want to pay money for that, which could perhaps balance the costs of the sync servers. Are the users that upset about getting their bookmarks read by others? :D

I usually have no reason to delete bookmarks, but sometimes I do change the folder structure, like moving folders from one place to the other or adding new ones. Initially, I was using only Xmarks, but a year ago (I think) I added Firefox sync as well, just to try it out, and I left it running as a backup service, just in case Xmarks fails. I think I have around 350 folders (and some are even 5 or more levels deep), based on the number of </H3> tags in the HTML export. I have no idea how to count them otherwise and I'm too lazy to write a parser now...

> Beyond that, we tend to eschew adding configuration UI for this kind of
> thing, because if it's visible and obvious it confuses users, and if it's
> not visible and obvious it might as well not exist, because people will only
> discover it when they've already encountered a problem.

Well, yeah, but data loss is usually not acceptable, so there should be some workaround, even for freaks who want one million bookmarks :) As a bare minimum, the client should be aware that it hasn't received a complete snapshot from the server, in which case it should either keep on trying or give up, but most definitely it should not try to rewrite the server data with its own. Since the bookmarks on the Android device have no structure (it doesn't allow you to create any folders & stuff, right?), then they should just be thrown somewhere in an "unsorted bookmarks" folder (maybe a new folder) during the first sync. Still, I think that you're giving a bit too much freedom to the client. By imposing a few sane constraints, some of the problems might go away.

Anyway, thank you very much for your time and good luck hunting bugs. I used to work for tech support myself, so I know what it's like...
Comment 22 Richard Newman [:rnewman] 2012-10-13 21:36:56 PDT
> Oh, yeah, now I remember reading that Firefox does encrypt bookmarks. That
> is one nice privacy feature, I have to say, and I am sure that some people
> do appreciate it, but, indeed, this complicates many things. Incremental
> encrypted backups are hard to get right... On the other hand, I find it
> strange that Mozilla doesn't want to sell at least some aggregated bookmarks
> data to 3rd parties.

We have a very strong set of privacy principles, outlined here:

http://www.mozilla.org/en-US/privacy/#our-principles

On a much more concrete level: everyone I know working on Sync — ops, server, or clients — has a very strong *personal* desire to do the right thing by our users (which includes ourselves!). Working on a not-for-profit organization like Mozilla has benefits when it comes to sleeping at night. :)


> I think some people would want to pay money for that,
> which could perhaps balance the costs of the sync servers. Are the users
> that upset about getting their bookmarks read by others? :D

You would be surprised at the kinds of things people store in their bookmarks!

 
> I usually have no reason to delete bookmarks, but sometimes I do change the
> folder structure, like moving folders from one place to the other or adding
> new ones. Initially, I was using only Xmarks, but a year ago (I think) I
> added Firefox sync as well, just to try it out, and I left it running as a
> backup service, just in case Xmarks fails. I think I have around 350 folders
> (and some are even 5 or more levels deep), based on the number of </H3> tags
> in the HTML export. I have no idea how to count them otherwise and I'm too
> lazy to write a parser now...

Interesting. Given that you should have no more than 3,000 records (2618 bookmarks, 350 folders), I am surprised that Android Sync was able to download 5,000.

Would you mind running some JS for me to see what counts you have?

Open a privileged web console (new tab, about:config, Tools > Web Console), and run this:

  https://developer.mozilla.org/en-US/docs/Firefox_Sync/Code_snippets#Count_types_of_bookmark_records

It'll print something like

[21:34:49.173] Deleted: 34, {"folder":37,"bookmark":561,"separator":2,"query":3}


> Since the bookmarks on the Android device have no structure
> (it doesn't allow you to create any folders & stuff, right?), then they
> should just be thrown somewhere in an "unsorted bookmarks" folder (maybe a
> new folder) during the first sync.

That's exactly what happens, but consider that even Android phones can — and do — have pre-existing bookmark structures, perhaps from previous Sync accounts or profile migration from XUL Fennec, and server data isn't trustworthy due to a whole pile of reasons (server bugs, client bugs, lack of transactionality in the protocol, etc.).

Syncing is a really hairy business, even disregarding performance, compatibility, error handling, and space constraints.

You just happen to have hit one of the intersection points of design constraints, schedule constraints, and unusual user behavior!


> Still, I think that you're giving a bit
> too much freedom to the client. By imposing a few sane constraints, some of
> the problems might go away.

There are lots and lots of things we would do differently if we had a couple of years to build a brand-new Sync protocol! (And believe me, this is way better than it was in 2010!)

Alas, we work with what we've got. :D
Comment 23 Mihai Todor 2012-10-14 07:24:38 PDT
(In reply to Richard Newman [:rnewman] from comment #22)

> Working on a not-for-profit organization like Mozilla has benefits when it comes
> to sleeping at night. :)

If you do get to sleep, that is ;)

> Would you mind running some JS for me to see what counts you have?

It's my pleasure (and many thanks for the cool code snippets. I had no idea about that page). Here's the output:

[16:13:43.713] Deleted: 0, {"bookmark":2623,"folder":339,"query":2,"separator":3} @ Web Console:24

> Syncing is a really hairy business, even disregarding performance, compatibility, error 
> handling, and space constraints.

Yeah, I agree now, based on this conversation. And it is starting to sound as an interesting challenge to design and enhance it. I'd love to work on such a project :)

> Alas, we work with what we've got.

It could have been worse. One year ago, I was working on fixing bugs in a big backend application for investment banks, designed way back in '93 - '95. Instead of refactoring it over the years, they preferred to just keep the existing core code written in C and just patch it. You don't want to know how it ended up looking...
Comment 24 Nick Alexander :nalexander 2012-10-25 08:18:08 PDT
> > Syncing is a really hairy business, even disregarding performance, compatibility, error 
> > handling, and space constraints.
> 
> Yeah, I agree now, based on this conversation. And it is starting to sound
> as an interesting challenge to design and enhance it. I'd love to work on
> such a project :)

Hi Mihai, I just realized I didn't respond to this.  We'd love your help!  You should definitely check out

http://whatcanidoformozilla.org/

for a broad view of what parts of the Mozilla project you might be interested in.

If you want to contribute to Android Sync, I'm working on some getting started documentation for the next few days.  At the moment, you can look at

https://wiki.mozilla.org/Services/NativeSync

for some scattered technical information about getting a build up and

https://bugzilla.mozilla.org/buglist.cgi?classification=Client Software;query_format=advanced;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;component=Android Sync;product=Mozilla Services

(ugh!) for a list of open Android Sync tickets.  A few are marked good first bug or mentor=nalexander and I am trying to make it easier to contribute to Android Sync as a whole.  Sorry for the delay responding.
Comment 25 Peter Lairo 2012-10-25 09:12:10 PDT
My workaround:

1. Spend 2 hours fixing the order of my bookmarks and bookmark folders.
2. Sync all bookmarks to my two Android devices (Nexus S and Nexus 7).
3. Turn OFF sync on my Android devices.
4. Live with outdated bookmarks on my Android devices until this bug is fixed.


(In reply to Richard Newman [:rnewman] from comment #22)
> Interesting. Given that you should have no more than 3,000 records (2618
> bookmarks, 350 folders)

Is that the limit? Is there a way for a *user* to see how close they are. I've been using Firefox since 1999 and probably have close to that many bookmarks/folders.

> https://developer.mozilla.org/en-US/docs/Firefox_Sync/
> Code_snippets#Count_types_of_bookmark_records

This give me: [18:07:00.851] SyntaxError: syntax error

> You just happen to have hit one of the intersection points of design
> constraints, schedule constraints, and unusual user behavior!

I don't think that having many bookmarks and folders should be considered "unusual user behavior", especially since Firefox has been around for years, and longtime users *will* have many bookmarks (and a smartphone).

> There are lots and lots of things we would do differently if we had a couple
> of years to build a brand-new Sync protocol!

Are you saying this "major" bug will not be fixed?
Comment 26 Mihai Todor 2012-10-25 09:19:53 PDT
@Nick: Thanks for the feedback. I will look more into the links you provided, and, hopefully, I will start contributing some code in my spare time. I was actually looking for such a challenge.

@Peter: I think you missed the part about running that script in a *privileged web console*. Basically, you open about:config or about:about or something similar and there you will be allowed to run that code in the console. Regarding Richard's comments, if you have the patience to go through this whole thread, I think you will understand that  the design of this feature is not perfect, but it sure is nice and it respects privacy. I am sure that they will improve it in time and, hopefully, they can find some time to look into the code to see if my logs actually uncover some bugs.

Regards,

Mihai
Comment 27 Peter Lairo 2012-10-25 09:34:49 PDT
(In reply to Peter Lairo from comment #25)
> My workaround:
> 1. Spend 2 hours fixing the order of my bookmarks and bookmark folders.

30 minutes into the tedious process of restoring my folder order, I see that many (almost all) of the folders that were erroneously moved into the Unsorted Bookmarks folder due to this bug are now *empty*! This is a serious dataloss issue. I've lost many bookmarks that I have collected during the past ten years - many of them very important to me.

Guys, this needs to be fixed ASAP. Some good suggestions have been made that do not require "a couple of years to build a brand-new Sync protocol", including putting up a visual indicator that a (initial) sync is in progress, and warning users when they exit while a sync is in progress that dataloss might (will) occur.
Comment 28 Peter Lairo 2012-10-25 09:40:43 PDT
(In reply to Mihai Todor from comment #26)
> @Peter: I think you missed the part about running that script in a
> *privileged web console*. Basically, you open about:config or about:about or
> something similar and there you will be allowed to run that code in the
> console. 

I did run the script in the about:config tab.

STR:
1. Open new tab and load about:config
2. Press Firefox/Nightly button / Web Developer / Web Console
3. At the bottom of the screen, after the ">", I pasted:
https://developer.mozilla.org/en-US/docs/Firefox_Sync/Code_snippets#Count_types_of_bookmark_records

Result: 
[18:38:47.154] SyntaxError: syntax error

I'm using Firefox nightly "19.0a1 (2012-10-24)" in Windows 7
Comment 29 Richard Newman [:rnewman] 2012-10-25 09:47:05 PDT
(In reply to Peter Lairo from comment #25)
> My workaround:
> 
> 1. Spend 2 hours fixing the order of my bookmarks and bookmark folders.
> 2. Sync all bookmarks to my two Android devices (Nexus S and Nexus 7).
> 3. Turn OFF sync on my Android devices.

This will not work if you have more than five thousand bookmarks and folders. If you have fewer than that, then you might as well leave Sync turned on.


> Is that the limit? Is there a way for a *user* to see how close they are.
> I've been using Firefox since 1999 and probably have close to that many
> bookmarks/folders.

Sync on Android will download no more than 5,000 items. If you have more than 5,000 bookmarks and folders, you'll see some jumbling. This was a deliberate tradeoff: the alternative is bad for different reasons.


> I don't think that having many bookmarks and folders should be considered
> "unusual user behavior", especially since Firefox has been around for years,
> and longtime users *will* have many bookmarks (and a smartphone).

Statistics says otherwise, but I should be clear: this is not a question of what we would *like* to do.


> > There are lots and lots of things we would do differently if we had a couple
> > of years to build a brand-new Sync protocol!
> 
> Are you saying this "major" bug will not be fixed?

You should probably read my thorough explanation in Comment 18.
Comment 30 Richard Newman [:rnewman] 2012-10-25 09:50:57 PDT
(In reply to Peter Lairo from comment #28)

> https://developer.mozilla.org/en-US/docs/Firefox_Sync/
> Code_snippets#Count_types_of_bookmark_records

Those code snippets are version-dependent, because they rely on code internals. They won't work on Nightly right now.
Comment 31 Peter Lairo 2012-10-25 10:18:52 PDT
(In reply to Richard Newman [:rnewman] from comment #30)
> (In reply to Peter Lairo from comment #28)
> 
> > https://developer.mozilla.org/en-US/docs/Firefox_Sync/
> > Code_snippets#Count_types_of_bookmark_records
> 
> Those code snippets are version-dependent, because they rely on code
> internals. They won't work on Nightly right now.

I just tried it with Firefox 16.0.1 and still get a syntax error.
Comment 32 Peter Lairo 2012-10-25 10:40:23 PDT
(In reply to Richard Newman [:rnewman] from comment #29)
> Sync on Android will download no more than 5,000 items. If you have more
> than 5,000 bookmarks and folders, you'll see some jumbling. This was a
> deliberate tradeoff: the alternative is bad for different reasons.

What could possibly be worse than losing bookmarks? If a user has many bookmarks, he's likely a geek. If he's a geek (like me), he likely has WLAN at home (or a "flat rate" data-plan[1]). If he has WLAN at home, he can sync *any* number of bookmarks on the initial sync.

[1] Most flat-rate plans in Germany are limited to x MB per month (mine is 500 MB/month for 10 €/m)). Then the speed is throttled to a nearly useless speed.

> You should probably read my thorough explanation in Comment 18.

Thanks for the pointer. I did. Below are my hopes:

(In reply to Richard Newman [:rnewman] from comment #18)
> * Remove the limit, and hope that your phone successfully syncs before it
> consumes your entire mobile data allowance.

Please do this!!! 

If need be, put up a dialog with the estimated data size and/or recommending WLAN and/or flat rate data plan.

> * Increase the limit, and orchestrate a sequence of syncs so as to get a
> whole 5-minute run at each of the big collections.

Yes, increase the limit (if removing the limit is too bold).

> * Batch downloads (Bug 730142), which will require some kind of similar
> tracking.

The total data amount of the batches is the same as for "no limit", so why bother?

> * Build a local cache of server data against which we do the actual bookmark
> syncing (two-phase sync). We wouldn't do any syncing against a part of the
> folder tree until that tree was complete.

Wouldn't that double the data volume (download all into cache + upload all onto server)?

> The combination of batch downloads and two-phase sync is probably the only
> way to 'solve' this problem, at the cost of delayed gratification (you
> wouldn't see any bookmarks in Fennec until we had all of them) and increased
> memory and CPU cost.

Batch + two-phase is fine by me (after "no limit"). 1000x better than dataloss.

> If we simply implement batching _and refuse to upload or reposition until
> we've downloaded everything_, we get an 80% solution. Flawed, but would work
> in the general case.

That's 99% better than dataloss for the 0.1% of Firefox users[2]. ;-)

[2] That's nearly a half-a-million users (450 million * 0.1%).
Comment 33 Richard Newman [:rnewman] 2012-10-25 12:35:21 PDT
(In reply to Peter Lairo from comment #32)
> (In reply to Richard Newman [:rnewman] from comment #29)
> > Sync on Android will download no more than 5,000 items. If you have more
> > than 5,000 bookmarks and folders, you'll see some jumbling. This was a
> > deliberate tradeoff: the alternative is bad for different reasons.
> 
> What could possibly be worse than losing bookmarks?

This limitation doesn't lose bookmarks: it loses *structure*. There's nothing in Sync on desktop or Android that will result in a bookmark being deleted unless we process a record that describes an explicit deletion. The worst that can happen is that some structure is moved to Unsorted Bookmarks.

This was better than not shipping bookmark sync.


> If he has WLAN at home, he can sync
> *any* number of bookmarks on the initial sync.

That's not true. It takes some amount of time, usually 80-400msec, to perform a bookmark insertion, let alone the rest of the time spent on a sync.

Out of the 300,000msec we have to work with, and assuming *zero* time spent on *anything* apart from database writes, and 80msec per write, we can insert no more than 3,750 bookmarks before Android kills our process. On a low-end Samsung, times are worse; on a top-end dual-core phone it'll be better.

Still think we can just remove the limit? I referred to 5,000 as a "sanity limit" for a reason.

> > * Batch downloads (Bug 730142), which will require some kind of similar
> > tracking.
> 
> The total data amount of the batches is the same as for "no limit", so why
> bother?

I don't think you understand the design constraints of this problem, or how batching applies.


> > * Build a local cache of server data against which we do the actual bookmark
> > syncing (two-phase sync). We wouldn't do any syncing against a part of the
> > folder tree until that tree was complete.
> 
> Wouldn't that double the data volume (download all into cache + upload all
> onto server)?

No.


> Batch + two-phase is fine by me (after "no limit"). 1000x better than
> dataloss.

Software engineering is the process of turning requirements and schedule constraints into software that maximizes the satisfaction of the stakeholders.

I think earlier comments in this thread explain why we don't already implement a complex but thorough solution, despite knowing that it would fix this problem (at the risk of introducing others).


> > If we simply implement batching _and refuse to upload or reposition until
> > we've downloaded everything_, we get an 80% solution. Flawed, but would work
> > in the general case.
> 
> That's 99% better than dataloss for the 0.1% of Firefox users[2]. ;-)

But also takes more time to build, and introduces complexity.

I know it's tempting, but please refrain from armchair quarterbacking this situation.
Comment 34 Mihai Todor 2012-10-25 13:03:31 PDT
Guys, let's stick to constructive arguments, since this complex model won't get rewritten in one day (yeah, it's much more complex than it seems) and let's focus on the fact that my logs revealed an inconsistency. I am really curious if it's really a bug and we should just provide all the required logs to help with the investigation, instead of complaining.
Comment 35 dragondrago 2012-10-26 22:02:59 PDT
Created attachment 675799 [details]
Logs from home PC
Comment 36 dragondrago 2012-10-26 22:03:48 PDT
Created attachment 675800 [details]
Logs from Android
Comment 37 dragondrago 2012-10-26 22:21:47 PDT
(My english is not very good so if u need extra explanation - write me). I have 2  computers - 1 laptop in work(Toshiba with Windows 7), 1 desktop PC (Windows 7) at home. Usually I update Firefox wit last versions. In last 10 days I obtained Samsung Galaxy Tab 2 (Android 4.0.3) and I installed Firefox on it before few days. One day at work I noticed that bookmarks are reordered (usually there are folders first and then unsorted bookmarks) - folders and unsoreted bookmarks were jumbled. i tried to find solution but meanwhile I see that some bookmarks are missing (fore sure some of folders were empty). I found that I can restore them with Xmarks add-on and restore version before 2 days. Then after sync all devices receive correct bookmarks. On same day or on next i made logs of home PC (see attached files) and today logs from Android (see attached file). Yestarday i saw that again I have problem on home PC. Now i see that some of bookmarks are duplicated (they are in folders and in unsorted). Total number of bookamrks is 88.

So in few words - I am not sure when and how it is happened.
Comment 38 Peter Lairo 2012-10-27 08:02:03 PDT
(In reply to Richard Newman [:rnewman] from comment #33)
> (In reply to Peter Lairo from comment #32)
> > 
> > What could possibly be worse than losing bookmarks?
> 
> This limitation doesn't lose bookmarks: it loses *structure*. 

I place bookmarks into folders (sometimes without looking at the URL or the title - but who can remember hundreds of URLs & titles weeks later anyways). I know I will *need* the bookmarks later. For instance, I created folder for my company called "good ideas". In this folder, I collected about 20 bookmarks over a period of 6 months. It contained bookmarks from all over the place and covering a wide range of topics related to my company. With this bug, the folder and all it contents got moved into the Unsorted Bookmarks folder; my "Good Ideas" folder is empty, and the bookmarks are *scattered* among a few thousand other bookmarks I had collected (and usually placed into folders!). There is *no way* I am able to find and identify the bookmarks I had placed into my "Good Ideas" folder. The same goes for about 50 other (now empty) folders. 

So, for all intents and purposes, this bug does "lose" bookmarks! Once that sinks in, I think you will realize how frustrated (angry) some four-hundred-and-fifty-thousand Firefox users are right now.

> Out of the 300,000msec we have to work with, and assuming *zero* time spent
> on *anything* apart from database writes, and 80msec per write, we can
> insert no more than 3,750 bookmarks before Android kills our process. On a
> low-end Samsung, times are worse; on a top-end dual-core phone it'll be
> better.

So it's an Android limit. OK. Thanks for explaining it.

> > Batch + two-phase is fine by me (after "no limit"). 1000x better than
> > dataloss.
> 
> Software engineering is the process of turning requirements and schedule
> constraints into software that maximizes the satisfaction of the
> stakeholders.

OK. Then do it! ;-)

> I know it's tempting, but please refrain from armchair quarterbacking this
> situation.

Well, at least this "armchair quarterbacking" made you aware that this bug is, from the users' perspective, a *dataloss* issue.
Comment 39 Mihai Todor 2012-11-23 16:26:11 PST
(In reply to Richard Newman [:rnewman] from comment #22)
> Interesting. Given that you should have no more than 3,000 records (2618
> bookmarks, 350 folders), I am surprised that Android Sync was able to
> download 5,000.
> 
> Would you mind running some JS for me to see what counts you have?

Dear Richard,

I have provided feedback for this request in comment 23 and I am curious if you were able to reconcile it with the logs I provided. Is this another bug? Do you need any other logs / feedback / help to sort this out?

Thanks!
Mihai
Comment 40 Nick Alexander :nalexander 2012-11-23 16:34:11 PST
(In reply to Mihai Todor from comment #39)
> (In reply to Richard Newman [:rnewman] from comment #22)
> > Interesting. Given that you should have no more than 3,000 records (2618
> > bookmarks, 350 folders), I am surprised that Android Sync was able to
> > download 5,000.
> > 
> > Would you mind running some JS for me to see what counts you have?
> 
> Dear Richard,
> 
> I have provided feedback for this request in comment 23 and I am curious if
> you were able to reconcile it with the logs I provided. Is this another bug?
> Do you need any other logs / feedback / help to sort this out?

Hi Mihai, it's more likely there is an error in the record counting/printing Android Sync logic.  I have a few minutes now, I'll investigate.
Comment 41 Mihai Todor 2012-11-23 16:40:52 PST
Thanks Richard! Also, I've noticed that there exists a "Places Maintenance" extension in some of the comments for one of the related bugs and, out of curiosity, I've installed it (current version, 1.3). Here's the output for "Deep Check":

> Integrity check
+ The database is sane
> Coherence check
+ The database is coherent
> Orphans expiration
+ Database cleaned up
> Vacuum
Initial database size is 30720 KiB
+ The database has been vacuumed
Final database size is 30720 KiB
> Statistics
Database size is 30720 KiB
user_version is 21
page_size is 32768
cache_size is 128
journal_mode is wal
synchronous is 1
History can store a maximum of 61107 unique pages
Table moz_places has 42694 records
Table moz_historyvisits has 57885 records
Table moz_inputhistory has 208 records
Table moz_hosts has 4011 records
Table moz_bookmarks has 3184 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 2 records
Table sqlite_sequence has 1 records
Table moz_favicons has 4431 records
Table moz_anno_attributes has 10 records
Table moz_annos has 983 records
Table moz_items_annos has 41 records
Table sqlite_stat1 has 15 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_hosts_1
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex

I did add more bookmarks in the mean time, but it looks like I'm still far away from 5000. Also, I have the History box unchecked in the Sync options, so I would say that those records don't count, right?
Comment 42 Richard Newman [:rnewman] 2012-11-23 17:05:16 PST
(In reply to Mihai Todor from comment #41)

Thanks for taking a look.

> I did add more bookmarks in the mean time, but it looks like I'm still far
> away from 5000.

Yup.

Note that you probably don't need to hit 5,000 for there to be a problem; you can likely also suffer in some cases if there's an unreliable network.

This is hypothetical; I haven't checked the code to see if this problem can occur, but it's an example of the intrinsic flaw in one-pass syncing of structured data.

Say we download a bunch of bookmarks onto your phone (the first 50 of 3,000, perhaps).

There's no parent folder for them yet (that's hanging around in position 2,000), so we put them in Unsorted Bookmarks -- they have to live somewhere. We can't just batch them in memory; 2,000 records is too much for the amount of heap we have to work with.

They get a modified time that's now later than their time on the server (because the local record was modified; that's what it means). We also bump the modified time of Unsorted Bookmarks, because we need to upload a record for that.

Now AT&T drops your data connection. The sync fails. We don't advance the sync timestamp, so we'll sync again a little later.

Next time we download the same records, they appear to be older than the ones we've stored locally -- the local ones 'win'. When the sync finishes, we've decided that those records must have been locally moved into Unsorted Bookmarks *after* they were originally moved into their correct parent folder, and we propagate that fact to your other devices.

There is no difference in the data model between this scenario and you moving a bookmark on your desktop then moving it on your phone: they both involve reconciling two records with different modified times.

There are other knock-on effects, too: we need to compose a consistent folder order for our partially filled Android bookmarks, and that will end up being propagated to other devices. Those other devices will try to figure out a sane order, given that they know of more bookmarks than the Android device, and you'll see jumbling.

We don't have transactional storage on Android (at least, not from Sync's side of the database), so we can't rollback if there's a failure. And in any case, transactionality isn't a good idea when you've got five minutes to work and the user is trying to browse! You'll get collisions all the time.

The only solution here is to restrict bidirectional syncing to a failure-proof local operation, limiting the failure-prone server interaction to an operation on a non-canonical cache. Anything else is -- like the current approach -- an approximation that works most of the time, with some failure modes.


> Also, I have the History box unchecked in the Sync options,
> so I would say that those records don't count, right?

History and bookmarks are unrelated in this instance, regardless.
Comment 43 Mihai Todor 2012-11-23 17:54:32 PST
Uh, that's really nasty :( Still, during the experiments, my phone was connected via WiFi to a home network that I like to consider very reliable. Nevertheless, since there are so many failure points in this model, it's really hard to take a guess at the root cause... Good luck with the new version.
Comment 44 Richard Newman [:rnewman] 2012-11-23 21:11:42 PST
(In reply to Mihai Todor from comment #43)
> Still, during the experiments, my phone was
> connected via WiFi to a home network that I like to consider very reliable.

Could be all manner of things; perhaps Android decided to interrupt the sync, perhaps the phone ran out of memory. We don't get a chance to do any kind of cleanup or recovery, so any intermediate changes we made just… stick.

> Nevertheless, since there are so many failure points in this model, it's
> really hard to take a guess at the root cause... Good luck with the new
> version.

Thanks!
Comment 45 Peter Lairo 2012-12-01 19:56:42 PST
FWIW: I had 7,000 bookmarks. After running the CheckPlaces add-on[1], I was able to reduce the number down to 3,300 bookmarks. That's still too many for Firefox's sync. My "solution" is that I was forced to turn off sync on *all* my Firefox devices (computers, laptop, tablet, and smartphone).

BTW: It seems mozilla.org has managed to **** off and drive away the developer of this very useful add-on: see http://www.andyhalford.com/

[1] http://mac.softpedia.com/get/Internet-Utilities/CheckPlaces.shtml
Comment 46 Richard Newman [:rnewman] 2012-12-01 21:40:52 PST
(In reply to Peter Lairo from comment #45)
> FWIW: I had 7,000 bookmarks. After running the CheckPlaces add-on[1], I was
> able to reduce the number down to 3,300 bookmarks. That's still too many for
> Firefox's sync.

Not strictly true, but you're certainly more vulnerable to some classes of problems than the typical 10-20 bookmark user.

> My "solution" is that I was forced to turn off sync on *all*
> my Firefox devices (computers, laptop, tablet, and smartphone).

You know, you could just turn off bookmark sync if you're not confident in its operation; you'll still get history, passwords, prefs, etc.

Bookmarks is by far the most complicated engine (with add-on sync a close second), and the only one which involves cross-record dependencies, which is the part that makes it vulnerable to failure.

Just uncheck "bookmarks" in Sync preferences on any device.

> BTW: It seems mozilla.org has managed to **** off and drive away the
> developer of this very useful add-on: see http://www.andyhalford.com/

FWIW, I've never had any involvement with, or seen any interaction with, Andy Halford. And as one of the longest-standing Sync client developers (second in line after it stopped being called Weave, and the longest-standing current developer), I feel fairly confident in saying that the Sync team as a whole has never had any interaction with him.

Anyway, let's keep this bug on track; let's leave project discussion, complaints, and speculation for IRC or mailing lists.
Comment 47 Peter Lairo 2012-12-02 15:48:43 PST
(In reply to Richard Newman [:rnewman] from comment #46)
> Not strictly true, but you're certainly more vulnerable to some classes of
> problems than the typical 10-20 bookmark user.

The following point is to make sure that the severity of this bug is not whitewashed:

"more"? That sounds like the understatement of the year. The real issue seems to be *how much* more, and what are the consequences when the bug kicks in? I'm guessing that I'm a "lot" more vulnerable and the consequences are "huge".

> You know, you could just turn off bookmark sync if you're not confident in
> its operation; you'll still get history, passwords, prefs, etc.

Thanks for the pointer. That takes care of 5% of my usage for sync; bookmarks (this bug) is 95% of my usage.

> Just uncheck "bookmarks" in Sync preferences on any device.

I couldn't find where to do this on my Android 4.1.2 phone and Android 4.2 tablet. The syncing seems to be all or nothing.

> FWIW, I've never had any involvement with, or seen any interaction with,
> Andy Halford.

Good to hear that. He might have had a problem with mozilla.org's add-ons department. Anyhow, it would probably be beneficial for the sync department to solicit his expertise. He might know things about syncing that is currently not available in moz's sync department (He already had a functioning sync add-on[1]; although I don't know how fine-grained his syncing was).

[1] http://mac.softpedia.com/get/Internet-Utilities/SyncPlaces.shtml

Good luck, and I hope you create a solution soon.
Comment 48 Richard Newman [:rnewman] 2012-12-02 16:33:05 PST
(In reply to Peter Lairo from comment #47)

> The following point is to make sure that the severity of this bug is not
> whitewashed:

To be clear: there is no need for you to attempt to change my opinions on this bug. More bug comments will not get any work done any faster, which is why I try to encourage people to respect the first rules of Bugzilla Etiquette (<https://bugzilla.mozilla.org/page.cgi?id=etiquette.html>) and stick to comments that contribute to building a solution.

There are known limitations in this, as with all software; the concern is always around resource allocation and scoping.

We have a good understanding of bookmark sync's limitations. The current solution works well enough for the vast, vast majority of users (and has done for several years, with a very low percentage of problems since 2010), and for the others it causes recoverable and non-critical problems (according to most people's view of "critical").

(Remember that the majority hardly use bookmarks, and the majority of the rest don't care about organization; instead they "star" bookmarks.)

That you don't fall into the former category, and that your view of "critical" is different, is regrettable, but reiterating how unhappy you feel isn't going to somehow jam three months' of work into the schedule in an instant.

That sounds harsh, but that's the way it is. Your particular requirements have only a small effect on the project as a whole; that's how communities work.

Sure, the programmer part of my brain wants to drop everything else, make some coffee, and start work on the perfect, optimal, bulletproof solution (I even have a pretty good idea of what to do). 

The senior engineer part insists on an evaluation of costs and benefits, given constrained resources. Bug 814331 is relatively urgent, because it wipes out a swath of risk at little cost; a total rewrite is not, and so I'll bear that in mind when discussing future work.

Software engineering involves conscious tradeoffs. That applied years ago, when desktop Sync was written; it applied in January, when Android Sync was written; and it applies now.


> > Just uncheck "bookmarks" in Sync preferences on any device.
> 
> I couldn't find where to do this on my Android 4.1.2 phone and Android 4.2
> tablet. The syncing seems to be all or nothing.

Android "Settings" > "Firefox Sync" > "What to sync". But the same setting applies wherever you set it, so you can do it on your desktop.


> Good luck, and I hope you create a solution soon.

Thank you!
Comment 49 Peter Lairo 2012-12-03 09:47:23 PST
(In reply to Richard Newman [:rnewman] from comment #48)
> I try to encourage people to respect the first rules of Bugzilla
> Etiquette (<https://bugzilla.mozilla.org/page.cgi?id=etiquette.html>)

I think that my comments do respect the Bugzilla Etiquette! They illuminated the heretofore unmentioned and (nearly) irreversible problem of users who have many bookmarks and who have organized them into folders (i.e. advanced users and long-time users). My comments therefore were both "constructive and helpful".

> There are known limitations in this, as with all software; the concern is
> always around resource allocation and scoping.

That is clear (and obvious to me). However, allocation and scoping is only effective when the severity of a need is also understood.

> for the others it causes recoverable and non-critical problems

If spending 40-80 hours on opening all 3,500 bookmarks, trying to remember which folder they were in, and re-saving each one into its former folder location, then I guess this bug can be called "recoverable and non-critical".

> reiterating how unhappy you feel isn't going to ...

My intent is not to state how "unhappy" I am; my intent is to illuminate the *severity* of this bug (however many users it may affect).

> Bug 814331 is relatively urgent, because it wipes out a swath of risk at little cost

Thanks for the pointer. That would be a good interim band-aid.

> Android "Settings" > "Firefox Sync" > "What to sync". 

That setting doesn't seem to exist on my Android phone (Firefox Mobile 17.0, Nexus S, Android 4.1.2).

> But the same setting applies wherever you set it, so you can do it on your desktop.

Good to know. Thanks!
Comment 50 Richard Newman [:rnewman] 2012-12-03 11:02:11 PST
(In reply to Peter Lairo from comment #49)

> I think that my comments do respect the Bugzilla Etiquette! They illuminated
> the heretofore unmentioned and (nearly) irreversible problem of users who
> have many bookmarks…

See Comment 18. With respect, you're adding nothing new to this discussion.

> My intent is not to state how "unhappy" I am; my intent is to illuminate the
> *severity* of this bug (however many users it may affect).

That's already understood. But remember that number of affected users is very significant.

> > Android "Settings" > "Firefox Sync" > "What to sync". 
> 
> That setting doesn't seem to exist on my Android phone (Firefox Mobile 17.0,
> Nexus S, Android 4.1.2).

Settings > Accounts & sync > Firefox Sync, on pre-JellyBean.
Comment 51 Peter Lairo 2012-12-03 12:05:06 PST
(In reply to Richard Newman [:rnewman] from comment #50)
> (In reply to Peter Lairo from comment #49)
> 
> > I think that my comments do respect the Bugzilla Etiquette! They illuminated
> > the heretofore unmentioned and (nearly) irreversible problem of users who
> > have many bookmarks…
> 
> See Comment 18. With respect, you're adding nothing new to this discussion.

With respect, comment #18 does not address the severity as I have, merely that about 0.1% of users are affected (that's 450,000 if Firefox's most loyal users!) and a description of sync's limitations and how hard it is to solve the problem (all good stuff, but it's not the severity).

> > My intent is not to state how "unhappy" I am; my intent is to illuminate the
> > *severity* of this bug (however many users it may affect).
> 
> That's already understood. But remember that number of affected users is
> very significant.

OK, the "quantity" (i.e. number of users affected)  is "very" significant (You are the one who decides how significant half-a-million users are). Hopefully, the "quality" (i.e. the severity) is also very significant.

> > > Android "Settings" > "Firefox Sync" > "What to sync". 
> > 
> > That setting doesn't seem to exist on my Android phone (Firefox Mobile 17.0,
> > Nexus S, Android 4.1.2).
> 
> Settings > Accounts & sync > Firefox Sync, on pre-JellyBean.

As stated, I use Jelly Bean (4.1.2 and 4.2), and there is no such option. Under System Settings / Firefox Sync, I only see:

+-----------------------+
| < Firefox Sync        |
|  Accounts             |
|    (/) MyUserName     |
|          Firefox [x]  | <-- this item is on a separate screen
|  Options              |     after clicking on MyUserName
|    Pair a Device.     |
+-----------------------+

The Menu button only offers: Sync now, Remove account, and Help.
Comment 52 Richard Newman [:rnewman] 2012-12-03 18:18:10 PST
(In reply to Peter Lairo from comment #51)

> As stated, I use Jelly Bean (4.1.2 and 4.2), and there is no such option.

Ah, coming in Firefox 19. Sorry, I use Aurora.
Comment 53 8472 2013-01-15 06:06:39 PST
Hi,

I've just ran into this problem as well, recently, after I've bought an tablet (nexus 7).
Have already asked about it here https://support.mozilla.org/en-US/questions/946851 before I discovered this bug thread.

Though I'm not a developer myself like I've noticed most of you are, I'm quite skilled with systems, using Linux mostly, thus feel free if any cooperation of mine might be required.

One significant note to what I've noticed Richard mentioned before, that mostly users with huge ammount of bookmarks are affected, I don't agree.
I've just tested this problem with another sync account of another family member of mine (created another account for him on my tablet and performed an sync), while this user has really small amount of bookmarks (if 50 items, than I'm perhaps exaggerating), and the order of bookmarks was broken too, just like with my own account (I'm having quite a lot items, but I don't know how much precisely).
Comment 54 Richard Newman [:rnewman] 2013-01-15 11:16:38 PST
(In reply to 8472 from comment #53)

> One significant note to what I've noticed Richard mentioned before, that
> mostly users with huge ammount of bookmarks are affected, I don't agree.

Problems are possible for anyone (even, in principle, desktop-only users). That's well understood.

Users with more than a few thousand bookmarks are simply very, very likely to have problems.

The proportion of users with 10,000 bookmarks who will see issues is approximately 100%.

The proportion of users with 50 bookmarks who will see issues is almost 0%, but non-zero.

There are simply several hundred times as many users with 50 bookmarks as there are with 10,000, so we'll see a spread of reports.

(I've seen this *once*, about a year ago, on one of my profiles with ~100 bookmarks, using a pre-release version.)

Bug 814331 causes us to refuse to sync bookmarks if you have too many, so we're simply left with the background hum of occasional failures, rather than the systemic issue.


> I've just tested this problem with another sync account of another family
> member of mine (created another account for him on my tablet and performed
> an sync), while this user has really small amount of bookmarks (if 50 items,
> than I'm perhaps exaggerating), and the order of bookmarks was broken too,
> just like with my own account (I'm having quite a lot items, but I don't
> know how much precisely).

How old was their sync account?
Comment 55 8472 2013-01-15 12:21:01 PST
(In reply to Richard Newman [:rnewman] from comment #54)
> > I've just tested this problem with another sync account of another family
> > member of mine (created another account for him on my tablet and performed
> > an sync), while this user has really small amount of bookmarks (if 50 items,
> > than I'm perhaps exaggerating), and the order of bookmarks was broken too,
> > just like with my own account (I'm having quite a lot items, but I don't
> > know how much precisely).
> 
> How old was their sync account?

both (mine and the other one) sync accounts we use were founded in around of September 2010
Comment 56 Richard Newman [:rnewman] 2013-01-15 14:44:05 PST
> both (mine and the other one) sync accounts we use were founded in around of
> September 2010

It's entirely possible that a bookmark record that causes unexpected behavior was just hangin' out on the server. It got processed by the new device, and caused a change that was re-propagated to your desktop.

This kind of issue is very hard to track down.
Comment 57 8472 2013-02-07 09:54:58 PST
Hi again,

Pls don't take me wrong, I'm just being curious.
This bug is already open since 2012/06, is already P1, but when do you expect this to get fixed?
thx
Comment 58 Richard Newman [:rnewman] 2013-02-07 15:15:47 PST
(In reply to 8472 from comment #57)
> Hi again,
> 
> Pls don't take me wrong, I'm just being curious.
> This bug is already open since 2012/06, is already P1, but when do you
> expect this to get fixed?
> thx

I do not expect significant further investment in the existing Sync codebase.

Any remaining issues with bookmark sync (such as a lack of robustness in handling some distributed failure scenarios) are not so much bugs as failings in design; that means they're not easy to fix, and instead require a new approach.

We had intended to invest time in a new iteration of Sync, which would allow us to fix issues like this without breaking compatibility (by breaking it once!). That's probably not going to happen; instead, there's an effort called PICL to take a run at browser data in the cloud from a different direction.

Take a look at http://groups.google.com/group/mozilla.dev.identity/ if you want to find out more about this.
Comment 59 8472 2013-02-07 20:00:12 PST
ok, any assumption, about when any even an replacement solution you've described, will be available?
Comment 60 Richard Newman [:rnewman] 2013-02-07 21:09:29 PST
(In reply to 8472 from comment #59)
> ok, any assumption, about when any even an replacement solution you've
> described, will be available?

I have too little control over the outcome, and not enough insight, to be able to offer any advice here. My best suggestion is to inquire at http://groups.google.com/group/mozilla.dev.identity/ .
Comment 61 Peter Lairo 2013-06-10 08:24:48 PDT
(In reply to Richard Newman [:rnewman] from comment #60)
> (In reply to 8472 from comment #59)
> > ok, any assumption, about when any even an replacement solution you've
> > described, will be available?
> 
> I have too little control over the outcome, and not enough insight, to be
> able to offer any advice here. My best suggestion is to inquire at
> http://groups.google.com/group/mozilla.dev.identity/ .

I could not find any mention of this problem in that newsgroup. Why would switching the sync login from the password manager to BrowserID have any positive impact on this data-loss bug?

I've posted this question in the newsgroup. Here's the Subject and link:

Will syncing large number of bookmarks (PC <--> Android) be handled here? (see Bug 769476)
news://news.mozilla.org:119/NOmdnQIcecGVcijMnZ2dnUVZ_vKdnZ2d@mozilla.org
Comment 62 Richard Newman [:rnewman] 2013-06-10 09:08:10 PDT
PiCL is not an alternative login to Sync; it's a new product. I know that they plan to use a different approach to exchanging changes, which should be resistant to some of the flaws in the design of Sync. And I've outlined some of the design parameters to those folks a few times.
Comment 63 Chris Karlof [:ckarlof] 2013-06-12 15:24:09 PDT
This bug is on the PICL radar. I can't make any promises right now on what size of hierarchical bookmarks we will promise to sync gracefully or what limits we will impose, but we are certainly designing it to take these situations into account.
Comment 64 Niels 2014-07-19 00:17:56 PDT
Looks like this Problem still exist. I had from one of my bookmarks two favorites stored. The Second was not Synced on my Android Device, and so he had every time change the ordner from this "unknown" favorite.
Comment 65 Kevin Brosnan [:kbrosnan] 2015-09-13 16:54:20 PDT
*** Bug 1204372 has been marked as a duplicate of this bug. ***
Comment 66 Ferry Huberts 2015-10-26 09:50:23 PDT
I have this too.
It doesn't matter if I reorder on a PC and then sync, Android Firefox keep overwriting that latest sync data
Comment 67 Ferry Huberts 2015-10-26 13:31:58 PDT
(In reply to Ferry Huberts from comment #66)
> I have this too.
> It doesn't matter if I reorder on a PC and then sync, Android Firefox keep
> overwriting that latest sync data

I disconnected the Android Firefox and still my bookmarks get reordered.
Then I disconnected the desktop Firefox, reconnected it, and again the bookmarks get reordered.
Quite nasty.
How to I get rid of this reordering? It messes with my muscle memory.
Comment 68 Ferry Huberts 2015-10-26 13:36:19 PDT
(In reply to Ferry Huberts from comment #67)
> (In reply to Ferry Huberts from comment #66)
> > I have this too.
> > It doesn't matter if I reorder on a PC and then sync, Android Firefox keep
> > overwriting that latest sync data
> 
> I disconnected the Android Firefox and still my bookmarks get reordered.
> Then I disconnected the desktop Firefox, reconnected it, and again the
> bookmarks get reordered.
> Quite nasty.
> How to I get rid of this reordering? It messes with my muscle memory.

Firefox 41.0.2
Fedora 23, x64
Comment 69 Richard Newman [:rnewman] 2015-10-26 16:27:41 PDT
(In reply to Ferry Huberts from comment #67)

> How to I get rid of this reordering? It messes with my muscle memory.

The issue is with the data in your Sync account, not with the devices themselves.

The solution in your case is to re-sync all of your bookmarks.

Add the Sync spinner to your toolbar. Firefox menu > Customize > drag the Sync spinner to your toolbar. This will allow you to see when a sync has completed, and also trigger a new one. You can drag it away again when you're done with these steps, if you like.


1. Make sure your desktop is on a reliable network.
2. On your desktop, uncheck Bookmarks in Sync Preferences. You should see the spinner spin then stop.
3. Check Bookmarks again. Wait for the spinner to stop.

Then, if you wish:

4. Make sure your Android device is on a reliable network.
5. Sign in to your Firefox account.
6. Wait five minutes before switching applications, just in case.
Comment 70 Ferry Huberts 2015-10-31 13:29:03 PDT
I've tried the things you suggested and several more.
I even remove the sync account from the Android device.

Sync keeps re-ordering my bookmarks.
Especially after I've made changes. Like move bookmarks into folders.
Comment 71 Richard Newman [:rnewman] 2015-11-02 10:16:18 PST
(In reply to Ferry Huberts from comment #70)

> Sync keeps re-ordering my bookmarks.
> Especially after I've made changes. Like move bookmarks into folders.

If you can file a new bug with your sync logs, ideally with bookmark sync logging set to Trace, that would help pin down what's happening.
Comment 72 Ferry Huberts 2015-11-02 12:12:01 PST
And how would I do that?
Comment 73 Ferry Huberts 2015-11-04 07:20:14 PST
And now this becomes even more wonderful:
I added some bookmarks and a while later some bookmarks folders were re-arranged.
Then went into the sync prefs and see my surprise: bookmark sync was turned OFF!
Clearly there is a lot wrong in this area.

Note You need to log in before you can comment on or make changes to this bug.