Closed Bug 769476 Opened 12 years ago Closed 4 years ago

bookmarks reordered on desktop after syncing with android

Categories

(Firefox for Android Graveyard :: Android Sync, defect, P2)

Product:
Firefox for Android Graveyard
Component:
Android Sync
Platform:
All
Android
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: ally, Unassigned)

References

(Depends on 1 open bug)

Details

(Whiteboard: [sync:bookmarks])

Attachments

(3 files)

Android and Firefox sync logs
107.74 KB, application/octet-stream
Details
Logs from home PC
21.19 KB, application/octet-stream
Details
Logs from Android
70.59 KB, text/plain
Details
There are multiple hypothesis about what the cause of this behavior is. We may have more than one issue in play, but we're guessing in the dark at this point. QA, Given keheldar's detailed replies, we'd like to know if you can create reliable reproductions of the jumbling issues. Detailed discussion (including comments from 2 sync engineers) & STR (serious props to kheldar) : https://support.mozilla.org/en-US/questions/927574 NB: To (most) users, bookmark reordering == data loss
OS: Mac OS X → Android
Hardware: x86 → All
Keywords: qawanted
We are very much guessing in the dark, but there are almost certainly two issues. An interrupted first sync -- partial data on the server -- is a very predictable source of reordering, and one that we're (broadly speaking) ignoring. A robust client will fix positions of records to ensure a contiguous set. That should only happen if a client uploads a folder but is interrupted before it can upload the children, which is an insignificant risk on all but a first sync. However, I've seen reordering on a mature Sync account, so a partial upload issue cannot be the whole story. I'm much more concerned about "I added an Android device to my existing stable Sync constellation, and now all of my bookmarks moved". That *could* be caused by a partial *download* (client doesn't receive all records but isn't informed of success). If a user has thousands of bookmarks, that could happen. But ordinarily it should not… hence the guessing in the dark.
Oh, and a difference in behavior between Android and desktop: when desktop sees partial data, it computes a new order locally, but -- as far as I can tell -- doesn't reupload a reordered folder unless something else changes (add or remove a child locally). Android does its best to ensure that the local and remote data are identical, so when it modifies a folder (including its child array) it uploads the modified record. Formerly, then, if you: * Sync desktop; quit before done * Sync some other device * Sync desktop again you'd see no issue, but if you: * Sync desktop; quit before done * Sync some other device * Modify the partial folder on other device * Sync desktop again you'd see reordering. If the other device is Fennec Native, you'll see possible reordering in either case. Mitigation: have desktop upload records in a strict partial-safe order. Probably the most partial-safe order for: root Toolbar A B C Menu F1 D E F2 G H is to leave folders until last, walking them leaf-to-trunk, but that imposes the maximal amount of churn on other clients (every record has to be applied then moved). Another mitigation is to ensure that records are proximal to their folders, so they have more chance of hitting the server together.
Walking the dog, five further late-night unvetted possibilities came to mind. This is the one I think is most likely. Desktop uploads bookmark records in an arbitrary order. The server will also return them in an arbitrary order -- *probably* with folders first, but who knows? If Android Sync fails to finish storing records -- some exception is thrown, or we're killed with OOM, or whatever -- it will have a sample of records from the other device. More importantly, it either won't reach a repositioning stage, or will have to handle partial data when it does. If it tries a sync again, it will re-download records, because our timestamp is still 0. But during the reconciling stage it will see that the local record was modified _later_ -- it made it to the database! -- and won't apply the incoming record. We'll finish the download, and then we'll upload our own computed records, with our own computed positions. I haven't explored this possibility yet, but it seems promising. Four more possibilities: Firstly, is desktop propagating position changes when they occur? Secondly, is it using the same method to indicate position changes as we are on Android? (There's a "pos" attribute to a record, and the children array on the parent.) Thirdly, is Sync on desktop uploading records with the correct children array? Fourthly, is Fennec preserving the positions of records? Imagine that a user adds a bunch of bookmarks to their toolbar. Each addition causes an instant sync. Then they rearrange the toolbar. No sync occurs. Their phone syncs, and -- for whatever reason -- touches the record on the server again. Desktop will download the toolbar record and undo the user's position changes. Or imagine if desktop is uploading the wrong children array, or an empty children array. Android will reconstruct one and upload it. Or if Fennec mutates the bookmark order after insertion, bad things will happen. This log from my Aurora profile, recorded as I moved a couple of toolbar entries around, might illustrate such: 1340950738001 Sync.Status DEBUG Status.service: success.status_ok => success.status_ok 1340950748082 Sync.Tracker.Bookmarks TRACE onItemMoved: 4416 1340950748082 Sync.Tracker.Bookmarks TRACE Adding changed ID: toolbar,1340950748 1340950748358 Sync.SyncScheduler DEBUG Global Score threshold hit, triggering sync. 1340950749077 Sync.Tracker.Bookmarks TRACE Saving json to disk: weave/changes/bookmarks.json 1340950760491 Sync.Tracker.Bookmarks TRACE onItemMoved: 4584 1340950760492 Sync.Tracker.Bookmarks TRACE Adding changed ID: toolbar,1340950760 1340950760579 Sync.SyncScheduler DEBUG Global Score threshold hit, triggering sync. 1340950761493 Sync.Tracker.Bookmarks TRACE Saving json to disk: weave/changes/bookmarks.json 1340950762485 Sync.SyncScheduler DEBUG Global Score threshold hit, triggering sync. 1340950767464 Sync.Status DEBUG Status.service: success.status_ok => success.status_ok 1340950767464 Sync.Status DEBUG Status.service: success.status_ok => success.status_ok 1340950767981 Sync.Status DEBUG Status.service: success.status_ok => success.status_ok 1340950767981 Sync.Status DEBUG Status.service: success.status_ok => success.status_ok 1340950768797 Sync.ErrorHandler DEBUG Beginning user-triggered sync. 1340950768801 Sync.Service DEBUG User-Agent: Firefox/15.0a2 FxSync/1.17.0.20120624042007. 1340950768801 Sync.Service INFO Starting sync at 2012-06-28 23:19:28 1340950768802 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score. 1340950768803 Sync.Service INFO In sync(). 1340950768803 Sync.Status INFO Resetting Status. 1340950768803 Sync.Status DEBUG Status.service: success.status_ok => success.status_ok 1340950769288 Sync.Resource DEBUG mesg: GET success 200 https://phx-sync542.services.mozilla.com/1.1/holygoat/info/collections 1340950769288 Sync.Resource DEBUG GET success 200 https://phx-sync542.services.mozilla.com/1.1/holygoat/info/collections versus: 1340951048047 Sync.Tracker.Bookmarks TRACE onItemAdded: 4654 1340951048048 Sync.Tracker.Bookmarks TRACE Adding changed ID: 0biQzJtu--x_,1340951048 1340951048048 Sync.Tracker.Bookmarks TRACE Adding changed ID: menu,1340951048 1340951048050 Sync.Tracker.Bookmarks TRACE onItemChanged: 4654, bookmarkProperties/description (anno) 1340951048155 Sync.SyncScheduler DEBUG Global Score threshold hit, triggering sync. 1340951048156 Sync.Service DEBUG User-Agent: Firefox/15.0a2 FxSync/1.17.0.20120624042007. 1340951048156 Sync.Service INFO Starting sync at 2012-06-28 23:24:08 1340951048158 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score. 1340951048159 Sync.Service INFO In sync(). That is, repositioning a bookmark does not (unlike addition) cause an instant sync to occur.
I am moderating testday for a couple hours this morning, then I'll attempt to repro rnewman's STR's in comment #2. I think those are what the user described; Their data in the cloud was mangled somehow. They *may* have shut down during initial sync. Reset from desktop fixes the problem. It sounds like the user has a significant amount of tool bar bookmarks.
(In reply to Tracy Walker [:tracy] from comment #4) > I am moderating testday for a couple hours this morning, then I'll attempt > to repro rnewman's STR's in comment #2. I think those are what the user > described; Their data in the cloud was mangled somehow. They *may* have > shut down during initial sync. Reset from desktop fixes the problem. It > sounds like the user has a significant amount of tool bar bookmarks. Yeah, you'll need to have more items (much greater than 100) than can be uploaded and applied in a single request (100 items). A bad network connection might help. I will see if I have time to try to write a test for my main hypothesis.
Tracy, any news?
Severity: normal → major
Priority: -- → P1
Sorry, I wasn't able to get to this that Friday. I've been on vacation since. I'll try to repro after s-c train testing.
I've been attempting various scenarios for the better part of yesterday afternoon and this morning. I have yet to trigger desktop BM reordering.
Hi guys, This just happened to me again today. I installed Firefox 16.0.1 (the non-beta version) on my HTC One X and I wanted to see if this bug is fixed, so I paired the device to my laptop (which is also running Firefox 16.0.1) and I let the sync do its job on the phone. Just after it finished, the order of my bookmarks on the laptop became an incomprehensible mess. I haven't looked at all of them, but it definitely messed up my bookmarks toolbar (changed folder order, moved subfolders directly into the toolbar, rearranged the bookmarks inside the folders, etc). Perhaps I should mention that I'm also using the Xmarks plugin as an extra backup (luckily). This is not cool, guys. I want to be able to use the sync functionality on my phone. Please let me know if I can be of further assistance. I can provide a dump of my bookmarks to let you test this issue. I can also provide whatever logs you want, if needed, just let me know.
PS: I'm a C++ software engineer, so don't be shy to go into technical details, if needed.
(In reply to Mihai Todor from comment #10) > haven't looked at all of them, but it definitely messed up my bookmarks > toolbar (changed folder order, moved subfolders directly into the toolbar, … You mean you had Toolbar Folder A Folder B Item 1 Item 2 and ended up with Toolbar Folder B Item 2 Item 1 Folder A ? If yes, then that's *very* odd. There's no straightforward way for bookmarks sync on either platform to cause that to occur -- it doesn't fall-back up the hierarchy at all, instead using the Unsorted Bookmarks folder as the default location. (That is, if somehow Folder A disappeared, Folder B would be found in Unsorted Bookmarks, not in the Toolbar.) Related questions, if the answer is yes: did you originally create Folder B in the toolbar, then move it into Folder A? How recently did you create that folder hierarchy? > Perhaps I should mention that I'm also using the Xmarks plugin as an extra > backup (luckily). For all we know, that could be the cause. > This is not cool, guys. I want to be able to use the sync functionality on > my phone. Please let me know if I can be of further assistance. I can > provide a dump of my bookmarks to let you test this issue. I can also > provide whatever logs you want, if needed, just let me know. If you can reproduce this, Trace logs for bookmark sync on desktop (see http://bit.ly/q6fBL1) and mobile (see http://160.twinql.com/how-to-file-a-good-android-sync-bug) would be excellent. We haven't been able to reproduce any symptoms. We have extensive tests covering the components of this system, which means that the bug is hiding in some (probably long-term) interaction between components.
Flags: needinfo?(mihai.todor85)
Hi Richard, Thank you very much for the quick reply. It seems that you are lucky today, and I was able to replicate it (to a certain degree). Here is the setup: 1. Firefox 16.0.1 on my personal laptop (Windows 7 x64). Extensions: - Adblock Plus 2.1.2 - Cookies Manager+ 1.5.1.1 - Element Hiding Helper for Adblock Plus 1.2.3 - Firebug 1.10.4 - FireFTP 2.0.7 - Flashblock 1.5.15.1 - gTranslate 0.9 - JavaScript Deobfuscator 1.6.3 - Live HTTP headers 0.17 - NoScript 2.5.7rc5 - Romanian spelling dictionary 1.12 - SQLite Manager 0.7.7 - Xmarks 4.1.3 2. HTC One X, Android 4.0.4 and HTC Sense 4.1, connected to the internet using WiFi, via Connectify, that is running on my laptop. I enabled the services.sync.log.appender.file.logOnSuccess option in Firefox on my laptop. I completely removed Firefox from my phone and installed Firefox 16.0.1 via Play Store. Afterwards, I started it. At this point, I connected the phone via USB cable to my laptop and I started the adb logger. Afterwards, I went to my phone and started setting up sync as instructed. I went on my laptop to introduce the required codes and, on my phone, Firefox instructed me that the sync will work in the background. I went to Settings > Accounts & sync > Firefox Sync and waited until it finished. After this, I stopped the ADB logger. Now, I waited for sync to finish messing up my bookmarks on the laptop, which happened in about 20 seconds or so. This time, the damage was smaller and I did not notice any updates to the folder structure anymore, just rearranged bookmarks. Earlier today I was very upset that it messed up my bookmarks again (I was a bit scared that I lost something) and I am not 100% sure if it really messed up the folder structure, but it definitely added some stuff inside the Bookmarks Toolbar (the main Bookmarks Toolbar folder), because I remember seeing it had more items than it was supposed to (I should’ve taken a dump of them for closer inspection and comparison). Perhaps it threw in there some bookmarks from my Bookmarks Menu, but I don’t remember what (my Unsorted Bookmarks folder is empty). Anyway, on my laptop, Firefox produced 4 sync logs for you to analyze. Please find them and the android log attached. I really hope you’ll be able to get to the bottom of this issue. If needed, I can also send you a dump of my bookmarks, but I don’t really want to upload them here for anybody to download, so please provide an email address where I should send them. I’ve been using Firefox for quite some time now, and I really like it, so if you need more feedback or help from me, please let me know and I will be happy to assist.
Flags: needinfo?(mihai.todor85)
> I’ve been using Firefox for quite some time now, and I really like it, so if > you need more feedback or help from me, please let me know and I will be > happy to assist. Hi Mihai, Glad you like Firefox enough to help us with this extremely frustrating bug! I'm going to look at your logs now, and I know rnewman will too.
6443:10-14 00:44:09.170 I/FxSync (26905): ServerSyncStage :: Reset timestamps for org.mozilla.gecko.sync.stage.AndroidBrowserBookmarksServerSyncStage@412a0bc8 6840:10-14 00:44:15.870 I/FxSync (26905): BrowserRepoSession :: BEGIN: creating record -> GUID map. 6842:10-14 00:44:15.920 I/FxSync (26905): BrowserRepoSession :: END: creating record -> GUID map. 9566:10-14 00:47:11.420 E/FxSync (26905): BrowserRepoSession :: Finish called but 2949 bookmark(s) have been placed in unsorted bookmarks and not been reparented. 9567:10-14 00:47:11.440 I/FxSync (26905): ServerSyncStage :: Received 5000 and sent 657 records in 182.28 seconds. What this means is that our download limit is being hit, with terrible results :( We have considered this -- see Bug 730142 for a possible solution -- and your experience suggests we need to make this higher priority. I'm going to let rnewman think about this before we make any concrete plans, though.
@Nick: Is everybody at Mozilla working on weekends? :) I really appreciate that you're taking the time to investigate this now, but it is not a big emergency, so please take your time. It's nice that you managed to locate a possible cause so quickly and I do confirm that I have a large number of bookmarks and a complex folder hierarchy in the Bookmarks Toolbar. The first time I tried FF Sync on my phone (more than 6 months ago), I lost some bookmarks and it really was frustrating. I haven't reactivated it until today and, sadly, I noticed the same issue. I should be more paranoic when it comes to backups :P
(In reply to Mihai Todor from comment #17) > @Nick: Is everybody at Mozilla working on weekends? :) We sometimes try not to :) > What this means is that our download limit is being hit, with terrible > results :( We have considered this -- see Bug 730142 for a possible > solution -- and your experience suggests we need to make this higher > priority. You're one of the 0.1% of users who have enough bookmarks that it's considered impractical to process them all. That very likely means you are *not* encountering this specific bug; you're encountering a known limitation. A limit is a necessary evil -- if for no reason other than Android will kill the Sync process if it takes more than five minutes to complete the whole sync operation on the phone. If we get killed like that, we'll keep retrying, so you'll have your Sync client pulling down five megs of bookmarks a few times a day. Unlike history, a partial download of bookmarks is destructive. What's happening here is something like this: * Desktop has uploaded some or all of its bookmarks to the server. We'll assume all for simplicity, but if it didn't there'll be additional corruption. * Android downloads 5,000 of them, sorted by importance. * About 3,000 of those bookmarks don't have their parent folder — that folder is outside of the set of 5,000 downloaded records. Those are moved into Unsorted Bookmarks, because there's nowhere else to put them. This could happen because we hit a limit, or it could happen because the server data is inconsistent -- a bad client could easily have uploaded junk, or we could just be seeing the server state in the middle of a sync over a bad network. * The phone constructs the sanest arrangement it can manage, such that each record has a parent, a consecutive position within that parent, and so on. * We then upload that structure to the server, so that other clients end up with the same set. (The alternative is that we *don't* upload records, but that means the contents of the phone won't get positioned correctly in the merge case, amongst other things.) You can think of this as the well-behaved client 'fixing' the corrupt server. * The desktop downloads the changed records and repositions/relocates accordingly. There are a few complex things we can do to try to guard against this case, but no perfect solution. * Remove the limit, and hope that your phone successfully syncs before it consumes your entire mobile data allowance. * Increase the limit, and orchestrate a sequence of syncs so as to get a whole 5-minute run at each of the big collections. * Try to track which folders haven't been reconciled locally, and don't upload those which have only been repositioned as a consequence of partial data. Maybe we'll get fixed up in the future. * Batch downloads (Bug 730142), which will require some kind of similar tracking. * Introduce some kind of state which will allow us to stretch a multi-part sync over multiple Android syncs. * Build a local cache of server data against which we do the actual bookmark syncing (two-phase sync). We wouldn't do any syncing against a part of the folder tree until that tree was complete. The combination of batch downloads and two-phase sync is probably the only way to 'solve' this problem, at the cost of delayed gratification (you wouldn't see any bookmarks in Fennec until we had all of them) and increased memory and CPU cost. If we simply implement batching _and refuse to upload or reposition until we've downloaded everything_, we get an 80% solution. Flawed, but would work in the general case. Unfortunately for you, we won't have time to work on this until the new year. The best advice I can give you is to begin by turning off bookmark sync, because it's guaranteed to not work for you. There are a few things you could do: * Find a way to import your bookmarks to Fennec. If the bookmark structure is initially the same on both devices, no corruption will occur. * Use a different tool for handling bookmarks — pinboard.in, for example. The Firefox bookmark UI (and data store, really) aren't optimized for users who have thousands or tens of thousands of bookmarks, and even if this whole thing would function at all, the Sync client on each end is doing a huge amount of work to manipulate a model of that bookmark structure each time you sync. Better to just store the data somewhere central.
(In reply to Richard Newman [:rnewman] from comment #18) Wow, thanks for the highly detailed reply, Richard. I was not expecting such rich feedback. While I am not familiar with the Firefox internals, here are my 2 cents, based on my experience with large scale applications: ---- * Build a local cache of server data against which we do the actual bookmark syncing (two-phase sync). We wouldn't do any syncing against a part of the folder tree until that tree was complete. ---- This seems like the sanest solution to me, and, to be honest, I fail to see why would anyone want to complicate things so much, considering the rest of the solutions that you mentioned above. I just exported all my bookmarks as JSON. The file has ~700KB and it contains at least 3 years of bookmarks (2618 items without the associated folders to be more precise - a surprisingly small number, I would say). If I archive it as zip without any special compression, I get ~160KB. Now, I think we can all agree that downloading a few megabytes (in the worse case I can imagine) on a mobile phone won't cause major headaches for anybody. Actually, I download & upload pictures much larger than that on a daily basis. At this point, I can think of only one pain in the ass: what if the user already has some bookmarks on his mobile phone and wants to merge them with the ones received from the server? I would answer this with one simple two choice dialogue: either loose 'em or let the phone churn on bookmark tree merging for a few minutes in a background thread. If this is such a big issue, the phone could upload the partial list to the server and ask it to perform merging (I can imagine some cute DDoS attacks in this scenario), but I doubt this is really needed, even on older phones. To make things even faster, why not send the entire bookmark store to the client (phone) as a native Firefox store (SQLite database, probably)? If properly archived, I doubt it will take up much more space than the JSON string and the client won't need to parse it. For the sake of argument, let's say that the user's internet speed is really slow and he has a couple of MB of bookmarks to sync and an older phone. Why not just warn him? "Hey, dude, it's going to take one hour to fetch all your stuff." Also, I can imagine some scalability issues with your servers, but I guess that's not the big problem here. Still, as an easy-to-implement alternative, I propose to have a "power-user setting" somewhere that enables bookmark freaks like me to get read-only bookmark access on certain devices. I find it pointless most of the times to bookmark stuff from my phone, considering that the mobile version doesn't provide any advanced bookmark management ( https://support.mozilla.org/en-US/questions/828180 ). This could also make sense as a sanity feature, if I sync my bookmarks on a work / public computer, where multiple people have access. I hope my feedback helps and I'll wait patiently for you to come up with a solution for this, even if it will take until next year :) I don't want to mess around with another bookmark manager, like pinboard.in, since it doesn't have a nice integration in the browser...
> I just exported all my bookmarks as JSON. The file has ~700KB and it > contains at least 3 years of bookmarks (2618 items without the associated > folders to be more precise - a surprisingly small number, I would say). If I > archive it as zip without any special compression, I get ~160KB. Bear in mind that Sync's records are structured and encrypted (which also means they don't compress well). My 500 bookmarks occupy about 500KB on the server. Out of interest, how often do you delete bookmarks, and how long have you had Sync enabled on your desktop? And how many folders do you have? > Now, I think we can all agree that downloading a few megabytes (in the worse > case I can imagine) on a mobile phone won't cause major headaches for > anybody. Actually, I download & upload pictures much larger than that on a > daily basis. That's not strictly true; it's a little more complex than that. For example, we support ARMv6 devices on flaky networks, and there are some overheads to consider, but moving on… > At this point, I can think of only one pain in the ass: what if the user > already has some bookmarks on his mobile phone and wants to merge them with > the ones received from the server? Which is always the case, with the exception of a deliberate Reset Sync. There's always merging to do. (The code paths don't even differ after a reset's initial wipe.) > I would answer this with one simple two > choice dialogue: either loose 'em or let the phone churn on bookmark tree > merging for a few minutes in a background thread. It's always a background thread, but consider that these are often phones with very limited processor and memory capacities, and Android allows us only 8MB of heap. > If this is such a big > issue, the phone could upload the partial list to the server and ask it to > perform merging (I can imagine some cute DDoS attacks in this scenario), but > I doubt this is really needed, even on older phones. Not possible; Sync uses client-side encryption. The server doesn't have the cleartext, and IMO should not. (Not to mention how much slower upload is from a mobile phone.) > To make things even faster, why not send the entire bookmark store to the > client (phone) as a native Firefox store (SQLite database, probably)? If > properly archived, I doubt it will take up much more space than the JSON > string and the client won't need to parse it. Firefox for Android uses a completely different storage backend, and a different version of SQLite. (Ignoring for a moment that this would require clients to maintain snapshots and store them somewhere, and that the protocol would have to be changed to support snapshotting, and that someone would have to pay for the storage….) > For the sake of argument, let's say that the user's internet speed is really > slow and he has a couple of MB of bookmarks to sync and an older phone. Why > not just warn him? "Hey, dude, it's going to take one hour to fetch all your > stuff." Also, I can imagine some scalability issues with your servers, but I > guess that's not the big problem here. Mostly that background sync operations don't have UI to do so, and we don't really know how long it'll take until we're part-way through. > Still, as an easy-to-implement alternative, I propose to have a "power-user > setting" somewhere that enables bookmark freaks like me to get read-only > bookmark access on certain devices. Honestly, doing this would pretty much entail solving the problem we're discussing -- incremental download of large bookmark collections with safe reconciling. All of the same persistence requirements (that is, if I see a folder record before I've seen all of its children, I need to keep it around to use in later parenting operations) still apply. There's no way to offload work to the server, because your data is stored as opaque encrypted blobs; we can't lazily query for a list of bookmarks by folder structure, for example. Beyond that, we tend to eschew adding configuration UI for this kind of thing, because if it's visible and obvious it confuses users, and if it's not visible and obvious it might as well not exist, because people will only discover it when they've already encountered a problem. > I hope my feedback helps and I'll wait patiently for you to come up with a > solution for this, even if it will take until next year :) I don't want to > mess around with another bookmark manager, like pinboard.in, since it > doesn't have a nice integration in the browser... Fair enough! Thanks for taking the time to write, and for your patience.
(In reply to Richard Newman [:rnewman] from comment #20) Oh, yeah, now I remember reading that Firefox does encrypt bookmarks. That is one nice privacy feature, I have to say, and I am sure that some people do appreciate it, but, indeed, this complicates many things. Incremental encrypted backups are hard to get right... On the other hand, I find it strange that Mozilla doesn't want to sell at least some aggregated bookmarks data to 3rd parties. I think some people would want to pay money for that, which could perhaps balance the costs of the sync servers. Are the users that upset about getting their bookmarks read by others? :D I usually have no reason to delete bookmarks, but sometimes I do change the folder structure, like moving folders from one place to the other or adding new ones. Initially, I was using only Xmarks, but a year ago (I think) I added Firefox sync as well, just to try it out, and I left it running as a backup service, just in case Xmarks fails. I think I have around 350 folders (and some are even 5 or more levels deep), based on the number of </H3> tags in the HTML export. I have no idea how to count them otherwise and I'm too lazy to write a parser now... > Beyond that, we tend to eschew adding configuration UI for this kind of > thing, because if it's visible and obvious it confuses users, and if it's > not visible and obvious it might as well not exist, because people will only > discover it when they've already encountered a problem. Well, yeah, but data loss is usually not acceptable, so there should be some workaround, even for freaks who want one million bookmarks :) As a bare minimum, the client should be aware that it hasn't received a complete snapshot from the server, in which case it should either keep on trying or give up, but most definitely it should not try to rewrite the server data with its own. Since the bookmarks on the Android device have no structure (it doesn't allow you to create any folders & stuff, right?), then they should just be thrown somewhere in an "unsorted bookmarks" folder (maybe a new folder) during the first sync. Still, I think that you're giving a bit too much freedom to the client. By imposing a few sane constraints, some of the problems might go away. Anyway, thank you very much for your time and good luck hunting bugs. I used to work for tech support myself, so I know what it's like...
> Oh, yeah, now I remember reading that Firefox does encrypt bookmarks. That > is one nice privacy feature, I have to say, and I am sure that some people > do appreciate it, but, indeed, this complicates many things. Incremental > encrypted backups are hard to get right... On the other hand, I find it > strange that Mozilla doesn't want to sell at least some aggregated bookmarks > data to 3rd parties. We have a very strong set of privacy principles, outlined here: http://www.mozilla.org/en-US/privacy/#our-principles On a much more concrete level: everyone I know working on Sync — ops, server, or clients — has a very strong *personal* desire to do the right thing by our users (which includes ourselves!). Working on a not-for-profit organization like Mozilla has benefits when it comes to sleeping at night. :) > I think some people would want to pay money for that, > which could perhaps balance the costs of the sync servers. Are the users > that upset about getting their bookmarks read by others? :D You would be surprised at the kinds of things people store in their bookmarks! > I usually have no reason to delete bookmarks, but sometimes I do change the > folder structure, like moving folders from one place to the other or adding > new ones. Initially, I was using only Xmarks, but a year ago (I think) I > added Firefox sync as well, just to try it out, and I left it running as a > backup service, just in case Xmarks fails. I think I have around 350 folders > (and some are even 5 or more levels deep), based on the number of </H3> tags > in the HTML export. I have no idea how to count them otherwise and I'm too > lazy to write a parser now... Interesting. Given that you should have no more than 3,000 records (2618 bookmarks, 350 folders), I am surprised that Android Sync was able to download 5,000. Would you mind running some JS for me to see what counts you have? Open a privileged web console (new tab, about:config, Tools > Web Console), and run this: https://developer.mozilla.org/en-US/docs/Firefox_Sync/Code_snippets#Count_types_of_bookmark_records It'll print something like [21:34:49.173] Deleted: 34, {"folder":37,"bookmark":561,"separator":2,"query":3} > Since the bookmarks on the Android device have no structure > (it doesn't allow you to create any folders & stuff, right?), then they > should just be thrown somewhere in an "unsorted bookmarks" folder (maybe a > new folder) during the first sync. That's exactly what happens, but consider that even Android phones can — and do — have pre-existing bookmark structures, perhaps from previous Sync accounts or profile migration from XUL Fennec, and server data isn't trustworthy due to a whole pile of reasons (server bugs, client bugs, lack of transactionality in the protocol, etc.). Syncing is a really hairy business, even disregarding performance, compatibility, error handling, and space constraints. You just happen to have hit one of the intersection points of design constraints, schedule constraints, and unusual user behavior! > Still, I think that you're giving a bit > too much freedom to the client. By imposing a few sane constraints, some of > the problems might go away. There are lots and lots of things we would do differently if we had a couple of years to build a brand-new Sync protocol! (And believe me, this is way better than it was in 2010!) Alas, we work with what we've got. :D
(In reply to Richard Newman [:rnewman] from comment #22) > Working on a not-for-profit organization like Mozilla has benefits when it comes > to sleeping at night. :) If you do get to sleep, that is ;) > Would you mind running some JS for me to see what counts you have? It's my pleasure (and many thanks for the cool code snippets. I had no idea about that page). Here's the output: [16:13:43.713] Deleted: 0, {"bookmark":2623,"folder":339,"query":2,"separator":3} @ Web Console:24 > Syncing is a really hairy business, even disregarding performance, compatibility, error > handling, and space constraints. Yeah, I agree now, based on this conversation. And it is starting to sound as an interesting challenge to design and enhance it. I'd love to work on such a project :) > Alas, we work with what we've got. It could have been worse. One year ago, I was working on fixing bugs in a big backend application for investment banks, designed way back in '93 - '95. Instead of refactoring it over the years, they preferred to just keep the existing core code written in C and just patch it. You don't want to know how it ended up looking...
> > Syncing is a really hairy business, even disregarding performance, compatibility, error > > handling, and space constraints. > > Yeah, I agree now, based on this conversation. And it is starting to sound > as an interesting challenge to design and enhance it. I'd love to work on > such a project :) Hi Mihai, I just realized I didn't respond to this. We'd love your help! You should definitely check out http://whatcanidoformozilla.org/ for a broad view of what parts of the Mozilla project you might be interested in. If you want to contribute to Android Sync, I'm working on some getting started documentation for the next few days. At the moment, you can look at https://wiki.mozilla.org/Services/NativeSync for some scattered technical information about getting a build up and https://bugzilla.mozilla.org/buglist.cgi?classification=Client Software;query_format=advanced;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;component=Android Sync;product=Mozilla Services (ugh!) for a list of open Android Sync tickets. A few are marked good first bug or mentor=nalexander and I am trying to make it easier to contribute to Android Sync as a whole. Sorry for the delay responding.
My workaround: 1. Spend 2 hours fixing the order of my bookmarks and bookmark folders. 2. Sync all bookmarks to my two Android devices (Nexus S and Nexus 7). 3. Turn OFF sync on my Android devices. 4. Live with outdated bookmarks on my Android devices until this bug is fixed. (In reply to Richard Newman [:rnewman] from comment #22) > Interesting. Given that you should have no more than 3,000 records (2618 > bookmarks, 350 folders) Is that the limit? Is there a way for a *user* to see how close they are. I've been using Firefox since 1999 and probably have close to that many bookmarks/folders. > https://developer.mozilla.org/en-US/docs/Firefox_Sync/ > Code_snippets#Count_types_of_bookmark_records This give me: [18:07:00.851] SyntaxError: syntax error > You just happen to have hit one of the intersection points of design > constraints, schedule constraints, and unusual user behavior! I don't think that having many bookmarks and folders should be considered "unusual user behavior", especially since Firefox has been around for years, and longtime users *will* have many bookmarks (and a smartphone). > There are lots and lots of things we would do differently if we had a couple > of years to build a brand-new Sync protocol! Are you saying this "major" bug will not be fixed?
@Nick: Thanks for the feedback. I will look more into the links you provided, and, hopefully, I will start contributing some code in my spare time. I was actually looking for such a challenge. @Peter: I think you missed the part about running that script in a *privileged web console*. Basically, you open about:config or about:about or something similar and there you will be allowed to run that code in the console. Regarding Richard's comments, if you have the patience to go through this whole thread, I think you will understand that the design of this feature is not perfect, but it sure is nice and it respects privacy. I am sure that they will improve it in time and, hopefully, they can find some time to look into the code to see if my logs actually uncover some bugs. Regards, Mihai
(In reply to Peter Lairo from comment #25) > My workaround: > 1. Spend 2 hours fixing the order of my bookmarks and bookmark folders. 30 minutes into the tedious process of restoring my folder order, I see that many (almost all) of the folders that were erroneously moved into the Unsorted Bookmarks folder due to this bug are now *empty*! This is a serious dataloss issue. I've lost many bookmarks that I have collected during the past ten years - many of them very important to me. Guys, this needs to be fixed ASAP. Some good suggestions have been made that do not require "a couple of years to build a brand-new Sync protocol", including putting up a visual indicator that a (initial) sync is in progress, and warning users when they exit while a sync is in progress that dataloss might (will) occur.
(In reply to Mihai Todor from comment #26) > @Peter: I think you missed the part about running that script in a > *privileged web console*. Basically, you open about:config or about:about or > something similar and there you will be allowed to run that code in the > console. I did run the script in the about:config tab. STR: 1. Open new tab and load about:config 2. Press Firefox/Nightly button / Web Developer / Web Console 3. At the bottom of the screen, after the ">", I pasted: https://developer.mozilla.org/en-US/docs/Firefox_Sync/Code_snippets#Count_types_of_bookmark_records Result: [18:38:47.154] SyntaxError: syntax error I'm using Firefox nightly "19.0a1 (2012-10-24)" in Windows 7
(In reply to Peter Lairo from comment #25) > My workaround: > > 1. Spend 2 hours fixing the order of my bookmarks and bookmark folders. > 2. Sync all bookmarks to my two Android devices (Nexus S and Nexus 7). > 3. Turn OFF sync on my Android devices. This will not work if you have more than five thousand bookmarks and folders. If you have fewer than that, then you might as well leave Sync turned on. > Is that the limit? Is there a way for a *user* to see how close they are. > I've been using Firefox since 1999 and probably have close to that many > bookmarks/folders. Sync on Android will download no more than 5,000 items. If you have more than 5,000 bookmarks and folders, you'll see some jumbling. This was a deliberate tradeoff: the alternative is bad for different reasons. > I don't think that having many bookmarks and folders should be considered > "unusual user behavior", especially since Firefox has been around for years, > and longtime users *will* have many bookmarks (and a smartphone). Statistics says otherwise, but I should be clear: this is not a question of what we would *like* to do. > > There are lots and lots of things we would do differently if we had a couple > > of years to build a brand-new Sync protocol! > > Are you saying this "major" bug will not be fixed? You should probably read my thorough explanation in Comment 18.
(In reply to Peter Lairo from comment #28) > https://developer.mozilla.org/en-US/docs/Firefox_Sync/ > Code_snippets#Count_types_of_bookmark_records Those code snippets are version-dependent, because they rely on code internals. They won't work on Nightly right now.
(In reply to Richard Newman [:rnewman] from comment #30) > (In reply to Peter Lairo from comment #28) > > > https://developer.mozilla.org/en-US/docs/Firefox_Sync/ > > Code_snippets#Count_types_of_bookmark_records > > Those code snippets are version-dependent, because they rely on code > internals. They won't work on Nightly right now. I just tried it with Firefox 16.0.1 and still get a syntax error.
(In reply to Richard Newman [:rnewman] from comment #29) > Sync on Android will download no more than 5,000 items. If you have more > than 5,000 bookmarks and folders, you'll see some jumbling. This was a > deliberate tradeoff: the alternative is bad for different reasons. What could possibly be worse than losing bookmarks? If a user has many bookmarks, he's likely a geek. If he's a geek (like me), he likely has WLAN at home (or a "flat rate" data-plan[1]). If he has WLAN at home, he can sync *any* number of bookmarks on the initial sync. [1] Most flat-rate plans in Germany are limited to x MB per month (mine is 500 MB/month for 10 €/m)). Then the speed is throttled to a nearly useless speed. > You should probably read my thorough explanation in Comment 18. Thanks for the pointer. I did. Below are my hopes: (In reply to Richard Newman [:rnewman] from comment #18) > * Remove the limit, and hope that your phone successfully syncs before it > consumes your entire mobile data allowance. Please do this!!! If need be, put up a dialog with the estimated data size and/or recommending WLAN and/or flat rate data plan. > * Increase the limit, and orchestrate a sequence of syncs so as to get a > whole 5-minute run at each of the big collections. Yes, increase the limit (if removing the limit is too bold). > * Batch downloads (Bug 730142), which will require some kind of similar > tracking. The total data amount of the batches is the same as for "no limit", so why bother? > * Build a local cache of server data against which we do the actual bookmark > syncing (two-phase sync). We wouldn't do any syncing against a part of the > folder tree until that tree was complete. Wouldn't that double the data volume (download all into cache + upload all onto server)? > The combination of batch downloads and two-phase sync is probably the only > way to 'solve' this problem, at the cost of delayed gratification (you > wouldn't see any bookmarks in Fennec until we had all of them) and increased > memory and CPU cost. Batch + two-phase is fine by me (after "no limit"). 1000x better than dataloss. > If we simply implement batching _and refuse to upload or reposition until > we've downloaded everything_, we get an 80% solution. Flawed, but would work > in the general case. That's 99% better than dataloss for the 0.1% of Firefox users[2]. ;-) [2] That's nearly a half-a-million users (450 million * 0.1%).
(In reply to Peter Lairo from comment #32) > (In reply to Richard Newman [:rnewman] from comment #29) > > Sync on Android will download no more than 5,000 items. If you have more > > than 5,000 bookmarks and folders, you'll see some jumbling. This was a > > deliberate tradeoff: the alternative is bad for different reasons. > > What could possibly be worse than losing bookmarks? This limitation doesn't lose bookmarks: it loses *structure*. There's nothing in Sync on desktop or Android that will result in a bookmark being deleted unless we process a record that describes an explicit deletion. The worst that can happen is that some structure is moved to Unsorted Bookmarks. This was better than not shipping bookmark sync. > If he has WLAN at home, he can sync > *any* number of bookmarks on the initial sync. That's not true. It takes some amount of time, usually 80-400msec, to perform a bookmark insertion, let alone the rest of the time spent on a sync. Out of the 300,000msec we have to work with, and assuming *zero* time spent on *anything* apart from database writes, and 80msec per write, we can insert no more than 3,750 bookmarks before Android kills our process. On a low-end Samsung, times are worse; on a top-end dual-core phone it'll be better. Still think we can just remove the limit? I referred to 5,000 as a "sanity limit" for a reason. > > * Batch downloads (Bug 730142), which will require some kind of similar > > tracking. > > The total data amount of the batches is the same as for "no limit", so why > bother? I don't think you understand the design constraints of this problem, or how batching applies. > > * Build a local cache of server data against which we do the actual bookmark > > syncing (two-phase sync). We wouldn't do any syncing against a part of the > > folder tree until that tree was complete. > > Wouldn't that double the data volume (download all into cache + upload all > onto server)? No. > Batch + two-phase is fine by me (after "no limit"). 1000x better than > dataloss. Software engineering is the process of turning requirements and schedule constraints into software that maximizes the satisfaction of the stakeholders. I think earlier comments in this thread explain why we don't already implement a complex but thorough solution, despite knowing that it would fix this problem (at the risk of introducing others). > > If we simply implement batching _and refuse to upload or reposition until > > we've downloaded everything_, we get an 80% solution. Flawed, but would work > > in the general case. > > That's 99% better than dataloss for the 0.1% of Firefox users[2]. ;-) But also takes more time to build, and introduces complexity. I know it's tempting, but please refrain from armchair quarterbacking this situation.
Flags: needinfo?
Guys, let's stick to constructive arguments, since this complex model won't get rewritten in one day (yeah, it's much more complex than it seems) and let's focus on the fact that my logs revealed an inconsistency. I am really curious if it's really a bug and we should just provide all the required logs to help with the investigation, instead of complaining.
Flags: needinfo?
Attached file Logs from home PC —
Attached file Logs from Android —
(My english is not very good so if u need extra explanation - write me). I have 2 computers - 1 laptop in work(Toshiba with Windows 7), 1 desktop PC (Windows 7) at home. Usually I update Firefox wit last versions. In last 10 days I obtained Samsung Galaxy Tab 2 (Android 4.0.3) and I installed Firefox on it before few days. One day at work I noticed that bookmarks are reordered (usually there are folders first and then unsorted bookmarks) - folders and unsoreted bookmarks were jumbled. i tried to find solution but meanwhile I see that some bookmarks are missing (fore sure some of folders were empty). I found that I can restore them with Xmarks add-on and restore version before 2 days. Then after sync all devices receive correct bookmarks. On same day or on next i made logs of home PC (see attached files) and today logs from Android (see attached file). Yestarday i saw that again I have problem on home PC. Now i see that some of bookmarks are duplicated (they are in folders and in unsorted). Total number of bookamrks is 88. So in few words - I am not sure when and how it is happened.
(In reply to Richard Newman [:rnewman] from comment #33) > (In reply to Peter Lairo from comment #32) > > > > What could possibly be worse than losing bookmarks? > > This limitation doesn't lose bookmarks: it loses *structure*. I place bookmarks into folders (sometimes without looking at the URL or the title - but who can remember hundreds of URLs & titles weeks later anyways). I know I will *need* the bookmarks later. For instance, I created folder for my company called "good ideas". In this folder, I collected about 20 bookmarks over a period of 6 months. It contained bookmarks from all over the place and covering a wide range of topics related to my company. With this bug, the folder and all it contents got moved into the Unsorted Bookmarks folder; my "Good Ideas" folder is empty, and the bookmarks are *scattered* among a few thousand other bookmarks I had collected (and usually placed into folders!). There is *no way* I am able to find and identify the bookmarks I had placed into my "Good Ideas" folder. The same goes for about 50 other (now empty) folders. So, for all intents and purposes, this bug does "lose" bookmarks! Once that sinks in, I think you will realize how frustrated (angry) some four-hundred-and-fifty-thousand Firefox users are right now. > Out of the 300,000msec we have to work with, and assuming *zero* time spent > on *anything* apart from database writes, and 80msec per write, we can > insert no more than 3,750 bookmarks before Android kills our process. On a > low-end Samsung, times are worse; on a top-end dual-core phone it'll be > better. So it's an Android limit. OK. Thanks for explaining it. > > Batch + two-phase is fine by me (after "no limit"). 1000x better than > > dataloss. > > Software engineering is the process of turning requirements and schedule > constraints into software that maximizes the satisfaction of the > stakeholders. OK. Then do it! ;-) > I know it's tempting, but please refrain from armchair quarterbacking this > situation. Well, at least this "armchair quarterbacking" made you aware that this bug is, from the users' perspective, a *dataloss* issue.
Depends on: 814331
Depends on: 814801
(In reply to Richard Newman [:rnewman] from comment #22) > Interesting. Given that you should have no more than 3,000 records (2618 > bookmarks, 350 folders), I am surprised that Android Sync was able to > download 5,000. > > Would you mind running some JS for me to see what counts you have? Dear Richard, I have provided feedback for this request in comment 23 and I am curious if you were able to reconcile it with the logs I provided. Is this another bug? Do you need any other logs / feedback / help to sort this out? Thanks! Mihai
(In reply to Mihai Todor from comment #39) > (In reply to Richard Newman [:rnewman] from comment #22) > > Interesting. Given that you should have no more than 3,000 records (2618 > > bookmarks, 350 folders), I am surprised that Android Sync was able to > > download 5,000. > > > > Would you mind running some JS for me to see what counts you have? > > Dear Richard, > > I have provided feedback for this request in comment 23 and I am curious if > you were able to reconcile it with the logs I provided. Is this another bug? > Do you need any other logs / feedback / help to sort this out? Hi Mihai, it's more likely there is an error in the record counting/printing Android Sync logic. I have a few minutes now, I'll investigate.
Thanks Richard! Also, I've noticed that there exists a "Places Maintenance" extension in some of the comments for one of the related bugs and, out of curiosity, I've installed it (current version, 1.3). Here's the output for "Deep Check": > Integrity check + The database is sane > Coherence check + The database is coherent > Orphans expiration + Database cleaned up > Vacuum Initial database size is 30720 KiB + The database has been vacuumed Final database size is 30720 KiB > Statistics Database size is 30720 KiB user_version is 21 page_size is 32768 cache_size is 128 journal_mode is wal synchronous is 1 History can store a maximum of 61107 unique pages Table moz_places has 42694 records Table moz_historyvisits has 57885 records Table moz_inputhistory has 208 records Table moz_hosts has 4011 records Table moz_bookmarks has 3184 records Table moz_bookmarks_roots has 5 records Table moz_keywords has 2 records Table sqlite_sequence has 1 records Table moz_favicons has 4431 records Table moz_anno_attributes has 10 records Table moz_annos has 983 records Table moz_items_annos has 41 records Table sqlite_stat1 has 15 records Index sqlite_autoindex_moz_inputhistory_1 Index sqlite_autoindex_moz_hosts_1 Index sqlite_autoindex_moz_bookmarks_roots_1 Index sqlite_autoindex_moz_keywords_1 Index sqlite_autoindex_moz_favicons_1 Index sqlite_autoindex_moz_anno_attributes_1 Index moz_places_faviconindex Index moz_places_hostindex Index moz_places_visitcount Index moz_places_frecencyindex Index moz_places_lastvisitdateindex Index moz_historyvisits_placedateindex Index moz_historyvisits_fromindex Index moz_historyvisits_dateindex Index moz_bookmarks_itemindex Index moz_bookmarks_parentindex Index moz_bookmarks_itemlastmodifiedindex Index moz_places_url_uniqueindex Index moz_places_guid_uniqueindex Index moz_bookmarks_guid_uniqueindex Index moz_annos_placeattributeindex Index moz_items_annos_itemattributeindex I did add more bookmarks in the mean time, but it looks like I'm still far away from 5000. Also, I have the History box unchecked in the Sync options, so I would say that those records don't count, right?
(In reply to Mihai Todor from comment #41) Thanks for taking a look. > I did add more bookmarks in the mean time, but it looks like I'm still far > away from 5000. Yup. Note that you probably don't need to hit 5,000 for there to be a problem; you can likely also suffer in some cases if there's an unreliable network. This is hypothetical; I haven't checked the code to see if this problem can occur, but it's an example of the intrinsic flaw in one-pass syncing of structured data. Say we download a bunch of bookmarks onto your phone (the first 50 of 3,000, perhaps). There's no parent folder for them yet (that's hanging around in position 2,000), so we put them in Unsorted Bookmarks -- they have to live somewhere. We can't just batch them in memory; 2,000 records is too much for the amount of heap we have to work with. They get a modified time that's now later than their time on the server (because the local record was modified; that's what it means). We also bump the modified time of Unsorted Bookmarks, because we need to upload a record for that. Now AT&T drops your data connection. The sync fails. We don't advance the sync timestamp, so we'll sync again a little later. Next time we download the same records, they appear to be older than the ones we've stored locally -- the local ones 'win'. When the sync finishes, we've decided that those records must have been locally moved into Unsorted Bookmarks *after* they were originally moved into their correct parent folder, and we propagate that fact to your other devices. There is no difference in the data model between this scenario and you moving a bookmark on your desktop then moving it on your phone: they both involve reconciling two records with different modified times. There are other knock-on effects, too: we need to compose a consistent folder order for our partially filled Android bookmarks, and that will end up being propagated to other devices. Those other devices will try to figure out a sane order, given that they know of more bookmarks than the Android device, and you'll see jumbling. We don't have transactional storage on Android (at least, not from Sync's side of the database), so we can't rollback if there's a failure. And in any case, transactionality isn't a good idea when you've got five minutes to work and the user is trying to browse! You'll get collisions all the time. The only solution here is to restrict bidirectional syncing to a failure-proof local operation, limiting the failure-prone server interaction to an operation on a non-canonical cache. Anything else is -- like the current approach -- an approximation that works most of the time, with some failure modes. > Also, I have the History box unchecked in the Sync options, > so I would say that those records don't count, right? History and bookmarks are unrelated in this instance, regardless.
Uh, that's really nasty :( Still, during the experiments, my phone was connected via WiFi to a home network that I like to consider very reliable. Nevertheless, since there are so many failure points in this model, it's really hard to take a guess at the root cause... Good luck with the new version.
(In reply to Mihai Todor from comment #43) > Still, during the experiments, my phone was > connected via WiFi to a home network that I like to consider very reliable. Could be all manner of things; perhaps Android decided to interrupt the sync, perhaps the phone ran out of memory. We don't get a chance to do any kind of cleanup or recovery, so any intermediate changes we made just… stick. > Nevertheless, since there are so many failure points in this model, it's > really hard to take a guess at the root cause... Good luck with the new > version. Thanks!
FWIW: I had 7,000 bookmarks. After running the CheckPlaces add-on[1], I was able to reduce the number down to 3,300 bookmarks. That's still too many for Firefox's sync. My "solution" is that I was forced to turn off sync on *all* my Firefox devices (computers, laptop, tablet, and smartphone). BTW: It seems mozilla.org has managed to **** off and drive away the developer of this very useful add-on: see http://www.andyhalford.com/ [1] http://mac.softpedia.com/get/Internet-Utilities/CheckPlaces.shtml
(In reply to Peter Lairo from comment #45) > FWIW: I had 7,000 bookmarks. After running the CheckPlaces add-on[1], I was > able to reduce the number down to 3,300 bookmarks. That's still too many for > Firefox's sync. Not strictly true, but you're certainly more vulnerable to some classes of problems than the typical 10-20 bookmark user. > My "solution" is that I was forced to turn off sync on *all* > my Firefox devices (computers, laptop, tablet, and smartphone). You know, you could just turn off bookmark sync if you're not confident in its operation; you'll still get history, passwords, prefs, etc. Bookmarks is by far the most complicated engine (with add-on sync a close second), and the only one which involves cross-record dependencies, which is the part that makes it vulnerable to failure. Just uncheck "bookmarks" in Sync preferences on any device. > BTW: It seems mozilla.org has managed to **** off and drive away the > developer of this very useful add-on: see http://www.andyhalford.com/ FWIW, I've never had any involvement with, or seen any interaction with, Andy Halford. And as one of the longest-standing Sync client developers (second in line after it stopped being called Weave, and the longest-standing current developer), I feel fairly confident in saying that the Sync team as a whole has never had any interaction with him. Anyway, let's keep this bug on track; let's leave project discussion, complaints, and speculation for IRC or mailing lists.
(In reply to Richard Newman [:rnewman] from comment #46) > Not strictly true, but you're certainly more vulnerable to some classes of > problems than the typical 10-20 bookmark user. The following point is to make sure that the severity of this bug is not whitewashed: "more"? That sounds like the understatement of the year. The real issue seems to be *how much* more, and what are the consequences when the bug kicks in? I'm guessing that I'm a "lot" more vulnerable and the consequences are "huge". > You know, you could just turn off bookmark sync if you're not confident in > its operation; you'll still get history, passwords, prefs, etc. Thanks for the pointer. That takes care of 5% of my usage for sync; bookmarks (this bug) is 95% of my usage. > Just uncheck "bookmarks" in Sync preferences on any device. I couldn't find where to do this on my Android 4.1.2 phone and Android 4.2 tablet. The syncing seems to be all or nothing. > FWIW, I've never had any involvement with, or seen any interaction with, > Andy Halford. Good to hear that. He might have had a problem with mozilla.org's add-ons department. Anyhow, it would probably be beneficial for the sync department to solicit his expertise. He might know things about syncing that is currently not available in moz's sync department (He already had a functioning sync add-on[1]; although I don't know how fine-grained his syncing was). [1] http://mac.softpedia.com/get/Internet-Utilities/SyncPlaces.shtml Good luck, and I hope you create a solution soon.
(In reply to Peter Lairo from comment #47) > The following point is to make sure that the severity of this bug is not > whitewashed: To be clear: there is no need for you to attempt to change my opinions on this bug. More bug comments will not get any work done any faster, which is why I try to encourage people to respect the first rules of Bugzilla Etiquette (<https://bugzilla.mozilla.org/page.cgi?id=etiquette.html>) and stick to comments that contribute to building a solution. There are known limitations in this, as with all software; the concern is always around resource allocation and scoping. We have a good understanding of bookmark sync's limitations. The current solution works well enough for the vast, vast majority of users (and has done for several years, with a very low percentage of problems since 2010), and for the others it causes recoverable and non-critical problems (according to most people's view of "critical"). (Remember that the majority hardly use bookmarks, and the majority of the rest don't care about organization; instead they "star" bookmarks.) That you don't fall into the former category, and that your view of "critical" is different, is regrettable, but reiterating how unhappy you feel isn't going to somehow jam three months' of work into the schedule in an instant. That sounds harsh, but that's the way it is. Your particular requirements have only a small effect on the project as a whole; that's how communities work. Sure, the programmer part of my brain wants to drop everything else, make some coffee, and start work on the perfect, optimal, bulletproof solution (I even have a pretty good idea of what to do). The senior engineer part insists on an evaluation of costs and benefits, given constrained resources. Bug 814331 is relatively urgent, because it wipes out a swath of risk at little cost; a total rewrite is not, and so I'll bear that in mind when discussing future work. Software engineering involves conscious tradeoffs. That applied years ago, when desktop Sync was written; it applied in January, when Android Sync was written; and it applies now. > > Just uncheck "bookmarks" in Sync preferences on any device. > > I couldn't find where to do this on my Android 4.1.2 phone and Android 4.2 > tablet. The syncing seems to be all or nothing. Android "Settings" > "Firefox Sync" > "What to sync". But the same setting applies wherever you set it, so you can do it on your desktop. > Good luck, and I hope you create a solution soon. Thank you!
(In reply to Richard Newman [:rnewman] from comment #48) > I try to encourage people to respect the first rules of Bugzilla > Etiquette (<https://bugzilla.mozilla.org/page.cgi?id=etiquette.html>) I think that my comments do respect the Bugzilla Etiquette! They illuminated the heretofore unmentioned and (nearly) irreversible problem of users who have many bookmarks and who have organized them into folders (i.e. advanced users and long-time users). My comments therefore were both "constructive and helpful". > There are known limitations in this, as with all software; the concern is > always around resource allocation and scoping. That is clear (and obvious to me). However, allocation and scoping is only effective when the severity of a need is also understood. > for the others it causes recoverable and non-critical problems If spending 40-80 hours on opening all 3,500 bookmarks, trying to remember which folder they were in, and re-saving each one into its former folder location, then I guess this bug can be called "recoverable and non-critical". > reiterating how unhappy you feel isn't going to ... My intent is not to state how "unhappy" I am; my intent is to illuminate the *severity* of this bug (however many users it may affect). > Bug 814331 is relatively urgent, because it wipes out a swath of risk at little cost Thanks for the pointer. That would be a good interim band-aid. > Android "Settings" > "Firefox Sync" > "What to sync". That setting doesn't seem to exist on my Android phone (Firefox Mobile 17.0, Nexus S, Android 4.1.2). > But the same setting applies wherever you set it, so you can do it on your desktop. Good to know. Thanks!
(In reply to Peter Lairo from comment #49) > I think that my comments do respect the Bugzilla Etiquette! They illuminated > the heretofore unmentioned and (nearly) irreversible problem of users who > have many bookmarks… See Comment 18. With respect, you're adding nothing new to this discussion. > My intent is not to state how "unhappy" I am; my intent is to illuminate the > *severity* of this bug (however many users it may affect). That's already understood. But remember that number of affected users is very significant. > > Android "Settings" > "Firefox Sync" > "What to sync". > > That setting doesn't seem to exist on my Android phone (Firefox Mobile 17.0, > Nexus S, Android 4.1.2). Settings > Accounts & sync > Firefox Sync, on pre-JellyBean.
(In reply to Richard Newman [:rnewman] from comment #50) > (In reply to Peter Lairo from comment #49) > > > I think that my comments do respect the Bugzilla Etiquette! They illuminated > > the heretofore unmentioned and (nearly) irreversible problem of users who > > have many bookmarks… > > See Comment 18. With respect, you're adding nothing new to this discussion. With respect, comment #18 does not address the severity as I have, merely that about 0.1% of users are affected (that's 450,000 if Firefox's most loyal users!) and a description of sync's limitations and how hard it is to solve the problem (all good stuff, but it's not the severity). > > My intent is not to state how "unhappy" I am; my intent is to illuminate the > > *severity* of this bug (however many users it may affect). > > That's already understood. But remember that number of affected users is > very significant. OK, the "quantity" (i.e. number of users affected) is "very" significant (You are the one who decides how significant half-a-million users are). Hopefully, the "quality" (i.e. the severity) is also very significant. > > > Android "Settings" > "Firefox Sync" > "What to sync". > > > > That setting doesn't seem to exist on my Android phone (Firefox Mobile 17.0, > > Nexus S, Android 4.1.2). > > Settings > Accounts & sync > Firefox Sync, on pre-JellyBean. As stated, I use Jelly Bean (4.1.2 and 4.2), and there is no such option. Under System Settings / Firefox Sync, I only see: +-----------------------+ | < Firefox Sync | | Accounts | | (/) MyUserName | | Firefox [x] | <-- this item is on a separate screen | Options | after clicking on MyUserName | Pair a Device. | +-----------------------+ The Menu button only offers: Sync now, Remove account, and Help.
(In reply to Peter Lairo from comment #51) > As stated, I use Jelly Bean (4.1.2 and 4.2), and there is no such option. Ah, coming in Firefox 19. Sorry, I use Aurora.
Depends on: bookmarksync
Whiteboard: [sync:bookmarks]
Hi, I've just ran into this problem as well, recently, after I've bought an tablet (nexus 7). Have already asked about it here https://support.mozilla.org/en-US/questions/946851 before I discovered this bug thread. Though I'm not a developer myself like I've noticed most of you are, I'm quite skilled with systems, using Linux mostly, thus feel free if any cooperation of mine might be required. One significant note to what I've noticed Richard mentioned before, that mostly users with huge ammount of bookmarks are affected, I don't agree. I've just tested this problem with another sync account of another family member of mine (created another account for him on my tablet and performed an sync), while this user has really small amount of bookmarks (if 50 items, than I'm perhaps exaggerating), and the order of bookmarks was broken too, just like with my own account (I'm having quite a lot items, but I don't know how much precisely).
(In reply to 8472 from comment #53) > One significant note to what I've noticed Richard mentioned before, that > mostly users with huge ammount of bookmarks are affected, I don't agree. Problems are possible for anyone (even, in principle, desktop-only users). That's well understood. Users with more than a few thousand bookmarks are simply very, very likely to have problems. The proportion of users with 10,000 bookmarks who will see issues is approximately 100%. The proportion of users with 50 bookmarks who will see issues is almost 0%, but non-zero. There are simply several hundred times as many users with 50 bookmarks as there are with 10,000, so we'll see a spread of reports. (I've seen this *once*, about a year ago, on one of my profiles with ~100 bookmarks, using a pre-release version.) Bug 814331 causes us to refuse to sync bookmarks if you have too many, so we're simply left with the background hum of occasional failures, rather than the systemic issue. > I've just tested this problem with another sync account of another family > member of mine (created another account for him on my tablet and performed > an sync), while this user has really small amount of bookmarks (if 50 items, > than I'm perhaps exaggerating), and the order of bookmarks was broken too, > just like with my own account (I'm having quite a lot items, but I don't > know how much precisely). How old was their sync account?
Flags: needinfo?(b5b5b5b5)
(In reply to Richard Newman [:rnewman] from comment #54) > > I've just tested this problem with another sync account of another family > > member of mine (created another account for him on my tablet and performed > > an sync), while this user has really small amount of bookmarks (if 50 items, > > than I'm perhaps exaggerating), and the order of bookmarks was broken too, > > just like with my own account (I'm having quite a lot items, but I don't > > know how much precisely). > > How old was their sync account? both (mine and the other one) sync accounts we use were founded in around of September 2010
Flags: needinfo?(b5b5b5b5)
> both (mine and the other one) sync accounts we use were founded in around of > September 2010 It's entirely possible that a bookmark record that causes unexpected behavior was just hangin' out on the server. It got processed by the new device, and caused a change that was re-propagated to your desktop. This kind of issue is very hard to track down.
Hi again, Pls don't take me wrong, I'm just being curious. This bug is already open since 2012/06, is already P1, but when do you expect this to get fixed? thx
(In reply to 8472 from comment #57) > Hi again, > > Pls don't take me wrong, I'm just being curious. > This bug is already open since 2012/06, is already P1, but when do you > expect this to get fixed? > thx I do not expect significant further investment in the existing Sync codebase. Any remaining issues with bookmark sync (such as a lack of robustness in handling some distributed failure scenarios) are not so much bugs as failings in design; that means they're not easy to fix, and instead require a new approach. We had intended to invest time in a new iteration of Sync, which would allow us to fix issues like this without breaking compatibility (by breaking it once!). That's probably not going to happen; instead, there's an effort called PICL to take a run at browser data in the cloud from a different direction. Take a look at http://groups.google.com/group/mozilla.dev.identity/ if you want to find out more about this.
ok, any assumption, about when any even an replacement solution you've described, will be available?
(In reply to 8472 from comment #59) > ok, any assumption, about when any even an replacement solution you've > described, will be available? I have too little control over the outcome, and not enough insight, to be able to offer any advice here. My best suggestion is to inquire at http://groups.google.com/group/mozilla.dev.identity/ .
Product: Mozilla Services → Android Background Services
(In reply to Richard Newman [:rnewman] from comment #60) > (In reply to 8472 from comment #59) > > ok, any assumption, about when any even an replacement solution you've > > described, will be available? > > I have too little control over the outcome, and not enough insight, to be > able to offer any advice here. My best suggestion is to inquire at > http://groups.google.com/group/mozilla.dev.identity/ . I could not find any mention of this problem in that newsgroup. Why would switching the sync login from the password manager to BrowserID have any positive impact on this data-loss bug? I've posted this question in the newsgroup. Here's the Subject and link: Will syncing large number of bookmarks (PC <--> Android) be handled here? (see Bug 769476) news://news.mozilla.org:119/NOmdnQIcecGVcijMnZ2dnUVZ_vKdnZ2d@mozilla.org
PiCL is not an alternative login to Sync; it's a new product. I know that they plan to use a different approach to exchanging changes, which should be resistant to some of the flaws in the design of Sync. And I've outlined some of the design parameters to those folks a few times.
This bug is on the PICL radar. I can't make any promises right now on what size of hierarchical bookmarks we will promise to sync gracefully or what limits we will impose, but we are certainly designing it to take these situations into account.
Priority: P1 → --
Looks like this Problem still exist. I had from one of my bookmarks two favorites stored. The Second was not Synced on my Android Device, and so he had every time change the ordner from this "unknown" favorite.
I have this too. It doesn't matter if I reorder on a PC and then sync, Android Firefox keep overwriting that latest sync data
(In reply to Ferry Huberts from comment #66) > I have this too. > It doesn't matter if I reorder on a PC and then sync, Android Firefox keep > overwriting that latest sync data I disconnected the Android Firefox and still my bookmarks get reordered. Then I disconnected the desktop Firefox, reconnected it, and again the bookmarks get reordered. Quite nasty. How to I get rid of this reordering? It messes with my muscle memory.
(In reply to Ferry Huberts from comment #67) > (In reply to Ferry Huberts from comment #66) > > I have this too. > > It doesn't matter if I reorder on a PC and then sync, Android Firefox keep > > overwriting that latest sync data > > I disconnected the Android Firefox and still my bookmarks get reordered. > Then I disconnected the desktop Firefox, reconnected it, and again the > bookmarks get reordered. > Quite nasty. > How to I get rid of this reordering? It messes with my muscle memory. Firefox 41.0.2 Fedora 23, x64
(In reply to Ferry Huberts from comment #67) > How to I get rid of this reordering? It messes with my muscle memory. The issue is with the data in your Sync account, not with the devices themselves. The solution in your case is to re-sync all of your bookmarks. Add the Sync spinner to your toolbar. Firefox menu > Customize > drag the Sync spinner to your toolbar. This will allow you to see when a sync has completed, and also trigger a new one. You can drag it away again when you're done with these steps, if you like. 1. Make sure your desktop is on a reliable network. 2. On your desktop, uncheck Bookmarks in Sync Preferences. You should see the spinner spin then stop. 3. Check Bookmarks again. Wait for the spinner to stop. Then, if you wish: 4. Make sure your Android device is on a reliable network. 5. Sign in to your Firefox account. 6. Wait five minutes before switching applications, just in case.
I've tried the things you suggested and several more. I even remove the sync account from the Android device. Sync keeps re-ordering my bookmarks. Especially after I've made changes. Like move bookmarks into folders.
(In reply to Ferry Huberts from comment #70) > Sync keeps re-ordering my bookmarks. > Especially after I've made changes. Like move bookmarks into folders. If you can file a new bug with your sync logs, ideally with bookmark sync logging set to Trace, that would help pin down what's happening.
And how would I do that?
And now this becomes even more wonderful: I added some bookmarks and a while later some bookmarks folders were re-arranged. Then went into the sync prefs and see my surprise: bookmark sync was turned OFF! Clearly there is a lot wrong in this area.
Severity: major → normal
Priority: -- → P1
See Also: → 1352947
Assignee: nobody → gkruglov
Will get to this and other ordering bugs once telemetry is over the line.
Assignee: gkruglov → nobody
Priority: P1 → P2
And the reordering happened to me again, twice, when adding two new machines. Big bummer. My workaround now is to first export all bookmarks, then add a new machine, then importing from the exported ones the make the order right again. Kind of defeating why sync is there.
(In reply to Ferry Huberts from comment #75) > And the reordering happened to me again, twice, when adding two new machines. > Big bummer. I'm curious what the About Sync add-on has to say: https://addons.mozilla.org/en-US/firefox/addon/about-sync/ You probably have some bad server data; disabling then re-enabling 'bookmarks' in your Sync preferences might make a difference.
Ok so I added another machine and let it sync fully on a stable network link. That seemed to work ok. It seems - to me - that sync has a problem with unstable links and/or incomplete syncs. You should really make the sync transactional (with transfer checksums) so that it's an all-or-nothing thing. You can do this per block to sync or for the complete sync, whatever your design dictates.
(In reply to Ferry Huberts from comment #77) > You should really make the sync transactional (with transfer checksums) so > that it's an all-or-nothing thing. Yes, this is well understood: see Bug 814801 and friends. Firefox for iOS does this correctly.
Product: Android Background Services → Firefox for Android
Taking into consideration the discussions above, I am removing the qawanted keyword.
Keywords: qawanted
We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: