545647 - Bookmarks Toolbar wrong sort order and missing elements

Status: RESOLVED FIXED

Description

[RNicoletto]

I did a fresh installation of Weave on both my Firefox Work profile (Firefox 3.6 en-US on Windows XP Professional SP3) and my Firefox Home profile (Firefox 3.6 it-IT on Windows 7 64-bit Home Premium). Weave client in the Work profile is set to override Weave data with local data; as opposite Weave client in the Home profile is set to override local data with Weave data. As a result of the synchronization I experienced:
[1] data loss in the Bookmarks Toolbar (in the Home profile);
[2] wrong element sorting in the Bookmarks Toolbar (in the Home profile).

Here you are two screenshots showing both the problems:
Home profile ==> http://img9.imageshack.us/img9/5910/firefoxweavetoolbarit.png
Work profile ==> http://img40.imageshack.us/img40/9822/firefoxweavetoolbaren.png

The lost Bookmark Toolbar element are:
[*] GMail ==> https://mail.google.com/
[*] Facebook ==> http://www.facebook.com/
[*] Inbox ==> http://www.inbox.com/

Weave log of my Firefox Work profile ==> http://pastebin.com/f32e43ecc
Weave log (censored) of my Firefox Home profile ==> http://pastebin.com/f307157b

Comment 1

[Ed Lee :Mardak (PTO Dec 22-30)]

It seems like the source computer uploaded 680 bookmark items and the receiving end processed all 680 of them, but somehow they didn't end up in the right spots. 

The reason why some bookmarks are out of order is that the missing item prevents them from creating the right complete order. Notice that digg, KT!, .. PWD are in order but are in the wrong place because Inbox is missing.

Can you find where the missing bookmarks were moved to? Unfortunately there isn't a good way to search for the bookmark and see what its containing folder is. And it seems like you have many bookmarks. Also, would you happen to still have the log from verbose.txt.1? (If you view the activity log, you can add a .1 to the location and view the rolled-over log.)

Comment 2

[Ed Lee :Mardak (PTO Dec 22-30)]

Oh. Did you have the work computer running Weave for a while before? And do you remember if you've moved any items, say Gmail or anything immediately before/after that bookmark? If so, this might be related to bug 544068.

Comment 3

[RNicoletto]

(In reply to comment #2)
> Oh. Did you have the work computer running Weave for a while before? And do you
> remember if you've moved any items, say Gmail or anything immediately
> before/after that bookmark? If so, this might be related to bug 544068.

Work profile is quite old with no recent changes to Bookmark Toolbar.
Home profile is fresh new one.
Weave has been installed recently, the above testcase was the 3rd attempt to synchronize my computers.

I'm going to retry the whole process and I'll post the verbose log.

Comment 4

[Mike Connor [:mconnor]]

Still waiting on the log...

Comment 5

[Jonny]

same problem here: 

- installed weave on ubuntu
- said to override existing stuff with stuff from server
--> bookmark toolbar missing over 90% of bookmarks (3 of ~100 appeared)
--> bookmarks menu missing EVERYTHING (all folders and subfolders and booksmarks i have there)
--> unsorted bookmarks appeared, BUT some subfolders from booksmarks menu appeared as regular folders in the unsorted bookmarks section, and it's a HUGE clusterfuck altogether

i am about 5 minutes away from going back to xmarks

Comment 6

[Jonny]

retried override from scratch
--> same result, all messed up beyond repair

Comment 7

[Mike Connor [:mconnor]]

what version of Weave is this?  That sounds a lot like the partial sync state, which puts you at Weave 1.1... basically we stick folders into unsorted until we have the full parent chain synced (i.e. if you have folder B inside of folder A, until folder A is fully synced it'll show up in unsorted.  once Weave syncs all bookmarks (it would sync in chunks) your full tree should be restored.

Weave 1.2 and higher syncs everything at once on desktop clients, in response to that situation...

Comment 8

[Jonny]

this IS 1.2.3 (latest) (or it was, it's xmarks now anyway)

Comment 9

[RejZoR]

I'm having a similar problem which is **** me off. It worked fine up until yesterday when i reinstalled my main system and also Firefox with Weave. All Toolbar bookmarks were gone and parts of it were scrambled between Unsorted bookmarks. If i try forcing the sync on my netbook, nothing changes (in form of also missing toolbar bookmarks on netbook). So apparently the bookmarks are still on server, they just don't move anywhere else. Going back to Xmarks as well...

Comment 11

[Cameron Dawson [:camd] (Away from Bugmail Dec 24-Jan 2)]

I am hitting this, too.  I'll attach screenshots of what's not syncing up.  I noticed it first on the toolbar, but also in the "Bookmarks Menu" it's wrong at this point.

I'm syncing between a mac and a windows 7 vm.  I can provide sync account details upon request.  See win7 bookmarks vs. mac bookmarks.

Comment 12

[Cameron Dawson [:camd] (Away from Bugmail Dec 24-Jan 2)]

Created attachment 480169 [details]
mac bookmarks

Comment 13

[Cameron Dawson [:camd] (Away from Bugmail Dec 24-Jan 2)]

Created attachment 480170 [details]
win7 bookmarks

Comment 14

[Cameron Dawson [:camd] (Away from Bugmail Dec 24-Jan 2)]

I think this is a blocking issue.  I have my bookmarks in specific order in specific folders as my personal workflow.  Having them change around frequently is frustrating and annoying.  I think it will seriously affect people's desire to use the feature.  It makes it feel unstable.  And me, personally, my bookmarks are very very important to me.

I have not noticed any bookmarks disappearing, though since the order is changing, I can't really tell if some are now gone.

I will say, however, that it does seem that if a bookmark is in a folder it will stay in that folder.  It's just the order within folders and the order of the folders themselves that is changing.

Comment 15

[Ragavan S [:rags]]

Can you please post detailed steps to reproduce? Would greatly help figuring out what's going on here.

Comment 16

[Dietrich Ayala (:dietrich)]

Marking blocking for investigation, since it's a dataloss issue to the extent that there's meaning in the order of the items.

The sync devs above seemed to have an idea about what's going on, but that's way back in May. Can we get a contemporary comment about why this might be happening?

Comment 17

[Mike Connor [:mconnor]]

We had one case (bookmarks and folders ended up in unsorted until we had the whole parent chain) that's long dead.  We don't have STR for this using current clients, which makes it hard reason about what might cause it.

I know that we had some fun issues at one point with 3.x clients where there could be bad DB states where parent info could be busted (i.e. the bm root was a child of a folder in the bm menu on my profile).  AIUI, Mak fixed a number of cases like this, and that means we won't get busted state from the DB... but that's 4.x only.

Can anyone reporting this issue reproduce after a Reset Sync using 4.0 trunk nightlies?

Comment 18

[Cameron Dawson [:camd] (Away from Bugmail Dec 24-Jan 2)]

I'll have to do more testing around with this.  But I think it may be related to doing a Sort by Name on a folder.  

As I recall what I did was to sort by name in a folder.  Then move a couple items.  The ones I moved explicitly stayed in their spots, but the other ones moved around.  Once I had moved each item by hand, then they started staying in place.  But I'll do more testing along these lines to see if I can prove this theory.

Comment 19

[Cameron Dawson [:camd] (Away from Bugmail Dec 24-Jan 2)]

OK, I did a reset sync on my Win7 and Mac and the bookmarks look like they're ordered identical now.  They weren't prior to the latest nightly update and reset sync.  So this issue may be resolved for me now.  I'll keep playing with it and report back.

Comment 20

[Mike Connor [:mconnor]]

At this point, I believe this is a non-issue for Fx4 users.  3.x users may be affected, but we don't have STR at this point, and that won't block Fx4.  3.x support will cease shortly after Fx4 release.

Comment 21

[Jonathan Baron]

Using nightly builds of Firefox 4.0b8pre for Linux, with sync (on home desktop, home laptop, and work desktop) I find my bookmarks constantly re-ordered, although nothing is lost.  I cannot figure out any pattern to the re-ordering, although it may be that the ones I don't use much go to the bottom.  This is a pain because I am used to using the keyboard to find certain bookmarks, e.g., alt-b, p, goes to the first bookmark folder starting with p, and then I continue the same way, but now that keeps moving around, even though I do use it.  For a while I was also ordering them alphabetically so that I could find things easily; that seems like a reasonable thing to do.

Comment 22

[Masayuki Nakano [:masayuki] (JST, +0900)(Away from Bugmail Dec 27-Jan 3))]

I can reproduce this bug on latest nightly builds. Even if I select "Replace All other devices with..." from a computer, another machine which has new profile receives wrong ordered bookmarks for bookmark toolbar.

Comment 23

[Masayuki Nakano [:masayuki] (JST, +0900)(Away from Bugmail Dec 27-Jan 3))]

I cannot reproduce this bug with clean profiles and new account.

On the other hand, my daily use account still has this bug. I enabled the debug log of sync. Then, the log said that:

2011-01-11 13:00:01	Store.Bookmarks      DEBUG	created bookmark 193 under 5 as ...

2011-01-11 13:00:02	Store.Bookmarks      DEBUG	created bookmark 235 under 5 as ...

2011-01-11 13:00:02	Store.Bookmarks      DEBUG	Reparenting orphans 193,196,198,199,201,203,208,212,213,215,218,221,223,229,231,235,237,241,242,247,249,250,252,260,263,264,268,269 to 271

On the latest bookmarks, the #235 is top of #271. However, the reparenting orphans inserts other items first. So, the stored array in bookmarks.js already lost the order. I'm not sure whether the order had been broken in server side or client side(bookmarks.js).

I think that if we cannot fix this bug at Fx4, we should document the way to restore the bookmark from automatically backed up file.

Comment 24

[Philipp von Weitershausen [:philikon]]

We definitely had an issue with bookmark reordering and reparenting. The reordering problem should be fixed with bug 623418. The reparenting problem is indeed as Masayuki describes in comment 23. We're fixing that in bug 618403.

Comment 25

[Masayuki Nakano [:masayuki] (JST, +0900)(Away from Bugmail Dec 27-Jan 3))]

(In reply to comment #24)
> We definitely had an issue with bookmark reordering and reparenting. The
> reordering problem should be fixed with bug 623418. The reparenting problem is
> indeed as Masayuki describes in comment 23. We're fixing that in bug 618403.

Bug 618403 was fixed, but I can still see the wrong ordered bookmarks when I sync them with new profile. The order on the server data isn't updated even when I reset the order from backup. This means that if the server data is damaged by some reason, the data will be never updated by restored data.

Comment 26

[Philipp von Weitershausen [:philikon]]

(In reply to comment #25)
> Bug 618403 was fixed, but I can still see the wrong ordered bookmarks when I
> sync them with new profile. The order on the server data isn't updated even
> when I reset the order from backup. This means that if the server data is
> damaged by some reason, the data will be never updated by restored data.

That's sadly true. We have bug 626796 on file for this. In the mean time you can choose Reset Sync to force a reupload.

Comment 27

[beingalink]

I had the issue of a mixed up bm toolbar because of sync. I did a reset of sync (let all data be overwritten with what's on the server) and enabled logging. As requested in #sync, I paste my sync-log here:

2011-01-26 17:36:01	Service.Main         INFO	Loading Weave @weave_version@
2011-01-26 17:36:01	Tracker.Bookmarks    TRACE	Loading json from disk: weave/changes/bookmarks.json
2011-01-26 17:36:01	Engine.Bookmarks     DEBUG	Engine initialized
2011-01-26 17:36:01	Engine.Forms         DEBUG	Engine initialized
2011-01-26 17:36:01	Engine.History       DEBUG	Engine initialized
2011-01-26 17:36:01	Engine.Passwords     DEBUG	Engine initialized
2011-01-26 17:36:01	Engine.Prefs         DEBUG	Engine initialized
2011-01-26 17:36:01	Engine.Tabs          DEBUG	Engine initialized
2011-01-26 17:36:01	Engine.Tabs          DEBUG	Resetting tabs last sync time
2011-01-26 17:36:01	Service.Main         INFO	Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b11pre) Gecko/20110126 Firefox/4.0b11pre
2011-01-26 17:36:01	Service.Main         DEBUG	Caching URLs under storage user base: https://phx-sync006.services.mozilla.com/1.0/beingalink/
2011-01-26 17:36:12	Service.Main         DEBUG	Clearing sync triggers.
2011-01-26 17:36:13	Service.Main         INFO	Logging in user beingalink
2011-01-26 17:36:14	Net.Resource         DEBUG	GET success 200 https://phx-sync006.services.mozilla.com/1.0/beingalink/info/collections
2011-01-26 17:36:14	Service.Main         DEBUG	Fetching global metadata record
2011-01-26 17:36:14	Net.Resource         DEBUG	GET success 200 https://phx-sync006.services.mozilla.com/1.0/beingalink/storage/meta/global
2011-01-26 17:36:14	Service.Main         DEBUG	Weave Version: @weave_version@ Local Storage: 5 Remote Storage: 5
2011-01-26 17:36:14	Service.Main         INFO	Sync key is up-to-date: no need to upgrade.
2011-01-26 17:36:14	Service.Main         DEBUG	Fetching and verifying -- or generating -- symmetric keys.
2011-01-26 17:36:14	Net.Resource         DEBUG	GET success 200 https://phx-sync006.services.mozilla.com/1.0/beingalink/info/collections
2011-01-26 17:36:14	Service.Main         INFO	Testing info/collections: {"clients":1296059438.43,"crypto":1291996263.01,"keys":1285342958.74,"meta":1292011384.38,"bookmarks":1296059485.01,"forms":1296059627.56,"history":1296059628.78,"passwords":1296055700.04,"prefs":1296059582.73,"tabs":1296059583.46}
2011-01-26 17:36:14	CollectionKeys       INFO	Testing for updateNeeded. Last modified: 0
2011-01-26 17:36:14	Service.Main         INFO	CollectionKeys reports that a key update is needed.
2011-01-26 17:36:15	Net.Resource         DEBUG	GET success 200 https://phx-sync006.services.mozilla.com/1.0/beingalink/storage/crypto/keys
2011-01-26 17:36:15	CollectionKeys       INFO	Updating collection keys...
2011-01-26 17:36:15	CollectionKeys       INFO	Setting CollectionKeys contents. Our last modified: 0, input modified: 1291996263.01.
2011-01-26 17:36:15	BulkKeyBundle        INFO	BulkKeyBundle being created for [default]
2011-01-26 17:36:15	CollectionKeys       INFO	Processing downloaded per-collection keys.
2011-01-26 17:36:15	CollectionKeys       INFO	Clearing CollectionKeys...
2011-01-26 17:36:15	CollectionKeys       INFO	Saving downloaded keys.
2011-01-26 17:36:15	CollectionKeys       INFO	Bumping last modified to 1291996263.01
2011-01-26 17:36:15	CollectionKeys       INFO	Collection keys updated.
2011-01-26 17:41:14	Tracker.Bookmarks    TRACE	onItemMoved: 4118
2011-01-26 17:41:14	Tracker.Bookmarks    TRACE	Adding changed ID: toolbar,1296060074
2011-01-26 17:41:15	Tracker.Bookmarks    TRACE	Saving json to disk: weave/changes/bookmarks.json
2011-01-26 17:41:21	Tracker.Bookmarks    TRACE	onItemMoved: 4082
2011-01-26 17:41:21	Tracker.Bookmarks    TRACE	Adding changed ID: toolbar,1296060081
2011-01-26 17:41:22	Tracker.Bookmarks    TRACE	Saving json to disk: weave/changes/bookmarks.json
2011-01-26 17:41:27	Tracker.Bookmarks    TRACE	onItemMoved: 4046
2011-01-26 17:41:27	Tracker.Bookmarks    TRACE	Adding changed ID: toolbar,1296060087
2011-01-26 17:41:28	Tracker.Bookmarks    TRACE	Saving json to disk: weave/changes/bookmarks.json

Comment 28

[Masayuki Nakano [:masayuki] (JST, +0900)(Away from Bugmail Dec 27-Jan 3))]

I cannot reset the broken bookmark order easily.

First, I restored bookmark on a machine which has correct order. Next, I did "Sync Now".

On another machine, I set the Sync Options as Replace all data on this computer with my Sync data. And I did "Sync Now". However, the broken order isn't corrected.

Finally, I removed the places.sqlite (after shutdown Fx) and I did "Sync Now" again, then, the machine received bookmarks with correct order.

I think this is too difficult for most users...

Comment 30

[Tracy Walker [:tracy]]

In some way, I think this is really a Firefox bug.  see bug 250787

The sort option is not a preference that can be synced.

Comment 31

[Ralf-Peter Rohbeck]

4.0RC2 on Debian squeeze on a slow system (EeePC 1005HA.)
Synchronization ("Replace all data on this computer with my Sync data") never downloads all bookmarks (about 8,100.) It always fails at some point. CPU is pegged at 100% for a considerable amount of time. Maybe the protocol times out due to performance issues? I'm thinking about vacuuming the sqlite DBs.

Comment 32

[Richard Newman [:rnewman]]

(In reply to comment #31)
> 4.0RC2 on Debian squeeze on a slow system (EeePC 1005HA.)
> Synchronization ("Replace all data on this computer with my Sync data") never
> downloads all bookmarks (about 8,100.) It always fails at some point. CPU is
> pegged at 100% for a considerable amount of time. Maybe the protocol times out
> due to performance issues? I'm thinking about vacuuming the sqlite DBs.

Ralf-Peter, please file a new bug in Firefox Sync : Backend, and attach your Sync log. We can easily dupe in Bugzilla, but it's very hard to manage two issues in the same bug. Thanks.

Comment 33

[Richard Newman [:rnewman]]

To summarize this ancient and gnarly bug:

* The old Sync bookmarks code was flawed in many ways. Philipp and I fixed lots and lots of bugs before Fx4, so issues around bookmarks disappearing or being reordered should be resolved. I think that covers the original issue.

* Imposed sort order doesn't get synced. Not much we can do about that. However, if you can reproduce some awful behavior in Firefox 4 or beyond, particularly involving sort + move + sync, then I'd love to see steps to reproduce... in a new bug.

RNicoletto, please reopen this if you're still seeing your original problem; otherwise, I'm closing this as FIXED, because I don't want to have to pick an individual bookmarks bug to dupe it to :D