[meta] Bookmarks dataloss (duplicated, reordered, moved, lost, attributes changed, etc) caused by Sync

RESOLVED WONTFIX

Status

()

defect
P3
normal
RESOLVED WONTFIX
9 years ago
9 months ago

People

(Reporter: philikon, Unassigned)

Tracking

(Depends on 4 bugs)

unspecified
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(blocking2.0 -)

Details

(Whiteboard: [sync:rigor][sync:bookmarks])

This is a meta bug tracking the various issues people are reporting with Sync 1.6 / Firefox 4.0b8.
Depends on: 621674
Depends on: 610501
Depends on: 621265, 618469
Depends on: 602937
Depends on: 621759
Depends on: 620889
Depends on: 620938
Depends on: 620901
Depends on: 617834
Depends on: 621782
Depends on: 617437, 617521
Depends on: 622448
Depends on: 622657

Updated

9 years ago
Blocks: 622588
No longer blocks: 622588
Depends on: 622588

Updated

9 years ago
No longer depends on: 618403

Updated

9 years ago
No longer depends on: 621489

Updated

9 years ago
No longer depends on: 610501
Depends on: 618403
Depends on: 621489
Depends on: 610501
Depends on: 623103
Depends on: 623295
Depends on: 623418
Depends on: 624677
blocking2.0: --- → ?
Not going to block on a meta bug, please nominate individual bugs instead.
blocking2.0: ? → -
Depends on: 545647
Depends on: 625681
Depends on: 626551
Depends on: 627169
Depends on: 627490
Depends on: 627497
Depends on: 632287
Depends on: 652519
Depends on: 682446
Whiteboard: [sync:rigor][sync:bookmarks]
Depends on: 926578

Comment 2

6 years ago
I did another test, just by re setting up sync.  The android device melted down due to the bookmarks it received, and subsequently the up-sync damaged my original bookmarks.
Depends on: 865129
Depends on: 956448
Depends on: 812348
Depends on: 859889
I leave my own machine with verbose logging enabled, just in case I spot this issue. It happens to me once a year or so. It just did.

An incremental fetch returned 257 modified bookmark records:

storage/bookmarks?newer=1403930192.75&full=1
=>
1403996503978	Sync.Engine.Bookmarks	INFO	Records: 257 applied, 257 successfully, 0 failed to apply, 0 newly failed to apply, 0 reconciled.

A bunch of folder records applied:

1403996502868	Sync.Store.Bookmarks	DEBUG	Applying record NRt8rIRDd4TQ
1403996502868	Sync.Store.Bookmarks	DEBUG	Local parent is 5lQz_glXhEtn
1403996502868	Sync.Store.Bookmarks	DEBUG	Record NRt8rIRDd4TQ is not an orphan.
1403996502869	Sync.Store.Bookmarks	DEBUG	Reparenting orphans  to 4980
1403996502874	Sync.Store.Bookmarks	DEBUG	Applying record O7v4Ols_TnMD
1403996502874	Sync.Store.Bookmarks	DEBUG	Local parent is SFIUyHkW7qPb
1403996502874	Sync.Store.Bookmarks	DEBUG	Record O7v4Ols_TnMD is not an orphan.

No orphans.

A bunch of non-folder records:

1403996503230	Sync.Store.Bookmarks	DEBUG	Applying record 7k3qvuFPy11P
1403996503230	Sync.Store.Bookmarks	DEBUG	Local parent is VF3wbfjLyJVz
1403996503230	Sync.Store.Bookmarks	DEBUG	Record 7k3qvuFPy11P is not an orphan.
1403996503233	Sync.Store.Bookmarks	DEBUG	Applying record 9itnzgJiL0Ok
1403996503233	Sync.Store.Bookmarks	DEBUG	Local parent is slf34ON7SOO-
1403996503234	Sync.Store.Bookmarks	DEBUG	Record 9itnzgJiL0Ok is not an orphan.
1403996503239	Sync.Store.Bookmarks	DEBUG	Applying record AyhmBi7Cv598
1403996503239	Sync.Store.Bookmarks	DEBUG	Local parent is mobile

and some special folders:

1403996503972	Sync.Store.Bookmarks	DEBUG	Applying record mobile
1403996503972	Sync.Store.Bookmarks	DEBUG	Processing special node: mobile
1403996503973	Sync.Store.Bookmarks	DEBUG	Applying record ngcvhzmHboB4
1403996503973	Sync.Store.Bookmarks	DEBUG	Local parent is B8WxDqcMMQqf
1403996503974	Sync.Store.Bookmarks	DEBUG	Record ngcvhzmHboB4 is not an orphan.
1403996503975	Sync.Store.Bookmarks	DEBUG	Reparenting orphans  to 5265
1403996503977	Sync.Store.Bookmarks	DEBUG	Applying record toolbar
1403996503977	Sync.Store.Bookmarks	DEBUG	Processing special node: toolbar

Those record applications resulted in reordering.


Interestingly, a client record had changed:

1403996501296	Sync.Engine.Clients	INFO	Records: 1 applied, 1 successfully, 0 failed to apply, 0 newly failed to apply, 0 reconciled.

so I think we can deduce that another client synced.



The immediately previous sync had failed for network reasons:

1403979724657	Sync.ErrorHandler	TRACE	Beginning stream copy to error-1403979724657.txt: 1403979724657
1403979724658	Sync.Service	DEBUG	Exception: Error: NS_ERROR_NET_TIMEOUT (resource://services-sync/resource.js:394:18) JS Stack trace: Res_get@resource.js:414:12 < Sync11Service.prototype._fetchInfo@service.js:540:14 < sync@enginesync.js:77:16 < onNotify@service.js:1271:7 < WrappedNotify@util.js:141:21 < WrappedLock@util.js:96:16 < _lockedSync@service.js:1261:12 < sync/<@service.js:1253:14 < WrappedCatch@util.js:70:16 < sync@service.js:1241:5
1403979724658	Sync.ErrorHandler	TRACE	Notifying weave:ui:sync:finish. Status.login is success.login. Status.sync is error.login.reason.network


10 logins that I hadn't used recently were downloaded and applied -- e.g., landfill.bugzilla.org, gogoinflight.com (I haven't flown for months). This really raises a red flag.

One login even threw because we tried to insert when it already existed (Bug 1031855):

1403996513250	Sync.Store.Passwords	DEBUG	Adding record {5df990fc-d78a-4566-be16-72006853a39a} resulted in exception [Exception... "This login already exists.'This login already exists.' when calling method: [nsILoginManager::addLogin]"  nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)"  location: "JS frame :: resource://gre/modules/services-sync/engines/passwords.js :: PasswordStore__create :: line 229"  data: no] Stack trace: PasswordStore__create()@resource://gre/modules/services-sync/engines/passwords.js:229 < Store.prototype.applyIncoming()@resource://services-sync/engines.js:329 < Store.prototype.applyIncomingBatch()@resource://services-sync/engines.js:297 < doApplyBatch()@resource://services-sync/engines.js:951 < doApplyBatchAndPersistFailed()@resource://services-sync/engines.js:966 < SyncEngine.prototype._processIncoming()@resource://services-sync/engines.js:1070 < SyncEngine.prototype._sync()@resource://services-sync/engines.js:1481 < WrappedNotify()@resource://services-sync/util.js:141 < Engine.prototype.sync()@resource://services-sync/engines.js:655 < _syncEngine()@resource://services-sync/stages/enginesync.js:199 < sync()@resource://services-sync/stages/enginesync.js:149 < onNotify()@resource://gre/modules/services-sync/service.js:1271 < WrappedNotify()@resource://services-sync/util.js:141 < WrappedLock()@resource://services-sync/util.js:96 < _lockedSync()@resource://gre/modules/services-sync/service.js:1261 < sync/<()@resource://gre/modules/services-sync/service.js:1253 < WrappedCatch()@resource://services-sync/util.js:70 < sync()@resource://gre/modules/services-sync/service.js:1241 < <file:unknown>


That site was last visited *in December*, so it should already have been synced long ago. That it existed with a different GUID is also bizarre.

So this looks like another client touched a bunch of records on the server based on very old state, and this client applied them, with predictably unhappy results.
(In reply to Richard Newman [:rnewman] from comment #3)

> An incremental fetch returned 257 modified bookmark records:

I should note that I have a little over 1000 bookmarks.
Was this intended only to track old sync problems, 
and is there any tracker bug, 
or need for a tracker bug relating to new sync ?

A very quick glance finds sync & bookmark related bugs: bug 950167, bug 978339, bug 974080, bug 1025989, bug 1046954, bug1050397, bug 1092722
The only thing changed in Sync is the auth layer, so this is the right bug for tracking bookmark issues.
Depends on: 1092722, 1046954

Updated

5 years ago
Depends on: 1124369
Depends on: 1125603
Depends on: 1182366

Updated

4 years ago
Depends on: 1228827
Depends on: 1230892
Depends on: 1164806, 1236209, 1163376
Summary: [meta] Bookmarks are duplicated, reordered and moved → [meta] Bookmarks dataloss (duplicated, reordered, moved, lost, attributes changed, etc) caused by Sync
Flags: firefox-backlog+
Priority: -- → P2
Depends on: 1245721
Removing priorities from all our meta sync bugs.
Priority: P2 → --
Depends on: 1234543

Comment 8

3 years ago
I've duplicated folders for History, Downloads and Tags on Firefox Developer Edition.

I have not however on Firefox Nightly.

Is this expected?
Flags: firefox-backlog+
Depends on: 974080
Priority: -- → P3
Hello, I think I know what is happening for bookmark sort broken by Android Sync.

I was able to reproduce issue while adding sync support for my personnal project: "A GNOME web browser"

I think is missing some code like this:
- When pushing a new record, it need to push parent with "children" array
- Take care of pusning parent record after all children are pushed

Example:
{'id': 'toolbar', 'type': 'folder', 'parentid': 'places', 'parentName': '', 'title': 'Barre personnelle', 'children': ['TE44RsH4ZPoL', 'XncDg9-VCj09', '7aogaBxve0hT', 'QrTnV2_3Sqg3', '_hRqIwWa2bEf', 'nm4iqlypoS4k', 'PDR9JW7OKzJF', '9J_r61iy3QGK', 'bvp3MOmXzD1F', 'Z1j4ThDYGB4V', 'tVz6m8kFgKbc', 'OjgmgPaI_qEA', 'P7I1IfNannnb', 'gMp1WKi47YFs']}
(In reply to Cédric Bellegarde from comment #9)

> I think is missing some code like this:
> - When pushing a new record, it need to push parent with "children" array
> - Take care of pusning parent record after all children are pushed

The sync clients on all platforms do indeed do this. There are a variety of interesting causes for this kind of issue, including partial writes (remedied by atomic batch uploads in Bug 1253051 and friends), partial reads (buffered downloads; Bug 1305563), client bugs (many solved by Bug 1258127), non-structural record application, extremely large folders (Bug 1321021), and many more.
Depends on: 1384487
(In reply to Richard Newman [:rnewman] from comment #10)
> Take care of pusning parent record after all children are pushed
> 
> The sync clients on all platforms do indeed do this. There are a variety of
> interesting causes for this kind of issue, including partial writes
> (remedied by atomic batch uploads in Bug 1253051 and friends), partial reads
> (buffered downloads; Bug 1305563), client bugs (many solved by Bug 1258127),
> non-structural record application, extremely large folders (Bug 1321021),
> and many more.

I think this is really an issue because: in Eolie Web Browser, I do not use folders but only tags but I have to deal with Firefox folders.

Updating a bookmark entry without updating "parentName", "parentid" and parent record should not remove the bookmark from parent, only update the entry.
(In reply to Cédric Bellegarde from comment #11) 
> Updating a bookmark entry without updating "parentName", "parentid" and
> parent record should not remove the bookmark from parent, only update the
> entry.

I haven't much context here - but who exactly is updating a bookmark record without specifying those fields?
>I haven't much context here - but who exactly is updating a bookmark record without specifying those fields?

I was wrong in my previous comment, this is not my issue.

My issue is that if you update a bookmark record without updating parent record (and it's children attribute), the bookmark position in parent is changed (by Firefox).
Cédric, is the bookmark record uploaded from a third-party client, or Firefox? Could you please post an example of the full JSON bookmark record that's being uploaded to the server? If you're changing the "parentid" in the bookmark, or omitting it entirely, Firefox will move it to "unfiled". JSON records need to specify all fields, including ones that haven't changed; we don't support "deltas" with only a subset of fields.
Flags: needinfo?(cedric.bellegarde)
As I sayed, I was wrong, I understand it's needed to set the parentid. And yes it's a third-party client.

No, the issue is that if you update a record with it's parentid but you need to update parent record (children attribute) to preserve order.
Flags: needinfo?(cedric.bellegarde)
(In reply to Cédric Bellegarde from comment #15)
> No, the issue is that if you update a record with it's parentid but you need
> to update parent record (children attribute) to preserve order.

Yes - for better or worse, that's how bookmarks work. Clients must update both, so this is WONTFIX.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
Component: Firefox Sync: Backend → Sync
Product: Cloud Services → Firefox
You need to log in before you can comment on or make changes to this bug.