Closed Bug 817445 Opened 12 years ago Closed 4 years ago

Bookmarks have disappeared and no new bookmarks can be created

Categories

(Firefox :: Sync, defect)

Product:
Firefox
Component:
Sync
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: roc, Unassigned)

References

(Depends on 1 open bug)

Details

(Whiteboard: [sync:bookmarks]) 

I've got my Firefox profile into a state where I can't create bookmarks anymore and all the bookmarks I used to have are gone.

Even after restarting, attempting to bookmark a page simply does nothing. Possibly related: Sync is no longer working; I get "Sync encountered an error while syncing: Unknown error. Sync will automatically retry this action."

This is a homemade build from changeset bc69705c162d (Nov 19).

In a debug build I get these warnings when I try to create a bookmark:
WARNING: NS_ENSURE_TRUE(NS_SUCCEEDED(rv) && (!isNull || aFolderId == 0)) failed: file z:/roc/mozilla-inbound/obj-ff-debug/toolkit/components/places/../../../../toolkit/components/places/nsNavBookmarks.cpp, line 1974
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file z:/roc/mozilla-inbound/obj-ff-debug/toolkit/components/places/../../../../toolkit/components/places/nsNavBookmarks.cpp, line 573
JavaScript error: resource://gre/modules/PlacesUtils.jsm, line 2226: NS_ERROR_ILLEGAL_VALUE: Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsITransactionManager.doTransaction]
we got some reports that Sync may corrupt bookmarks applying an unwanted change to roots... bug 791562 is adding protection against such actions and I know the Services team is looking into that as well. Are you syncing bookmarks with Android?

Something you may try for now, is to rename your places.sqlite to places.sqlite.corrupt, and on restart it should restore bookmarks from the last json backup.

If you're comfortable with that I would like to get a copy of the corrupt database through mail, otherwise I can probably give you some queries to run on it, through SQLite Manager add-on.

For example, do you see any NULL rows with this?
SELECT root_name, id, parent, title from moz_bookmarks_roots r
LEFT JOIN moz_bookmarks b ON r.folder_id = b.id
PS: restoring from json will lose history, so far no way around that, apart manual editing of the db, that I could probably do.

If you should still have all the roots, we can probably run some different checks on the database to verify its integrity.
The first warning
  NS_ENSURE_TRUE(NS_SUCCEEDED(rv) && (!isNull || aFolderId == 0),
                 NS_ERROR_INVALID_ARG);
happens because isNull is true and aFolderId is 2. I guess
the folder where we're trying to add the bookmark doesn't exist.

Indeed, I don't have a bookmarks toolbar folder, or any other folder other than "Unsorted Bookmarks".

Are we supposed to recover from a situation where these things have been deleted/lost?
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #3)
> Are we supposed to recover from a situation where these things have been
> deleted/lost?

well, it's a bit more complicated, in the UI we don't expose the root folders exactly cause we don't want the user to mess them up. We instead expose "links" to them, these links are restored when the Library is opened if they are corrupt. So far this code worked fine, maybe you could have hit a case where it doesn't?
The user should never be able to put bookmarks into a non-working state from the UI.

On the other side, if something (our UI can't, so must be an add-on, or synchronization, or manual editing) corrupts the real roots in the database, this is a data coherency breakage for which we don't have a fallback. IIRC there is a bug filed to create one.
I have to look at the db to exactly tell what happened, running the query in comment 1 would already be a good start though.
(In reply to Marco Bonardo [:mak] from comment #1)
> we got some reports that Sync may corrupt bookmarks applying an unwanted
> change to roots... bug 791562 is adding protection against such actions and
> I know the Services team is looking into that as well. Are you syncing
> bookmarks with Android?

Yes, but I haven't changed any bookmarks in Android for a while.

> Something you may try for now, is to rename your places.sqlite to
> places.sqlite.corrupt, and on restart it should restore bookmarks from the
> last json backup.

I'll do that.

> If you're comfortable with that I would like to get a copy of the corrupt
> database through mail, otherwise I can probably give you some queries to run
> on it, through SQLite Manager add-on.

I'll send you the corrupt profile.
(In reply to Marco Bonardo [:mak] from comment #1)
> Something you may try for now, is to rename your places.sqlite to
> places.sqlite.corrupt, and on restart it should restore bookmarks from the
> last json backup.

Doing this did not restore anything.

I don't see any places-related JSON files in my profile.
there should be a bookmarkbackups folder in your profile, with json in it.
So, looking at your database, it's quite clear something (if you just use Sync I can only guess it's that, if you use other add-ons would be good to add them to the investigation) decided to _delete_ your toolbar, menu and unsorted roots... This is pretty bad since it's unrecoverable.
We keep 10 days or more of backups, so I hope of your jsons is good.

We should port bug 791562 to all branches asap.
(In reply to Marco Bonardo [:mak] from comment #8)
> there should be a bookmarkbackups folder in your profile, with json in it.

also, once you have generated a new clean places.sqlite, you should be able to open the library and select a backup from the Import and backup / Restore menu if the automated process should not reimport it on startup (strange, could be the last backup was already corrupt)
(In reply to Marco Bonardo [:mak] from comment #9)
> So, looking at your database, it's quite clear something (if you just use
> Sync I can only guess it's that, if you use other add-ons would be good to
> add them to the investigation)

I don't use any relevant addons. Is there already a Sync bug filed covering this?

> We keep 10 days or more of backups, so I hope of your jsons is good.

I do have backed-up bookmarks:
-rw-r--r-- 1 roc Administrators 65679 Oct 25 01:50 bookmarkbackups/bookmarks-2012-10-25.json
-rw-r--r-- 1 roc Administrators 65679 Oct 26 00:11 bookmarkbackups/bookmarks-2012-10-26.json
-rw-r--r-- 1 roc Administrators 66309 Oct 27 10:18 bookmarkbackups/bookmarks-2012-10-27.json
-rw-r--r-- 1 roc Administrators 66094 Oct 31 13:31 bookmarkbackups/bookmarks-2012-10-31.json
-rw-r--r-- 1 roc Administrators 66094 Nov  2 01:20 bookmarkbackups/bookmarks-2012-11-02.json
-rw-r--r-- 1 roc Administrators 14621 Nov  8 02:51 bookmarkbackups/bookmarks-2012-11-07.json
-rw-r--r-- 1 roc Administrators 14621 Nov 15 10:41 bookmarkbackups/bookmarks-2012-11-14.json
-rw-r--r-- 1 roc Administrators 14621 Nov 22 16:23 bookmarkbackups/bookmarks-2012-11-22.json
-rw-r--r-- 1 roc Administrators  1119 Nov 28 03:21 bookmarkbackups/bookmarks-2012-11-28.json
-rw-r--r-- 1 roc Administrators  1119 Dec  3 11:22 bookmarkbackups/bookmarks-2012-12-03.json
-rw-r--r-- 1 roc Administrators  1119 Dec  4 00:02 bookmarkbackups/bookmarks-2012-12-04.json

Looks like something happened between Nov 2 and Nov 8 and then got worse between Nov 22 and Nov 28. I definitely did not intentionally delete the bookmarks that were deleted between Nov 2 and Nov 8. (But I'm not sure how I failed to notice the change since Nov 8...)
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #11)
> I don't use any relevant addons. Is there already a Sync bug filed covering
> this?

yes, bug 817232

> Looks like something happened between Nov 2 and Nov 8 and then got worse
> between Nov 22 and Nov 28. I definitely did not intentionally delete the
> bookmarks that were deleted between Nov 2 and Nov 8. (But I'm not sure how I
> failed to notice the change since Nov 8...)

Yes, I also find scary losing control over my local bookmarks, though off-hand I can't tell what happened in those dates, you may compare the jsons to check that and let us know if that was a dataloss or rather some normal operation.

Maybe Richard could be interested in getting a Sync log, needinfo-ing him.
Flags: needinfo?(rnewman)
(In reply to Marco Bonardo [:mak] from comment #12)
> Yes, I also find scary losing control over my local bookmarks, though
> off-hand I can't tell what happened in those dates, you may compare the
> jsons to check that and let us know if that was a dataloss or rather some
> normal operation.

I did check. As I said in comment #11, this was definitely not deliberate, some bookmarks disappeared that I rely on. I.e., it was dataloss.

I just created a new profile and synced it. The new profile got all my old bookmarks (without my restoring from any json), which is good; Sync still has all my data. In my corrupt profile, even though I moved my corrupt places.sqlite out of the way, Sync still doesn't put my old bookmarks back, although it has now put back the default bookmarks that my new profile had!

So now I'm really confused.
I looked at about:sync-log. I have three logs Nov 25-27 showing just failed attempts to sync due to network unavailable. Then on Nov 28 I have a log where the bookmarks fail to sync, looking like this:

1354061453280	Sync.Engine.Bookmarks	INFO	9 outgoing items pre-reconciliation
1354061453672	Sync.Engine.Bookmarks	WARN	Got exception "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsINavBookmarksService.getItemType] Stack trace: _buildGUIDMap()@resource://services-sync/engines/bookmarks.js:233 < resource://services-sync/engines/bookmarks.js:357 < _mapDupe()@resource://services-sync/engines/bookmarks.js:313 < _createRecord()@resource://services-sync/engines/bookmarks.js:397 < _reconcile()@resource://services-sync/engines.js:1166 < resource://services-sync/engines.js:872 < resource://services-sync/record.js:625 < Channel_onDataAvail()@resource://services-sync/resource.js:532 < <file:unknown>" building GUID map. Skipping all other incoming items.
1354061453672	Sync.Engine.Bookmarks	WARN	Reconciliation failed: aborting incoming processing.
1354061453673	Sync.Collection	DEBUG	mesg: GET success 200 https://phx-sync155.services.mozilla.com/1.1/rocallahan/storage/bookmarks?newer=1354057853&full=1
1354061453673	Sync.Collection	DEBUG	GET success 200 https://phx-sync155.services.mozilla.com/1.1/rocallahan/storage/bookmarks?newer=1354057853&full=1
1354061453673	Sync.Engine.Bookmarks	DEBUG	Records that failed to apply: mobile
1354061453674	Sync.Status	DEBUG	Status for engine bookmarks: error.engine.reason.unknown_fail
1354061453674	Sync.Status	DEBUG	Status.service: success.status_ok => error.sync.failed_partial
1354061453674	Sync.ErrorHandler	DEBUG	bookmarks failed: Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsINavBookmarksService.getItemType] Stack trace: _buildGUIDMap()@resource://services-sync/engines/bookmarks.js:233 < resource://services-sync/engines/bookmarks.js:357 < _mapDupe()@resource://services-sync/engines/bookmarks.js:313 < _createRecord()@resource://services-sync/engines/bookmarks.js:397 < _reconcile()@resource://services-sync/engines.js:1166 < resource://services-sync/engines.js:872 < resource://services-sync/record.js:625 < Channel_onDataAvail()@resource://services-sync/resource.js:532 < <file:unknown>

Looks like the bookmarks were already corrupt at that point. So I don't have any logs of a successful sync before the bookmarks were corrupt, sorry.
I managed to fix my profile by disconnecting from Sync, quitting, deleting places.sqlite, restarting, and re-setting up Sync.
> 1354061453280	Sync.Engine.Bookmarks	INFO	9 outgoing items pre-reconciliation
> 1354061453672	Sync.Engine.Bookmarks	WARN	Got exception "Component returned
> failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE)
> [nsINavBookmarksService.getItemType] Stack trace:
> _buildGUIDMap()@resource://services-sync/engines/bookmarks.js:233 <

That's exactly what happens when a bookmark root is missing.


> Looks like the bookmarks were already corrupt at that point. So I don't have
> any logs of a successful sync before the bookmarks were corrupt, sorry.

If Sync deleted the root, and nothing else went wrong, it would look like a successful sync. Without logOnSuccess = true, no log would be kept.


Sync shouldn't generate a deleted root record, so I'm puzzled how this is happening. I'll continue to look into it.
Flags: needinfo?(rnewman)
Maybe you could add code where we generate these records to check that we're not deleting the root, and prevent it from being generated. And possibly send telemetry notifications when that check is triggered.
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #17)
> Maybe you could add code where we generate these records to check that we're
> not deleting the root, and prevent it from being generated. And possibly
> send telemetry notifications when that check is triggered.

Considering only Mozilla clients, there are two platforms/codebases in which this could happen. Alas, neither of those have any hooks into telemetry for anything at all.

Fortunately, code reading should be enough to assess whether anything like this could happen; I just haven't had time to get to it yet.

Bug 817232 covers refusing to apply root deletions on desktop. Android already does something similar.
(In reply to Richard Newman [:rnewman] from comment #16)
> Sync shouldn't generate a deleted root record, so I'm puzzled how this is
> happening. I'll continue to look into it.

Heh, I'm puzzled as well, I did a check of the Places code and there's no place we may directly delete them with a query. the only left thing is an API call, but since we don't expose the roots in the UI I can't think of a way.
I'm pretty confident that Android Sync can't be directly at fault here.

Firstly, we only produce deleted records if they're already deleted in the database.

Secondly, prior to syncing bookmarks we run checkAndBuildSpecialGUIDs:

https://github.com/mozilla-services/android-sync/blob/develop/src/main/java/org/mozilla/gecko/sync/repositories/android/AndroidBrowserBookmarksDataAccessor.java#L196

which ensures that all of our expected roots exist *and are non-deleted*.

So something else weird must be happening: desktop Sync is doing something wrong, or desktop Sync is reconciling two records together such that a root is being merged with another, or there's another source of corruption (what happens if we try to move a root away from the Places root?).
(In reply to Richard Newman [:rnewman] from comment #20)
> (what
> happens if we try to move a root away from the Places root?).

It's basically the same as deleting it, everything stops working.
Bug 819262 has some tests to try to get an incoming record to reconcile with a local root, which could result in a move.

It does result in a reconciling match… but it doesn't cause a move, because the real local GUID is not the 'magic' GUID that we use on the wire for roots, so there's essentially nothing to move.

Still looking…
I'm going to move this bug into the Sync component. Marco, please make sure there are bugs open somewhere for any robustness stuff you want to add -- e.g., preventing root moves!
Component: Bookmarks & History → Firefox Sync: Backend
Product: Firefox → Mozilla Services
Depends on: bookmarksync
Whiteboard: [sync:bookmarks]
(In reply to Richard Newman [:rnewman] from comment #25)
> I'm going to move this bug into the Sync component. Marco, please make sure
> there are bugs open somewhere for any robustness stuff you want to add --
> e.g., preventing root moves!

bug 791562
I just had the same (or very similar thing happen).

Windows 7 x86_64
Firefox 20.0.1 (downloaded, not built)
Potential plugin relations: Smartest Bookmarks Bar 1.11, Stylish 1.3.2

I have a lot of bookmarks. Well, had.  The puny scrolling arrow at the bottom of my bookmark menu was insufficient and I grew frustrated. I installed SMB1.11 (above) thinking it might give me some scrollbars on the side of the menu list, but it reconfigured the actual menubar and that menu only (one of you mentioned that changing or deleting the bar may have been a factor).  I installed them on Tuesday, and used Stylish until I noticed it this morning. I disabled SMB immediately after realizing it didn't do what I wanted and seemed fishy.

I know that I had bookmarks yesterday afternoon.  I have tried the steps listed in the thread: deleting places.sqlite, resetting and unlinking sync then restarting FF, I even tried an extension (Places Maintenance 1.3) that fixes places.sqlite files (It said both my original and recreated databases were "sane." The old database is 41MB and the new one is 10.  The bookmarks manager will not import from a backup json, not even one from february.  

I noticed something was wrong yesterday when sync kept failing to finish.  I didn't notice the bookmarks until today.  Firefox on both my android tablet and phone say there are no Desktop bookmarks.  Between that and the sqlite file, I think maybe they were just deleted and that the file wasn't corrupt (assuming the DB checker is telling the truth.

When I get home tonight, I will see if my computers there have bookmarks and if they can be restored.  If so, I will just delete my work Firefox installation and try again.  I'll keep the install and profile around in case someone may want to do an autopsy.
My bookmarks on my home computer (which never had any of those plugins) are gone.  I can't restore from any of the backups.  It's the same as the other machine: 
Library>Imports&Backups>Restore>[date]

and nothing happens.  Not even a half second of pretending to process something.
sorry for the rapid replies.  I thought I had included this in my original note.  The errors I started getting in my synch logs (on both machines) are similar to this:

Sync.Store.Bookmarks	DEBUG	Failed to reparent item. Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsINavBookmarksService.moveItem] Stack trace: _reparentItem()@resource://gre/modules/services-sync/engines/bookmarks.js:603 < BStore_update()@resource://gre/modules/services-sync/engines/bookmarks.js:839 < Store_applyIncoming()@resource://services-sync/engines.js:268 < BStore_applyIncoming()@resource://gre/modules/services-sync/engines/bookmarks.js:548 < applyIncomingBatch()@resource://services-sync/engines.js:234 < doApplyBatch()@resource://services-sync/engines.js:813 < resource://services-sync/engines.js:924 < resource://gre/modules/services-sync/record.js:625 < Channel_onDataAvail()@resource://gre/modules/services-sync/resource.js:542 < <file:unknown>
We determined that this is nothing to do with Android Sync, and possibly nothing to do with Sync at all. We don't know what the fundamental cause is, but current Firefox includes code to defend against disappearing roots. If they're still disappearing, that would be weird.

The reason why importing a backup fails is that the DB is semantically corrupt, despite being structurally sound.

(Marco, you should probably add a check for missing roots to the Places Maintenance add-on...)



paul: you know about Bookmarks > Show All Bookmarks, right?

Regarding missing bookmarks at home: did you look in Unsorted Bookmarks? The trace in Comment 29 indicates that Sync isn't able to move bookmarks into the right place, which is a different manifestation of Bug 623295, and indicates that you have a corrupt DB at home.
(In reply to Richard Newman [:rnewman] from comment #30)
> (Marco, you should probably add a check for missing roots to the Places
> Maintenance add-on...)

It couldn't do much, no parts of Places can initialize if the roots are broken so the add-on doesn't get the connection. Getting a separate write connection would be scary for me, even if feasible.
At this point the best thing we can do is fix bug 478912 to make bookmarks init more tolerant with some added code complication.
That won't prevent dataloss but at least the product will keep working.

As you know also the Places APIs have protection against the removal of roots, so the only thing that may cause the issue is direct delete queries to the database. I'd really would like to have the list of add-ons installed in these profiles to verify if by chance some of them is doing direct querying.
I took bug 478912, will work on it next week.
I know about Bookmarks > Show All Bookmarks in the sense that it is where I am attempting to restore old bookmarks.  If you meant something more than that, maybe not.  

If my database is corrupt at home then it is also corrupt on 2 other machines then (and possibly 2 android devices as well). They all exhibit the same behavior with the bookmark toolbar being weird and sometimes not being able to create new bookmarks. And all happened on the same day (the file list of the auto json backups is consistently the same size for weeks going back, then change on the same day to about 2/3 and have been the same since.) Only one machine had the  two plugins I described earlier, so if those are related, then Synch had to have propagated the corruption one of them caused.

If it is the database that is corrupt, shouldn't I be able to restore the json  backup into a fresh database?  I think the backups might be corrupt, and even backups from 2 months ago won't import into a fresh install of firefox (fresh as in no file or directory or key on my computer had a reference to anything related to firefox). The json files are not empty, and seem to be valid json (I don't know the proper format of a bookmark backup to validate that).  

If Sync didn't cause it, I believe it at least spread it.  3 computers and two android devices lost bookmarks (but not history or passwords).  My computers all updated to FF 20.0.1 a few days before the sync failures started and a week before the bookmarks were removed from the three of them.  Could there have been some incompatibilities between versions? Some of these profiles are a couple fears old but a lot of the lost data has been hopping around my firefox installations for a very long time. I might try cleaning my home machine and trying to install FF19 and restoring a json backup from before.


Questions:

1) I don't suppose the Sync server has a backup from 2-3 weeks ago?  Is there a way for me to see what is on the Synch server? I'd like to know if this data exists and is just currently unavailable or if it is truly lost. As I mentioned, when I tell Sync to replace all local data with what Sync has, it warns me I am going to lose a few hundred bookmarks.  

2) I query my database, and it looks like there is bookmark related data in it.  Is there a utility version of "Export Bookmarks to XML" that can I can run manually? What about a standalone version of "Restore" that will export to HTML instead of overwriting. I'd rather have an HTML bookmark page than lose everything. I'd think the dev team might have something like this for testing and troubleshooting.
I installed FF19.0.2.  I tried restoring one of the old backup .jsons and it failed silently like all the other attempts.

One of the three machines I mentioned above is running ubuntu.  It's bookmarks are also missing, but when I try to restore a backup, I get the "unable to process backup" notification. The Windows versions fail silently.

I haven't actually logged into this machine's gui in about three months. It has been running the whole time, so Firefox would still have been able to Sync whenever it wanted. With this being on two platforms, I really do suspect that Sync was involved.  The Ubuntu firefox has no plugins or extensions except what came with it, so that removes my suspicion that a common plugin had affected my two windows machines, and I doubt a virus/trojan/malware. 

So, could this mean that whatever this is has corrupted my backups?  Could Synch have synced the corruption from one database to the other installations of firefox? The backup I tried on ubuntu was from December, way before this was going on. I supposed it could be that my database has been corrupted for months and just now reached some critical point.  Maybe they are unrelated.  The database was semi-corrupt, as in usable for navigation but making bad backups, and it went unnoticed until something else deleted the bookmarks?
These are the plugins from my home machine.  I'll add the ones on the work machine monday. My ubuntu machine has only standard/default plugins. Versions are as of now, I don't know what they were a week ago.

Adblock Plus
Amazon Search
Bookmarklet (disabled)
Context Search
Download Statusbar
feedly (installed a few weeks ago)
Firebug
Forecastfox
Fox to Phone
Greasmonkey
Pocket
Saved Password Editor (disabled)
Search IMDB
Smartest Bookmarks Bar (recent installation)
Stylish (recent installation, but have used before)
Modify Headers(disabled)

Also, in my immediately prior reply (yeah, sorry for so many, just trying to give as much info as possible), I meant to add that the Ubuntu FF is still v17, so that scratches the version update as the cause unless it was also propagated by Sync.
(In reply to paul from comment #33)
> I know about Bookmarks > Show All Bookmarks in the sense that it is where I
> am attempting to restore old bookmarks.  If you meant something more than
> that, maybe not.  

I meant "it shows all of your bookmarks in a big usable window, rather than scrolling through a crummy menu".

If you're scrolling through a big Bookmarks menu, you're probably doing something wrong: either browse All Bookmarks, use the Bookmarks Toolbar, or use the Awesomebar.

> If my database is corrupt at home then it is also corrupt on 2 other
> machines then (and possibly 2 android devices as well). They all exhibit the
> same behavior with the bookmark toolbar being weird and sometimes not being
> able to create new bookmarks.

"Sometimes" is an odd sign. If your DB is corrupt, you won't be able to do *anything* bookmark related, and your toolbar will always be blank.

Can you explain what you mean by "sometimes"?

> since.) Only one machine had the  two plugins I described earlier, so if
> those are related, then Synch had to have propagated the corruption one of
> them caused.

Sync also syncs add-ons, if you had that turned on.

> If it is the database that is corrupt, shouldn't I be able to restore the
> json  backup into a fresh database?

Yes.

> I think the backups might be corrupt,
> and even backups from 2 months ago won't import into a fresh install of
> firefox (fresh as in no file or directory or key on my computer had a
> reference to anything related to firefox). The json files are not empty, and
> seem to be valid json (I don't know the proper format of a bookmark backup
> to validate that).  

Marco might be able to help, if you're willing to share a backup with him.

> If Sync didn't cause it, I believe it at least spread it.

If a chunk of bookmarks got marked as deleted, that would be propagated.

If your Sync account got migrated between nodes, and a machine that had lost part of its bookmarks DB was the first to sync, then the other machines will pick up that state.


> 1) I don't suppose the Sync server has a backup from 2-3 weeks ago?  Is
> there a way for me to see what is on the Synch server? I'd like to know if
> this data exists and is just currently unavailable or if it is truly lost.
> As I mentioned, when I tell Sync to replace all local data with what Sync
> has, it warns me I am going to lose a few hundred bookmarks.

There are no backups on the Sync server. It's a tool for moving data between machines, not a backup service, I'm afraid. Think of it as a whiteboard.
(In reply to Richard Newman [:rnewman] from comment #36)
> 
> I meant "it shows all of your bookmarks in a big usable window, rather than
> scrolling through a crummy menu".
> 
> If you're scrolling through a big Bookmarks menu, you're probably doing
> something wrong: either browse All Bookmarks, use the Bookmarks Toolbar, or
> use the Awesomebar.

Fair enough.  I knew about it but always thought of it as a maintenance interface rather than something for "daily use". Never made the connection.  If I am able to restore my bookmarks or get a large number again' I'll keep that in mind.  Thanks for the thinking reset.

> "Sometimes" is an odd sign. If your DB is corrupt, you won't be able to do
> *anything* bookmark related, and your toolbar will always be blank.
> 
> Can you explain what you mean by "sometimes"?

It's ambiguous because I was trying different combinations of actions.  Different combinations of resetting Sync, deleting places.sqlite, removing or manually selecting json backup files.  I should have kept better track of which ones did what.  I'll try to do that for further troubleshooting.  I'm not ready to give up on recovering this data.

One thing I am pretty sure of is that trying to restore one of the recent backups on a FRESH(complete uninstall and scrub)  install did not let me restore the automatic .json backups from the last 3-4 months.

My auto .json backups at home have data (~110k - ~130k) but cannot be imported into a new placess database.  I think I remember (but won't swear) that trying re-created the toolbar and ability to add new bookmarks.  These files have data that appears to be bookmark data (url's I recognize).

My auto .json backups at work are all 1-2k, and cannot be imported.  They have data that looks like structure without bookmark data.  This is true for the files from a month ago (before I was aware of the problem).  I don't know if that means This install hase been saving bad backups for a while before I realizd it or if something corrupted them after.  I found a manually created .json backup from about a year ago.  It seems to be from before I added Sync at work. I restored it and it worked as expected, but it's only work-related bookmarks (not the years of history). I then restored a recent auto backup, and it erased all bookmarks.  Re-restoring the manual one worked again. I'll try to import one of the larger home-backups on my work install and see what that does.


 
> Sync also syncs add-ons, if you had that turned on.

It does, but they never made it from work to home.  So those two were never synched to my home install.  I don't know if the communication break was Work->Sync server or Sync Server->Home. But I'd imagine that they either coincided with or are related to the corrupted database causing Sync to fail.


> > If it is the database that is corrupt, shouldn't I be able to restore the
> > json  backup into a fresh database?
> 
> Yes.

As mentioned, I was able to do so with a very old manual one, but not the auto ones from the last few months.

 
> > I think the backups might be corrupt,
> > and even backups from 2 months ago won't import into a fresh install of
> > firefox (fresh as in no file or directory or key on my computer had a
> > reference to anything related to firefox). The json files are not empty, and
> > seem to be valid json (I don't know the proper format of a bookmark backup
> > to validate that).  
> 
> Marco might be able to help, if you're willing to share a backup with him.

If he's willing, I'd like to send a couple from each machine (the last one from the series of one file size and the first one of the new size).

> > If Sync didn't cause it, I believe it at least spread it.
> 
> If a chunk of bookmarks got marked as deleted, that would be propagated.
> 
> If your Sync account got migrated between nodes, and a machine that had lost
> part of its bookmarks DB was the first to sync, then the other machines will
> pick up that state.

Waht I meant is that it apears to have synched more than just deleted bookmarks, unless the deleted bookmarks are what corrupted the other databases. I'm not quite sure what you guys are talking about with bookmark init and what the roots, missing or otherwise, are.

> There are no backups on the Sync server. It's a tool for moving data between
> machines, not a backup service, I'm afraid. Think of it as a whiteboard.

I know it wasn't meant to be a VCS, but I had assumed that Mozilla backs up the server and it's data againts failure.  Whether those backups are easy to get to and interact with is what I was asking.
(In reply to paul from comment #37)

> If he's willing, I'd like to send a couple from each machine (the last one
> from the series of one file size and the first one of the new size).

Flagging as needinfo.

> Waht I meant is that it apears to have synched more than just deleted
> bookmarks, unless the deleted bookmarks are what corrupted the other
> databases.

Nothing Sync does directly should cause corruption. We don't know what the root cause is here. It does seem in some way to be linked to Sync (though non-Sync users have also seen roots disappear).

> > There are no backups on the Sync server. It's a tool for moving data between
> > machines, not a backup service, I'm afraid. Think of it as a whiteboard.
> 
> I know it wasn't meant to be a VCS, but I had assumed that Mozilla backs up
> the server and it's data againts failure.  Whether those backups are easy to
> get to and interact with is what I was asking.

Sync was designed so that all permanent state lives in the client. The servers can be cheap and unreliable (to a point), and it doesn't matter -- they're just a channel for clients to talk across space and time. There are no backups -- if a client detects inconsistency, it just wipes the server and starts over.

We regularly migrate users between nodes without any data persistence. Clients can reupload what they need to share.
Flags: needinfo?(mak77)
(In reply to Richard Newman [:rnewman] from comment #38)
> (In reply to paul from comment #37)
> 
> > If he's willing, I'd like to send a couple from each machine (the last one
> > from the series of one file size and the first one of the new size).
> 
> Flagging as needinfo.

I'm fine with getting the bookmarks backups through mail, just don't send me the 1-2KB backups since those are pointless. It's clear the backup system read empty roots (due to a reset or loss) and backed up those, even if I don't think that's possible on a corrupt db, likely more on a freshly restored empty db.

Regarding the correct restore procedure, you should backup all of your profile folders, make a clean uninstall-install of firefox and create a totally new profile (don't reuse the old one for whatever reason).
After this proper clean install you may try restoring old json backups through the "restore from file" option in the Library, and pointing to the old profile backups, you may guess from the backups files size a good one (if there's a large size decrease that may indicate the beginning of the problem, unless it started so much time ago that all of the backups are corrupt).
Flags: needinfo?(mak77)
I thought I had replied to to this already.

That is the procedure I have tried.  It seems that all of the backups I had going back to January are corrupt.  I opened one in a .json editor to validate structure. When I formatted it and saved, it nearly doubled in size.  On formatting, it went from 1 line to 5600 lines and 127000 characters to 229000 characters long. 5600 newlines doesn't account for that.
Don't forget spacing and indentation.
Component: Firefox Sync: Backend → Sync
Product: Cloud Services → Firefox

This just happened to me after updating from 75 to 76. Thanks a lot. 5 years of bookmarks gone.

Rich, I'm so sorry that happened to you!

Would you mind following the instructions here to check if you have any automatic backups showing in the Library view? (You can also look inside the bookmarkbackups folder in your profile directory—which you can open from a button next to "Profile Folder" in about:support—and also please check if there's a places.sqlite.corrupt file, while you're there).

If you're having trouble locating the backups, they're corrupted on disk, or you see a places.sqlite.corrupt, please open a new bug, and we can investigate more there. This eight-year old bug is almost certainly a different underlying issue, and the errors it's mentioning are from code that's long been removed.

Thanks, and sorry again!

Flags: needinfo?(miserlou)

Thank you for your swift reply! It looks like there are backups there, which is great news! Thanks!

Flags: needinfo?(miserlou)

There was also no .corrupt file, so I think I can restore a backup and get back to business. Thanks again!

Great! I'm so happy you found the backups, and thanks so much for getting back to us!

I'll go ahead and close out this bug. INCOMPLETE isn't a satisfying resolution, especially to a problem involving database corruption. But, for this specific bug, most of the bookmarks code has changed since it was filed eight years ago, and Sync added checks not to corrupt roots (it also validates that they're correct before syncing, and Places now repairs wrong roots on startup). Let's use new bugs for anything else that comes up.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.