Update to Firefox 61.0 killed all my bookmarks and also the backups are unreadable

VERIFIED FIXED in Firefox 61

Status

()

P1
normal
VERIFIED FIXED
9 months ago
8 months ago

People

(Reporter: Bjoern, Assigned: standard8)

Tracking

({dataloss, regression})

61 Branch
mozilla63
dataloss, regression
Points:
---
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox-esr60 unaffected, firefox61+ verified, firefox62 verified, firefox63 verified)

Details

(Whiteboard: [fxsearch])

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

9 months ago
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
Build ID: 20180621125625

Steps to reproduce:

Update to Firefox 61


Actual results:

Since the update yesterday, all my 586 bookmarks are gone with the wind.
If I try to use the restore function it says the backup files cannot be restored (while the file is not corrupt, it can be decompressed fine to a valid JSON file).
Also taking places.sqlite from a Veeam backup of 2 days ago where everything was fine does not change a thing.


Expected results:

I should not have lost all my bookmarks unrecoverably.
(Reporter)

Comment 1

9 months ago
After I downgraded FF to 60.0.2 by restoring the installation directory from a Veeam full backup and restored the places.sqlite from yesterday, all my bookmarks are back as far as I can see.
(Reporter)

Comment 2

9 months ago
Colleagues that updated to 61 did not have that problem, I hope you can help me find out what the problem is in my setup. Starting over with a fresh profile is not really an option for me.
Component: General → Places
Product: Firefox → Toolkit
Would you mind sending us a copy of your places.sqlite and the latest corrupt backup through private email for analysis?
Or at least the json backup.

It's clear that 61 migration hit some data corruption in your database, that can happen because there are so many different corruption cases in the wild, though the backups should not really be broken.

You can either mail mak AT mozilla.com or standard8 AT mozilla.com
Flags: needinfo?(Bjoern)
(Reporter)

Comment 4

9 months ago
As I'm not sure whether I have sensitive data in the bookmarks, e. g. access tokens in URLs, I'd rather not share my places.sqlite if possible, but if you have SQL queries to investigate the issue, I'd happily execute them.
Flags: needinfo?(Bjoern)
(Assignee)

Comment 5

9 months ago
It might take longer / be harder, but we can give it a go.

It is probably also if you set up either a duplicate profile for testing Firefox 61, or create a new one and take across just the places.sqlite.

Some ideas/questions:

- Which localisation of Firefox are you running?

- Was it just bookmarks that were lost? Was history still intact?

- What is displayed in the "Show all Bookmarks" (ctrl-shift-B) window in Firefox 61? Do you see the normal items, e.g. History, Downloads, Tags, All Bookmarks and then Bookmarks Toolbar, Bookmarks Menu and Other Bookmarks underneath All Bookmarks?

- In Firefox 61, if you go to the three-line menu button -> Help -> Troubleshooting Information, then scroll down to the Places Database section and select "Verify Integrity". Please copy & paste that output here.

- In Firefox 61, if you go to the three-line menu button, then Web Developer followed by "Browser Console" (not Web Console), and then attempt to re-import one of the backups, do any messages appear in the console at the time of import? If so, can you paste them here.

- On the pre-upgraded version of places.sqlite, can you run this query:

SELECT * FROM moz_bookmarks WHERE guid IN ('root________', 'menu________', 'toolbar_____', 'tags________', 'unfiled_____', 'mobile______') ORDER by id
(Assignee)

Updated

9 months ago
Flags: needinfo?(Bjoern)
Open Scratchpad, from the Environment menu pick Browser and run this

(async function () {
  let db = await PlacesUtils.promiseDBConnection();
  let rows = await db.execute(`
    SELECT
      (SELECT count(*) FROM moz_bookmarks_deleted),
      (SELECT count(*) FROM moz_keywords),
      (SELECT count(*) FROM moz_bookmarks WHERE syncChangeCounter > 0),
      (SELECT count(*) FROM moz_places WHERE url_hash NOTNULL),
      (SELECT count(*) FROM moz_icons),
      (SELECT count(*) FROM moz_pages_w_icons),
      (SELECT count(*) FROM moz_icons_to_pages),
      (SELECT count(*) FROM moz_items_annos),
      (SELECT count(*) FROM moz_anno_attributes)
  `);
  console.log("moz_bookmarks_deleted: " + rows[0].getResultByIndex(0));
  console.log("moz_keywords: " + rows[0].getResultByIndex(1));
  console.log("unsynced bookmarks: " + rows[0].getResultByIndex(2));
  console.log("urls: " + rows[0].getResultByIndex(3));
  console.log("icons: " + rows[0].getResultByIndex(4));
  console.log("pages: " + rows[0].getResultByIndex(5));
  console.log("icon rels: " + rows[0].getResultByIndex(6));
  console.log("annos: " + rows[0].getResultByIndex(7));
  console.log("attributes: " + rows[0].getResultByIndex(8));
  rows = await db.execute(`
    SELECT guid, id, parent, position FROM moz_bookmarks WHERE guid IN (
      "root________", "menu________", "toolbar_____", "unfiled_____", "mobile______", "tags________"
    )`);
  rows.forEach(r => console.log(`${r.getResultByIndex(0)}, ${r.getResultByIndex(1)}, ${r.getResultByIndex(2)} ${r.getResultByIndex(3)}`));
})();

then please post here the log from the browser console.

Another interesting thing would be to try running Places Maintenance from about:support, and checking if update after it still fails.
Then, it may be useful to check if the Browser Console reports any errors when trying to restore one of those broken json backups.
(Reporter)

Comment 7

9 months ago
> It might take longer / be harder, but we can give it a go.

If that's ok with you, it for me, as I can for now work with the 60.0.2 until we found the problem

> - Which localisation of Firefox are you running?

German

> - Was it just bookmarks that were lost? Was history still intact?

I think so, but I'm not 100% sure, will try again later, I'm at work currently

> - On the pre-upgraded version of places.sqlite, can you run this query:

Here the result:

> id,type,fk,parent,position,title,keyword_id,folder_type,dateAdded,lastModified,guid,syncStatus,syncChangeCounter
> 1,2,,0,0,"",,,1327331352111000,1529566583864000,root________,1,1
> 2,2,,73,4,Lesezeichen-Menü,,,1327331352111000,1519610859064000,menu________,2,0
> 3,2,,73,3,Lesezeichen-Symbolleiste,,,1327331352111000,1524818362612000,toolbar_____,2,0
> 4,2,,1,1,Schlagwörter,,,1327331352111000,1444911535029000,tags________,1,1
> 5,2,,73,5,Weitere Lesezeichen,,,1327331352111000,1529566583864000,unfiled_____,2,0
> 557,2,,1,3,Mobile Lesezeichen,,,1489510061588000,1521408073767000,mobile______,2,0

> Open Scratchpad, from the Environment menu pick Browser and run this

@::mak you mean in the working or in the broken version?
I guess in the broken one, as in the working one there are no moz_icons, moz_pages_w_icons or moz_icons_to_pages tables

In the working one the result (done through SQL tool) is

> moz_bookmarks_deleted,moz_keywords,unsynced_bookmarks,urls,annos,attributes
> 0,12,13,81616,64,12

For other stuff I should try in 61, I'll do the update again later when I'm at home.
(Reporter)

Comment 8

9 months ago
For the last one it is

> guid,id,parent,position
> menu________,2,73,4
> mobile______,557,1,3
> root________,1,0,0
> tags________,4,1,1
> toolbar_____,3,73,3
> unfiled_____,5,73,5
(Reporter)

Comment 9

9 months ago
And I just did the integrity check on about:support, that is what you meant, isn't it?
Here the result:

> > Task: checkIntegrity
> + The database is sane
> > Task: invalidateCaches
> + The caches have been invalidated
> > Task: checkCoherence
> + The database is coherent
> > Task: expire
> + Database cleaned up
> > Task: vacuum
> + Initial database size is 40960KiB
> + The database has been vacuumed
> + Final database size is 40960KiB
> > Task: stats
> + Places.sqlite size is 40960KiB
> + Favicons.sqlite size is 19520KiB
> + pragma_user_version is 43
> + pragma_page_size is 32768
> + pragma_cache_size is -2048
> + pragma_journal_mode is wal
> + pragma_synchronous is 1
> + History can store a maximum of 147391 unique pages
> + Table moz_places has 81616 records
> + Table moz_historyvisits has 160134 records
> + Table moz_inputhistory has 639 records
> + Table moz_bookmarks has 592 records
> + Table moz_keywords has 12 records
> + Table sqlite_sequence has 1 records
> + Table moz_annos has 189 records
> + Table moz_anno_attributes has 12 records
> + Table moz_items_annos has 64 records
> + Table sqlite_stat1 has 19 records
> + Table moz_hosts has 5178 records
> + Table moz_bookmarks_deleted has 0 records
> + Index sqlite_autoindex_moz_inputhistory_1
> + Index sqlite_autoindex_moz_keywords_1
> + Index sqlite_autoindex_moz_anno_attributes_1
> + Index sqlite_autoindex_moz_hosts_1
> + Index sqlite_autoindex_moz_bookmarks_deleted_1
> + Index moz_places_hostindex
> + Index moz_places_visitcount
> + Index moz_places_frecencyindex
> + Index moz_places_lastvisitdateindex
> + Index moz_historyvisits_placedateindex
> + Index moz_historyvisits_fromindex
> + Index moz_historyvisits_dateindex
> + Index moz_bookmarks_itemindex
> + Index moz_bookmarks_parentindex
> + Index moz_bookmarks_itemlastmodifiedindex
> + Index moz_places_url_hashindex
> + Index moz_places_guid_uniqueindex
> + Index moz_bookmarks_guid_uniqueindex
> + Index moz_annos_placeattributeindex
> + Index moz_items_annos_itemattributeindex
> + Index moz_keywords_placepostdata_uniqueindex
> + Index moz_bookmarks_dateaddedindex
> > Task: _refreshUI
(Assignee)

Comment 10

9 months ago
(In reply to Björn Kautler from comment #8)
> For the last one it is
> 
> > guid,id,parent,position
> > menu________,2,73,4
> > mobile______,557,1,3
> > root________,1,0,0
> > tags________,4,1,1
> > toolbar_____,3,73,3
> > unfiled_____,5,73,5

Thanks, the interesting part here is that the menu/toolbar/unfiled have parents of 73, when it should be 1.

I think you'll find that if you:

- upgrade to 61 again
- run the Verify Integrity (as you did above)
- restart Firefox (just in case)
- Try access bookmarks

I think it will correct those parents for you and recover.

Verify integrity forces running of the background maintenance tasks, one of which is fixing the parents - we've seen a few users that have had this problem before.
(Reporter)

Comment 11

9 months ago
Ok, with the last comment I had some hopes, so tried it right away, unfortunately that's not the solution yet.

Here some more of the requested information:

> - Was it just bookmarks that were lost? Was history still intact?

History seems to be intact

> - What is displayed in the "Show all Bookmarks" (ctrl-shift-B) window in Firefox 61? Do you see the normal items, e.g. History, Downloads, Tags, All Bookmarks and then Bookmarks Toolbar, Bookmarks Menu and Other Bookmarks underneath All Bookmarks?

I see (roughly translated)
- History
- Downloads
- Keywords or Tags, dunno how it is in English (Schlagwörter in German)
- All Bookmarks
  - Bookmark Toolbar
  - Bookmark Menu
  - Other Bookmarks
  - Mobile Bookmarks

Only ones with content are History, Downloads and Mobile Bookmarks, the others are empty

> - In Firefox 61, if you go to the three-line menu button -> Help -> Troubleshooting Information, then scroll down to the Places Database section and select "Verify Integrity". Please copy & paste that output here.

This is the output from when I did it after upgrading and before restarting, as advised in your last comment:

> > Task: checkIntegrity
> + The database is sane
> > Task: invalidateCaches
> + The caches have been invalidated
> > Task: checkCoherence
> + The database is coherent
> > Task: expire
> + Database cleaned up
> > Task: vacuum
> + Initial database size is 40960KiB
> + The database has been vacuumed
> + Final database size is 40960KiB
> > Task: stats
> + Places.sqlite size is 40960KiB
> + Favicons.sqlite size is 19520KiB
> + pragma_user_version is 47
> + pragma_page_size is 32768
> + pragma_cache_size is -2048
> + pragma_journal_mode is wal
> + pragma_synchronous is 1
> + History can store a maximum of 151030 unique pages
> + Table moz_places has 81338 records
> + Table moz_historyvisits has 160140 records
> + Table moz_inputhistory has 636 records
> + Table moz_bookmarks has 45 records
> + Table moz_keywords has 0 records
> + Table sqlite_sequence has 1 records
> + Table moz_annos has 169 records
> + Table moz_anno_attributes has 4 records
> + Table moz_items_annos has 1 records
> + Table sqlite_stat1 has 19 records
> + Table moz_hosts has 5177 records
> + Table moz_bookmarks_deleted has 547 records
> + Table moz_meta has 0 records
> + Index sqlite_autoindex_moz_inputhistory_1
> + Index sqlite_autoindex_moz_keywords_1
> + Index sqlite_autoindex_moz_anno_attributes_1
> + Index sqlite_autoindex_moz_hosts_1
> + Index sqlite_autoindex_moz_bookmarks_deleted_1
> + Index moz_places_hostindex
> + Index moz_places_visitcount
> + Index moz_places_frecencyindex
> + Index moz_places_lastvisitdateindex
> + Index moz_historyvisits_placedateindex
> + Index moz_historyvisits_fromindex
> + Index moz_historyvisits_dateindex
> + Index moz_bookmarks_itemindex
> + Index moz_bookmarks_parentindex
> + Index moz_bookmarks_itemlastmodifiedindex
> + Index moz_places_url_hashindex
> + Index moz_places_guid_uniqueindex
> + Index moz_bookmarks_guid_uniqueindex
> + Index moz_annos_placeattributeindex
> + Index moz_items_annos_itemattributeindex
> + Index moz_keywords_placepostdata_uniqueindex
> + Index moz_bookmarks_dateaddedindex
> > Task: _refreshUI

- In Firefox 61, if you go to the three-line menu button, then Web Developer followed by "Browser Console" (not Web Console), and then attempt to re-import one of the backups, do any messages appear in the console at the time of import? If so, can you paste them here.

The error shown is:

> Failed to restore bookmarks from C:\Users\bkautler\AppData\Roaming\Mozilla\Firefox\Profiles\lq5on55a.default\bookmarkbackups\bookmarks-2018-06-29_586_8F1a9LNs7DzmSu5orZwWQQ==.jsonlz4: TypeError: searchBookmark is null BookmarkJSONUtils.jsm:103
> importFromFile resource://gre/modules/BookmarkJSONUtils.jsm:103:7
> throw self-hosted:1236:9 

> Open Scratchpad, from the Environment menu pick Browser and run this

There are still no moz_icons, moz_pages_w_icons or moz_icons_to_pages tables.
Are these maybe from post-61?
Or is this due to failed migration?
Again without them, the result now is

> moz_bookmarks_deleted,moz_keywords,unsynced_bookmarks,urls,annos,attributes
> 0,0,2,81338,1,4

And the second query now after the integrity check:

> guid,id,parent,position
> menu________,805,1,2
> mobile______,557,1,1
> root________,1,0,0
> tags________,4,1,0
> toolbar_____,806,1,3
> unfiled_____,807,1,4
Flags: needinfo?(Bjoern)
moz_icons and friends are in an attached database, it doesn't matter that they aren't in places.sqlite, that's why I suggested using scratchpad instead of direct queries.

Looks like integrity fixed your roots in 61. Did you restart the browser and retry after the integrity check? Is it still broken?
(Assignee)

Comment 13

9 months ago
So we've gone from:

> guid,id,parent,position
> menu________,2,73,4
> mobile______,557,1,3
> root________,1,0,0
> tags________,4,1,1
> toolbar_____,3,73,3
> unfiled_____,5,73,5

to:

> menu________,805,1,2
> mobile______,557,1,1
> root________,1,0,0
> tags________,4,1,0
> toolbar_____,806,1,3
> unfiled_____,807,1,4

That's kinda worrying. It shouldn't have needed to recreate those folders... which implies we ripped them out for some reason. I can't yet reproduce it though.

Can you run this query on the pre-upgrade version of the database please? It should give us the full structure of moz_bookmarks, but not your personal info (maybe attach the results as a file, attach link is just below the bug headers before the comments start).

SELECT guid, id, parent, position, type, fk, dateAdded, lastModified FROM moz_bookmarks ORDER BY id
(Reporter)

Comment 14

9 months ago
> moz_icons and friends are in an attached database, it doesn't matter that they aren't in places.sqlite, that's why I suggested using scratchpad instead of direct queries.

Ah sorry, my bad then.
Do you know what Scratchpad is called in German localization?
Either I executed it in the wrong place or it doesn't work:

> ReferenceError: PlacesUtils is not defined

> Looks like integrity fixed your roots in 61. Did you restart the browser and retry after the integrity check? Is it still broken?

Yes, unfortunately

> Can you run this query on the pre-upgrade version of the database please? It should give us the full structure of moz_bookmarks, but not your personal info (maybe attach the results as a file, attach link is just below the bug headers before the comments start).
> 
> SELECT guid, id, parent, position, type, fk, dateAdded, lastModified FROM moz_bookmarks ORDER BY id
(Reporter)

Comment 15

9 months ago
(Assignee)

Comment 16

9 months ago
Requesting tracking: We believe this is a dataloss issue, that is caused by places database migration. It happens to a few users that have databases where the root folders are incorrectly parented, and we were trying to fix them.

The causes of this in the past have been attributed to the XMarks add-on, we don't know if there are other causes.

The only way to restore bookmarks would be for the user to import from an old backup (bookmarks are backed up in the profile), if there is one that is valid & can be imported.
status-firefox61: --- → affected
status-firefox-esr60: --- → unaffected
tracking-firefox61: --- → ?
Keywords: dataloss
(Assignee)

Comment 17

9 months ago
(In reply to Björn Kautler from comment #15)
> Created attachment 8988746 [details]
> SELECT_____guid______id______parent_____.csv

This looks like it was on the post-upgrade database, could you try the pre-upgrade one please?
Flags: needinfo?(Bjoern)
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P1
Whiteboard: [fxsearch]
Ryan I'm ni you to make you aware of this problem, we may be breaking certain bookmark databases, Mark has an idea about the cause. What's the best course of action, should we block upgrades and try to get a fix asap?
Flags: needinfo?(ryanvm)
(Reporter)

Comment 19

9 months ago
Ups, sorry, accidentally opened the wrong file, here the correct one
Attachment #8988746 - Attachment is obsolete: true
Flags: needinfo?(Bjoern)
(Reporter)

Comment 20

9 months ago
> The causes of this in the past have been attributed to the XMarks add-on, we don't know if there are other causes.

Probably, I never heard of XMarks and doubt I used it in the past.
(In reply to Marco Bonardo [::mak] from comment #18)
> Ryan I'm ni you to make you aware of this problem, we may be breaking
> certain bookmark databases, Mark has an idea about the cause. What's the
> best course of action, should we block upgrades and try to get a fix asap?

Do we have any data to suggest what the frequency of the issue is?
tracking-firefox61: ? → +
Flags: needinfo?(ryanvm)
no, we don't have data about frequency of this.
(Assignee)

Comment 23

9 months ago
This is an example database based on Firefox 60. I manually simulated the broken root structure.

When this is loaded into 61, all the folders are empty (rather than having the "default" bookmarks in). New bookmarks and other bookmark actions will work correctly though.
Assignee: nobody → standard8
(Assignee)

Updated

9 months ago
Attachment #8988767 - Attachment is patch: false
Attachment #8988767 - Attachment mime type: text/plain → application/x-sqlite3
See Also: → bug 1472241
(Reporter)

Comment 25

9 months ago
Do you need anything more from me or is it clear already, as you constructed a commit?
And if everything is clear, is there anything I can do, like fixing my places DB?
Or should I just wait until you release a new FF version that has the fix in it?
It's clear enough thank you, we'll try to release a fix in the next 61.0.1 minor release.
You can follow the status of the bug here.
(Reporter)

Comment 27

9 months ago
Ok, great, thanks.
Many thanks for the fast and great help and fix.

One more question, does the status of the bug here change when the release is rolled out, or do I have to watch the about dialog about when it is safe to update for me?
The status of the bug changes when we land things in trees, then there is still some delay before a release.
We'll take care of letting you know when it's safe to upgrade.
Blocks: 1310295
Keywords: regression
(Assignee)

Comment 29

9 months ago
mozreview-review
Comment on attachment 8988770 [details]
Bug 1472127 - Update to Firefox 61.0 may wrongly remove incorrectly parented roots.

https://reviewboard.mozilla.org/r/253958/#review260714

This looks good, I've tested it locally against my test version and it does the right thing and everything keeps working as far as I can tell. The test looks good as well.
Attachment #8988770 - Flags: review?(standard8) → review+

Comment 30

9 months ago
Pushed by mak77@bonardo.net:
https://hg.mozilla.org/integration/autoland/rev/86bbfe5a3a39
Update to Firefox 61.0 may wrongly remove incorrectly parented roots. r=standard8
Comment on attachment 8988770 [details]
Bug 1472127 - Update to Firefox 61.0 may wrongly remove incorrectly parented roots.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1310295
[User impact if declined]: If the bookmark roots have wrong parents (data coherence problem), we wrongly wipe any bookmark in that root.
[Is this code covered by automated tests?]: yes
[Has the fix been verified in Nightly?]: not yet, verified locally by me and Mark
[Needs manual test from QE? If yes, steps to reproduce]: yes, there's a test broken database attached in the bug
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: it's a one-line condition restricting a query, so it has a smaller reach and a smaller risk than the original unconditioned query.
[String changes made/needed]: none
Attachment #8988770 - Flags: approval-mozilla-release?
Attachment #8988770 - Flags: approval-mozilla-beta?
Comment on attachment 8988770 [details]
Bug 1472127 - Update to Firefox 61.0 may wrongly remove incorrectly parented roots.

OK, let's take this for beta 5, sounds like it helps us avoid a potentially bad problem for users. If we're able to reproduce this problem in QA (maybe with the add-on mentioned) then it would be nice to verify the fix in nightly or beta.
Attachment #8988770 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Flags: qe-verify+

Comment 33

9 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/86bbfe5a3a39
Status: NEW → RESOLVED
Last Resolved: 9 months ago
status-firefox63: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
status-firefox62: --- → affected
Comment on attachment 8988770 [details]
Bug 1472127 - Update to Firefox 61.0 may wrongly remove incorrectly parented roots.

Taking for 61.0.1 also.
Attachment #8988770 - Flags: approval-mozilla-release? → approval-mozilla-release+

Comment 35

9 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/01156ddc21ac
status-firefox62: affected → fixed
Flags: in-testsuite+

Comment 36

9 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-release/rev/ced0b5dc71eb
status-firefox61: affected → fixed
status-firefox-esr52: --- → unaffected
Managed to reproduce on using the attachment from step 23, on  win 10x64; Ffx 60->61.0 update.
Checked on win 10x64, macOS 10.13.4, Ubuntu 16.04 LTS, and after the upgrade to 61.0.1 the issue was no longer reproducible.
Fix confirmed with 62.0b5  and 63.0a1 (2018-07-04) as well.
Status: RESOLVED → VERIFIED
status-firefox61: fixed → verified
status-firefox62: fixed → verified
status-firefox63: fixed → verified
Flags: qe-verify+

Comment 38

9 months ago
Not sure if this will help, but I just had what might or might not be the same problem. I run Linux Mint 18.3 on both my desktop and laptop, both with Firefox 60.0.2, and with the bookmarks synced. This morning I upgraded the desktop to Firefox 61.0. When I started it, the bookmark toolbar displayed empty. I tried restoring the most recent bookmark backup, and after a restart or two the bookmark toolbar was populated, but with a lot of bookmarks missing. The toolbar displayed the correct set of bookmarks and folders, but the folders were missing most of their content.

Looking at the desktop backups in the restore dialog, all files were listed with 18 to 21 items. On my laptop, the backups are listed with 444 items. Looking at the file names in the backup folder, I can confirm that the item counts are 18 to 21 in the file names, and the file lengths are around ~2K (versus ~45K on the laptop). What I find singularly odd is that all the backup files on the desktop are too small, even though I've had those 444 bookmarks on the desktop for a long time.

I'm also a bit confused about the dates on the backup files. On the laptop, where the backups are intact, the date in the file name matches the time stamp on the file in all but one case. One file has a time stamp of April 11 of this year, but the file name is "bookmarks-2018-05-10_444_...". Did May come early this year? The files on the desktop have dates in their names going back to 2015 (and, again, all are too small to be correct for the indicated dates) and time stamps going back no further than this February. In one case, the file has 2018-04-25 in its name but a time stamp from Feb. 24.

I've manually restored the bookmarks from the laptop to the desktop, so Firefox 61 on the desktop is no longer "symptomatic". I still have the funky backup files on the desktop, if that helps. Also, I ran the verification task on the places database in 61, and I think it's okay. Here's the output, FWIW:

> Task: checkIntegrity
+ The database is sane
> Task: invalidateCaches
+ The caches have been invalidated
> Task: checkCoherence
+ The database is coherent
> Task: expire
+ Database cleaned up
> Task: vacuum
+ Initial database size is 71680KiB
+ The database has been vacuumed
+ Final database size is 71680KiB
> Task: stats
+ Places.sqlite size is 71680KiB
+ Favicons.sqlite size is 29536KiB
+ pragma_user_version is 47
+ pragma_page_size is 32768
+ pragma_cache_size is -2048
+ pragma_journal_mode is wal
+ pragma_synchronous is 1
+ History can store a maximum of 143923 unique pages
+ Table moz_places has 143461 records
+ Table moz_historyvisits has 279422 records
+ Table moz_inputhistory has 36 records
+ Table moz_bookmarks has 447 records
+ Table moz_keywords has 0 records
+ Table sqlite_sequence has 1 records
+ Table moz_annos has 573 records
+ Table moz_anno_attributes has 8 records
+ Table moz_items_annos has 74 records
+ Table sqlite_stat1 has 17 records
+ Table moz_hosts has 6749 records
+ Table moz_bookmarks_deleted has 0 records
+ Table moz_meta has 1 records
+ Index sqlite_autoindex_moz_inputhistory_1
+ Index sqlite_autoindex_moz_keywords_1
+ Index sqlite_autoindex_moz_anno_attributes_1
+ Index sqlite_autoindex_moz_hosts_1
+ Index sqlite_autoindex_moz_bookmarks_deleted_1
+ Index moz_places_hostindex
+ Index moz_places_visitcount
+ Index moz_places_frecencyindex
+ Index moz_places_lastvisitdateindex
+ Index moz_historyvisits_placedateindex
+ Index moz_historyvisits_fromindex
+ Index moz_historyvisits_dateindex
+ Index moz_bookmarks_itemindex
+ Index moz_bookmarks_parentindex
+ Index moz_bookmarks_itemlastmodifiedindex
+ Index moz_places_url_hashindex
+ Index moz_places_guid_uniqueindex
+ Index moz_bookmarks_guid_uniqueindex
+ Index moz_annos_placeattributeindex
+ Index moz_items_annos_itemattributeindex
+ Index moz_keywords_placepostdata_uniqueindex
+ Index moz_bookmarks_dateaddedindex
> Task: _refreshUI

Comment 39

9 months ago
Forgot one detail: after finding the empty bookmark tool bar, and before restoring a backup, I tried hitting the sync button, which accomplished nothing visible (unless that's what mucked up the backup files).
(Reporter)

Comment 40

9 months ago
@mak this seems to only have fixed my problem partly. :-(
Or is the fix not in 61.0.1?
As it was out I tried upgrading again.
This time it seemed to have ben worked better (bookmark toolbar was filled).
But when I looked at the auto-backup files, 61.0.1 only put 41 bookmarks there, not 586, the file also is just 5.4 KiB, not 69.0 KiB like the one before.
And restoring the backup still yields the same error, that the backup file is corrupt.

I assume I can trust in you not giving my places and backup file to anyone else, not abusing it and deleting it as soon as this is resolved?
Then I would send it to you now for further investigation.
(Assignee)

Comment 41

9 months ago
(In reply to Björn Kautler from comment #40)
> @mak this seems to only have fixed my problem partly. :-(
> Or is the fix not in 61.0.1?
> As it was out I tried upgrading again.
> This time it seemed to have ben worked better (bookmark toolbar was filled).

Björn, thanks for reaching out again. I'm glad that the upgrade worked better.

> I assume I can trust in you not giving my places and backup file to anyone
> else, not abusing it and deleting it as soon as this is resolved?
> Then I would send it to you now for further investigation.

Yes, we would do that. You can either send it/them to myself or Marco. Probably the best way to send it is via https://send.firefox.com/ - this will encrypt the file prior to uploading it, and give you a link that can be used to only download once.

Send the link(s) to either myself or Marco via the email addresses you can see clicking on our names in the bug.

At a guess, the places.sqlite from before you upgraded would be the best thing for us to look at if you still have it.

If the issue is in Firefox we'll probably file a new bug to handle this - it is likely a different issue to the upgrade one.
(Reporter)

Comment 42

9 months ago
Ah, nice upload service, thanks for that.
Mail is sent.
Yes, I have the one before the upgrade, as I again restored 60.0.2 from previous days Veeam Backup.
Don't trust too much the "corrupt backup" dialog we show, we show it too often (even if a single query import fails), Mark will file a bug for that, it's a longstanding problem.
Upgrading to 61.0.1 is unlikely to make things worse than 60, please follow the mail advice Mark sent you and let us know if it helps this specific case.

(In reply to Paul A. Rubin from comment #38)
> Not sure if this will help, but I just had what might or might not be the
> same problem. I run Linux Mint 18.3 on both my desktop and laptop, both with
> Firefox 60.0.2, and with the bookmarks synced. This morning I upgraded the
> desktop to Firefox 61.0. When I started it, the bookmark toolbar displayed
> empty.

It could be the same issue, please restore a backup of places.sqlite (as it was before the upgrade to 61), and then upgrade to 61.0.1 that was just released.
After the upgrade, go to about:support and run Places Database integrity check.
Then restart the browser.

If things still look strange, please file a separate bug so we can help with your specific case.

Comment 44

9 months ago
Sorry, I can't find a backup of places.sqlite. If Firefox does periodic backups of it, they're hiding in a location not obvious to me.

FWIW, I did a system upgrade on my laptop that automatically installed FF 61.0. That went fine (bookmark toolbar visible on restart, no missing bookmarks). Since I restored a bookmark backup from my laptop to my desktop, my desktop has all the correct bookmarks as well, so I don't have any remaining issues here. I think the big question is what caused the bookmark backups on the desktop to only retain the top three bookmarks in each folder.

Comment 45

8 months ago
I have the same problem. Firefox version 61.0.1 (64bit) delete all the saved bookmark on toolbar. I stopped the sync in one unused PC so I have been able to make a backup on a json file. I have been also able to recover from a firefox recovery point. Which in any case show different number on different PC. The problem has become very frustating I do a restore and the next day I have all the bookmark deleted again. 
I have two chance:
1) restore once more and stop the sync until a fix will be released, I do not like this option!
2) use another browser, I do not like this option!

Any suggestion?
(In reply to Pinob from comment #45)
> I have the same problem.

If you upgraded directly from 60.x to 61.0.1, it's not the same bug, since this bug is fixed in 61.0.1.
If you upgraded to 61.0.0 and THEN to 61.0.1, then unfortunately the only possibility may be to restore a backup or manually change the database.

Please file a separate bug where we can investigate your case.

Json backups happen automatically, you can find them from the Import and Save menu in the Library window (Show All Bookmarks) and they are stored in the profile/bookmarkbackups folder (the profile folder is reachable from the about:support page).

To be able to help, we'll need you to send us privately your places.sqlite file, you can send that using https://send.firefox.com/, that will encrypt the file prior to uploading it, and give you a link that can be used to only download it once.

But as I said, please file your own bug, we can't coalesce different causes into the same bug report.
You need to log in before you can comment on or make changes to this bug.