Closed Bug 1498725 Opened Last year Closed Last year

Firefox updated; most of my LARGE collection of sites was lost.

Categories

(Firefox :: Bookmarks & History, defect)

62 Branch
defect
Not set

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: robertmiles, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0

Steps to reproduce:

I updated my previous version (about a month old) version of Firefox Quantum to Firefox Quantum 62.0.3 (64-bit).


Actual results:

Most or all of my VERY LARGE collection of sites (estimated at 40,000 sites; several levels of folders, with total number of folders almost 1,000) was lost.  At least two levels of folders remained); however, the few remaining sites appeared to be added by the Firefox update, and not from my previous collection of sites


Expected results:

All of the previous sites, and all of the previous folders, should be visible in the new version.  Sites were NOT previously saved with Firefox Sync - I was repeatedly unable to log in to ask about whether Firefox Sync has limits on the number of folders and the number of sites.
Summary: Firefox updated; ost of my LARGE collection of sites was lost. → Firefox updated; most of my LARGE collection of sites was lost.
Did you already follow https://support.mozilla.org/en-US/kb/recover-lost-or-missing-bookmarks ?
Component: Untriaged → Bookmarks & History
(In reply to Matthias Versen [:Matti] from comment #1)
> Did you already follow
> https://support.mozilla.org/en-US/kb/recover-lost-or-missing-bookmarks ?

Robert, did this help?
Flags: needinfo?(robertmiles)
It gave me some things to try, but none of them helped.

The "My Google Bookmarks are missing" section contains a bad link, so I can't tell if that link should have gone to something that would help.

There is only one profile, which appears to still contain all of the folders but very few of these folders still contain anything other than lower level folders.  I have deleted some of these now empty folders.

Other information I have found since reporting the problem:

I had previously been using a different synchronizer program to maintain a second copy of the bookmarks on my other desktop, until the company providing it shut down their server and dropped all support for that program many months ago.

Therefore, there was a second copy of most of the bookmarks - at least those stored before they dropped support.  At least some of those entered since then on the first desktop were lost, though. Installing the same update on the second desktop does not appear to have lost any of the bookmarks on that desktop.
Flags: needinfo?(robertmiles)
Robert, have you been able to restore from an older backup?

If you need to know how, please follow the "Restoring from backups" section here: https://support.mozilla.org/kb/restore-bookmarks-from-backup-or-move-them#w_restoring-from-backups

Note: there will likely be several in the list - you might need to look for the largest one rather than the latest.
Flags: needinfo?(robertmiles)
You're assuming, incorrectly, that there is a backup.
Flags: needinfo?(robertmiles)
Firefox should be saving backups approximately every day when idle. It will keep a maximum of 15 files. If there aren't any being saved, then either you've set "browser.bookmarks.max_backups" to 0, or there is a corruption problem in the database which has also stopped backups happening.

We have in the past seen corruption when XMarks has been used, I'm thinking this could be another case of it.

Do you have any backups of your computer (and hence Firefox profile) from before you upgrade to 62? If so, we may be able to analyse the database to see what went wrong.

In the meantime, please can you also try running the places verification: 

* Go to Help -> Troubleshooting Information
* Scroll down the page to find the "Places Database" section (it is near the bottom).
* Click the "Verify Integrity" button, this may take a few seconds to run.

Please copy & paste the output here.
Flags: needinfo?(robertmiles)
I found no sign that I have set browser.bookmarks.max_backups, so it should still be at the default value.

Where does Firefox put these backups?  Are the old ones still saved if the caches have been invalidated, even if no new one can be written?

My previous synchronizer was Xmarks.

Some very old backups, a few years old, from before my backup device stopped working.  At least the most recent year of those backups are inaccessible, but still before the last successful Xmarks synchronization.

> Task: checkIntegrity
+ The places.sqlite database is sane
+ The favicons.sqlite database is sane
> Task: invalidateCaches
+ The caches have been invalidated
> Task: checkCoherence
+ The database is coherent
> Task: expire
- The task queue was cleared by an error in another task.
> Task: originFrecencyStats
- The task queue was cleared by an error in another task.
> Task: vacuum
- The task queue was cleared by an error in another task.
> Task: stats
- The task queue was cleared by an error in another task.
> Task: _refreshUI
- The task queue was cleared by an error in another task.
Flags: needinfo?(robertmiles)
XMarks is sadly known to have caused various corruptions of the database in the past, and that's likely what happened here. Due to that corruption Firefox is unable to create backups and likely the update failed. If you wish you can send your places.sqlite file to Mark or me through private mail and Firefox Send and we may be able to suggest some query to fix it.

Backups are usually inside a bookmarkbackups folder in the profile folder (easily reachable from the about:support page).
And actually, the file you should send is the places.sqlite.corrupt file that is likely present in your profile folder, since the new places.sqlite doesn't contain your bookmarks.
No places.sqlite.corrupt file found.

I did find:

places.sqlite       46080 KB
places.sqlite-shm   32 KB
places.sqlite-wal   5541 KB

Are any of these useful?

I found the bookmarkbackups folder, with 15 backups in it.  Those from March and April are much larger than the more recent ones, but still smaller than places.sqlite.

I have upgraded to Firefox 63.0.

From a previous message about restoring bookmarks:

Click the 57 library icon , then click Bookmarks. Scroll to the bottom and click Show all Bookmarks.
In the Library window, click the Import and Backup button and then select Restore.

The first two sentences worked.  There was no Import and Backup button for the third sentence.
For the third sentence, Import and Backup appeared on the second line of the header, but not as a button.  Clicking on the triangle just to the right of it, gave me a list of 15 backups I could restore.

I'm not ready to choose which to restore, since I'm most interested in the sites saved since the last successful Xmarks sync, not the many more sites which Xmarks synced onto my other desktop.
Store those backup files apart for now, so they won't be removed. Then you can always try restoring them and see. Note the restore process may take a while, and it doesn't have a progress bar (there's a bug filed about that), so I'd suggest to let it alone for a while...
Flags: needinfo?(robertmiles)
Verify Integrity:

> Integrity check
+ The database is sane
> Coherence check
+ The database is coherent
> Orphans expiration
+ Database cleaned up
> Vacuum
Initial database size is 1696 KiB
+ The database has been vacuumed
Final database size is 1696 KiB
> Statistics
Database size is 1696 KiB
user_version is 36
page_size is 32768
cache_size is -2048
journal_mode is wal
synchronous is 1
History can store a maximum of 66607 unique pages
Table moz_places has 1576 records
Table moz_historyvisits has 1945 records
Table moz_inputhistory has 0 records
Table moz_bookmarks has 6 records
Table moz_keywords has 0 records
Table sqlite_sequence has 1 records
Table moz_favicons has 5 records
Table moz_annos has 0 records
Table moz_anno_attributes has 1 records
Table moz_items_annos has 1 records
Table sqlite_stat1 has 14 records
Table moz_hosts has 238 records
Table moz_bookmarks_deleted has 0 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index sqlite_autoindex_moz_hosts_1
Index sqlite_autoindex_moz_bookmarks_deleted_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_hashindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Index moz_keywords_placepostdata_uniqueindex
Flags: needinfo?(robertmiles)
Hi Robert, I'm sorry for the delay in responding. Which version of Firefox did you generate the output in comment 13? That seems like either a strange old version, or the database version wasn't updated correctly.

It might be better to start off from scratch - remove places.sqlite from the profile, and then use whichever most complete backup you have (make sure you let import finish), and see if you get any further issues from there.
Flags: needinfo?(robertmiles)
Information lost on what version was used for comment 13.  I have now upgraded to Firefox Quantum 63.0.3 (64 bit), since the info in comment was generated.
Flags: needinfo?(robertmiles)
So is it all working now?

I'm not sure there's much we can do for this, seeing as we don't really know what caused it or why it occurred.
I would say that the problem is not getting worse, rather than saying it is all working now.

Of the 15 backups found above, those for 2/21/2018 and 3/2/2018 are about the same size as the backups on my other computer (about 2.1 MB), so I assume that Xmarks synchronized these properly.  The one for 4/17/2018 is about 1.4 MB, and all those since are either 32 KB or 34 KB.

Xmarks support was dropped 5/1/2018, so the problem occurred before they dropped support and I uninstalled the Xmarks program.

Note - the dates of backups on the computer with the problem suggests that a backup will be made automatically only on days when I have made some change to the URLs list since the previous backup.
Are you getting new backups made, and are they looking about the right sort of size?
Flags: needinfo?(robertmiles)
Computer with the problem currently will not boot, so I'm currently unable to check.
Flags: needinfo?(robertmiles)
Ok, thank you for the response. For now, I'll close this as incomplete. If there's more information on an actual problem, we can always re-open or handle it in a new bug.
Status: UNCONFIRMED → RESOLVED
Closed: Last year
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.