1146253 - Nightly build 2015-03-21 completely erases history

Status: RESOLVED WONTFIX

(Firefox :: General, defect)

Description

[Francesco Lodolo [:flod]]

Updated Nightly this morning to 39.0a1 (2015-03-22)

After restarting Firefox I noticed that autocomplete in the address bar wasn't working anymore, also noticed missing icons in the boormarks bar, and figured out that history was completely empty.

Restored a profile from backup (2015-03-19), started Nightly, still empty history.

Restored the same profile again, started it with Developer Edition, everything works as expected. Started once again in Nightly, empty history. Tried again with Developer Edition, empty history. So Nightly is actively erasing my history.

Comment 1

[Tim Taubert [:ttaubert] (inactive)]

Marco, any idea what that could be? Did we land anything that could cause this?

Comment 2

[Francesco Lodolo [:flod]]

I have a second Mac running 10.10.2 (like the other one). 

I checked and it was running Nightly 2015-03-20, updated to 2015-03-22, exactly the same behavior: Firefox restarts and I only have bookmarks, no history.

Comment 3

[Francesco Lodolo [:flod]]

Some more notes.

It's not related to add-ons. Opened the profile in Firefox Developer Edition, removed all add-ons, started in Nightly -> empty history.

places.sqlite goes from 73.4 MB to 10,5 MB

Comment 4

[Francesco Lodolo [:flod]]

(In reply to Francesco Lodolo [:flod] from comment #3)
> places.sqlite goes from 73.4 MB to 10,5 MB

If I browser with this profile for a while, it maintains websites in history even after a restart. If I shut down Nightly, replace places.sqlite with my backup, and restart, I still get the empty history.

Comment 5

[Francesco Lodolo [:flod]]

Some more tests.

2015-03-21 has the same issue with history.
2015-03-20 doesn't delete history, but for some reasons can't restore my previous session.

To have a "working" environment I had to:
* Restore my profile's backup.
* Start it with Developer Edition, where I can restore my previous session (ton of pinned tabs).
* Start the same profile with Nightly 2015-03-20, still have my session and history.
* Do not update to 2015-03-22

Comment 6

[Marco Bonardo [:mak]]

[Tracking Requested - why for this release]:

We landed the keywords schema migration, could be Francesco had a keyword that fails a constraint check.

Francesco would it be a problem to send me your backed up places.sqlite, or alternatively a dump of the moz_keywords and moz_bookmarks tables from places.sqlite

Comment 7

[Marco Bonardo [:mak]]

alternatively, I can provide a query to run against the backup places.sqlite that should allow me to rebuild it without exposing privacy data

Comment 8

[Francesco Lodolo [:flod]]

(In reply to Marco Bonardo [::mak] from comment #7)
> alternatively, I can provide a query to run against the backup places.sqlite
> that should allow me to rebuild it without exposing privacy data

I think the query would be ideal, eventually if it doesn't help we can try with the tables.

Comment 9

[Marco Bonardo [:mak]]

So, I found a bug that is pretty bad, but it's not what is causing the failure, I filed that apart.

Comment 10

[Marco Bonardo [:mak]]

[8763] WARNING: The SQL statement 'SELECT place_id FROM moz_keywords' could not be compiled due to an error: no such column: place_id: file /builds/slave/m-cen-m64-d-000000000000000000/build/src/storage/src/mozStorageConnection.cpp, line 1059
[8763] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /builds/slave/m-cen-m64-d-000000000000000000/build/src/storage/src/mozStorageConnection.cpp, line 1408
[8763] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8052000B: file /builds/slave/m-cen-m64-d-000000000000000000/build/src/toolkit/components/places/Database.cpp, line 1561
[8763] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8052000B: file /builds/slave/m-cen-m64-d-000000000000000000/build/src/toolkit/components/places/Database.cpp, line 740

We are adding the column just a few rows before, and that addition does NOT fail, so this is a mistery.
I'll brainstorm over it a little bit hoping to get an idea.

Comment 11

[Marco Bonardo [:mak]]

ok, the warning is unrelated, cause it's generated by storage when we check for the column existence. The problem is just this query

1556   rv = mMainConn->ExecuteSimpleSQL(NS_LITERAL_CSTRING(
1557     "UPDATE moz_places SET foreign_count = "
1558     "(SELECT count(*) FROM moz_bookmarks WHERE fk = moz_places.id) + "
1559     "(SELECT count(*) FROM moz_keywords WHERE place_id = moz_places.id) "
1560   ));
1561   NS_ENSURE_SUCCESS(rv, rv);

Comment 12

[Marco Bonardo [:mak]]

OK, after a long debugging session (thank you flod!) we figured that the problem is caused by a corrupt places.sqlite database.
We rebuilt a clean database out of the corrupt one and verified that it properly migrates to the new version.

What happens is that due to a bad file coruption, the query in comment 11 fails to count the number of rows of one of the tables (likely moz_bookmarks). Since we fail during database initialization, the code evaluates this as an unrecoverable error and replaces the database with a sane one where bookmarks are restored from the last known json backup.
Unfortunately we don't have a history backup (bug 431274) so history gets lost in the process.
The process is "correct" in the sense first of all it tries to fix the profile so the user can continue using Firefox. The history loss is nothing we can fix atm, due to lack of resources.

There is a procedure to try to recover a corrupt places.sqlite, but it's currently not documented anywhere, so I'll create an mdn page documenting it.
It's ideally possible to automate this procedure in Firefox, but that requires a non-trivial amount of time and resources.

This bug will end up being resolved as a WONTFIX after I create the documentation page, cause there's no simple path to restore the database completely from a corrupt file and implementing one is not plausible at the moment.

Comment 13

[Marco Bonardo [:mak]]

https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/places.sqlite_Database_Troubleshooting

Comment 14

[Liz Henry (:lizzard) (relman/hg->git project)]

Untracking since this issue seems to have been from a corrupt profile. 

Marco, what was the bug you mention filing in comment 9?  Is this a general problem that we caused with the keywords schema migration?

Comment 15

[Marco Bonardo [:mak]]

Sorry, I should have untracked this :/
The bug I said in comment 9 was bug 1146299 that is already tracked (and fixed) in Firefox 39.
Thanks.