1173356 - Total failure to persist browser.db state across relaunches

Status: RESOLVED FIXED

(Firefox for iOS :: Data Storage, defect)

Description

[Stephan Leroux [:sleroux]]

STR:

1. Launch app with code from latest master in Xcode onto iOS simulator (https://github.com/mozilla/firefox-ios/commit/1763aed1292e841ae5121ac03d3d7b0cfc2d3327)
2. Navigate to a few websites to populate the history
3. Re-run the application from Xcode
4. Previous history doesn't show up anymore

Comment 1

[Richard Newman [:rnewman]]

No Sync?

Got a device log?

Comment 2

[Stephan Leroux [:sleroux]]

I just reproduced this in my simulator. Here's the console log:

[DEBUG] Did open IPv4 listening socket 6
[DEBUG] Did open IPv6 listening socket 7
[INFO] GCDWebServer started on port 53591 and reachable at http://localhost:53591/
2015-06-10 11:24:17.257 [Debug] [BrowserDB.swift:65] init(files:): Initializing BrowserDB.
2015-06-10 11:24:17.258 [Debug] [BrowserDB.swift:124] createOrUpdate: Create or update tableList version 1.
2015-06-10 11:24:17.259 [Debug] [BrowserDB.swift:78] createTable(_:table:): Try create tableList version 1
2015-06-10 11:24:17.260 [Debug] [BrowserDB.swift:124] createOrUpdate: Create or update logins version 1.
2015-06-10 11:24:17.260 [Debug] [SQLiteLogins.swift:55] exists: 0 logins tables exist.
2015-06-10 11:24:17.260 [Debug] [BrowserDB.swift:78] createTable(_:table:): Try create logins version 1
2015-06-10 11:24:17.261 Client[57051:2996286] Warning: Unable to create restoration in progress marker file
2015-06-10 11:24:17.282 [Debug] [BrowserDB.swift:124] createOrUpdate: Create or update favicons version 0.
2015-06-10 11:24:17.282 [Debug] [BrowserDB.swift:78] createTable(_:table:): Try create favicons version 0
2015-06-10 11:24:17.282 [Info] [BrowserTable.swift:64] init(): SQLite version: 3.8.5 (3008005).
2015-06-10 11:24:17.282 [Debug] [BrowserDB.swift:124] createOrUpdate: Create or update BROWSER version 4.
2015-06-10 11:24:17.283 [Debug] [BrowserTable.swift:248] exists: 0 tables exist. Expected 4
2015-06-10 11:24:17.283 [Debug] [BrowserDB.swift:78] createTable(_:table:): Try create BROWSER version 4
2015-06-10 11:24:17.362 [Debug] [Profile.swift:341] beginTimedHistorySync(): Starting history sync timer.
[DEBUG] Did open connection on socket 14
[DEBUG] Did connect
[DEBUG] Did start background task
[DEBUG] Connection received 350 bytes on socket 14
[DEBUG] Connection aborted with status code 405 on socket 14
[DEBUG] Connection sent 113 bytes on socket 14
[DEBUG] Did close connection on socket 14
[VERBOSE] [::1c1e:d158:0:0:53591] ::314f:d0dc:a79c:ed:53592 405 "GET /about/home/" (350 | 113)
2015-06-10 11:24:17.598 [Debug] [Profile.swift:170] onLocationChange: Ignoring navigation for nil.
[DEBUG] Did open connection on socket 17
DEBUG: readerModeMessageHandler message: {
    Type = ReaderModeStateChange;
    Value = Unavailable;
}
[DEBUG] Connection received 311 bytes on socket 17
2015-06-10 11:24:17.611 [Debug] [BrowserViewController.swift:1497] readerMode(_:didChangeRead[DEBUG] Connection aborted with status code 405 on socket 17
erModeState:forBrowser:): New readerModeState: Unavailable
[DEBUG] Connection sent 113 bytes on socket 17
[DEBUG] Did close connection on socket 17
[VERBOSE] [::1c1e:d159:0:0:53591] ::314f:d0dc:a79c:ed:53593 405 "GET /favicon.ico" (311 | 113)
[DEBUG] Did disconnect
[DEBUG] Did end background task

Comment 3

[Richard Newman [:rnewman]]

The DB tables aren't persisting across restarts. Which is weird.

Can you grab your browser.db out of the Simulator and run PRAGMA integrity_check?

Suggestions:

* We should explicitly WAL checkpoint after a createOrUpdate in BrowserTable.
* We should explicitly sqlite3_finalize our statements, rather than doing it in a deinit. The sooner that gets started, the less likely we'll miss a finalize due to an app kill.

Comment 4

[Richard Newman [:rnewman]]

DB plz!

Comment 5

[Stephan Leroux [:sleroux]]

Attachment: profile.profile.zip

I attached the profile I have along with browser.db you were looking for. Also ran integrity_check on it:

sqlite> PRAGMA integrity_check;
ok

Comment 6

[Wesley Johnston (:wesj)]

Looks like the tablelist isn't being created...

Comment 7

[Richard Newman [:rnewman]]

12:31:21 <@rnewman> sleroux: opening browser.db causes the WAL to replay, so it jumps to 76KB
12:31:26 <@rnewman> and all the tables are there

This looks totally valid when opened locally.

Comment 8

[Richard Newman [:rnewman]]

Full WAL checkpointing doesn't help. This isn't limited to history, either:

2015-06-10 13:02:29.062 [Debug] [BrowserDB.swift:72] init(files:): Initializing BrowserDB.
2015-06-10 13:02:29.065 [Debug] [BrowserDB.swift:134] createOrUpdate: Create or update tableList version 1.
2015-06-10 13:02:29.067 [Debug] [BrowserDB.swift:85] createTable(_:table:): Try create tableList version 1
2015-06-10 13:02:29.068 [Debug] [SwiftData.swift:119] transaction: Op in transaction succeeded. Committing.
2015-06-10 13:02:29.069 [Debug] [BrowserDB.swift:134] createOrUpdate: Create or update logins version 1.
2015-06-10 13:02:29.069 [Debug] [SQLiteLogins.swift:55] exists: 0 logins tables exist.
2015-06-10 13:02:29.070 [Debug] [BrowserDB.swift:85] createTable(_:table:): Try create logins version 1
2015-06-10 13:02:29.070 [Debug] [SwiftData.swift:119] transaction: Op in transaction succeeded. Committing.
...


so for some reason on relaunch the DB is just blank.

Comment 9

[Richard Newman [:rnewman]]

Somehow we're getting .bak files created when setup seems to complete.

Comment 10

[Richard Newman [:rnewman]]

Attachment: Pull req.

Inside-out bugfixing: this fixes the SQL, and adds logging that would have shown the bug.

Comment 11

[Richard Newman [:rnewman]]

Stefan: if Bug 1171732 got into Build 20, we'll need to respin a Build 21.

Comment 12

[Wesley Johnston (:wesj)]

Comment on attachment 8620728 [details] [review]
Pull req.

Doh. Thanks.

Comment 13

[Stephan Leroux [:sleroux]]

Comment on attachment 8620728 [details] [review]
Pull req.

Looks good! Works on my simulator and device now.

Comment 14

[Richard Newman [:rnewman]]

https://github.com/mozilla/firefox-ios/commit/acfe2dcd4ec9a2f1bc0faea9fb35fc273defbd82

Comment 15

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Ok we will need to do a new build then!