874171 - (SQLite3.7.17) Upgrade to SQLite 3.7.17

Status: RESOLVED FIXED

(Core :: SQLite and Embedded Database Bindings, defect)

Description

[Boaz Dodin]

SQLite version 3.7.17 is a regularly schedule maintenance release.
There are many bug fixes in this version, includes a fix for potential database corruption bug:
http://www.sqlite.org/src/info/e636a050b7

According to SQLite - "due to the large number of fixes, all users are encouraged to upgrade when possible":
http://www.sqlite.org/news.html

Comment 1

[Ryan VanderMeulen [:RyanVM]]

SQLite Release 3.7.17 On 2013-05-20 (3.7.17)

* Add support for memory-mapped I/O.
* Add the sqlite3_strglob() convenience interface.
* Assigned the integer at offset 68 in the database header as the Application ID for when SQLite is used as an application file-format. Added the PRAGMA application_id command to query and set the Application ID.
* Report rollback recovery in the error log as SQLITE_NOTICE_RECOVER_ROLLBACK. Change the error log code for WAL recover from SQLITE_OK to SQLITE_NOTICE_RECOVER_WAL.
* Report the risky uses of unlinked database files and database filename aliasing as SQLITE_WARNING messages in the error log.
* Added the SQLITE_TRACE_SIZE_LIMIT compile-time option.
* Increase the default value of SQLITE_MAX_SCHEMA_RETRY to 50 and make sure that it is honored in every place that a schema change might force a statement retry.
* Add a new test harness called "mptester" used to verify correct operation when multiple processes are using the same database file at the same time.
* Enhance the extension loading mechanism to be more flexible (while still maintaining backwards compatibility) in two ways:
  * If the default entry point "sqlite3_extension_init" is not present in the loadable extension, also try an entry point "sqlite3_X_init" where "X" is based on the shared library filename. This allows every extension to have a different entry point, which allows them to be statically linked with no code changes.
  * The shared library filename passed to sqlite3_load_extension() may omit the filename suffix, and an appropriate architecture-dependent suffix (".so", ".dylib", or ".dll") will be added automatically. 
* Added many new loadable extensions to the source tree, including amatch, closure, fuzzer, ieee754, nextchar, regexp, spellfix, and wholenumber. See header comments on each extension source file for further information about what that extension does.
* Enhance FTS3 to avoid using excess stack space when there are a huge number of terms on the right-hand side of the MATCH operator. A side-effect of this change is that the MATCH operator can only accommodate 12 NEAR operators at a time.
* Enhance the fts4aux virtual table so that it can be a TEMP table.
* Added the fts3tokenize virtual table to the full-text search logic.
* Query planner enhancement: Use the transitive property of constraints to move constraints into the outer loops of a join whenever possible, thereby reducing the amount of work that needs to occur in inner loops.
* Discontinue the use of posix_fallocate() on unix, as it does not work on all filesystems.
* Improved tracing and debugging facilities in the Windows VFS.
* Bug fix: Fix a potential database corruption bug in shared cache mode when one database connection is closed while another is in the middle of a write transaction. Ticket e636a050b7
* Bug fix: Only consider AS names from the result set as candidates for resolving identifiers in the WHERE clause if there are no other matches. In the ORDER BY clause, AS names take priority over any column names. Ticket 2500cdb9be05
* Bug fix: Do not allow a virtual table to cancel the ORDER BY clause unless all outer loops are guaranteed to return no more than one row result. Ticket ba82a4a41eac1.
* Bug fix: Do not suppress the ORDER BY clause on a virtual table query if an IN constraint is used. Ticket f69b96e3076e.
* Bug fix: The command-line shell gives an exit code of 0 when terminated using the ".quit" command.
* Bug fix: Make sure PRAGMA statements appear in sqlite3_trace() output.
* Bug fix: When a compound query that uses an ORDER BY clause with a COLLATE operator, make sure that the sorting occurs according to the specified collation and that the comparisons associate with the compound query use the native collation. Ticket 6709574d2a8d8.
* Bug fix: Makes sure the authorizer callback gets a valid pointer to the string "ROWID" for the column-name parameter when doing an UPDATE that changes the rowid. Ticket 0eb70d77cb05bb2272
* Bug fix: Do not move WHERE clause terms inside OR expressions that are contained within an ON clause of a LEFT JOIN. Ticket f2369304e4
* Bug fix: Make sure an error is always reported when attempting to preform an operation that requires a collating sequence that is missing. Ticket 0fc59f908b

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Attachment: Upgrade to SQLite 3.7.17 - Mozilla changes

https://tbpl.mozilla.org/?tree=Try&rev=66b46892b9bc

Comment 3

[Boaz Dodin]

Are we going to use memory-mapped I/O?

According to SQLite http://sqlite.org/mmap.html there are important advantages do use it:
 * Many operations, especially I/O intensive operations, can be much faster since content does need to be copied between kernel space and user space. In some cases, performance can nearly double.
 * The SQLite library may need less RAM since it shares pages with the operating-system page cache and does not always need its own copy of working pages.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

That question is better handled as a follow-up bug to this one.

Comment 5

[Marco Bonardo [:mak]]

Comment on attachment 752321 [details] [diff] [review]
Upgrade to SQLite 3.7.17 - Mozilla changes

Review of attachment 752321 [details] [diff] [review]:
-----------------------------------------------------------------

nice bugfixes in this version, I also filed a couple dependencies about simple things we may add/drop in our config based on changes, and replied to the chance to use mmap IO (for now it's unlikely we can just enable it blindly).

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1879a11685d1
https://hg.mozilla.org/integration/mozilla-inbound/rev/2aa8aefdbe90

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/1879a11685d1
https://hg.mozilla.org/mozilla-central/rev/2aa8aefdbe90