1162643 - (SQLite3.8.10.1) Upgrade to SQLite 3.8.10.1

Status: RESOLVED FIXED

(Toolkit :: Storage, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

SQLite Release 3.8.10 On 2015-05-07

* Added the sqldiff.exe utility program for computing the differences between two SQLite database files.
* Added the y format string to the matchinfo() function of FTS3.
* Performance improvements for ORDER BY, VACUUM, CREATE INDEX, PRAGMA integrity_check, and PRAGMA quick_check.
* Fix many obscure problems discovered while SQL fuzzing.
* Identify all methods for important objects in the interface documentation. (example)
* Made the American Fuzzy Lop fuzzer a standard part of SQLite's testing strategy.
* Add the ".binary" and ".limits" commands to the command-line shell.
* Make the "dbstat" virtual table part of standard builds when compiled with the SQLITE_ENABLE_DBSTAT_VTAB option. 

Richard, we recently upgraded to 3.8.9 across a number of older releases in order to pick up the fuzzing fixes it contained. Is it worth taking this release across all of them too?

Comment 1

[D. Richard Hipp]

The 3.8.10 does fix several arcane bugs (mostly assertion faults) that the AFL fuzzer picked up.  But none of this are important to FF, I don't believe.

On the other hand, the reason we released 3.8.10 early in the first place is to provide the SQLITE_ENABLE_DBSTAT_VTAB compile-time option, for the DBSTAT virtual table, which Ben Turner wants to use in order to derive better estimates on when a VACUUM of the IndexedDB data would be helpful.

Also, the performance improvements result in about 7.5% fewer CPU cycles for our tests.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

OK, sounds like we can just land it on mozilla-central and let it go through the regular release process then. Thanks!

Comment 3

[Ben Turner (not reading bugmail, use the needinfo flag!)]

And we need to add the SQLITE_ENABLE_DBSTAT_VTAB define to our build config.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Attachment: Upgrade to SQLite 3.8.10

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4aa82c4df27

Comment 5

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Comment on attachment 8602878 [details] [diff] [review]
Upgrade to SQLite 3.8.10

This will need another sqlite3_compileoption_used check in configure.in

Comment 6

[D. Richard Hipp]

Ugh.  In the rush to get the release out, we failed to add SQLITE_ENABLE_DBSTAT_VTAB to the sqlite3_compileoption_used() method.  Bummer.  Does that throw a wrench into things?

Comment 7

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to D. Richard Hipp from comment #6)
> Ugh.  In the rush to get the release out, we failed to add
> SQLITE_ENABLE_DBSTAT_VTAB to the sqlite3_compileoption_used() method. 
> Bummer.  Does that throw a wrench into things?

Oh, hm. Well, I guess we can test for it some other way?

Comment 8

[D. Richard Hipp]

You mean test at run-time?  Just run:  "CREATE VIRTUAL TABLE temp.dbstat USING dbstat;"  If it works, then SQLITE_ENABLE_DBSTAT_VTAB is on.  If it fails, then it isn't.

Comment 9

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to D. Richard Hipp from comment #8)
> You mean test at run-time?

Well, sorta. We need to test it at configure time (that's half run-time and half build-time!). We need to make sure that any system-supplied sqlite was built with SQLITE_ENABLE_DBSTAT_VTAB, otherwise we make the build fail.

Comment 10

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Attachment: 8602878: Upgrade to SQLite 3.8.10

Comment 11

[Ben Turner (not reading bugmail, use the needinfo flag!)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=590da1db33e9

Comment 12

[Marco Bonardo [:mak]]

would it be a big deal to release a 3.8.10.1 with updated sqlite3_compileoption_used?

Fwiw, we could even take a patched 3.8.10, if the same fix will be in the next Sqlite release.
We don't usually patch Sqlite, but it's ok to do that when there's already a fix upstream that is riding the train to the next release.

Comment 13

[Marco Bonardo [:mak]]

oh wait, that wouldn't do for system sqlite, nevermind.

Comment 14

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Mak, the updated configure.in changes I attached allow us to detect this without sqlite3_compileoption_used support, so we don't really need a new version, right?

Comment 15

[Marco Bonardo [:mak]]

I saw that, it is still far more complex than it would be with proper sqlite3_compileoption_used support.
So I'm asking what's the price to have that fixed in a .1 version, to compare that with the price of this more complex check that regardless we should replace as soon as we get a fixed sqlite3_compileoption_used. If it's easy to get .1 version I don't think it's worth the workaround and fullow-up bug to remove it.

Comment 16

[D. Richard Hipp]

SQLite version 3.8.10.1 is a patch release that fixes sqlite3_compileoption_used() so that it reports on SQLITE_ENABLE_DBSTAT_VTAB as it should.

Comment 17

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to D. Richard Hipp from comment #16)

Thanks!

Comment 18

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Attachment: Upgrade to SQLite 3.8.10.1

Comment 19

[Marco Bonardo [:mak]]

Comment on attachment 8604193 [details] [diff] [review]
Upgrade to SQLite 3.8.10.1

Review of attachment 8604193 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 20

[Ben Turner (not reading bugmail, use the needinfo flag!)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c6ea21c091e9

Comment 21

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c6ea21c091e9

Comment 22

[Arthur K. (he/him)]

SQLite 3.8.10.2 was released on the 5/20/2015 with the below fix being the only update:

- Fix an index corruption issue introduced by version 3.8.7. An index with a TEXT key can be corrupted by an INSERT into the corresponding table if the table has two nested triggers that convert the key value to INTEGER and back to TEXT again. Ticket 34cd55d68e0

Comment 23

[Arthur K. (he/him)]

(In reply to Arthur K. from comment #22)
> SQLite 3.8.10.2 was released on the 5/20/2015 with the below fix being the
> only update:
> 
> - Fix an index corruption issue introduced by version 3.8.7. An index with a
> TEXT key can be corrupted by an INSERT into the corresponding table if the
> table has two nested triggers that convert the key value to INTEGER and back
> to TEXT again. Ticket 34cd55d68e0

Ugh, sorry for the churn. Saw too late that this was already reported.