1339321 - (SQLite3.17.0) Upgrade to SQLite 3.17.0

Status: RESOLVED FIXED

(Core :: SQLite and Embedded Database Bindings, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

+++ This bug was initially created as a clone of Bug #1328161 +++

2017-02-13 - Release 3.17.0

SQLite version 3.17.0 is a regularly scheduled maintenance release.

Most of the changes in this release are performance optimizations. Optimizations to the R-Tree extension are especially noticeable.

In this release, the default size of the lookaside buffer allocated for each database connection is increased from 64,000 to 120,000 bytes. This provides improved performance on many common workloads in exchange for a small increase in memory usage. Applications that value a small memory footprint over raw speed can change the lookaside buffer size back to its old value (or to zero) using the SQLITE_DEFAULT_LOOKASIDE compile-time option, or the sqlite3_config(SQLITE_CONFIG_LOOKASIDE) start-time setting, or the sqlite3_db_config(SQLITE_DBCONFIG_LOOKASIDE) run-time setting.

Full changelog:
http://sqlite.org/releaselog/3_17_0.html

Comment 1

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e293652cb9852f971b2953f823385d9c12103fdf

Comment 2

[Marco Bonardo [:mak]]

rs=me

Comment 3

[Marco Bonardo [:mak]]

Note, we may want to uplift this, at least to Aurora, per https://bugzilla.mozilla.org/show_bug.cgi?id=1309696#c29

The specific change:
"Fixed performance problems and potential stack overflows when creating views from multi-row VALUES clauses with hundreds of thousands of rows."
Looks like on MacOS that limit is much lower than hundred thousands, and it's just a few thousands.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b3404ec6a4f0
Upgrade to SQLite 3.17.0. r=mak

Comment 6

[Ryan VanderMeulen [:RyanVM]]

(In reply to Marco Bonardo [::mak] from comment #3)
> Note, we may want to uplift this, at least to Aurora, per
> https://bugzilla.mozilla.org/show_bug.cgi?id=1309696#c29

Whether it's via uplifting the 3.17.0 release or getting a custom 3.15.3 build, I still think we should do something for 52 as well since it's going to be the next ESR release.

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b3404ec6a4f0

Comment 8

[Marco Bonardo [:mak]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #6)
> Whether it's via uplifting the 3.17.0 release or getting a custom 3.15.3
> build, I still think we should do something for 52 as well since it's going
> to be the next ESR release.

3.16 and 3.17 were mostly maintenance releases, without large breaking changes, so we could even try to uplift this. It's not something we should do normally, but considering the large amount of perf and stability improvements in these versions, plus the crash fix, it sounds like a good idea.

We should also try to fix bug 1309696 separately, so that in case we should ever need to undo this, we'll have a separate proper fix.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Attachment: Upgrade SQLite to version 3.17.0

(In reply to Wes Kocher (:KWierso) from comment #7)
> https://hg.mozilla.org/mozilla-central/rev/b3404ec6a4f0

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: crashes from bug 1309696
[Is this code covered by automated tests?]: yes, both in our tree and extensively upstream
[Has the fix been verified in Nightly?]: as much as a third-party library update can be :)
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: not really
[Why is the change risky/not risky?]: as Marco pointed out, none of the recent SQLite updates have had any major breaking changes in them. We also have a lot of in-tree tests that exercise SQLite and I'm confident that any major issues would be caught by them.
[String changes made/needed]: none

Comment 10

[Mike Hommey [:glandium]]

A note for future updates: the single sqlite.c source is, in essence, a unified build of the normal sqlite multi-file source. Since we have unified builds of our own, we could import the multi-file source and use UNIFIED_SOURCES.

Comment 11

[Marco Bonardo [:mak]]

(In reply to Mike Hommey [:glandium] from comment #10)
> A note for future updates: the single sqlite.c source is, in essence, a
> unified build of the normal sqlite multi-file source. Since we have unified
> builds of our own, we could import the multi-file source and use
> UNIFIED_SOURCES.

Apart from the obvious benefit of not blocking the UI when one tries to see part of the Sqlite code in dxr, what are the advantages?

Comment 12

[Marco Bonardo [:mak]]

note: for uplift we'll also need the upstream patch for bug 1340028. We can take upstream patches, we just don't take mozilla-only patches.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

I just landed an import of the upstream patch on inbound with Marco's blessing. I'll fold it into the uplift in this bug when/if it's approved.

Comment 14

[Julien Cristau [:jcristau]]

Comment on attachment 8837610 [details]
Upgrade SQLite to version 3.17.0

update sqlite to fix some recursion issue on osx, aurora53+, beta52+

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/34611d3b1fb6

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/eeb452d52bd3

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/eeb452d52bd3