530667 - (SQLite3.6.22) Upgrade to SQLite 3.6.22

Status: RESOLVED FIXED

(Toolkit :: Storage, defect)

Description

[Shawn Wilsher :sdwilsh]

We'll need this to pick up a fix to how SQLITE_SECURE_DELETE works (so it works right in all cases), as well as fixing the performance regression we hit when upgrading to 3.6.20.

Comment 1

[Shawn Wilsher :sdwilsh]

3.6.22 is out very soon.  I got a copy from drh and it passed the try server, so we are going to take that instead.

Comment 2

[Shawn Wilsher :sdwilsh]

Attachment: Mozilla Changes

rs=mconnor for SQLite library upgrades.

Comment 3

[Shawn Wilsher :sdwilsh]

Pushed this to the try server Monday without issue.  Trying mozilla-central today.

Comment 4

[Shawn Wilsher :sdwilsh]

Lets hope the try server is right. :)

http://hg.mozilla.org/mozilla-central/rev/17c42b6b184c
http://hg.mozilla.org/mozilla-central/rev/b2d8a922901c

Comment 5

[Shawn Wilsher :sdwilsh]

I'd like to start talking about getting this into 1.9.2.  This release has a number of stability fixes, plus a bunch of performance wins for us (especially on shutdown).  Right now we have 3.6.16.1 on 1.9.2.  This has baked on mozilla-central for two weeks without any issues reported against it.

Comment 6

[Mike Beltzner [:beltzner, not reading bugmail]]

OK, so let's start talking. I'll go first:

 - what is the state of automated testing for SQLite usage in Firefox?
 - where have we gotten caught when doing these upgrades before?
 - have we thought if this will have any adverse effects on mobile?

Comment 7

[Shawn Wilsher :sdwilsh]

(In reply to comment #6)
>  - what is the state of automated testing for SQLite usage in Firefox?
Better than most of the code base.  The recent LCOV reports look suspect because I know there are tests for some of the stuff that it says is not tested, but you can view that report here:
http://people.mozilla.com/~mnandigama/codecoverage_html/storage/src/index.html
Additionally, SQLite has 100% branch test coverage, so the likelihood of introducing a regression is low (although still possible).  See http://www.sqlite.org/testing.html for more details on their testing procedures.

>  - where have we gotten caught when doing these upgrades before?
Performance regressions, but that is easy enough to back out.

>  - have we thought if this will have any adverse effects on mobile?
It should not.  In fact, it should give us a performance and stability win across the board.

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Thunderbird would also like to see SQLite 3.6.22 on 1.9.2 to be able to minimize forkage.

We released TB 3.1a1 with 3.6.22, and TB 3.0 with 3.6.20 if that provides any reassurance.  (I'd go with the insane level of testing the SQLite people perform, myself.)

Comment 9

[Mike Beltzner [:beltzner, not reading bugmail]]

:asuth when are you planning on shipping a milestone off of 1.9.2? Trying to determine if we need this for 1.9.2.2 or can wait for a future one.

I don't think this blocks, per se, but we're definitely interested. Is there a changelist that drivers can look over?

Comment 10

[Shawn Wilsher :sdwilsh]

(In reply to comment #9)
> I don't think this blocks, per se, but we're definitely interested. Is there a
> changelist that drivers can look over?
http://www.sqlite.org/changes.html

1.9.2 is running SQLite 3.6.16.1.

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #9)
> :asuth when are you planning on shipping a milestone off of 1.9.2? Trying to
> determine if we need this for 1.9.2.2 or can wait for a future one.

Thunderbird 3.1b1 is shipping this week-ish and is using a mozilla-1.9.2 release branch where we have patched 3.6.22 into place.  (The reason for the patching is actually for the storage component more than SQLite; but since we already had to patch...)

Thunderbird 3.1b2 will ship at the end of the March or early April.  We will no longer be using a specially patched mozilla-1.9.2 release for storage purposes and will instead be using a copy of mozilla-central's storage transplanted into comm-central.  That landing will likely happen this week so that our nightlies are usable.  We will transplant SQLite 3.6.22 as well if it's not already in mozilla-1.9.2 since that will eliminate a lot of crash-stats debates for us.

It's not a big deal if we have to do it, it's just preferable not to have to do it.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 420363 [details] [diff] [review]
Mozilla Changes

Pretty pretty please can we land SQLite 3.6.22 for 1.9.2.3?

The motivation is that Thunderbird needs a more recent release of SQLite than 3.6.16.1 and it turns out that my claim that we could easily fork it into comm-central underestimated the difficulty.  We would otherwise need to turn to a release branch.

As discussed earlier on the bug, the risks of upgrading the SQLite version are very low.

Comment 13

[Nomis101]

Since today there is SQLite 3.6.23 in m-c, see Bug 551260. So wouldn't it be better to update 1.9.2.3 to 3.6.23? Or is this to early?

Comment 14

[Shawn Wilsher :sdwilsh]

No, it hasn't backed in a beta release so we do not know for sure how stable it is.

Comment 15

[Mark Banner (:standard8)]

Just to make an extension to comment 8 - Thunderbird has shipped 3.6.22 since 3.0.2 (since Gecko 1.9.1.8) and we've not seen any significant problems as a result.

Comment 16

[Shawn Wilsher :sdwilsh]

(In reply to comment #15)
> Just to make an extension to comment 8 - Thunderbird has shipped 3.6.22 since
> 3.0.2 (since Gecko 1.9.1.8) and we've not seen any significant problems as a
> result.
I am going to prod beltzner in person about this today since here is on-site.  I won't be able to land it until next week, but I bet asuth would be happy to do so since Thunderbird really wants it.

Comment 17

[Andrew Sutherland [:asuth] (he/him)]

beltzner, can you provide an ETA on when 1.9.2 drivers would be willing to accept 3.6.22 into mozilla-1.9.2?  If it's not imminently, can we get approval to change the sqlite.def file to expose a function we need for our comm-central-only backport of mozStorage?

Comment 18

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 420363 [details] [diff] [review]
Mozilla Changes

a=beltzner for 1.9.2.4 who is hoping like hell this isn't the bug that comes back to bite him in the ass

Comment 19

[Andrew Sutherland [:asuth] (he/him)]

Attachment: SQLite 3.6.22 mozilla changes for mozilla-1.9.2 v1

This is effectively the same patch as for trunk except that we also update the sqlite.def file to export sqlite3_db_mutex.

Stuff that trunk has that mozilla-1.9.2 does not have with these changes:
1) On trunk the library has been renamed to "mozsqlite3" (versus "sqlite3") and also has a SHORT_LIBNAME because of os/2.
2) On trunk "NO_PROFILE_GUIDED_OPTIMIZE = 1" is only the case for SOLARIS_SUNPRO_CC, whreas on mozilla-1.9.2 it is the case for everybody.
3) On trunk we have more thorough configure.in checks when system SQLite is requested.

I believe this is all correct, but it bears explicitly calling out so beltzner does not hunt me down.

Comment 20

[Shawn Wilsher :sdwilsh]

Comment on attachment 437131 [details] [diff] [review]
SQLite 3.6.22 mozilla changes for mozilla-1.9.2 v1

>--- Shawn Wilsher <[e-mail]>, 10/2009
>+-- Shawn Wilsher <[e-mail]>, 01/2010
You should update the date to this month, and put your name since you are actually doing that upgrade.

r=sdwilsh

Comment 21

[Andrew Sutherland [:asuth] (he/him)]

I landed this on mozilla-1.9.2 and think it's sticking.  (There are a few outstanding ridiculous slow Windows optimized builds yet to report in, but they are unlikely to have anything interesting other than random oranges.)  I also needed to remove that pesky obsolete test_storage_combined_sharing.js unit test that sdwilsh gave r+ to remove long ago when the Thunderbird release branches ran into the same trouble:

SQLite 3.6.22:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/89f29d9aea91
nuked test:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/c2a5d27c0c5b

beltzner, Thunderbird thanks you for your approval granting.  As a downstream consumer of stable mozilla trees, we do have a nice set of chompers.

Comment 22

[Serge Gautherie (:sgautherie)]

(In reply to comment #21)
> SQLite 3.6.22:
> http://hg.mozilla.org/releases/mozilla-1.9.2/rev/89f29d9aea91

+
http://hg.mozilla.org/comm-central/rev/fa38ca676dd2