583611 - (SQLite3.7.1) Upgrade to SQLite 3.7.1

Status: RESOLVED FIXED

(Toolkit :: Storage, defect)

Description

[Shawn Wilsher :sdwilsh]

It's not out yet, but we've got a few bugs that will be fixed by upgrading.

Comment 1

[Shawn Wilsher :sdwilsh]

Asking for blocking so we can fix at least the two issues that are now dependent on this bug.

Comment 2

[Shawn Wilsher :sdwilsh]

Note: The SQLITE_MAX_PAGE_SIZE compile-time option is now silently ignored. The maximum page size is hard-coded at 65536 bytes.

So we can stop passing that in at compile time.

Comment 3

[Shawn Wilsher :sdwilsh]

Performance results (more being computed as I post this):
http://tinyurl.com/2c6q2ac

Still seeing the Ts shutdown regressions (expected), but we should be able to make those go away with work in bug 585713.  Going to wait a little bit, and then land.

Comment 4

[Shawn Wilsher :sdwilsh]

(In reply to comment #2)
> Note: The SQLITE_MAX_PAGE_SIZE compile-time option is now silently ignored. The
> maximum page size is hard-coded at 65536 bytes.
Note that we don't actually use this, so this comment is not important.

Comment 5

[Shawn Wilsher :sdwilsh]

Attachment: Mozilla Changes

Comment 6

[Shawn Wilsher :sdwilsh]

Attachment: SQLite Changes

Comment 7

[Shawn Wilsher :sdwilsh]

Looks good on try: http://hg.mozilla.org/try/rev/4e13658951d3

Comment 8

[Shawn Wilsher :sdwilsh]

Getting a strange error on the try server only for OS X 64bit Debug builds:
alive test failed
=== Output ===
python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log
 in dir /builds/slave/tryserver-macosx64-debug/build/obj-firefox/_leaktest (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'leaktest.py', '--', '--trace-malloc', 'malloc.log', '--shutdown-leaks=sdleak.log']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-x11no2/Render
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-SsFRcV/:0
  HOME=/Users/cltbld
  LC_ALL=C
  LOGNAME=cltbld
  MINIDUMP_STACKWALK=/builds/slave/tryserver-macosx64-debug/tools/breakpad/osx64/minidump_stackwalk
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=obj-firefox
  PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/tryserver-macosx64-debug/build/obj-firefox/_leaktest
  PYTHONPATH=/tools/buildbot/lib/python2.6/site-packages:/tools/twisted/lib/python2.6/site-packages/:/tools/twisted-core/lib/python2.6/site-packages:/tools/zope-interface/lib/python2.6/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-2mNMDn/Listeners
  TMPDIR=/var/folders/7I/7I253dv+HLesSBUBGCX08E+++TM/-Tmp-/
  USER=cltbld
  XPCOM_DEBUG_BREAK=stack-and-abort
 closing stdin
 using PTY: True
INFO | automation.py | Application pid: 95944
WARNING: NS_ENSURE_TRUE(zipFile) failed: file /builds/slave/tryserver-macosx64-debug/build/modules/libjar/nsJAR.cpp, line 165
WARNING: Ignoring duplicate observer.: file /builds/slave/tryserver-macosx64-debug/build/modules/libpref/src/nsPrefBranch.cpp, line 724
WARNING: couldn't get schema version!: file /builds/slave/tryserver-macosx64-debug/build/extensions/cookie/nsPermissionManager.cpp, line 270
WARNING: 1 sort operation has occurred for the SQL statement '0x10b59f278'.  See https://developer.mozilla.org/En/Storage/Warnings details.: file /builds/slave/tryserver-macosx64-debug/build/storage/src/mozStoragePrivateHelpers.cpp, line 138
++DOCSHELL 0x10d5087d0 == 1
++DOMWINDOW == 1 (0x10d538228) [serial = 1] [outer = 0x0]
++DOCSHELL 0x10b4053a0 == 2
++DOMWINDOW == 2 (0x10b409698) [serial = 2] [outer = 0x0]
WARNING: Context has no global.: file /builds/slave/tryserver-macosx64-debug/build/dom/base/nsJSEnvironment.cpp, line 2518
++DOMWINDOW == 3 (0x10b49e438) [serial = 3] [outer = 0x10b409630]
WARNING: Context has no global.: file /builds/slave/tryserver-macosx64-debug/build/dom/base/nsJSEnvironment.cpp, line 2518
++DOMWINDOW == 4 (0x1112daa28) [serial = 4] [outer = 0x10d5381c0]
WARNING: Unable to retrieve pref: plugins.unloadASAP: file /builds/slave/tryserver-macosx64-debug/build/modules/plugin/base/src/nsPluginHost.cpp, line 316
************************************************************
* Call to xpconnect wrapped JSObject produced this error:  *
[Exception... "'[JavaScript Error: "enum.getNext().gBrowser is undefined" {file: "file:///builds/slave/tryserver-macosx64-debug/build/obj-firefox/dist/MinefieldDebug.app/Contents/MacOS/components/nsBrowserGlue.js" line: 100}]' when calling method: [nsIObserver::observe]"  nsresult: "0x80570021 (NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS)"  location: "JS frame :: resource://services-sync/ext/Observers.js :: anonymous :: line 122"  data: yes]
************************************************************
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
WARNING: loaded the same script twice (bug 392650): file /builds/slave/tryserver-macosx64-debug/build/content/xul/document/src/nsXULPrototypeCache.cpp, line 284
++DOCSHELL 0x10cbe13a0 == 3
++DOMWINDOW == 5 (0x10cbe3bd8) [serial = 5] [outer = 0x0]
++DOCSHELL 0x10c900000 == 4
++DOMWINDOW == 6 (0x10c900618) [serial = 6] [outer = 0x0]
Chrome file doesn't exist: /builds/slave/tryserver-macosx64-debug/build/obj-firefox/dist/MinefieldDebug.app/Contents/MacOS/chrome/toolkit/content/global/autocomplete.css
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012: file /builds/slave/tryserver-macosx64-debug/build/layout/style/Loader.cpp, line 2092
++DOCSHELL 0x10fd34400 == 5
WARNING: NS_ENSURE_TRUE(browserChrome) failed: file /builds/slave/tryserver-macosx64-debug/build/docshell/base/nsDocShell.cpp, line 10597
WARNING: Something wrong when creating the docshell for a frameloader!: file /builds/slave/tryserver-macosx64-debug/build/content/base/src/nsFrameLoader.cpp, line 1440
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /builds/slave/tryserver-macosx64-debug/build/content/base/src/nsFrameLoader.cpp, line 275
++DOCSHELL 0x10fdb6ec0 == 6
++DOMWINDOW == 7 (0x10fdb9b28) [serial = 7] [outer = 0x0]
WARNING: Context has no global.: file /builds/slave/tryserver-macosx64-debug/build/dom/base/nsJSEnvironment.cpp, line 2518
++DOMWINDOW == 8 (0x10ffb44a8) [serial = 8] [outer = 0x10fdb9ac0]

command timed out: 3600 seconds without output, killing pid 95943
process killed by signal 9
program finished with exit code -1
elapsedTime=4412.715618

Comment 9

[Shawn Wilsher :sdwilsh]

I filed bug 590224 about the above issue.  I've seen it every time I've pushed SQLite to the try server, but then other changesets without the SQLite upgrade have also seen it.  I think we can land.

Comment 10

[Shawn Wilsher :sdwilsh]

And in case anyone asks, yes, I'm aware that SQLite 3.7.2 came out.  We don't need to take it, so I'm going to move forward with 3.7.1.

Comment 11

[Shawn Wilsher :sdwilsh]

http://hg.mozilla.org/mozilla-central/rev/49e08d483b1b
http://hg.mozilla.org/mozilla-central/rev/c6838f6cb68a

Comment 12

[Shawn Wilsher :sdwilsh]

Per discussions with LegNeato, we need this on 1.9.2 to land bug 573492

Comment 13

[LegNeato]

Marking as blocking. Please ask for approvals when needed.

Comment 14

[Shawn Wilsher :sdwilsh]

Attachment: 1.9.2 patch

Comment 15

[Daniel Veditz [:dveditz]]

Comment on attachment 478309 [details] [diff] [review]
1.9.2 patch

What about 3.5.x users?

Comment 16

[LegNeato]

Comment on attachment 478309 [details] [diff] [review]
1.9.2 patch

a=LegNeato for 1.9.2.11.

Comment 17

[Shawn Wilsher :sdwilsh]

(In reply to comment #15)
> What about 3.5.x users?
I didn't think we wanted it for that.  I can make a patch for that too (the risk should be about the same).

Comment 18

[Shawn Wilsher :sdwilsh]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/d4f62917d563

Comment 19

[Shawn Wilsher :sdwilsh]

OK, so due to Apple contributed code to make SQLite work properly over NFS (which we enable), we can't actually run on OS X 10.4.  Apple doesn't care about 10.4 which puts us in a crappy situation.  We don't want to turn on WAL until we get it on a security branch, but in order to do that, we'd have to break NFS support on branch.

rock * hard place
     ^ us ):

Comment 20

[Shawn Wilsher :sdwilsh]

Also, backed out in http://hg.mozilla.org/releases/mozilla-1.9.2/rev/97ad6aaa7a21

Comment 21

[Daniel Veditz [:dveditz]]

I don't fully understand comment 19. If NFS didn't work ("properly"?) on OS X before Apple's 10.4-breaking patch then does it matter if we turn NFS off on the branch (for mac-only, of course)?

Comment 22

[Shawn Wilsher :sdwilsh]

(In reply to comment #21)
> I don't fully understand comment 19. If NFS didn't work ("properly"?) on OS X
> before Apple's 10.4-breaking patch then does it matter if we turn NFS off on
> the branch (for mac-only, of course)?
I'm still investigating what to do here, but AFAIK, it works correctly in the old code, but Apple contributed a bunch of new code.  I'll post back here when I get more information.

Comment 23

[Shawn Wilsher :sdwilsh]

And to elaborate, I'm working with the SQLite team to get a fix which will be 3.7.1 + minimal changes for it to compile and work correctly on 10.4.

Comment 24

[Shawn Wilsher :sdwilsh]

This is the current change that should let us work with 10.4 again:
http://www.sqlite.org/src/ci/44deaaefee
This should only remove a performance optimization when you have two clients on different hosts talking to the same database at the same time, which we should never have.

I'm going to put this through try to make sure it's all good.  Should I file a new bug this then on branch, or keep it here?

Comment 25

[LegNeato]

Here's fine, please land.

Comment 26

[Shawn Wilsher :sdwilsh]

Attachment: 1.9.2 patch v1.1

With the fix mentioned in comment 24.  Can't test this since try doesn't let us target 10.4 though, so we have to try by landing...

Comment 27

[LegNeato]

Yep, I've approved for landing "blind".

Comment 28

[Shawn Wilsher :sdwilsh]

OK...landed away!
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/09649109fcea