840377 - sqlite: infinite loop during enabling wifi

Status: RESOLVED FIXED

(Core :: Storage: IndexedDB, defect)

Description

[Gregor Wagner [:gwagner]]

#19241 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19242 0x4071030c in resolveSelectStep (pWalker=0x48a91b34, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19243 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a91b34, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19244 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19245 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19246 0x4071030c in resolveSelectStep (pWalker=0x48a91c0c, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19247 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a91c0c, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19248 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19249 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19250 0x4071030c in resolveSelectStep (pWalker=0x48a91ce4, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19251 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a91ce4, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19252 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19253 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19254 0x4071030c in resolveSelectStep (pWalker=0x48a91dbc, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19255 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a91dbc, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19256 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19257 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19258 0x4071030c in resolveSelectStep (pWalker=0x48a91e94, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19259 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a91e94, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19260 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19261 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19262 0x4071030c in resolveSelectStep (pWalker=0x48a91f6c, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19263 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a91f6c, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19264 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
---Type <return> to continue, or q <return> to quit---
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19265 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19266 0x4071030c in resolveSelectStep (pWalker=0x48a92044, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19267 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a92044, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19268 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19269 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19270 0x4071030c in resolveSelectStep (pWalker=0x48a9211c, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19271 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a9211c, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19272 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19273 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19274 0x4071030c in resolveSelectStep (pWalker=0x48a921f4, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19275 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a921f4, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19276 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19277 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19278 0x4071030c in resolveSelectStep (pWalker=0x48a922cc, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:72966
#19279 0x4070eec0 in sqlite3WalkSelect (pWalker=0x48a922cc, p=0x49acdb08) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:71948
#19280 0x407107a4 in sqlite3ResolveSelectNames (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:73228
#19281 0x4072da8c in sqlite3SelectPrep (pParse=0x4b5a0c08, p=0x49acdb08, pOuterNC=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97118
#19282 0x4072e03e in sqlite3Select (pParse=0x4b5a0c08, p=0x49acdb08, pDest=0x48a924e8) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:97383
#19283 0x40730cf8 in codeTriggerProgram (pParse=0x4b5a0c08, pStepList=0x4b5d7048, orconf=5)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:99162
#19284 0x40730f24 in codeRowTrigger (pParse=0x4aedac08, pTrigger=0x4c1b06d8, pTab=0x47fcdb88, orconf=5)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:99291
#19285 0x4073101c in getRowTrigger (pParse=0x4aedac08, pTrigger=0x4c1b06d8, pTab=0x47fcdb88, orconf=5)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:99348
#19286 0x40731190 in sqlite3TriggerColmask (pParse=0x4aedac08, pTrigger=0x4c1b06d8, pChanges=0x0, isNew=0, tr_tm=3, pTab=0x47fcdb88, orconf=5)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:99513
#19287 0x4071e24c in sqlite3GenerateRowDelete (pParse=0x4aedac08, pTab=0x47fcdb88, iCur=0, iRowid=11, count=0, pTrigger=0x4c1b06d8, onconf=5)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:84967
#19288 0x40723fdc in sqlite3GenerateConstraintChecks (pParse=0x4aedac08, pTab=0x47fcdb88, baseCur=0, regRowid=2, aRegIdx=0x49acdf08, rowidChng=0, 
    isUpdate=0, overrideError=5, ignoreDest=-1, pbMayReplace=0x48a9270c) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:89368
---Type <return> to continue, or q <return> to quit---
#19289 0x4072367a in sqlite3Insert (pParse=0x4aedac08, pTabList=0x49ace688, pList=0x49acf588, pSelect=0x0, pColumn=0x49ace788, onError=5)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:88936
#19290 0x4073c0b8 in yy_reduce (yypParser=0x4ac45808, yyruleno=172) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:109551
#19291 0x4073d84c in sqlite3Parser (yyp=0x4ac45808, yymajor=1, yyminor=..., pParse=0x4aedac08)
    at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:110332
#19292 0x4073e4a0 in sqlite3RunParser (pParse=0x4aedac08, 
    zSql=0x425fe540 "INSERT OR REPLACE INTO object_data (object_store_id, key_value, data, file_ids) VALUES (:osid, :key_value, :data, :file_ids)", 
    pzErrMsg=0x48a92960) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:111169
#19293 0x40728ed2 in sqlite3Prepare (db=0x49aaf808, 
    zSql=0x425fe540 "INSERT OR REPLACE INTO object_data (object_store_id, key_value, data, file_ids) VALUES (:osid, :key_value, :data, :file_ids)", 
    nBytes=-1, saveSqlFlag=1, pReprepare=0x0, ppStmt=0x4b5d9518, pzTail=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:93310

#19294 0x407290f6 in sqlite3LockAndPrepare (db=0x49aaf808, 
    zSql=0x425fe540 "INSERT OR REPLACE INTO object_data (object_store_id, key_value, data, file_ids) VALUES (:osid, :key_value, :data, :file_ids)", 
    nBytes=-1, saveSqlFlag=1, pOld=0x0, ppStmt=0x4b5d9518, pzTail=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:93402
#19295 0x4072923a in sqlite3_prepare_v2 (db=0x49aaf808, 
    zSql=0x425fe540 "INSERT OR REPLACE INTO object_data (object_store_id, key_value, data, file_ids) VALUES (:osid, :key_value, :data, :file_ids)", 
    nBytes=-1, ppStmt=0x4b5d9518, pzTail=0x0) at /Volumes/2mac/gaia/b2g18/db/sqlite3/src/sqlite3.c:93478
#19296 0x417273a2 in mozilla::storage::Connection::prepareStatement (this=0x47fcd7c0, aSQL=..., _stmt=0x4b5d9518)
    at /Volumes/2mac/gaia/b2g18/storage/src/mozStorageConnection.cpp:806

#19297 0x41729b02 in mozilla::storage::Statement::initialize (this=0x4b5d9500, aDBConnection=0x47fcd7c0, aSQLStatement=...)
    at /Volumes/2mac/gaia/b2g18/storage/src/mozStorageStatement.cpp:144
#19298 0x41727f1a in mozilla::storage::Connection::CreateStatement (this=0x47fcd7c0, aSQLStatement=..., _stmt=0x48a92ac8)
    at /Volumes/2mac/gaia/b2g18/storage/src/mozStorageConnection.cpp:1104
#19299 0x4123eb00 in mozilla::dom::indexedDB::IDBTransaction::GetCachedStatement (this=0x4aeea7f0, aQuery=...)
    at /Volumes/2mac/gaia/b2g18/dom/indexedDB/IDBTransaction.cpp:410

#19300 0x4123ba54 in mozilla::dom::indexedDB::IDBTransaction::GetCachedStatement<125> (this=0x4aeea7f0, aQuery=...)
    at ../../dist/include/mozilla/dom/indexedDB/IDBTransaction.h:124
#19301 0x41236c5c in DoDatabaseWork (this=0x4be4f820, aConnection=0x47fcd7c0) at /Volumes/2mac/gaia/b2g18/dom/indexedDB/IDBObjectStore.cpp:2822
#19302 0x41217264 in mozilla::dom::indexedDB::AsyncConnectionHelper::Run (this=0x4be4f820)
    at /Volumes/2mac/gaia/b2g18/dom/indexedDB/AsyncConnectionHelper.cpp:300

#19303 0x41254774 in mozilla::dom::indexedDB::TransactionThreadPool::TransactionQueue::Run (this=0x47fcd640)
    at /Volumes/2mac/gaia/b2g18/dom/indexedDB/TransactionThreadPool.cpp:639
#19304 0x41b639aa in nsThreadPool::Run (this=0x47f8c6a0) at /Volumes/2mac/gaia/b2g18/xpcom/threads/nsThreadPool.cpp:187
#19305 0x41b618dc in nsThread::ProcessNextEvent (this=0x483d4100, mayWait=true, result=0x48a92e97)
    at /Volumes/2mac/gaia/b2g18/xpcom/threads/nsThread.cpp:620
---Type <return> to continue, or q <return> to quit---
#19306 0x41b1b5d2 in NS_ProcessNextEvent_P (thread=0x483d4100, mayWait=true)
    at /Volumes/2mac/gaia/b2g18/nooptunagibuild/xpcom/build/nsThreadUtils.cpp:237
#19307 0x41b60cfe in nsThread::ThreadFunc (arg=0x483d4100) at /Volumes/2mac/gaia/b2g18/xpcom/threads/nsThread.cpp:258
#19308 0x405d1254 in _pt_root (arg=0x47f989b0) at /Volumes/2mac/gaia/b2g18/nsprpub/pr/src/pthreads/ptthread.c:156
#19309 0x4005be18 in __thread_entry (func=0x405d119d <_pt_root>, arg=0x47f989b0, tls=<value optimized out>) at bionic/libc/bionic/pthread.c:217
#19310 0x4005b96c in pthread_create (thread_out=<value optimized out>, attr=0xbee18a00, start_routine=0x405d119d <_pt_root>, arg=0x47f989b0)
    at bionic/libc/bionic/pthread.c:357

more GDB info:
http://www.pastebin.mozilla.org/2132692

Comment 1

[Blake Kaplan (:mrbkap) (inactive)]

One thing my bumbling gdb session showed was that the Select object that's passed around inside of the sqlite code had a null pSrc member. In sqlite's selectExpander function, there's an if check that suggests that this should never happen (...). As far as I could tell by looking around, this suggests an undetected malloc failure, but I wasn't able to prove that happening.

Comment 2

[Andrew Overholt [:overholt]]

This blocks a blocker and thus needs to block.

Comment 3

[Andrew Overholt [:overholt]]

Gregor, if you're too busy, maybe Ben can take this when he's back from PTO.

Comment 4

[Gregor Wagner [:gwagner]]

Yeah I don't think I can do much here. This seems to be a sqlite bug.

Comment 5

[D. Richard Hipp]

I can make a reasonable guess at how to fix the infinite loop.  But to really fix it right, we need to be able to reproduce the problem, and thus verify that it really has been fixed and that we don't get a regression in the future.  I'm having trouble doing that.

I see the SQL statement that is being prepared in the GDB output.  ("INSERT OR REPLACE INTO object_data (object_store_id, key_value, data, file_ids) VALUES (:osid, :key_value, :data, :file_ids)".)  But I'm having trouble figuring out which database this is running against, so that I cannot determine what the database schema is.  Clearly there are some triggers involved, and I suspect those are very important in reproducing the problem, so access to the schema is important.

Where do I need to look to find the schema for the database file that is preparing the INSERT statement when the loop occurs?

Comment 6

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to D. Richard Hipp from comment #5)
> I see the SQL statement that is being prepared in the GDB output.  ("INSERT
> OR REPLACE INTO object_data (object_store_id, key_value, data, file_ids)
> VALUES (:osid, :key_value, :data, :file_ids)".)

That's an IndexedDB statement for storing data via the add/put method.

> Where do I need to look to find the schema for the database file that is
> preparing the INSERT statement when the loop occurs?

https://mxr.mozilla.org/mozilla-central/source/dom/indexedDB/OpenDatabaseHelper.cpp#149

That function creates the schema for all of our current databases.

Comment 7

[D. Richard Hipp]

The SQLite check-in at http://www.sqlite.org/src/info/b7222a2bd0 should prevent the problem from ever recurring.  Nevertheless, we would still very much like to better understand the problem and to generate test cases to reproduce it if possible.  We'll continue to investigate using the schema information provided in Comment 6 above.

Question: Should we push the patch above into a rush release - 3.7.15.3 or 3.7.14.2?  Or, since this is seems to be a very obscure problem, can you wait until the next regular release cycle?

Comment 8

[Ben Turner (not reading bugmail, use the needinfo flag!)]

We could also just patch mozilla-b2g18 manually and wait for the regular release on mozilla-central. Gregor, how often do you see this?

Comment 9

[Gregor Wagner [:gwagner]]

(In reply to ben turner [:bent] from comment #8)
> We could also just patch mozilla-b2g18 manually and wait for the regular
> release on mozilla-central. Gregor, how often do you see this?

I hit this issue twice now during debugging bug 832385 and really hammering the wifi enable-disable button. Bug 832385 and this one might be related but we currently don't know.

We are more likely to hit bug 832385 so it's hard to say how often I see this bug. I would say around every 1000-3000 times I press the wifi button.

Comment 10

[Andrew Overholt [:overholt]]

We can't block here due to the rarity of reproduction.  If it becomes more easily reproducible, let's reconsider.

Comment 11

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Attachment: Patch for b2g18

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Not a regression
User impact if declined: Occasional infinite hangs
Testing completed: SQLite team has integrated it into their codebase so all of their testing on it should be sufficient
Risk to taking this patch (and alternatives if risky): Tiny patch, will be included in next SQLite version, should be safe
String or UUID changes made by this patch: None

Comment 12

[Ben Turner (not reading bugmail, use the needinfo flag!)]

I suggest we patch b2g18 with this tiny fix and then wait for the next SQLite version to get it on trunk.

Comment 13

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Ryan, can you shepherd this into the right branches? Thanks!

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/f614daf98a6e