627300 - Intermittent failure in places/tests/queries/test_tags.js | test failed (with xpcshell return code: -1073741819) OR xpcshell return code: 1, probably accessing a dead scope

Status: RESOLVED FIXED

(Toolkit :: Places, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

The root cause of the assertion is related to bug 611563 which is where the JS engine gets angry because some JS code tries to access properties on a dead window.  Since this is an xpcshell test, the likely answer is it is accessing some other dead scope rather than a window.

This test should probably try and make sure all pending events are consumed before the test completes.

Right now, 2 out of 2 runs of the windows debug builds on the places branch are hitting this after my push of the fix for 624051.  Because of the nature of the fix (an additional asynchronous operation is enqueued instead of potentially blocking the main thread, giving the asynchronous thread time to run its queue to zero), it greatly increases the potential for this kind of race to occur.

I am logging this as an intermittent failure because the problem could easily occur without the fix for bug 624051 and so it seems worth exposing this to the autostarring magic so people don't log against the other bugs in the test.  I will, however, be backing out my patch if I can't see an obvious way to fix the test quickly or it turns out I lack authority to perform a push under bustage-fixing rules or such.

TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\places\tests\queries\test_tags.js | test failed (with xpcshell return code: -1073741819), see following log:
PROCESS-CRASH | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\places\tests\queries\test_tags.js | application crashed (minidump found)

TEST-PASS | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/places/tests/queries/test_tags.js | [makeQuery : 770] false == false
TEST-PASS | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/places/tests/queries/test_tags.js | [setsAreEqual : 814] 1 == 1
TEST-PASS | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/places/tests/queries/test_tags.js | [null : 821] true == true
TEST-PASS | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/places/tests/queries/test_tags.js | [null : 822] true == true
TEST-INFO | (xpcshell/head.js) | test 2 finished
### ERROR: SymGetModuleInfo64: The specified module could not be found.

TEST-INFO | (xpcshell/head.js) | test 1 finished
TEST-INFO | (xpcshell/head.js) | exiting test
TEST-INFO | (xpcshell/head.js) | test 1 pending
### ERROR: SymGetModuleInfo64: The specified module could not be found.

TEST-PASS | (xpcshell/head.js) | 123 (+ 0) check(s) passed
WARNING: nsExceptionService ignoring thread destruction after shutdown: file e:/builds/moz2_slave/plc-w32-dbg/build/xpcom/base/nsExceptionService.cpp, line 197
Assertion failure: obj->containsSlot(slot), at e:/builds/moz2_slave/plc-w32-dbg/build/js/src/jsinterp.cpp:5333
<<<<<<<
PROCESS-CRASH | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\places\tests\queries\test_tags.js | application crashed (minidump found)

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

This made a Linux debug angry too, it just has a different error code.

Rev3 Fedora 12 places debug test xpcshell
http://tinderbox.mozilla.org/showlog.cgi?log=Places/1295501512.1295503050.21855.gz&fulltext=1

and the windows ones:
Rev3 WINNT 6.1 places debug test xpcshell
http://tinderbox.mozilla.org/showlog.cgi?log=Places/1295499758.1295503006.21762.gz&fulltext=1

Rev3 WINNT 6.1 places debug test xpcshell
http://tinderbox.mozilla.org/showlog.cgi?log=Places/1295495593.1295498960.3422.gz&fulltext=1

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

This is the bad guy:

(gdb) call DumpJSStack()
0 _run_next_test() ["/home/visbrero/rev_control/hg/mozilla-central/obj-firefox-debug/_tests/xpcshell/toolkit/components/places/tests/queries/head_queries.js -> file:///home/visbrero/rev_control/hg/mozilla-central/obj-firefox-debug/_tests/xpcshell/toolkit/components/places/tests/head_common.js":651]
    this = [object BackstagePass @ 0x96a8b0 (native @ 0x63c288)]
1 anonymous() ["/home/visbrero/rev_control/hg/mozilla-central/testing/xpcshell/head.js":406]
    this = [object Object]
2 <TOP LEVEL> ["<unknown>":0]
    <failed to get 'this' value>

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

It appears the test bookkeeping is wrong in this file or the places test framework in general.  Looking at the logs of the failed cases, it is clearly terminating the test early.  A successful run claims something like 582 checks, whereas the failed cases claim numbers like 196 and 123.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

test_tags.js is missing a wrapping do_test_pending so the test count is constantly hitting zero and setting _quit to true in xpcshell's head.js.  The only reason the test works at all is because xpcshell tries real hard to drain the event loops before quitting.

I will push a fix shortly.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

pushed to places with the justification that this is a bustage fix for my push of 624051 which greatly increased the chance of orange, and that it's happening on a lightly trafficked integration branch:
http://hg.mozilla.org/projects/places/rev/79348d4f7586

I am marking this fixed-in-places and not RESOLVED/FIXED because the orange could easily happen anywhere and could still affect trunk.

This should be marked fixed when places gets merged across.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Removing dependency on the JS bug, which I also mis-typed.  (I meant bug 611653.)

Comment 7

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

thank you for the fix, Shawn had a fix for this in bug 606966, we will patch run_next_test, but your effort is appreciated and avoided us going back to check the cause of this orange!

Comment 8

[Shawn Wilsher :sdwilsh]

D'oh, yeah...I had this fixed it just hadn't landed yet.  Thanks asuth!

Comment 9

[Shawn Wilsher :sdwilsh]

http://hg.mozilla.org/mozilla-central/rev/79348d4f7586