466800 - Gloda Unit Tests timing out on tinderbox occasionally.

Status: VERIFIED FIXED

(MailNews Core :: Database, defect)

Description

[Mark Banner (:standard8)]

We're getting occasional timeouts of the gloda tests on tinderbox:

gmake[3]: Leaving directory `/Volumes/Build/trunk-osx/build/objdir-tb/mailnews/db/gloda/components'
gmake[3]: Entering directory `/Volumes/Build/trunk-osx/build/objdir-tb/mailnews/db/gloda/test'
TEST-PASS | ../../../../mozilla/_tests/xpcshell-simple/test_mailnewsglobaldb/unit/test_cleanup_msf_databases.js | all tests passed
TinderboxPrint: TUnit<br/>344/0
buildbot.slave.commands.TimeoutError: command timed out: 1800 seconds without output, killing pid 853
TinderboxPrint: check <em class="testfail">timeout</em><br/>

Considering this is 1800 seconds without output, then its a little excessive. I'm guessing its not test_cleanup_msf_database.js that is failing, but one of the others.

This seems to have started occurring around the time bug 466636 and bug 465579 were checked in.


It could just be that unit tests aren't being tidy on shutdown, but I'd rather see if we've got time for an investigation before beta 1 just in case its something more serious.

Comment 1

[Mark Banner (:standard8)]

Looks like we're not going to fix these for b1, though I'd really like to get a fix asap as intermittent timeouts on tinderboxes are annoying.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: v1 event driven indexing should not get confused by local folders we have already entered, unit tests should go fast

The indexer was ignoring a message in a local folder because it didn't realize it was local (due to a failure to properly initialize folderIsLocal).  If the indexing of the previous message completed and indexing shutdown before we got the notification for the second message, everything would be okay because the _worker_messageIndex loop would update folderIsLocal.

With this change we can remove throttling from the unit test picture.  Because the unit tests rely on indexing to conclude to know when to run their check logic, I don't think we actually need to run the tests with throttling enabled... that code path is already getting a workout.

Comment 3

[Mark Banner (:standard8)]

I pushed this (http://hg.mozilla.org/comm-central/rev/d4da48162a86) so that we can hopefully stabilise the mac tests a bit.

Comment 4

[Mark Banner (:standard8)]

This caused test failures on Linux at least. So I backed out: http://hg.mozilla.org/comm-central/rev/51612322e680

Full log:

http://tinderbox.mozilla.org/showlog.cgi?tree=Thunderbird&errorparser=unittest&logfile=1228229913.1228230200.24556.gz&buildtime=1228229913&buildname=Linux%20comm-central%20check&fulltext=1

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Attachment: v2 same logic as v1, but add some exception handlers to the test framework to fail with a reason

So, I don't particularly understand why the linux test failed there.  I've added some exception handlers that explicitly invoke do_throw in case something was silently failing and screwing us up.  I think r=bienvenu still holds, but given that I don't know what is happening on the linux box, some form of review is in order.  I'd love a try server...

Comment 6

[Mark Banner (:standard8)]

Comment on attachment 351326 [details] [diff] [review]
v2 same logic as v1, but add some exception handlers to the test framework to fail with a reason

I've just tested this on my linux VM and it seems to work fine. Lets try it again...

Comment 7

[Mark Banner (:standard8)]

This seems to have stayed green this time.

Patch checked in: http://hg.mozilla.org/comm-central/rev/4f274d3e3d11

Comment 8

[Mark Banner (:standard8)]

This seems to have fixed the timeouts on Mac. Linux is still failing occasionally (though not with a timeout), so I've raised bug 468050 for that.