513999 - mochitest-plain hanging during shutdown

Status: RESOLVED FIXED

(Core :: General, defect, P1)

Description

[Benjamin Smedberg]

Starting at 2009/08/30 19:12:43 the mochitest-plain suite is exhibiting a new kind of "random" failure: the suite is completing, but during shutdown the browser apparently hangs, or at least stops responding for 300 seconds.

This has happened for at least 50% of the Linux unittest builds since then, and some of the separate-Mochitest builds.

I'm going to tentatively blame bug 512328 (upgrade liboggplay), because this seems to have started with http://hg.mozilla.org/mozilla-central/rev/fb6d235b9efb and that patch has some fun changes in terms of which functions are called holding locks, etc.

It's possible that some of the immediately prior landings triggered this instead: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=aa3a30f3f1e8&tochange=fb6d235b9efb

Unless you have better options, I'd like the liboggplay changes backed out for a day to see if that makes the hang/orange go away.

Comment 1

[Reed Loden [:reed]]

Isn't this just bug 503623?

Comment 2

[Benjamin Smedberg]

No. In that case the test may hang, but it will not hang after all the tests appear to have completed (on shutdown).

Comment 3

[Chris Pearce [:cpearce (Not reading bugmail)]]

Backed out bug 512328: http://hg.mozilla.org/mozilla-central/rev/e8b3bd49771e

Comment 4

[Chris Pearce [:cpearce (Not reading bugmail)]]

Due dilligence:

Linux mozilla-central unit test on 2009/09/01 12:55:56
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251834956.1251842516.27964.gz

Linux mozilla-central unit test on 2009/09/01 12:03:16
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251831796.1251839570.28029.gz

Linux mozilla-central unit test on 2009/09/01 09:50:30
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251823830.1251831956.9021.gz

Linux mozilla-central unit test on 2009/09/01 09:44:09
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251823449.1251831213.424.gz

Linux mozilla-central unit test on 2009/09/01 08:15:21
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251818121.1251826303.8091.gz

Linux mozilla-central unit test on 2009/09/01 08:15:21
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251818121.1251826303.8091.gz

Linux mozilla-central unit test on 2009/08/31 09:35:39
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251736539.1251747606.591.gz

Linux mozilla-central unit test on 2009/08/30 19:12:43
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251684763.1251695339.25850.gz

Linux mozilla-central unit test on 2009/08/30 19:12:43
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251684763.1251695339.25850.gz

Checkin:
Sun Aug 30 19:09:40 2009 -0700	fb6d235b9efb	Chris Pearce — Bug 512398 - fix liboggplay update (rev f4087c5ac148) which accidentally removed liboggplay's os2_semaphore.*. r=doublec

Comment 5

[Chris Pearce [:cpearce (Not reading bugmail)]]

After backing out, we added more unit test boxes to the tinderbox and we had:

2 Unit Test boxes pass:
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251855248.1251864050.4245.gz
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251855271.1251864823.12407.gz

2 Unit Test boxes otherwise randomly fail:
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251847131.1251855592.10372.gz
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1251855065.1251862141.16036.gz

4 Mochites boxes otherwise randomly fail:
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1251850954.1251854033.25511.gz
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1251857620.1251860737.860.gz
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1251859524.1251861874.13258.gz
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1251860357.1251863060.26016.gz

1 Everything-else box pass:
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1251857620.1251859737.22571.gz

1 Everything-else box otherwise randomly fail:
* http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1251850954.1251852981.14132.gz

None failed with the timeout that this bug was filed for. We'd best keep an eye out for further failures, but it seems probable that the liboggplay update was the cause.

Comment 6

[Chris Pearce [:cpearce (Not reading bugmail)]]

Closing bug. No more occurrences since the backout. Looks like it's caused by the audio hardware clock not advancing in nsOggDecodeStateMachine::PlayFrame(). That explains why it only occurs under linux too.