Closed Bug 499161 Opened 14 years ago Closed 14 years ago

packaged unittest runs can load libraries from other random unittest builds on the build slave, causing random failures

Categories

(Testing :: Mochitest, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mstange, Assigned: catlee)

Details

(Keywords: fixed1.9.1)

Attachments

(2 files)

I've seen two occurences of this problem; one with a mochitest from bug 471281  and the other with a crashtest from bug 492186.

Details on the mochitest case:

http://hg.mozilla.org/mozilla-central/rev/b116b49459f8 checked in a fix to arcTo stuff and changed the tests. These tests passed on the normal Firefox unittest box, but failed on the mochitest box:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245263253.1245264816.16573.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245261825/firefox-3.6a1pre.en-US.mac.dmg

The next run passed:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245264054.1245266769.19988.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245262441/firefox-3.6a1pre.en-US.mac.dmg

And the next run failed again:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245265900.1245267488.21136.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245262650/firefox-3.6a1pre.en-US.mac.dmg

Then the patch and the test changes were backed out:
http://hg.mozilla.org/mozilla-central/rev/ac55c0c1c34a
and the machine was green for a while... until

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245279898.1245281588.18871.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245278138/firefox-3.6a1pre.en-US.mac.dmg

failed when the disabled tests started to pass unexpectedly.
All these things only happened on the Mac boxes, until

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245335889.1245338355.31941.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-unittest/1245333067/firefox-3.6a1pre.en-US.linux-i686.tar.bz2

reported the same "unexpected pass" on a Linux box.

Details on the crashtest case:

http://hg.mozilla.org/mozilla-central/rev/7fcb443a08ce fixed a crash and added a crashtest. This crashtest passed on the normal box, but failed on the Mac everythingelse box:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245278334.1245279720.15019.gz
http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245275333/firefox-3.6a1pre.en-US.mac.dmg
I think it's unlikely that flaky tests are responsible for this phenomenon:
 - The failing tests passed consistently on the unittest boxes on the Firefox
   tree.
 - I've never seen the "unexpected pass" on the arcTo test before; if the
   arcTo tests were flaky, they wouldn't have started to act wonky only after
   the backout.
 - jrmuizel says that intermittent failures on the arcTo test don't make sense.
The arcTo tests that fail are exactly the ones where the expected result was changed. In these cases the failures seem like they would come from running old builds on a new test suite.
Component: Release Engineering → Mochitest
Product: mozilla.org → Testing
QA Contact: release → mochitest
Version: other → Trunk
catlee was able to determine that automation.py was using the original DIST_BIN in this call to environment():
http://mxr.mozilla.org/mozilla-central/source/build/automation.py.in#588

Which means that sometimes builds would find a build in the original build path and load libxul et. al from there, giving you a build that functions like that other build. Badness ensues.
Summary: Some test runs on the Firefox-Unittest tree seem to run on the wrong build → packaged unittest runs can load libraries from other random unittest builds on the build slave, causing random failures
Assignee: nobody → catlee
Comment on attachment 383970 [details] [diff] [review]
Set xrePath when creating the environment for the main process

What a stupid mistake!
Attachment #383970 - Flags: review?(ted.mielczarek) → review+
Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/a4a15aa37fa5

We should land this on 1.9.1 too.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
I hate to say it, but there has been another unexplained failure, this time it's the reftest from bug 481614:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245377401.1245379142.11871.gz
See also bug 481614 comment 15 / 16.
This is still happening on at least reftests.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #384099 - Flags: review?(ted.mielczarek)
Comment on attachment 384099 [details] [diff] [review]
Pass in xrePath to runApp for reftests

Another good catch, thanks!
Attachment #384099 - Flags: review?(ted.mielczarek) → review+
Pushed the second patch to m-c:
http://hg.mozilla.org/mozilla-central/rev/383949cfb7dc
Status: REOPENED → RESOLVED
Closed: 14 years ago14 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.