Closed Bug 499161 Opened 14 years ago Closed 14 years ago
packaged unittest runs can load libraries from other random unittest builds on the build slave, causing random failures
I've seen two occurences of this problem; one with a mochitest from bug 471281 and the other with a crashtest from bug 492186. Details on the mochitest case: http://hg.mozilla.org/mozilla-central/rev/b116b49459f8 checked in a fix to arcTo stuff and changed the tests. These tests passed on the normal Firefox unittest box, but failed on the mochitest box: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245263253.1245264816.16573.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245261825/firefox-3.6a1pre.en-US.mac.dmg The next run passed: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245264054.1245266769.19988.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245262441/firefox-3.6a1pre.en-US.mac.dmg And the next run failed again: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245265900.1245267488.21136.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245262650/firefox-3.6a1pre.en-US.mac.dmg Then the patch and the test changes were backed out: http://hg.mozilla.org/mozilla-central/rev/ac55c0c1c34a and the machine was green for a while... until http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245279898.1245281588.18871.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245278138/firefox-3.6a1pre.en-US.mac.dmg failed when the disabled tests started to pass unexpectedly. All these things only happened on the Mac boxes, until http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245335889.1245338355.31941.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-unittest/1245333067/firefox-3.6a1pre.en-US.linux-i686.tar.bz2 reported the same "unexpected pass" on a Linux box. Details on the crashtest case: http://hg.mozilla.org/mozilla-central/rev/7fcb443a08ce fixed a crash and added a crashtest. This crashtest passed on the normal box, but failed on the Mac everythingelse box: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245278334.1245279720.15019.gz http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx-unittest/1245275333/firefox-3.6a1pre.en-US.mac.dmg
I think it's unlikely that flaky tests are responsible for this phenomenon: - The failing tests passed consistently on the unittest boxes on the Firefox tree. - I've never seen the "unexpected pass" on the arcTo test before; if the arcTo tests were flaky, they wouldn't have started to act wonky only after the backout. - jrmuizel says that intermittent failures on the arcTo test don't make sense.
The arcTo tests that fail are exactly the ones where the expected result was changed. In these cases the failures seem like they would come from running old builds on a new test suite.
Component: Release Engineering → Mochitest
Product: mozilla.org → Testing
QA Contact: release → mochitest
Version: other → Trunk
catlee was able to determine that automation.py was using the original DIST_BIN in this call to environment(): http://mxr.mozilla.org/mozilla-central/source/build/automation.py.in#588 Which means that sometimes builds would find a build in the original build path and load libxul et. al from there, giving you a build that functions like that other build. Badness ensues.
Summary: Some test runs on the Firefox-Unittest tree seem to run on the wrong build → packaged unittest runs can load libraries from other random unittest builds on the build slave, causing random failures
Comment on attachment 383970 [details] [diff] [review] Set xrePath when creating the environment for the main process What a stupid mistake!
Attachment #383970 - Flags: review?(ted.mielczarek) → review+
Pushed to m-c: http://hg.mozilla.org/mozilla-central/rev/a4a15aa37fa5 We should land this on 1.9.1 too.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
I hate to say it, but there has been another unexplained failure, this time it's the reftest from bug 481614: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox-Unittest/1245377401.1245379142.11871.gz See also bug 481614 comment 15 / 16.
This is still happening on at least reftests.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 384099 [details] [diff] [review] Pass in xrePath to runApp for reftests Another good catch, thanks!
Attachment #384099 - Flags: review?(ted.mielczarek) → review+
Pushed the second patch to m-c: http://hg.mozilla.org/mozilla-central/rev/383949cfb7dc
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
Pushed both patches to 1.9.1: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/814fd57632f9 http://hg.mozilla.org/releases/mozilla-1.9.1/rev/5d021be5fac5
You need to log in before you can comment on or make changes to this bug.