Closed Bug 499161 Opened 15 years ago Closed 15 years ago

packaged unittest runs can load libraries from other random unittest builds on the build slave, causing random failures


(Testing :: Mochitest, defect)

Not set


(Not tracked)



(Reporter: mstange, Assigned: catlee)


(Keywords: fixed1.9.1)


(2 files)

I've seen two occurences of this problem; one with a mochitest from bug 471281  and the other with a crashtest from bug 492186.

Details on the mochitest case: checked in a fix to arcTo stuff and changed the tests. These tests passed on the normal Firefox unittest box, but failed on the mochitest box:

The next run passed:

And the next run failed again:

Then the patch and the test changes were backed out:
and the machine was green for a while... until

failed when the disabled tests started to pass unexpectedly.
All these things only happened on the Mac boxes, until

reported the same "unexpected pass" on a Linux box.

Details on the crashtest case: fixed a crash and added a crashtest. This crashtest passed on the normal box, but failed on the Mac everythingelse box:
I think it's unlikely that flaky tests are responsible for this phenomenon:
 - The failing tests passed consistently on the unittest boxes on the Firefox
 - I've never seen the "unexpected pass" on the arcTo test before; if the
   arcTo tests were flaky, they wouldn't have started to act wonky only after
   the backout.
 - jrmuizel says that intermittent failures on the arcTo test don't make sense.
The arcTo tests that fail are exactly the ones where the expected result was changed. In these cases the failures seem like they would come from running old builds on a new test suite.
Component: Release Engineering → Mochitest
Product: → Testing
QA Contact: release → mochitest
Version: other → Trunk
catlee was able to determine that was using the original DIST_BIN in this call to environment():

Which means that sometimes builds would find a build in the original build path and load libxul et. al from there, giving you a build that functions like that other build. Badness ensues.
Summary: Some test runs on the Firefox-Unittest tree seem to run on the wrong build → packaged unittest runs can load libraries from other random unittest builds on the build slave, causing random failures
Assignee: nobody → catlee
Comment on attachment 383970 [details] [diff] [review]
Set xrePath when creating the environment for the main process

What a stupid mistake!
Attachment #383970 - Flags: review?(ted.mielczarek) → review+
Pushed to m-c:

We should land this on 1.9.1 too.
Closed: 15 years ago
Resolution: --- → FIXED
I hate to say it, but there has been another unexplained failure, this time it's the reftest from bug 481614:
See also bug 481614 comment 15 / 16.
This is still happening on at least reftests.
Resolution: FIXED → ---
Attachment #384099 - Flags: review?(ted.mielczarek)
Comment on attachment 384099 [details] [diff] [review]
Pass in xrePath to runApp for reftests

Another good catch, thanks!
Attachment #384099 - Flags: review?(ted.mielczarek) → review+
Pushed the second patch to m-c:
Closed: 15 years ago15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.