Open Bug 878870 Opened 11 years ago Updated 2 years ago

check in mochitest bisector scripts

Categories

(Testing :: Mochitest, defect)

x86_64
Linux
defect

Tracking

(Not tracked)

People

(Reporter: mmc, Unassigned)

Details

According to ctalbert over IRC, jmaher has some scripts that essentially does https://wiki.mozilla.org/Auto-tools/Projects/Mochi_Bisector?title=Auto-tools/Projects/Mochi_Bisector. Please check them in somewhere.
I have these tools up on my bitbucket account:
https://bitbucket.org/jmaher/bleedthrough

This is more of a brute force method to go test by test and look for side effects.  In fact this could be modified without too much trouble to bisect.  If we find something like this useful, I would be happy to work towards cleaning this up and checking it into m-c if folks desire.
Given how large the mochitest bundles are and how little information the logs provide, this seems like a nobrainer to me :)

Flipping through flags on https://tbpl.mozilla.org/?tree=Try it seems that many people are trying to debug particular mochitest bundles.
This was written for a specific purpose...If we could have some better specs on how users would like to use this tool in general, those could turn into action items.  I had let this run for a week on a few different machines, it takes a long time as it stands.  For bisecting, it could do a good job overnight!
Here's my current effort, if it helps. 

1) Write a patch, push to try https://tbpl.mozilla.org/?tree=Try&rev=405b4b9a3d10
2) mochitest5 is failing on Ubuntu debug.
3) Clicking on the orange 5 reveals failing leakchecks, but no clue to which test leaks
4) Clicking on "Analyze leak" (https://tbpl.mozilla.org/php/getLeakAnalysis.php?id=23681272) says 5 DOM windows leaked, again with no clue to which test leaks

I am guessing based on the description of the original wiki (https://wiki.mozilla.org/Auto-tools/Projects/Mochi_Bisector?title=Auto-tools/Projects/Mochi_Bisector) and looking at flags that other people pass to tryserver, that this kind of problem is not uncommon. Having a bisector either integrated into try, that tells you the minimum set of tests that show the problem, or that you could run manually, would be great.

What do you think?
Running a bisector like this on try would be tough.  Our jobs are limited to a specific run time (I believe 1 hour), and then buildbot will kill them.  If we were bisecting a debug browser chrome session on windows, we could probably get a second iteration in before hitting the limit.  

Running it locally would be good, although the error might now show up locally.  Finding an error for leaks would be easy to look for (I parse the log after the test run).  Making a more general tool use try server to kick off builds and slowly narrow down the changeset would be slow, the overhead of queueing up jobs and waiting for builds is not very usable.

If you could seed this the start by knowing what error to look for, we could avoid the initial run.  That would probably make try server more useful.  Also dealing with random oranges complicate matters greatly, likewise the many test cases which cause other tests to behave differently.

For the specific issue you are debugging, does it reproduce locally?
Deprecating Testing :: Infrastructure (and setting needinfo).
Component: Infrastructure → Mochitest
Flags: needinfo?(mmc)
Hello Joel and Mark,

I am sorry for the late reply.

For the specific issue I was debugging, it did not reproduce locally. It was a memory leak that triggered under a race condition. I eventually fixed the problem with desk checking. I still would have appreciated a bisector that ran locally, even if it took a long time :)

Thanks,
Monica
Flags: needinfo?(mmc)
This is coming up again for another teammate who is having problems reproducing test failures on try.
there is no easy way to bisect on try server, is this problem your team mate is experiencing able to reproduce locally?
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.