Closed Bug 694741 Opened 8 years ago Closed 8 years ago

Move Android mochitest test-paths into the source tree

Categories

(Testing :: Mochitest, defect, blocker)

ARM
Android
defect
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED
mozilla10

People

(Reporter: philor, Assigned: jmaher)

References

Details

(Whiteboard: [android_tier_1][inbound])

Attachments

(1 file)

Bug 691411 moved js/src/xpconnect to js/xpconnect, the kind of path move we've done hundreds of times over the years. Unfortunately, the way that now breaks Android mochitests looks just like a way that they break anyway, so despite having starred thousands of Android test failures, it took me five times of saying that they were just a known failure before I realized that they were actually broken and trying to load a --test-path that no longer existed.

There is absolutely no way that we can expect developers who have starred maybe tens of Android test failures to recognize that, so given the current state of things, we have to expect to repeat the bug 691411 / bug 694716 thing over and over, as developers don't realize their try failure is real, land, everyone else rebases around the move, it gets merged to other trees, and then finally someone notices that a particular Android mochitest hunk has failed quite a few times in a row.

I'm also not sure how we could do it in a world with mozilla-central/mozilla-aurora/mozilla-beta/mozilla-release, even if someone changing paths on mozilla-central knew about the Android dependency. Would that then give us a merge-day dependency on a buildbot reconfig, having to change a list of "if (branch == 'mozilla-central' || branch == 'mozilla-aurora')" conditions as paths change trains?

I don't know *what* to put in the code tree, or how to have buildbot use it, but I do know that having it there would make it possible for a developer to fix his bustage after he finally discovers it, without needing to enrage someone from releng on a Friday evening, would make it possible for changes to automatically move along release trains, and given the way bsmedberg has always described the requirements for being a tier 1 test suite, would take Android mochitests from their current state of not being acceptable as a tier 1 suite because of their external dependency on another repo to being just another normal perfectly fine suite.
traditional mochitests on desktop just run everything (in 5 chunks) and do not use directory names.  For Android, we have so many tests which do not run (e10s, no applicable to fennec, resource limitations, bugs in fennec, bugs in the tests) that we are only running tests which we can ensure the entire directory runs.  This will not change anytime soon.

I really don't understand why if a test fails on try server, a developer will not take a look at it and if they don't know what is wrong ask somebody.  If that is not happening, then what needs to happen in order to make that the case?  

The last 5 pushes I have done to try where I have built Android and done tests have resulted in an average of 1 failure per push on all Android tests.
Did you look at mozilla-central for the push where this landed and the next couple of pushes?
Oh, never mind, I already mentioned that in comment 0.

Okay, so given a failure mode which looks exactly like a common intermittent failure, how can we make it so that developers will not just ignore it?

Maybe retry every Android failure on try automatically, until it has failed 10 times?
<rant>
Oranges only account for 45% of the intermittent failures and that is all the mobile team is interested in.  Additionally, there is a bug where we crash (bug 691073) and the mobile team takes no interest in fixing it or helping other teams who are debugging it get the debugging tools working.
</rant>

To solve the problem in the bug title, we need to branch the buildbot configs for each branch on android.  Assuming we could get manifests or tests fixed landed today and all mozilla-central mochitests ran, we still couldn't run them until buildbot configs supported different configs per branch.

I think we can get a manifest in place for android mochitests without a lot of trouble to resolve this.  The problem is this manifest solution will need to be on all branches, hence the need for better buildbot configs on android.
I don't understand (possibly because of my handwaving understanding of buildbot) why having the test-paths in the source tree requires per-branch configs.

If you hadn't been around to point out that we could just do all of js/ in bug 694716, then the current scheme would have required it: had poor bholley realized what try was telling him and gone looking for instructions on how to change a path, wouldn't the instructions have been "well, we have a single path for every tree on every branch where we run Android, so you either need to land your little 'hg mv' not only at the same time as a releng reconfig, but also land it at the same time on every single tree that's a mozilla-central branch, and on m-a and m-b and m-r, or, perhaps you should just realize that Android testing has a stranglehold on the current layout of the tree, and it cannot be changed."

If instead buildbot knew to (more handwaving) look for a file in the packaged tests that the build had put there, which being from a build on a particular tree at a particular rev would already be branched, and knew to read that file that the build had copied from /testing/mochitest/android_chunks.ini and look within that for

[hunk_2] = ['dom/src/json/test'], ['dom/src/jsurl/test'], ['dom/tests/mochitest/dom-level0'], ['js/jsd/test'], ['js/src/xpconnect/tests/mochitest'], ['js/xpconnect/tests/mochitest']

then his instructions would switch from "you need to land your move on every single branch and twig all at once and at the same time as a buildbot reconfig" to "just change /testing/mochitest/android_chunks.ini in your patch that moves things around."
(In reply to Joel Maher (:jmaher) from comment #4)

> I think we can get a manifest in place for android mochitests without a lot
> of trouble to resolve this.  The problem is this manifest solution will need
> to be on all branches, hence the need for better buildbot configs on android.
I agree we can do a manifest here.  So, we would need to branch the buildbot configs if we did this because the same buildbot configs run for nightly, aurora, beta, and release?  I thought that each of those trees had their own configs.

Why can't we just put a manifest into place for nightly, then roll those changes through the system?
this patch only exposes the directories in the json file to mochitests.  I haven't tested this on a production environment, but if this list looks good we can put it in staging.

To run this we would do:
python runtestsremote.py --run-only-tests=android.json --total-chunks=8 --this-chunk=1 --chunk-by-dir

instead of:
python runtestsremote.py --test-path=...


NOTE: the buidlbot configs would have to change for m-c unless this and another patch were to land on all branches.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #567446 - Flags: review?(ctalbert)
Attachment #567446 - Flags: feedback?(bear)
Attachment #567446 - Flags: feedback?(bear) → feedback+
Comment on attachment 567446 [details] [diff] [review]
add a manifest of tests to run for android mochitests (1.0)

So freaking awesome.
Attachment #567446 - Flags: review?(ctalbert) → review+
Whiteboard: [android_tier_1] → [android_tier_1][inbound]
https://hg.mozilla.org/mozilla-central/rev/104f9c9469be
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla10
You need to log in before you can comment on or make changes to this bug.