Closed Bug 815416 Opened 12 years ago Closed 12 years ago

Excludetest list of all layout test failures in layout for b2g.json


(Testing :: Mochitest, defect)

Windows 7
Not set


(firefox19 fixed, firefox20 fixed, b2g18 fixed)

Tracking Status
firefox19 --- fixed
firefox20 --- fixed
b2g18 --- fixed


(Reporter: martijn.martijn, Assigned: martijn.martijn)




(3 files)

Attached file b2g.json
I've attached a b2g.json file that made a succesful mochitest run inside the layout directory on the b2g emulator.

There might still be intermittent failures lurking around, since I only reran the whole mochtitest run in the layout directory, once.
This is an include list of all layout tests that are passing on the b2g emulator.
337 files and 32702 mochitests running in total.
Thanks Martijn. I've pushed this to try to see how it fares:

Before we can land this, we'll need to increase the number of mochitest chunks used by B2G (I'll file a bug for this).   Additionally, we should modify this to do the inverse of what it's doing, I think...that is, exclude all the tests that fail, rather than include the tests that pass.  The reason for this is so that so new tests added automatically get run for B2G and don't require changes to b2g.json.
Depends on: 819793
The try results are in:

chunk 1:  28 fails
chunk 2:  a timeout then a crash!
chunk 3:  some test timeouts followed by a buildbot timeout

Martijn, if you can add the 28 fails from chunk 1 to this list, we can at least get that running in TBPL.  It may take some more runs to excise all the timeout-related failures.
The last try run tried to run *all* the tests, due to a syntax error in b2g.json.  I've fixed that and pushed again:
(In reply to Jonathan Griffin (:jgriffin) from comment #4)
> The last try run tried to run *all* the tests, due to a syntax error in
> b2g.json.  I've fixed that and pushed again:

This mochitest run looks green to me.
I see that you used an include list, instead of an exclude list. I thought you preferred the exclude list?
Awesome!  Yes, I just used your patch; it would take some additional work to make an exclude list based on this include list.
Well, the first thing I posted here in this bug is an exclude list.
It should basically already be the opposite of this include list.
We should favor "include" lists for now, IMO. Reason is that I don't have buy-in that new tests added need to be B2G compliant (or added to an exclude). So we can't just blanket-include a directory or any given checkin might fail it.

As soon as I hear from overholt that they're willing to make tests B2G-compliant for r+, we can (and should) flip that so tests are opt-out instead of opt-in.
I'd still be in favor of using exclude lists.  If someone adds a test that isn't B2G-compatible, it will fail a try/inbound run, and they'll either be forced to fix it or add it as an exclude.

I think that's better than having people write tests and assume it will be run on B2G, when in fact that wouldn't actually be the case.
To be more clear: most developers do not know about our mobile-specific whitelists.
Pushed the first patch (the blacklist one) to try:
Almost green!

Updated list and pushed to try again:
A green run!  I'm going to land this version on inbound, if no one objects by tomorrow.
For the record, this is the patch I'm landing on inbound
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
Assignee: nobody → martijn.martijn
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
You need to log in before you can comment on or make changes to this bug.