Excludetest list of all layout test failures in layout for b2g.json

RESOLVED FIXED in Firefox 19

Status

RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: martijn.martijn, Assigned: martijn.martijn)

Tracking

unspecified
mozilla20
x86
Windows 7
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox19 fixed, firefox20 fixed, b2g18 fixed)

Details

Attachments

(3 attachments)

(Assignee)

Description

6 years ago
Created attachment 685380 [details]
b2g.json

I've attached a b2g.json file that made a succesful mochitest run inside the layout directory on the b2g emulator.

There might still be intermittent failures lurking around, since I only reran the whole mochtitest run in the layout directory, once.
(Assignee)

Updated

6 years ago
Blocks: 781696
(Assignee)

Comment 1

6 years ago
Created attachment 690099 [details]
Include list of all layout tests that are passing

This is an include list of all layout tests that are passing on the b2g emulator.
337 files and 32702 mochitests running in total.
Thanks Martijn. I've pushed this to try to see how it fares:

https://tbpl.mozilla.org/?tree=Try&rev=d8efad2677c3

Before we can land this, we'll need to increase the number of mochitest chunks used by B2G (I'll file a bug for this).   Additionally, we should modify this to do the inverse of what it's doing, I think...that is, exclude all the tests that fail, rather than include the tests that pass.  The reason for this is so that so new tests added automatically get run for B2G and don't require changes to b2g.json.
Depends on: 819793
The try results are in:

chunk 1:  28 fails
chunk 2:  a timeout then a crash!
chunk 3:  some test timeouts followed by a buildbot timeout

Martijn, if you can add the 28 fails from chunk 1 to this list, we can at least get that running in TBPL.  It may take some more runs to excise all the timeout-related failures.
The last try run tried to run *all* the tests, due to a syntax error in b2g.json.  I've fixed that and pushed again: https://tbpl.mozilla.org/?tree=Try&rev=e5d6f1a973cc
(Assignee)

Comment 5

6 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #4)
> The last try run tried to run *all* the tests, due to a syntax error in
> b2g.json.  I've fixed that and pushed again:
> https://tbpl.mozilla.org/?tree=Try&rev=e5d6f1a973cc

This mochitest run looks green to me.
I see that you used an include list, instead of an exclude list. I thought you preferred the exclude list?
Awesome!  Yes, I just used your patch; it would take some additional work to make an exclude list based on this include list.
(Assignee)

Comment 7

6 years ago
Well, the first thing I posted here in this bug is an exclude list.
It should basically already be the opposite of this include list.
We should favor "include" lists for now, IMO. Reason is that I don't have buy-in that new tests added need to be B2G compliant (or added to an exclude). So we can't just blanket-include a directory or any given checkin might fail it.

As soon as I hear from overholt that they're willing to make tests B2G-compliant for r+, we can (and should) flip that so tests are opt-out instead of opt-in.
I'd still be in favor of using exclude lists.  If someone adds a test that isn't B2G-compatible, it will fail a try/inbound run, and they'll either be forced to fix it or add it as an exclude.

I think that's better than having people write tests and assume it will be run on B2G, when in fact that wouldn't actually be the case.
To be more clear: most developers do not know about our mobile-specific whitelists.
Pushed the first patch (the blacklist one) to try:
https://tbpl.mozilla.org/?tree=Try&rev=1bf8acc6ae99
Blocks: 793045
Almost green!

Updated list and pushed to try again:

https://tbpl.mozilla.org/?tree=Try&rev=82bc20256b6e
A green run!  I'm going to land this version on inbound, if no one objects by tomorrow.
Created attachment 693595 [details] [diff] [review]
Enable layout mochitests for b2g,

For the record, this is the patch I'm landing on inbound
https://hg.mozilla.org/integration/mozilla-inbound/rev/7c4c1bbf4d88
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
https://hg.mozilla.org/mozilla-central/rev/7c4c1bbf4d88
Assignee: nobody → martijn.martijn
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
https://hg.mozilla.org/releases/mozilla-aurora/rev/d421c0772331
https://hg.mozilla.org/releases/mozilla-b2g18/rev/b5633541a9ab
status-b2g18: --- → fixed
status-firefox19: --- → fixed
status-firefox20: --- → fixed
Whiteboard: [automation-needed-in-aurora][automation-needed-in-b2g18]
You need to log in before you can comment on or make changes to this bug.