815416 - Excludetest list of all layout test failures in layout for b2g.json

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Martijn Wargers (dead)]

Attachment: b2g.json

I've attached a b2g.json file that made a succesful mochitest run inside the layout directory on the b2g emulator.

There might still be intermittent failures lurking around, since I only reran the whole mochtitest run in the layout directory, once.

Comment 1

[Martijn Wargers (dead)]

Attachment: Include list of all layout tests that are passing

This is an include list of all layout tests that are passing on the b2g emulator.
337 files and 32702 mochitests running in total.

Comment 2

[Jonathan Griffin (:jgriffin)]

Thanks Martijn. I've pushed this to try to see how it fares:

https://tbpl.mozilla.org/?tree=Try&rev=d8efad2677c3

Before we can land this, we'll need to increase the number of mochitest chunks used by B2G (I'll file a bug for this).   Additionally, we should modify this to do the inverse of what it's doing, I think...that is, exclude all the tests that fail, rather than include the tests that pass.  The reason for this is so that so new tests added automatically get run for B2G and don't require changes to b2g.json.

Comment 3

[Jonathan Griffin (:jgriffin)]

The try results are in:

chunk 1:  28 fails
chunk 2:  a timeout then a crash!
chunk 3:  some test timeouts followed by a buildbot timeout

Martijn, if you can add the 28 fails from chunk 1 to this list, we can at least get that running in TBPL.  It may take some more runs to excise all the timeout-related failures.

Comment 4

[Jonathan Griffin (:jgriffin)]

The last try run tried to run *all* the tests, due to a syntax error in b2g.json.  I've fixed that and pushed again: https://tbpl.mozilla.org/?tree=Try&rev=e5d6f1a973cc

Comment 5

[Martijn Wargers (dead)]

(In reply to Jonathan Griffin (:jgriffin) from comment #4)
> The last try run tried to run *all* the tests, due to a syntax error in
> b2g.json.  I've fixed that and pushed again:
> https://tbpl.mozilla.org/?tree=Try&rev=e5d6f1a973cc

This mochitest run looks green to me.
I see that you used an include list, instead of an exclude list. I thought you preferred the exclude list?

Comment 6

[Jonathan Griffin (:jgriffin)]

Awesome!  Yes, I just used your patch; it would take some additional work to make an exclude list based on this include list.

Comment 7

[Martijn Wargers (dead)]

Well, the first thing I posted here in this bug is an exclude list.
It should basically already be the opposite of this include list.

Comment 8

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

We should favor "include" lists for now, IMO. Reason is that I don't have buy-in that new tests added need to be B2G compliant (or added to an exclude). So we can't just blanket-include a directory or any given checkin might fail it.

As soon as I hear from overholt that they're willing to make tests B2G-compliant for r+, we can (and should) flip that so tests are opt-out instead of opt-in.

Comment 9

[Jonathan Griffin (:jgriffin)]

I'd still be in favor of using exclude lists.  If someone adds a test that isn't B2G-compatible, it will fail a try/inbound run, and they'll either be forced to fix it or add it as an exclude.

I think that's better than having people write tests and assume it will be run on B2G, when in fact that wouldn't actually be the case.

Comment 10

[(not currently active) Ted Mielczarek]

To be more clear: most developers do not know about our mobile-specific whitelists.

Comment 11

[Jonathan Griffin (:jgriffin)]

Pushed the first patch (the blacklist one) to try:
https://tbpl.mozilla.org/?tree=Try&rev=1bf8acc6ae99

Comment 12

[Jonathan Griffin (:jgriffin)]

Almost green!

Updated list and pushed to try again:

https://tbpl.mozilla.org/?tree=Try&rev=82bc20256b6e

Comment 13

[Jonathan Griffin (:jgriffin)]

A green run!  I'm going to land this version on inbound, if no one objects by tomorrow.

Comment 14

[Jonathan Griffin (:jgriffin)]

Attachment: Enable layout mochitests for b2g,

For the record, this is the patch I'm landing on inbound

Comment 15

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7c4c1bbf4d88

Comment 16

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/7c4c1bbf4d88

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/d421c0772331
https://hg.mozilla.org/releases/mozilla-b2g18/rev/b5633541a9ab