959709 - Add web-platform-tests to buildbot config

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[James Graham [:jgraham]]

In bug 945222 I am working on a test runner for the web-platform-tests. An early version of this along with a mozharness script has now landed on ash, but I presume to try it out I need to add it to the buildbot configs (and in any case I need to in order to land the change).

AIUI I need to look in http://hg.mozilla.org/build/buildbot-configs/mozilla-tests/config.py. It seems straightforward to add it to the list of testsuites and the list of jobs for each slave type (at the moment only for the linux ones), but I'm not sure if anything more is needed e.g. something to limit it to ash.

Comment 1

[Jonathan Griffin (:jgriffin)]

Yes, that's about right.  You should enable the tests on cedar, rather than ash, if you're using a new mozharness script; cedar is used for greening up test suites; ash for testing out mozharness changes that could impact production if they're pushed live, but that wouldn't be the case if you're creating a brand new mozharness script.

You can add them to a specific tree using something like the block here:

http://hg.mozilla.org/build/buildbot-configs/file/f9df7bca20c3/mozilla-tests/config.py#l1613

Comment 2

[James Graham [:jgraham]]

So if I land this on cedar, I have to land the untested (on infrastructure) mozharness script in production?

Comment 3

[Jonathan Griffin (:jgriffin)]

Yes, but a new mozharness script that isn't being used by anything else won't cause any problems except on cedar, if something goes wrong.

If you're modifying mozharness core code, though, we should test it on ash first.

Comment 4

[James Graham [:jgraham]]

Attachment: Add web-platform-tests to cedar

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 8363117 [details] [diff] [review]
Add web-platform-tests to cedar

I would have preferred an explicit set like this:
http://hg.mozilla.org/build/buildbot-configs/file/7c742fd377c3/mozilla-tests/config.py#l1558

but it looks like cedar is special cased all over already.

Comment 6

[Aki Sasaki (not active)]

(This will add these tests to macosx64, linux, linux64, and linux64-asan.)

Comment 7

[James Graham [:jgraham]]

Could be too many platform variants for initial testing I guess. jgriffin?

Comment 8

[Jonathan Griffin (:jgriffin)]

I think since we're just running them on cedar, the platform list is fine.

But, I believe we'll also need code similar to http://hg.mozilla.org/build/buildbot-configs/file/7c742fd377c3/mozilla-tests/config.py#l744 where we specify which config files to use.

Additionally, we don't want to schedule these tests on both Fedora and Ubuntu for linux, so we should add some code to specifically exclude Fedora slave_platform variants.

Comment 9

[Aki Sasaki (not active)]

Oh, yeah, that's right, sorry.

Comment 10

[James Graham [:jgraham]]

Attachment: Add web-platform-tests to buildbot config for cedar

Comment 11

[Jordan Lund (:jlund)]

bbot-cfg patch is in production! :)

Comment 12

[Jonathan Griffin (:jgriffin)]

These are running in cedar; I think we can open new bugs for scheduling them elsewhere when needed.