Run Puppeteer unit tests for all the changes on trunk
Categories
(Remote Protocol :: Agent, task, P1)
Tracking
(firefox83 fixed)
| Tracking | Status | |
|---|---|---|
| firefox83 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Whiteboard: [puppeteer-beta2-mvp])
Attachments
(2 files)
With all the changes related to bug 1607474 we now have a green Puppeteer job and can track regressions or improvements in our code base. Like for bug 1628578 we seem to had a patch which fixed a platform issue, and caused a test to permanently pass now.
We saw this by accident and it would be good to detect that earlier. As such I would like to see the jobs running regularly again for each and every commit. Also note that the duration of the job has been dropped from 30min to 15min, and we are working towards making it even faster.
Joel, I assume nothing else blocks us from re-enabling the test job again?
|
||
Comment 1•4 years ago
|
||
does this run on all platforms, I only see evidence of linux on m-c? could it run on shippable builds, not opt for autoland? I assume SETA will pick it up and reduce the frequency, but it will be sheriffed and need to be green prior to merging.
|
Assignee | |
Comment 2•4 years ago
|
||
I thought that these jobs were always running based on shippable builds. At least that was when the tests were running before the patch on bug 1610611 landed. Whereby that also only lists opt builds. Hm. I will have to check.
We would love to run it across platforms if you don't have any objections. So far CI covers Linux only, and I do the runs locally on Mac. For Windows it's totally untested.
For the landing it will indeed have to wait until bug 1628578 has been fixed.
|
|
Assignee | |
Comment 3•4 years ago
|
||
Oh, and as it looks like for shippable builds we have to run the job on Ubuntu 18.04. That currently doesn't work due to bug 1623128. So this is a hard requirement?
|
|
Assignee | |
Comment 4•4 years ago
|
||
Hm, whereby pup depends on linux64/opt, which should be shippable, or?
https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ci/source-test/remote.yml#l9
https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ci/source-test/kind.yml#l45
|
|
Assignee | |
Comment 5•4 years ago
|
||
Andrew, I assume that source tree kind of tests can only be run on Linux? Does it mean we would have to move it to its own test task? Or should it be a toolchain task? It would clearly depend on the Firefox build.
|
|
||
Comment 6•4 years ago
|
||
what is the timeline for fixing the tests to run on 1804? shippable isn't a hard requirement, but it reduces complexity of the matrix we run- ideally if we have shippable for a platform we only run tests on shippable for autoland.
I don't have objections to osx/windows- as long as it is stable enough to be tier-1
|
||
Comment 7•4 years ago
|
||
No, source-test can run on Windows/Mac as well and they can optionally depend on builds. However I don't believe there are any non-Linux source-test tasks using builds at the moment, so there may be rough edges that need fixing.
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 9•4 years ago
|
||
Dependencies are fixed so lets get it running for shippable builds on autoland, and on central.
|
|
Assignee | |
Comment 10•4 years ago
|
||
So I actually have a problem here. I cannot figure out how to run the Puppeteer jobs via shippable builds for trunk, but just opt build on try. Maybe the latter could be optional for try even, but would help us a lot with development when not having to wait that long on CI results.
Andrew, is that possible without any transforms and such? Here is our source test definition:
https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/remote.yml
|
|
||
Comment 11•4 years ago
|
||
Good idea!
I think you'll need to set up both pgo and opt builds in the require-build key, and then use a transform to set the platform key based on the current project. Alternatively you might be able to key the platform key by-project.. though I can't think of any other places we key by project, so you may need to set that up in the schema.
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 12•4 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #11)
I think you'll need to set up both
pgoandoptbuilds in therequire-buildkey, and then use a transform to set theplatformkey based on the current project. Alternatively you might be able to key theplatformkeyby-project.. though I can't think of any other places we key by project, so you may need to set that up in the schema.
I finally got back to this bug, and I checked how other source tests make use of different builds on try vs trunk. So I have seen a use-case in python.yml for Mochitest harness tests:
When I make use of this logic I get the opt builds when running the mach try fuzzy command. Not sure how to actually test it for mozilla-central.
Here a try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6d1d26e1566804592401a1a1f458d742e119eecb
Andrew, does that look ok?
|
|
Assignee | |
Comment 13•4 years ago
|
||
Instead of running the tests only for changes under /remote
they need to be executed for each changeset on trunk.
|
|
Assignee | |
Comment 14•4 years ago
|
||
Lets move the needinfo to the actual review in Phabricator.
|
||
Comment 15•4 years ago
|
||
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/2f38e26a143b [remote] Enable puppeteer job for all builds on trunk. r=ahal
|
|
Assignee | |
Comment 16•4 years ago
|
||
|
|
||
Comment 17•4 years ago
|
||
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b3d6a68c4df8 [remote] Don't run accidentally enabled Puppeteer Fission jobs on Trunk. r=ahal
|
||
Comment 18•4 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/2f38e26a143b
https://hg.mozilla.org/mozilla-central/rev/b3d6a68c4df8
Description
•