1701674 - Bump Puppeteer (non-Fission) tests to Tier 1

Status: RESOLVED FIXED

(Remote Protocol :: Agent, task, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

We now ship Puppeteer support for Firefox beta and release. Also the tests have been proven as stable. As such we would propose to bump the tier level to 1.

Note that because of bug 1701168 some tests are currently busted.

Sebastian, I assume that we should best wait until that regression has been fixed?

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Yes, please switch to Tier 1 when there are no permanent and frequent failures. Please set pup-fis as tier 2 because it doesn't run by default.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

Yes, the Fission tests will be kept as tier2.

Comment 3

[Chris Peterson [:cpeterson]]

Adding [not-a-fission-bug] whiteboard tag because this bug is about non-Fission tests.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

Attachment: Bug 1701674 - Bump non-Fission pup test jobs to Tier 1.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

Sebastian, I have seen that we don't have documentation about how to disable tests or mark them expected fail. I assume that this would be needed for Tier1? If yes, is there a central place where that information is usually stored or referenced from?

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

In tree similar to web-platform-tests would be suitable.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

Attachment: Bug 1701674 - [remote] Add handling of expectation data to readme.

Depends on D112816

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/22ab8a1a6946
Bump non-Fission pup test jobs to Tier 1. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/24d461ee009d
[remote] Add handling of expectation data to readme. r=remote-protocol-reviewers,jdescottes

Comment 9

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/22ab8a1a6946
https://hg.mozilla.org/mozilla-central/rev/24d461ee009d