1776774 - Some tests that do not run on beta or release are run on esr

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Geoff Brown [:gbrown]]

Some tests have been disabled on beta and release -- perhaps to save time or $? -- but still run on esr. That can lead to unexpected test failures on esr.

An example: Most (all?) osx/debug tests do not run on beta or release but do run on esr-102.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Leaving the tests running on ESR may have been an intentional decision, but I don't remember for sure. Joel might because I believe he drove this effort originally.

Comment 2

[Geoff Brown [:gbrown]]

Comparing esr to release tests, the main differences seem to be:

• esr does not run android tests (well known / OK)
• esr does not run many browsertime that are run on release (probably OK?)
• esr runs many debug tests on linux/windows/mac that are not run on release

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

thanks for looking into this.

debug tests- intentional (this was part of a cost saving effort from 2020).

browsertime- last time I touched perf we only ran on autoland and m-c. Let me redirect to Dave.

Comment 4

[Geoff Brown [:gbrown]]

For debug tests, bug 1652881 says "We should leave ESR alone as that is a long lived branch and could find issues (although it hasn't historically)." but I think we should re-evaluate that in light of the on-going test failures on esr.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

I'd rather we figure out why we've got these leaks instead, as this could very well be a real quality issue at play.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #5)

I'd rather we figure out why we've got these leaks instead, as this could very well be a real quality issue at play.

Could we do both? I don't mind if someone investigates these leaks, but the strategy of "run some tests on esr but not on release or beta" seems problematic: We had esr-specific test failures in both esr91 and esr102, and confusion about how esr-specific failures are possible like https://bugzilla.mozilla.org/show_bug.cgi?id=1771785#c4.

I think we should either run debug tests on esr/release/beta, or not run debug tests on esr/release/beta.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

we could add debug back to beta- I don't see a point for release as we are really only focused on shippable. ESR has a long runtime and there shouldn't be issues when we first merge, but the chance 6 months later of merging changes into ESR that our extra tests running will find a regression is pretty high. If we miss something on release (or even beta) we have new code that is green coming from central every few weeks.

Comment 8

[Geoff Brown [:gbrown]]

Attachment: Bug 1776774 - Run debug tests on beta; r=

Add non-android debug builds/tests back to mozilla-beta, to catch
post-central leaks and debug-only test failures before reaching esr.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/281ce521c6af
Run debug tests on beta; r=jmaher

Comment 10

[bszekely]

https://hg.mozilla.org/mozilla-central/rev/281ce521c6af

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #3)

browsertime- last time I touched perf we only ran on autoland and m-c. Let me redirect to Dave.

We recently started running browsertime tests on release (see bug 1753658). The documentation at https://firefox-source-docs.mozilla.org/testing/perfdocs/raptor.html shows where each Raptor (browsertime) test is running, however it doesn't include ESR. We don't have alerts enabled for ESR and I'm not aware of anyone monitoring performance there, so I suggest we stop running performance tests against this channel. What do you think sparky?

Comment 12

[Greg Mierzwinski [:sparky]]

I agree, we should stop running the perf tests on ESR.

Comment 13

[Geoff Brown [:gbrown]]

Comment on attachment 9283465 [details]
Bug 1776774 - Run debug tests on beta; r=

Beta/Release Uplift Approval Request

• User impact if declined: Extra tests will not be run until next merge.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Very low risk to product: Only affects taskcluster. Only adds debug desktop tests to mozilla-beta.
  However, will likely result in new test failures on mozilla-beta.
• String changes made/needed: none
• Is Android affected?: No

Comment 14

[Geoff Brown [:gbrown]]

(In reply to Greg Mierzwinski [:sparky] from comment #12)

I agree, we should stop running the perf tests on ESR.

Sorry, I think my original message was lost in the discussion:

esr does not run many browsertime that are run on release (probably OK?)

That is, we are not currently running browsertime on esr and I wanted to check that was OK. It sounds like it is intentional: no need for changes.

Thanks!

Comment 15

[Donal Meehan [:dmeehan]]

Will take a look at uplifting this to beta next week. Would not like to add new failures before the long weekend.

Comment 16

[Donal Meehan [:dmeehan]]

Comment on attachment 9283465 [details]
Bug 1776774 - Run debug tests on beta; r=

Approved for 103.0b6, thanks.

Comment 17

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/318b8c70ed5c