1554493 - Fission tests shouldn't run with try syntax

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Razvan Maries]

This is the push on central where the fission test were triggered by jya here: https://tools.taskcluster.net/groups/aMKGIcntSPK_4LeN8cZjlg/tasks/MlJE47HXRjO55rPKdXWYtA/details

TH push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=8388f1163e54676201327b95dcb1da7b2f920b39&searchStr=w-fis

It seems that the fission tests should not be enabled nor run by default.

Based on https://bugzilla.mozilla.org/show_bug.cgi?id=1553527#c9 they should only run on try.

Comment 1

[Andreea Pavel [:apavel]]

From IRC:

nika> apavel|sheriffduty: hey - if you're seeing Fission tests they're not supposed to be visible or on by default
5:36 PM They're super orange
5:36 PM (or red(
<nika> There may have been a mistake adding them which makes them visible.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/893e1d6cad95
Don't run fission tests on Try because they are permafailing. a=cleaner-try-pushes

Comment 3

[Andrew Halberstadt [:ahal]]

I'm very confused at how those tasks appeared on that central push (and equally confused why removing the "try" project seems to have fixed it). If I run mach taskgraph target locally (which uses mozilla-central's parameters by default), I don't see fission tasks listed. This means that "fission" isn't targeted for central (as expected) and shouldn't show up on any central pushes.

Adding them to try was intentional since the only way to select them would have been to use mach try fuzzy --full or mach try chooser --full. In theory, this shouldn't have any impact on whether or not they run on central, but clearly something is afoot.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

As far as I understood, they got manually added on central (triggered by jya in comment 0) to check if the failures on jya's Try push were from his push or also occur on central.

The issue from my POV is that Try pushes with -u all or a try syntax requesting wpt jobs will yield permafailing W-fis jobs.

Comment 5

[Jean-Yves Avenard [:jya]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #4)

As far as I understood, they got manually added on central (triggered by jya in comment 0) to check if the failures on jya's Try push were from his push or also occur on central.

Yes, I confirm.

Following the massive presence of orange, and having failed to realise those were fission tests I ran the same test on central to determine if my changes were the cause or if it was already present on central and missed.

The issue from my POV is that Try pushes with -u all or a try syntax requesting wpt jobs will yield permafailing W-fis jobs.

indeed.

While their are known to fail, they should only be enabled if explicitly asked for it.

Comment 6

[Andrew Halberstadt [:ahal]]

Ok, this makes sense.

Fwiw they do only show up when explicitly requested when using one of the more modern try selectors (fuzzy, chooser, etc). But try syntax doesn't have this ability. The proper solution is to fix bug 1400295 and bring mach try syntax in-line with the other selectors. But I don't have time to work on that at the moment, so in the meantime I can add "yet another exception" to the try syntax parser. This means it won't be possible to schedule fission tasks with try syntax, but that's probably fine.

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1554493 - [ci] Improve some formatting in try_option_syntax.py, r?jmaher

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1554493 - [ci] Don't run 'fission' tests with try syntax, r?jmaher

Depends on D32700

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1554493 - [ci] Re-enable fission tests on try, r?jmaher

Depends on D32701

Comment 10

[Andrew Halberstadt [:ahal]]

Try run with no -fis tasks:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0f1bdda7358d4cc52d65155864b874be1c0b5601

Comment 11

[Tom Prince [:tomprince]]

(In reply to Andrew Halberstadt [:ahal] from comment #3)

Adding them to try was intentional since the only way to select them would have been to use mach try fuzzy --full or mach try chooser --full.

Note that doing this doesn't currently affect what is selectable via mach try fuzzy, as that uses a mozilla-central graph. It would affect try syntax, but if it is going to be filtered out there anyway, it won't have an effect

---

One thing I would have thought would caused these tests to not run via syntax is this logic to skip tests that are not tier-1 if they are other variants that are, but given that -fis is added to the name, all the tests are going to be tier 3.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5925360fcc6c
[ci] Improve some formatting in try_option_syntax.py, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/c584c415ca35
[ci] Don't run 'fission' tests with try syntax, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/4caaa2fcdf35
[ci] Re-enable fission tests on try, r=jmaher

Comment 13

[Mats Palmgren (inactive)]

FYI, "-u all" still shows these broken tests:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b9e2e628cf8e0eb9fa38a001b43d9bc1430809f
It's rather confusing when a whitespace change shows up as orange across the board...

Comment 14

[Andrew Halberstadt [:ahal]]

The fix hasn't merged to central yet.

Comment 15

[Mats Palmgren (inactive)]

Ah, I misread "integration" in the push links above as "inbound".
OK, I'll try again later after it's merged...

Comment 16

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/5925360fcc6c
https://hg.mozilla.org/mozilla-central/rev/c584c415ca35
https://hg.mozilla.org/mozilla-central/rev/4caaa2fcdf35

Comment 17

[Andrew Halberstadt [:ahal]]

Not sure why leave-open was set, but I don't think there's anything else to do here.

Comment 18

[Chris Peterson [:cpeterson]]

Retroactively moving fixed bugs whose summaries mention "Fission" (or other Fission-related keywords) but are not assigned to a Fission Milestone to an appropriate Fission Milestone.

This will generate a lot of bugmail, so you can filter your bugmail for the following UUID and delete them en masse:

0ee3c76a-bc79-4eb2-8d12-05dc0b68e732