1506310 - Add support for optionally_keyed_by by-release-type to mozharness extra-options

Status: VERIFIED FIXED

Description

[Bob Clary [:bc:]]

After discussing the proposed patch in bug 1499352 attachment 9023986 [details] [diff] [review] with tomprince, we decided on the better approach would be to add the ability to optionally key the mozharness extra-options by release-type. It would support the ability to specify extra-options like:

raptor-tp6-1-firefox:
    description: "Raptor tp6-1 on Firefox"
    try-name: raptor-tp6-1-firefox
    treeherder-symbol: Rap(tp6-1)
    run-on-projects: built-projects
    max-run-time: 1800
    mozharness:
        extra-options:
            by-release-type:
                beta:
                    - --test=raptor-tp6-1
                    - --is--release-build
                release.*:
                    - --test=raptor-tp6-1
                    - --is--release-build
                esr.*:
                    - --test=raptor-tp6-1
                    - --is--release-build
                default:
                    - --test=raptor-tp6-1

Though, I wonder if projects would be an equivalent choice.

In the light of how verbose this would be and the fact it would be required on every test description I think I would just end up using 

if config.params['project'] in ('mozilla-beta', 'mozilla-release') or \
   config.params['project'].startswith('mozilla-esr'):
    extra_options.append('--is-release-build')

in https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#508

Feel free to wontfix this.

Comment 2

[Tom Prince [:tomprince]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cccda5cba8a853d94ed3f78349461010728ed4c4

Comment 3

[Tom Prince [:tomprince]]

Attachment: Bug 1506310: Remove unused branch_name passed from taskcluster to talos; r?bc

Comment 4

[Tom Prince [:tomprince]]

Attachment: Bug 1506310: Determine whether builds under test need signed extenions by release type; r?bc,ahal

Comment 5

[Intermittent Failures Robot]

154 failures in 585 pushes (0.263 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* try: 154

Platform breakdown:
* linux64: 32
* windows10-64: 18
* linux64-qr: 31
* windows10-64-qr: 18
* osx-10-10: 19
* windows10-64-ux: 18
* windows7-32: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1506310&startday=2018-11-16&endday=2018-11-16&tree=all

Comment 6

[Andreea Pavel [:apavel]]

Since all occurrences are only on try based on OF, removing disable recommended tag.

Comment 7

[Intermittent Failures Robot]

155 failures in 2917 pushes (0.053 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* try: 155

Platform breakdown:
* windows7-32: 18
* windows10-64-qr: 18
* linux64: 32
* osx-10-10: 20
* windows10-64: 18
* linux64-qr: 31
* windows10-64-ux: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1506310&startday=2018-11-12&endday=2018-11-18&tree=all

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a1fc429db01a
Remove unused branch_name passed from taskcluster to talos; r=jmaher
https://hg.mozilla.org/integration/autoland/rev/b1bec3572f17
Determine whether builds under test need signed extenions by release type; r=bc,ahal

Comment 9

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/a1fc429db01a
https://hg.mozilla.org/mozilla-central/rev/b1bec3572f17

Comment 10

[Eliza Balazs [:ebalazs_]]

This is fixed in today's beta simulations:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=9874af9fcaa8b20952dc65c4869a3ce1b0c0a01e
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=c301160f061b04ea32ff02956869ee2b6f242765

Comment 11

[Bob Clary [:bc:]]

Thanks for taking this on Tom! I appreciate your proactive work on this.

Comment 12

[Bob Clary [:bc:]]

Tom: It appears --is-release-build is being set even for mozilla-central builds:
<https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215084358&repo=mozilla-central&lineNumber=64-73>

Is there something different we need to do in order to only do it for beta, release or esr?

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f2833f7caddf
Fix typo in determining whether builds under test need signed extenions by release type; r=me

Comment 14

[Tom Prince [:tomprince]]

There was a typo in the code to evaluate `require-signed-extensions`.

Comment 15

[Andrei Ciure[:andrei_ciure]]

https://hg.mozilla.org/mozilla-central/rev/f2833f7caddf

Comment 16

[Bob Clary [:bc:]]

Hah! I don't know how many times I must of looked at that patch and *never* saw it. Looks good now. Thanks!

Comment 17

[Tom Prince [:tomprince]]

It took me grepping for the code that handled the 'by-release-type` and not finding it, to figure out that it was a typo. 😀

Comment 18

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/8c64f4130d86
https://hg.mozilla.org/releases/mozilla-esr60/rev/57c6bca5255e
https://hg.mozilla.org/releases/mozilla-esr60/rev/9bbffcaaefb9