1832974 - "That's a lot of tests selected" should give actual number of tests selected, so that I can know by how much to reduce --rebuild

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P2)

Description

[Markus Stange [:mstange]]

I wanted to run ./mach try perf --rebuild 20 with "Pageload (essential) desktop firefox" and "Speedometer 3 desktop firefox" but I got the following message:

That's a lot of tests selected (300)!
These tests won't be triggered. If this was unexpected, please file a bug in Testing :: Performance.

I would like to know by how much I need to reduce the rebuild count in order to go below the limit.

However, this message always prints 300, rather than the actual number of tests selected:

https://searchfox.org/mozilla-central/rev/0accfbfb0381e431651b40065c0e08c249100fba/tools/tryselect/selectors/perf.py#1130-1136

if (len(selected_tasks) * rebuild) > MAX_PERF_TASKS:
    print(
        "That's a lot of tests selected (%s)!\n"
        "These tests won't be triggered. If this was unexpected, "
        "please file a bug in Testing :: Performance." % MAX_PERF_TASKS
    )
    return None

Can we make it print the number of tests selected and the max? And maybe also the number of rebuilds that would make it go below the limit?

Comment 1

[Andrej (:aglavic)]

Attachment: Bug 1832974 - Changing unclear output message when too many tests are selected. r?#perftest

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e19f10f18cc2
Changing unclear output message when too many tests are selected. r=perftest-reviewers,afinder

Comment 3

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/e19f10f18cc2