1630675 - Stop appending test names reported to Perfherder with -cold and add the pageload type to extra options instead

Status: RESOLVED FIXED

(Testing :: Performance, enhancement, P1)

Description

[Greg Mierzwinski [:sparky]]

In browsertime tests, we should start using much shorter test names than the raptor-webext variants. We are doing well here so far, but one thing we should do as soon as possible is changing how the test names get changed depending on the pageload type.

For instance, when we run a cold pageload test on amazon, the test name is transformed to amazon-cold, and the warm variant stays as amazon. Instead, we should be adding the cold and warm specifiers to the extra options and leaving the test name alone.

:bebe, do you know someone that could take on this work? I think we will have to coordinate it with some dashboard changes since the cold pageload test names will change.

Comment 1

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Sparky, mind if you link here as see also the bug(s) that are adding -cold to test names?

Comment 2

[Greg Mierzwinski [:sparky]]

:alexandrui, we add -cold to the test name in a couple places, we need to make sure that we only modify the browsertime tests and not the existing raptor-webext tests:
(1) https://dxr.mozilla.org/mozilla-central/source/testing/raptor/raptor/manifest.py#426
(2) https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/raptor.py#115

Comment 3

[Alexandru Ionescu (needinfo me) [:alexandrui]]

so far I am getting

Traceback (most recent call last):
  File "/home/alexandru.ionescu/workspace/mozilla-central/taskcluster/mach_commands.py", line 367, in show_taskgraph
    tg = getattr(tgg, graph_attr)
  File "/home/alexandru.ionescu/workspace/mozilla-central/taskcluster/taskgraph/generator.py", line 153, in full_task_graph
    return self._run_until('full_task_graph')
  File "/home/alexandru.ionescu/workspace/mozilla-central/taskcluster/taskgraph/generator.py", line 359, in _run_until
    k, v = next(self._run)
  File "/home/alexandru.ionescu/workspace/mozilla-central/taskcluster/taskgraph/generator.py", line 273, in _run
    raise Exception("duplicate tasks with label " + task.label)
Exception: duplicate tasks with label test-windows7-32/opt-browsertime-tp6-profiling-firefox-amazon-e10s

When we get -cold out of browsertime tasks, they become their warm versions in terms of naming. We have to have a unique naming for each config version of the test. How do we deal with that?

I am going to submit to phab the current patch and talk there about it.

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1630675 - Replace browsertime -cold test suffix with a cold extra option. r?#perftest,sparky

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

The changes are already made in testing/raptor/raptor/manifest.py. The only thing missing now is adding cold into the extra options.

Sparky, can I kindly ask you to take over this? Doesn't look like a time consuming task and I see that you have more context than me o checking the changes are proper.

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Nevermind, I'll do it. I added a comment here: https://phabricator.services.mozilla.com/D73035#2389697 Could you please respond?

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This isn't related to the task names, it's about the test names that are reported to Perfherder.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c9321b6bb35f
Replace browsertime -cold test suffix with a cold extra option. r=sparky,perftest-reviewers

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c9321b6bb35f