Open Bug 1602893 Opened 1 year ago Updated 9 days ago

Can't run visual metrics on retriggered browsertime tasks

Categories

(Testing :: Raptor, defect, P2)

Version 3
defect

Tracking

(Not tracked)

People

(Reporter: sparky, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Whiteboard: [perf:workflow])

We currently have visual metrics running on browsertime but when we retrigger those browsertime tests, there are no new vismet tasks created for it, see here for a sample "add-new" task which retriggers a browsertime test, but doesn't add a new vismet task for it: https://firefox-ci-tc.services.mozilla.com/tasks/P6v8YHTOTzO6do85ER3RXg/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FP6v8YHTOTzO6do85ER3RXg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log

It came from this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=5ba6228ac0ee383b783734c386631856a339f0f2&searchStr=add-new&selectedJob=280540364

:ahal, I've added you as a CC to this bug in case you have any thoughts/ideas about this issue. For context, the vismet tasks are currently created dynamically with this transform: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/visual_metrics_dep.py#21

A run-visual-metrics attribute dictates which tasks should have one created for them: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#1023

This problem is worse than I originally thought. I can schedule multiple vismet tasks, but each of them only processed the results from the first btime task that was created for them. (We can't retrigger, and we can't schedule multiple runs). So for me to be able to analyze the vismet data, I would have to make one push per trial which is a bit much. Here's a task where I tried it with 50 retriggers: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=20528cc4f915039c970db9adacbb784491e2c01a

We have a partial solution here (thanks to :aki for the help):

  1. Select a browsertime test task (not a vismet task) that you want to retrigger.
  2. In the pop-up menu, select the ... and click on Custom Action.
  3. Pick the retrigger action if it wasn't already selected.
  4. Set downstream to true.
  5. Enter the number of times it should be retriggered in times.
  6. Retrigger!

This is the only method we have to retrigger these tasks. The --rebuild doesn't work but it would be cool if we could eventually get that to work with these tasks. Also note that the drop-down arrow on the push that provides the Custom Push Action option doesn't work for this either.

I've added that information to the wiki as well: https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Browsertime

I don't know if this workflow works anymore because you can't do retriggers on jobs that succeed.

Last I checked (last week), it still works for the browsertime tests. Are you referring to raptor-browsertime or mozperftest?

Using ./mach try fuzzy --retry N and getting stale data is a big footgun still.

Whiteboard: [perf:workflow]
Depends on: 1677559
You need to log in before you can comment on or make changes to this bug.