We currently have visual metrics running on browsertime but when we retrigger those browsertime tests, there are no new vismet tasks created for it, see here for a sample "add-new" task which retriggers a browsertime test, but doesn't add a new vismet task for it: https://firefox-ci-tc.services.mozilla.com/tasks/P6v8YHTOTzO6do85ER3RXg/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FP6v8YHTOTzO6do85ER3RXg%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log
It came from this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=5ba6228ac0ee383b783734c386631856a339f0f2&searchStr=add-new&selectedJob=280540364
:ahal, I've added you as a CC to this bug in case you have any thoughts/ideas about this issue. For context, the vismet tasks are currently created dynamically with this transform: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/visual_metrics_dep.py#21
A run-visual-metrics attribute dictates which tasks should have one created for them: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/raptor.yml#1023
This problem is worse than I originally thought. I can schedule multiple vismet tasks, but each of them only processed the results from the first btime task that was created for them. (We can't retrigger, and we can't schedule multiple runs). So for me to be able to analyze the vismet data, I would have to make one push per trial which is a bit much. Here's a task where I tried it with 50 retriggers: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=20528cc4f915039c970db9adacbb784491e2c01a
We have a partial solution here (thanks to :aki for the help):
This is the only method we have to retrigger these tasks. The --rebuild doesn't work but it would be cool if we could eventually get that to work with these tasks. Also note that the drop-down arrow on the push that provides the Custom Push Action option doesn't work for this either.
Custom Push Action
I've added that information to the wiki as well: https://wiki.mozilla.org/TestEngineering/Performance/Raptor/Browsertime
I don't know if this workflow works anymore because you can't do retriggers on jobs that succeed.
Last I checked (last week), it still works for the browsertime tests. Are you referring to raptor-browsertime or mozperftest?
1 failures in 3410 pushes (0.0 failures/push) were associated with this bug in the last 7 days.
For more details, see:
Using ./mach try fuzzy --retry N and getting stale data is a big footgun still.
./mach try fuzzy --retry N