461012 - Try-talos missed perf regression in patch, which mozilla-central-talos did catch

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

In bug#449443, sdwilsh submitted a patch to try-talos, and the results looked fine. However, when he landed the same patch on mozilla-central, the mozilla-central-talos results showed a perf regression. 

Its hard to recreate this problem, but what can we do on try-talos to catch this kinda problem in future? 

Filing this bug to track this, as irc/email is too easy to lose...

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

When the try-talos run was done, what were the results compared to ?

Comment 2

[Shawn Wilsher :sdwilsh]

I had done a clean build of m-c at the time, and I also looked at past runs with patches that didn't affect perf.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From triage today, there are also some other issues to look at:

- no comparison numbers per run / no baseline numbers
- skips queued builds, which is being tracked in bug#457885
- long time since last reboot of slaves, which also causes variance
- only have one slave per platform, so cant cross-compare trio-of-results as in m-c talos.

Pushing to p3/future.

Comment 4

[alice nodelman [:alice] [:anode]]

try talos:

- no longer skips queued builds
- reboots after every test run
- has multiple slaves per platform and runs each test twice

Going to call this fixed from the list in comment #3.

Comment 5

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.