1470851 - Disable build_metrics tests from devedition builds

Status: RESOLVED INCOMPLETE

(Tree Management :: Perfherder, task, P3)

Description

[Ionuț Goldan [:igoldan]]

We shouldn't be tracking tests like these: https://treeherder.mozilla.org/perf.html#/alerts?id=13980&filter=devedit

Comment 1

[Ionuț Goldan [:igoldan]]

Attachment: Bug 1470851 - Disable build_metrics for devedition builds

Comment 2

[(not currently active) Ted Mielczarek]

So we explicitly enabled build metrics for all builds in bug 1362148. I understand that tracking regressions on all build types isn't important, but I don't think we should be putting this functionality back in piecemeal.

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #2)
> So we explicitly enabled build metrics for all builds in bug 1362148. I
> understand that tracking regressions on all build types isn't important, but
> I don't think we should be putting this functionality back in piecemeal.

We have plans for enabling new perf tests and before that, we must turn off those that are of no use. We concluded build_metrics on devedition builds are just duplicated results that consumes us time.

Comment 4

[(not currently active) Ted Mielczarek]

I'd like gps' opinion on this as the build module owner since he made the change to enable metrics in all builds.

Comment 5

[Ionuț Goldan [:igoldan]]

Attachment: Bug 1470851 - Disable build_metrics for devedition builds

Comment 6

[Ionuț Goldan [:igoldan]]

Attachment: Bug 1470851 - Disable build_metrics for devedition builds

Comment 7

[Gregory Szorc [:gps]]

I think it is useful to have build metrics in as many places as possible.

Unfortunately, having the build metrics [in Perfherder] means that any regressions create Perfherder alerts. That creates a burden on performance sheriffing.

I'd like to think that the compromise position here is us recording the build metrics [in Perfherder], but either not generating alerts for them when we deem them not appropriate (a technical solution) or "ignoring" the alerts when they do arise (a people solution).

The former is more robust, as it requires us to annotate which alerts aren't "relevant." The latter requires people to remember which alerts are "relevant" and we'll inevitably have incorrect triaging due to human error and relevant alerts won't be seen by the appropriate build system people.

Fortunately, Perfherder supports suppressing alerts. Search for "shouldAlert" in the repo. e.g. https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/testing/mozharness/mozharness/mozilla/building/buildbase.py#1482.

I think we should devise a way to define "shouldAlert" for build metrics on a per-task or per-repo/project/build configuration basis. That feels like the best solution here.

Comment 8

[Ionuț Goldan [:igoldan]]

Much older than 18 months, thus closing it as INCOMPLETE.