Open Bug 1470851 Opened Last year Updated 5 months ago

Disable build_metrics tests from devedition builds

Categories

(Tree Management :: Perfherder, enhancement, P3, minor)

enhancement

Tracking

(Not tracked)

People

(Reporter: igoldan, Unassigned)

Details

Attachments

(3 obsolete files)

So we explicitly enabled build metrics for all builds in bug 1362148. I understand that tracking regressions on all build types isn't important, but I don't think we should be putting this functionality back in piecemeal.
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #2)
> So we explicitly enabled build metrics for all builds in bug 1362148. I
> understand that tracking regressions on all build types isn't important, but
> I don't think we should be putting this functionality back in piecemeal.

We have plans for enabling new perf tests and before that, we must turn off those that are of no use. We concluded build_metrics on devedition builds are just duplicated results that consumes us time.
I'd like gps' opinion on this as the build module owner since he made the change to enable metrics in all builds.
Flags: needinfo?(gps)
I think it is useful to have build metrics in as many places as possible.

Unfortunately, having the build metrics [in Perfherder] means that any regressions create Perfherder alerts. That creates a burden on performance sheriffing.

I'd like to think that the compromise position here is us recording the build metrics [in Perfherder], but either not generating alerts for them when we deem them not appropriate (a technical solution) or "ignoring" the alerts when they do arise (a people solution).

The former is more robust, as it requires us to annotate which alerts aren't "relevant." The latter requires people to remember which alerts are "relevant" and we'll inevitably have incorrect triaging due to human error and relevant alerts won't be seen by the appropriate build system people.

Fortunately, Perfherder supports suppressing alerts. Search for "shouldAlert" in the repo. e.g. https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/testing/mozharness/mozharness/mozilla/building/buildbase.py#1482.

I think we should devise a way to define "shouldAlert" for build metrics on a per-task or per-repo/project/build configuration basis. That feels like the best solution here.
Flags: needinfo?(gps)
Priority: -- → P3
Assignee: igoldan → nobody
Attachment #9003767 - Attachment is obsolete: true
Attachment #9003767 - Flags: review?(ted)
Attachment #9013531 - Attachment is obsolete: true
Attachment #9013531 - Flags: review?(ted)
Attachment #9013530 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.