1370421 - Bucketize metrics for build variations

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

I'm pretty sure bug 1362148 added redundant reporting for some build variations, such as the minimum build toolchains. We should look for build tasks reporting into the same Perfherder bucket and add metadata to the mozharness config to differentiate the build configuration.

Comment 1

[Gregory Szorc [:gps]]

Yeah, bug 1362148 definitely resulted in some redundancy. I'll clean this up.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1370421 - Put base-toolchains builds into own perfherder bucket;

Perfherder metrics are by default put in a bucket according to their
platform name. e.g. linux64/opt. Similar jobs emitting otherwise
identical metric names must differentiate the metrics by inserting
extra metadata.

Before this commit, we defined extra metadata in mozharness
sub-configs, as appropriate. As long as the sub-config remembers
to define the metadata, we're good.

However, some tasks don't use mozharness sub-configs. Instead, they
create variations in the taskgraph. The base toolchains builds
are an example. These share the same platform name (e.g. linux64/opt)
and the base mozharness config. Therefore there's no way to
distinguish these builds as being "base toolchains" from mozharness.

This commit teaches mozharness to look for an environment variable
defining extra perfherder keys to use to bucket results. We define
this variable in taskgraph for relevant tasks.

This commit stops short of implementing a linter in taskgraph. Ideally,
we would detect tasks that share the same base build mozconfig and
don't define PERFHERDER_EXTRA_OPTIONS. This would prevent writing
perfherder data to the same bucket. However, I'm not sure the best
way to implement this, as AFAICT taskgraph doesn't yet have a good
layer to define and inject linters. Perfect is the enemy of good:
this can be done in a follow-up.

Review commit: https://reviewboard.mozilla.org/r/146428/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146428/

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bug 1370421 - Define Perfherder bucket for asan-fuzzing;

linux64-asan-fuzzing/opt and linux64-asan/opt currently share the same
platform and mozharness config and therefore write to the same Perfherder
bucket.

This commit differentiates asan-fuzzing so it writes to its own bucket.

Review commit: https://reviewboard.mozilla.org/r/146436/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146436/

Comment 4

[Gregory Szorc [:gps]]

Attachment: Bug 1370421 - Differentiate Perfherder buckets for Linux64 ASAN builds;

Previously, these jobs were both writing to the same Perfherder
bucket keyed off linux64/asan. We introduce 2 new buckets, each with
the actual "opt" or "debug" configuration in their name.

Review commit: https://reviewboard.mozilla.org/r/146438/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146438/

Comment 5

[Gregory Szorc [:gps]]

Attachment: Bug 1370421 - Verify we don't write Perfherder data to same bucket;

It is really easy to copy and paste taskgraph or mozharness configs
and cause Perfherder data to be written to the same bucket, resulting
in non-useful metrics collection.

This commit adds a taskgraph transform for the "build" kind that
attempts to look for multiple build jobs writing to the same
Perfherder bucket.

It isn't perfect. But it has already flushed out some jobs writing
to the same bucket and therefore producing bimodal Perfherder data.

Review commit: https://reviewboard.mozilla.org/r/146440/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/146440/

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8875060 [details]
Bug 1370421 - Put base-toolchains builds into own perfherder bucket;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/146428/diff/1-2/

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8875060 [details]
Bug 1370421 - Put base-toolchains builds into own perfherder bucket;

https://reviewboard.mozilla.org/r/146428/#review150906

::: testing/mozharness/mozharness/mozilla/building/buildbase.py:2063
(Diff revision 2)
>          if build_metrics:
>              perfherder_data['suites'].append(build_metrics)
>          perfherder_data['suites'].extend(self._load_sccache_stats())
>  
>          # Ensure all extra options for this configuration are present.
>          for opt in self.config.get('perfherder_extra_options', []):

Can we move all of the perfherder_extra_options flags from mozharness configs into PERFHERDER_EXTRA_OPTIONS in Taskcluster in the future?

Comment 8

[Michael Shal [:mshal]]

Comment on attachment 8875075 [details]
Bug 1370421 - Define Perfherder bucket for asan-fuzzing;

https://reviewboard.mozilla.org/r/146436/#review150908

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8875076 [details]
Bug 1370421 - Differentiate Perfherder buckets for Linux64 ASAN builds;

https://reviewboard.mozilla.org/r/146438/#review150910

Comment 10

[Michael Shal [:mshal]]

Comment on attachment 8875077 [details]
Bug 1370421 - Verify we don't write Perfherder data to same bucket;

https://reviewboard.mozilla.org/r/146440/#review150912

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8875060 [details]
Bug 1370421 - Put base-toolchains builds into own perfherder bucket;

https://reviewboard.mozilla.org/r/146428/#review150906

> Can we move all of the perfherder_extra_options flags from mozharness configs into PERFHERDER_EXTRA_OPTIONS in Taskcluster in the future?

Possibly. I considered doing that. But I think there are some gotchas to that approach and I want to spend a bit of time looking at things before I commit to it. Either way, it's follow-up territory.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/378499da2030
Put base-toolchains builds into own perfherder bucket; r=mshal
https://hg.mozilla.org/integration/autoland/rev/d2289ec1f6d1
Define Perfherder bucket for asan-fuzzing; r=mshal
https://hg.mozilla.org/integration/autoland/rev/9a60cc7a38d1
Differentiate Perfherder buckets for Linux64 ASAN builds; r=mshal
https://hg.mozilla.org/integration/autoland/rev/0911969994df
Verify we don't write Perfherder data to same bucket; r=mshal

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/378499da2030
https://hg.mozilla.org/mozilla-central/rev/d2289ec1f6d1
https://hg.mozilla.org/mozilla-central/rev/9a60cc7a38d1
https://hg.mozilla.org/mozilla-central/rev/0911969994df