Closed Bug 1507334 Opened 10 months ago Closed 8 months ago

Taskcluster implementation of three-tiered PGO builds

Categories

(Firefox Build System :: General, enhancement)

enhancement
Not set

Tracking

(firefox66 fixed)

RESOLVED FIXED
mozilla66
Tracking Status
firefox66 --- fixed

People

(Reporter: cmanchester, Assigned: mshal)

References

Details

Attachments

(4 files, 1 obsolete file)

This bug covers the taskgraph machinery we'll need to implement PGO builds as three separate tasks.

The first is a regular build with (at least) MOZ_PROFILE_GENERATE set. Its output is the instrumented build. The second is a test-like task that will run the instrumented build via profileserver.py and upload merged.profdata (the profile) as well as jarlog/en-US.log (I'm not entirely sure this is relevant but it seems to be a part of the current flow). The third is a build that consumes the profile (sets MOZ_PROFILE_USE).

These things should be relatively straightforward to invoke individually but we weren't sure of the details of how the tasks need to be set up. Mike volunteered to find appropriate guidance and look into this.
(In reply to Chris Manchester (:chmanchester) from comment #0)
> I'm not entirely sure this is relevant but it seems to be a part of the current flow

It is, but I though the file name was different. It's used to reorder the omni.ja files according to the order its contents are used.
Depends on: 1507338
Depends on: 1507344
Support for running profileserver.py multiple times was originally added
in bug 663251 and subsequently removed in bug 672563. Since then we've
been copying around the argument to profileserver.py from mozconfigs in
buildbot-configs, to mozconfigs in mozilla-central, to client.mk, and
ultimately to Makefile.in. It no longer does anything and can be
removed.

MozReview-Commit-ID: DsZkInTOyU

Bug 1507334 - Add mozconfigs for 3-stage PGO; r?firefox-build-system-reviewers

These two mozconfigs are used for the 1st and 3rd stages of PGO.
'profile-generate' is used for the 1st stage to generate a firefox
binary with instrumentation enabled. Since it is only used for getting
profile data, we don't run tests against it or need crashreporter
symbol.s

The 'profile-use' mozconfig is for the 3rd and final stage were we
produce the fully optimized build. The profiling information and jarlog
are input artifacts produced by the 2nd stage (running the
profileserver).

MozReview-Commit-ID: JGwSM1OqZFH

Bug 1507334 - Taskcluster configurations for 3-tier PGO; r?firefox-build-system-reviewers

Two new kinds are introduced - 'instrumented-build' and
'generate-profile'. The instrumented-build kind is almost identical to
the build kind, except it can be used earlier in the task graph. The
3-tier PGO process becomes:

instrumented-build -> generate-profile -> build

The final build stage is identical to any other build, except it has
the 'use-pgo' flag set to True in its task definition. This flag causes
the transforms to add the instrumented-build and generate-profile tasks
to the taskgraph.

MozReview-Commit-ID: 9M5YXjDo4sc

Bug 1507334 - Disable 3-tier Linux PGO until bug 1516114 is fixed; r?firefox-build-system-reviewers

With 3-tier PGO enabled, we seem to hit bug 1516114 more frequently than
we do with the standard all-in-one PGO. It probably makes sense for us
to wait until that bug is fixed before fully switching it over,
otherwise we will have to mess with retriggering parts of the taskgraph.

The bulk of the 3-tier PGO work has been to support macOS and Android,
so we can revisit Linux after those are completed.

MozReview-Commit-ID: BEFZA32UDhn
Support for running profileserver.py multiple times was originally added
in bug 663251 and subsequently removed in bug 672563. Since then we've
been copying around the argument to profileserver.py from mozconfigs in
buildbot-configs, to mozconfigs in mozilla-central, to client.mk, and
ultimately to Makefile.in. It no longer does anything and can be
removed.
These two mozconfigs are used for the 1st and 3rd stages of PGO.
'profile-generate' is used for the 1st stage to generate a firefox
binary with instrumentation enabled. Since it is only used for getting
profile data, we don't run tests against it or need crashreporter
symbol.s

The 'profile-use' mozconfig is for the 3rd and final stage were we
produce the fully optimized build. The profiling information and jarlog
are input artifacts produced by the 2nd stage (running the
profileserver).

Depends on D15748
Two new kinds are introduced - 'instrumented-build' and
'generate-profile'. The instrumented-build kind is almost identical to
the build kind, except it can be used earlier in the task graph. The
3-tier PGO process becomes:

instrumented-build -> generate-profile -> build

The final build stage is identical to any other build, except it has
the 'use-pgo' flag set to True in its task definition. This flag causes
the transforms to add the instrumented-build and generate-profile tasks
to the taskgraph.

Depends on D15749
With 3-tier PGO enabled, we seem to hit bug 1516114 more frequently than
we do with the standard all-in-one PGO. It probably makes sense for us
to wait until that bug is fixed before fully switching it over,
otherwise we will have to mess with retriggering parts of the taskgraph.

The bulk of the 3-tier PGO work has been to support macOS and Android,
so we can revisit Linux after those are completed.

Depends on D15750
Attachment #9034468 - Attachment is obsolete: true
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1f487879b4d7
Remove '10' argument to profileserver.py; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers
https://hg.mozilla.org/integration/autoland/rev/6f12b99d521c
Add mozconfigs for 3-stage PGO; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers
https://hg.mozilla.org/integration/autoland/rev/35396e93307e
Taskcluster configurations for 3-tier PGO; r=tomprince,firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers
https://hg.mozilla.org/integration/autoland/rev/6692901d43cf
Disable 3-tier Linux PGO until bug 1516114 is fixed; r=firefox-build-system-reviewers,chmanchester#firefox-build-system-reviewers
Regressions: 1553065
Blocks: 935637
You need to log in before you can comment on or make changes to this bug.