Closed Bug 1352113 Opened 8 years ago Closed 6 years ago

implement shippable builds

Categories

(Release Engineering :: General, enhancement, P3)

enhancement

Tracking

(firefox68 fixed)

RESOLVED FIXED
Tracking Status
firefox68 --- fixed

People

(Reporter: mozilla, Assigned: Callek)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Whiteboard: [releng:q12019])

Attachments

(42 files, 1 obsolete file)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
Shippable builds will most likely require taskcluster nightlies... so we *could* proceed with linux and android at this point, or wait until osx and windows are on TC. This is a writeup of https://github.com/mozilla/build-relengdocs/blob/master/future/index.rst#shippable-builds and https://groups.google.com/forum/#!topic/mozilla.release.engineering/MRfHnhaVI48 . Let's: - rename 'nightly' builds to 'shippable' builds - get rid of 'pgo' builds - have a second promotion graph to ship shippable builds - (possibly rename 'depend' to 'ci', but that doesn't have full agreement.) promotion graph =============== - move the actual shipping tasks (beetmover, balrog, pushapk) to a second promotion graph. These should depend on the build or signing tasks in the build graph as their upstreamArtifacts. - the decision task for the second promotion graph needs to be able find the latest successful shippable graph, possibly through dummy task indexes, on mozilla-central and mozilla-aurora. On mozilla-beta, mozilla-release, and mozilla-esr, we can specify the shippable graph/tasks to use. scheduling ========== - shippable builds should periodically on integration branches, with backfill capability. - every 4 hours? Ideally let's not automatically re-run a shippable build on revision X if one has already successfully been triggered. - both depend and shippable builds should be choosable using try syntax. - on central/aurora/beta/release/esr*, let's build shippable builds on push and drop depend opt builds.
:jmaher, periodic shippable builds on integration branches may be problematic if talos still needs to run in chronological order. I know that was the case in the graph.m.o days; is that still the case with perfherder?
Flags: needinfo?(jmaher)
perfherder stores data by the revision, not by the date the data was generated, so if I understand the definition of chronological order in reference to the question, then this will not be problematic :)
Flags: needinfo?(jmaher)
Priority: -- → P3
Depends on: 1416538
Component: General Automation → General
Whiteboard: [releng:q32018]
Blocks: 1501898
Whiteboard: [releng:q32018] → [releng:q12019]
Assignee: nobody → bugspam.Callek

This commit adds a new build for OSX since there is no current PGO build type for OSX.
And calls it shippable, this mirrors that of the nightly.

This was needed since when we have job-defaults and later on a test set for
run-on-projects: by-test-platform: ...
We were ending up with both the list of by-* being extended but also
any array in that list also being extended (like default: ['a', 'b']
was getting extended to also have the new values for default)

This is not only usually wrong but very very likely not what the author wanted.

When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.

This sets all of the shippable tests to not run in the places where they would otherwise.
This patch will be effectively undone later in the patchset.

Without this the shippable builds take >20 seconds to compute the prune through new_as_old_is_high_value, locally. With this it is near instant.

Attachment #9049521 - Attachment description: Bug 1352113 - Shippable Builds - Do not run merge on by-* values. r=aki → Bug 1352113 - Shippable Builds - Do not run merge on by-* values. r=tomprince

This does not change the substance of what is passed down to transforms.

raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.

TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.

Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.

This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.

I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.

We are adding shippable-qr to beta because of replacing nightly too.

Autoland and inbound should have the same sets of tasks.

beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.

central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable

This is useful in order to not have to run linux64/opt on push, especially on autoland/inbound
when we need a source test. It is also required if we remove the linux64-pgo build type entirely.

Attachment #9050513 - Attachment description: Bug 1352113 - Shippable Builds - Do not do linux32/opt build on integration. → Bug 1352113 - Shippable Builds - Do not do linux32/opt build on integration. r=aki

Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.

Attachment #9050286 - Attachment description: Bug 1352113 - Shippable Builds - Remove windows pgo entirely. → Bug 1352113 - Shippable Builds - Remove windows pgo entirely. r=aki,jmaher
Attachment #9050287 - Attachment description: Bug 1352113 - Shippable Builds - Do not do linux pgo either. → Bug 1352113 - Shippable Builds - Do not do linux pgo either. r=aki,jmaher
Attachment #9050289 - Attachment description: Bug 1352113 - Shippable Builds - Don't run OSX opt builds by default on integration trees. → Bug 1352113 - Shippable Builds - Don't run OSX opt builds by default on integration trees. r=aki

Jmaher indicated we do not have the test capacity to incur this as a duplicated set

Attachment #9050291 - Attachment description: Bug 1352113 - Shippable Builds - make only-for-attributes accept any() instead of all() in the set. → Bug 1352113 - Shippable Builds - make only-for-attributes accept any() instead of all() in the set. r=aki
Attachment #9050290 - Attachment description: Bug 1352113 - Shippable Builds - Do not run opt-only OSX tests on integration trees. → Bug 1352113 - Shippable Builds - Do not run opt-only OSX tests on integration trees. r=jmaher
Keywords: leave-open
Pushed by jwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/0fe5ffc10ff8 Shippable Builds - Do not run merge on by-* values. r=tomprince https://hg.mozilla.org/integration/autoland/rev/e7eeb55c71ba Shippable Builds - Add index builder, but don't do any index's yet. r=aki https://hg.mozilla.org/integration/autoland/rev/1269ffd8a411 Shippable Builds - OSX Build. r=aki https://hg.mozilla.org/integration/autoland/rev/37f63cecb702 Shippable Builds - Add windows32 shippable. r=aki https://hg.mozilla.org/integration/autoland/rev/333ad0658d94 Shippable Builds - Add windows64 shippable. r=aki https://hg.mozilla.org/integration/autoland/rev/f59eb3b8d0b3 Shippable Builds - Add windows64 aarch64 shippable. r=aki https://hg.mozilla.org/integration/autoland/rev/b764eae9ff5a Shippable Builds - Add Linux32 shippable builds. r=aki https://hg.mozilla.org/integration/autoland/rev/4a98810cec0a Shippable Builds - Add linux64 shippable builds. r=aki https://hg.mozilla.org/integration/autoland/rev/b81435806374 Shippable Builds - Make sure we sign shippable builds. r=aki https://hg.mozilla.org/integration/autoland/rev/5de8fab632bc Shippable Builds - Add test run-on-projects to avoid scheduling shippable just yet. r=jmaher https://hg.mozilla.org/integration/autoland/rev/1fae9a0c3d4f Shippable Builds - OSX tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/6dfa69851019 Shippable Builds - Linux32 Tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/8f22b80c544e Shippable Builds - Linux64 Tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/dc7a3140939e Shippable Builds - Win7 Tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/3e1475bc7fd5 Shippable Builds - Win10 Tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/ea10d6f34744 Shippable Builds - add shippable qr variants for osx, linux and windows. r=jmaher https://hg.mozilla.org/integration/autoland/rev/fa9cc5631313 Shippable Builds - Refactor SETA old->new handling a bit, to make it clearer. r=jmaher https://hg.mozilla.org/integration/autoland/rev/a734011ca98e Shippable Builds - Significantly speedup SETA computation by using sets. r=jmaher https://hg.mozilla.org/integration/autoland/rev/2eed45e23116 Shippable Builds - Add more new/old conversions for SETA. r=jmaher https://hg.mozilla.org/integration/autoland/rev/f3040492fcbf Shippable Builds - add shippable index. r=aki https://hg.mozilla.org/integration/autoland/rev/26580c78cfb0 Shippable Builds - Consolidate raptor-chrome options up to job-defaults to reduce boilerplate spread throughout. r=jmaher

Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable

Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.

In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable

Attachment #9053321 - Attachment description: Bug 1352113 - amend debugger tests presets. r=ahal → Bug 1352113 - Shippable Builds - amend debugger tests presets. r=ahal
Pushed by jwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4a2e544fea0d Shippable Builds - allow raptor tests to run for shippable (ends up scheduling shippable builds, due to deps. r=jmaher https://hg.mozilla.org/integration/autoland/rev/01699fb72384 Shippable Builds - utilize run-on-projects more for previously excluded tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/456538d78b36 Shippable Builds - Start explicitly running shippable jobs per-push. r=aki https://hg.mozilla.org/integration/autoland/rev/6044aedac9c4 Shippable Builds - Adjust tests to align shippable sets with what was on pgo (where pgo exists) and opt. r=jmaher https://hg.mozilla.org/integration/autoland/rev/694eac65a72e Shippable Builds - Switch l10n jobs to be based on shippable builds. r=aki https://hg.mozilla.org/integration/autoland/rev/b31cbe91bdf7 Shippable Builds - Make source-test depend on shippable instead of pgo. r=aki https://hg.mozilla.org/integration/autoland/rev/fc16a3ecfe68 Shippable Builds - Remove windows pgo entirely. r=jmaher,aki https://hg.mozilla.org/integration/autoland/rev/dd2eddef8b43 Shippable Builds - Do not do linux32/opt build on integration. r=aki https://hg.mozilla.org/integration/autoland/rev/686c228e3579 Shippable Builds - Do not run linux32/opt tests, we now run them for linux32-shippable/opt. r=jmaher https://hg.mozilla.org/integration/autoland/rev/f64dfaf86a2e Shippable Builds - Do not do linux pgo either. r=jmaher,aki https://hg.mozilla.org/integration/autoland/rev/7cefe92f88d5 Shippable Builds - Don't run OSX opt builds by default on integration trees. r=aki https://hg.mozilla.org/integration/autoland/rev/b95b3f4e5243 Shippable Builds - Do not run opt-only OSX tests on integration trees. r=jmaher https://hg.mozilla.org/integration/autoland/rev/d7deec98601a Shippable Builds - Do not run any macosx64/opt or macosx64-qr/opt tests, leaving only shippable. r=jmaher https://hg.mozilla.org/integration/autoland/rev/6b65273fab78 Shippable Builds - amend debugger tests presets. r=ahal https://hg.mozilla.org/integration/autoland/rev/4b027c970719 Shippable Builds - make only-for-attributes accept any() instead of all() in the set. r=aki https://hg.mozilla.org/integration/autoland/rev/51ab82722846 Shippable Builds - Support shippable where 'nightly' is used. r=aki https://hg.mozilla.org/integration/autoland/rev/3e2b6a495e8c Remove nightly build types replaced by shippable. r=aki https://hg.mozilla.org/integration/autoland/rev/4b9413d05816 Shippable Builds - Update index routes for shippable. Try to be backwards compat where possible. r=aki https://hg.mozilla.org/integration/autoland/rev/4943b23813fe Shippable Builds - Account for new declarative artifacts work. r=mtabara
Backout by csabou@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/aea38cfc632a Backed out 19 changesets for causing upload symbol bustages. CLOSED TREE

This will avoid needing to rebuild the shippable builds are part of nightly
graphs, while we work on nightly promotion.

Attachment #9053775 - Attachment description: Bug 1352113 - Shippable Builds - Do FULL symbol uploads only on promote phase → Bug 1352113 - Shippable Builds - Do FULL symbol uploads only on promote phase. r=aki

On Wed, March 27, 2019, 1:44 PM GMT+2, by jwood@mozilla.com.
Revisions: D22832 diff 81613 ← D22833 diff 81614 ← D22834 diff 81615 ← D23122 diff 81616 ← D23123 diff 81617 ← D23124 diff 81618 ← D23125 diff 81619 ← D23248 diff 81620 ← D23383 diff 81621 ← D23126 diff 81622 ← D23128 diff 81623 ← D23129 diff 81624 ← D23382 diff 81625 ← D23437 diff 81626 ← D23130 diff 81627 ← D24699 diff 81628 ← D24700 diff 81629 ← D24829 diff 81630 ← D24831 diff 81631 ← D24999 diff 81645
Details: We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again. (255, 'applying /tmp/tmpnLJo_c\npatching file taskcluster/ci/release-snap-repackage/kind.yml\nHunk #2 FAILED at 58\n1 out of 2 hunks FAILED -- saving rejects to file taskcluster/ci/release-snap-repackage/kind.yml.rej\nabort: patch failed to apply', '')

Flags: needinfo?(bugspam.Callek)
Pushed by jwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ece439cc2d6d Shippable Builds - allow raptor tests to run for shippable (ends up scheduling shippable builds, due to deps. r=jmaher https://hg.mozilla.org/integration/autoland/rev/cc1547a13f15 Shippable Builds - utilize run-on-projects more for previously excluded tests. r=jmaher https://hg.mozilla.org/integration/autoland/rev/d7d2f0b6d43a Shippable Builds - Start explicitly running shippable jobs per-push. r=aki https://hg.mozilla.org/integration/autoland/rev/129505a33947 Shippable Builds - Adjust tests to align shippable sets with what was on pgo (where pgo exists) and opt. r=jmaher https://hg.mozilla.org/integration/autoland/rev/0d73807e7e2b Shippable Builds - Switch l10n jobs to be based on shippable builds. r=aki https://hg.mozilla.org/integration/autoland/rev/9cf5c1259a85 Shippable Builds - Make source-test depend on shippable instead of pgo. r=aki https://hg.mozilla.org/integration/autoland/rev/28eabae429db Shippable Builds - Remove windows pgo entirely. r=jmaher,aki https://hg.mozilla.org/integration/autoland/rev/b19d25e352b7 Shippable Builds - Do not do linux32/opt build on integration. r=aki https://hg.mozilla.org/integration/autoland/rev/4b333865518e Shippable Builds - Do not run linux32/opt tests, we now run them for linux32-shippable/opt. r=jmaher https://hg.mozilla.org/integration/autoland/rev/c919da7ce3d6 Shippable Builds - Do not do linux pgo either. r=jmaher,aki https://hg.mozilla.org/integration/autoland/rev/1e2f8a5f178e Shippable Builds - Don't run OSX opt builds by default on integration trees. r=aki https://hg.mozilla.org/integration/autoland/rev/1eae3d963d1b Shippable Builds - Do not run opt-only OSX tests on integration trees. r=jmaher https://hg.mozilla.org/integration/autoland/rev/9c524a5f06ea Shippable Builds - Do not run any macosx64/opt or macosx64-qr/opt tests, leaving only shippable. r=jmaher https://hg.mozilla.org/integration/autoland/rev/b91304f4d69e Shippable Builds - amend debugger tests presets. r=ahal https://hg.mozilla.org/integration/autoland/rev/8a1e1068d6ba Shippable Builds - make only-for-attributes accept any() instead of all() in the set. r=aki https://hg.mozilla.org/integration/autoland/rev/f75cdebc4e7b Shippable Builds - Support shippable where 'nightly' is used. r=aki https://hg.mozilla.org/integration/autoland/rev/76562c9a4269 Remove nightly build types replaced by shippable. r=aki https://hg.mozilla.org/integration/autoland/rev/72988abda760 Shippable Builds - Update index routes for shippable. Try to be backwards compat where possible. r=aki https://hg.mozilla.org/integration/autoland/rev/1de2be687d8f Shippable Builds - Account for new declarative artifacts work. r=mtabara https://hg.mozilla.org/integration/autoland/rev/dfcd5e5299f7 Shippable Builds - Don't use enable-full-crashsymbols attribute when we explicitly disable symbols for a job. r=aki
Flags: needinfo?(bugspam.Callek)
Pushed by mozilla@hocat.ca: https://hg.mozilla.org/integration/autoland/rev/cc36ced7b323 Re-use builds from push graph in nightly graphs; r=Callek
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1540132
No longer blocks: 1540132
Depends on: 1540132
No longer depends on: 1540941
Regressions: 1540941
Blocks: 1563663
See Also: → 1614970
Attachment #9053775 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: