Status

enhancement
P3
normal
RESOLVED FIXED
2 years ago
17 days ago

People

(Reporter: aki, Assigned: Callek)

Tracking

(Depends on 1 bug, Blocks 2 bugs)

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox68 fixed)

Details

(Whiteboard: [releng:q12019])

Attachments

(43 attachments)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
Shippable builds will most likely require taskcluster nightlies... so we *could* proceed with linux and android at this point, or wait until osx and windows are on TC.

This is a writeup of https://github.com/mozilla/build-relengdocs/blob/master/future/index.rst#shippable-builds and https://groups.google.com/forum/#!topic/mozilla.release.engineering/MRfHnhaVI48 .

Let's:
- rename 'nightly' builds to 'shippable' builds

- get rid of 'pgo' builds

- have a second promotion graph to ship shippable builds

- (possibly rename 'depend' to 'ci', but that doesn't have full agreement.)

promotion graph
===============

- move the actual shipping tasks (beetmover, balrog, pushapk) to a second promotion graph.  These should depend on the build or signing tasks in the build graph as their upstreamArtifacts.

    - the decision task for the second promotion graph needs to be able find the latest successful shippable graph, possibly through dummy task indexes, on mozilla-central and mozilla-aurora.  On mozilla-beta, mozilla-release, and mozilla-esr, we can specify the shippable graph/tasks to use.

scheduling
==========

- shippable builds should periodically on integration branches, with backfill capability.

    - every 4 hours?  Ideally let's not automatically re-run a shippable build on revision X if one has already successfully been triggered.

- both depend and shippable builds should be choosable using try syntax.

- on central/aurora/beta/release/esr*, let's build shippable builds on push and drop depend opt builds.
:jmaher, periodic shippable builds on integration branches may be problematic if talos still needs to run in chronological order.  I know that was the case in the graph.m.o days; is that still the case with perfherder?
Flags: needinfo?(jmaher)
perfherder stores data by the revision, not by the date the data was generated, so if I understand the definition of chronological order in reference to the question, then this will not be problematic :)
Flags: needinfo?(jmaher)
Priority: -- → P3
Duplicate of this bug: 932211
Depends on: 1416538
Component: General Automation → General
Whiteboard: [releng:q32018]
Blocks: 1501898
Whiteboard: [releng:q32018] → [releng:q12019]
Assignee: nobody → bugspam.Callek

This commit adds a new build for OSX since there is no current PGO build type for OSX.
And calls it shippable, this mirrors that of the nightly.

This was needed since when we have job-defaults and later on a test set for
run-on-projects: by-test-platform: ...
We were ending up with both the list of by-* being extended but also
any array in that list also being extended (like default: ['a', 'b']
was getting extended to also have the new values for default)

This is not only usually wrong but very very likely not what the author wanted.

When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.

This sets all of the shippable tests to not run in the places where they would otherwise.
This patch will be effectively undone later in the patchset.

Without this the shippable builds take >20 seconds to compute the prune through new_as_old_is_high_value, locally. With this it is near instant.

Attachment #9049521 - Attachment description: Bug 1352113 - Shippable Builds - Do not run merge on by-* values. r=aki → Bug 1352113 - Shippable Builds - Do not run merge on by-* values. r=tomprince

This does not change the substance of what is passed down to transforms.

raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.

TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.

Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.

This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.

I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.

We are adding shippable-qr to beta because of replacing nightly too.

Autoland and inbound should have the same sets of tasks.

beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.

central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable

This is useful in order to not have to run linux64/opt on push, especially on autoland/inbound
when we need a source test. It is also required if we remove the linux64-pgo build type entirely.

Attachment #9050513 - Attachment description: Bug 1352113 - Shippable Builds - Do not do linux32/opt build on integration. → Bug 1352113 - Shippable Builds - Do not do linux32/opt build on integration. r=aki

Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.

Attachment #9050286 - Attachment description: Bug 1352113 - Shippable Builds - Remove windows pgo entirely. → Bug 1352113 - Shippable Builds - Remove windows pgo entirely. r=aki,jmaher
Attachment #9050287 - Attachment description: Bug 1352113 - Shippable Builds - Do not do linux pgo either. → Bug 1352113 - Shippable Builds - Do not do linux pgo either. r=aki,jmaher
Attachment #9050289 - Attachment description: Bug 1352113 - Shippable Builds - Don't run OSX opt builds by default on integration trees. → Bug 1352113 - Shippable Builds - Don't run OSX opt builds by default on integration trees. r=aki

Jmaher indicated we do not have the test capacity to incur this as a duplicated set

Attachment #9050291 - Attachment description: Bug 1352113 - Shippable Builds - make only-for-attributes accept any() instead of all() in the set. → Bug 1352113 - Shippable Builds - make only-for-attributes accept any() instead of all() in the set. r=aki
Attachment #9050290 - Attachment description: Bug 1352113 - Shippable Builds - Do not run opt-only OSX tests on integration trees. → Bug 1352113 - Shippable Builds - Do not run opt-only OSX tests on integration trees. r=jmaher
Keywords: leave-open
Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0fe5ffc10ff8
Shippable Builds - Do not run merge on by-* values. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/e7eeb55c71ba
Shippable Builds - Add index builder, but don't do any index's yet. r=aki
https://hg.mozilla.org/integration/autoland/rev/1269ffd8a411
Shippable Builds - OSX Build. r=aki
https://hg.mozilla.org/integration/autoland/rev/37f63cecb702
Shippable Builds - Add windows32 shippable. r=aki
https://hg.mozilla.org/integration/autoland/rev/333ad0658d94
Shippable Builds - Add windows64 shippable. r=aki
https://hg.mozilla.org/integration/autoland/rev/f59eb3b8d0b3
Shippable Builds - Add windows64 aarch64 shippable. r=aki
https://hg.mozilla.org/integration/autoland/rev/b764eae9ff5a
Shippable Builds - Add Linux32 shippable builds. r=aki
https://hg.mozilla.org/integration/autoland/rev/4a98810cec0a
Shippable Builds - Add linux64 shippable builds. r=aki
https://hg.mozilla.org/integration/autoland/rev/b81435806374
Shippable Builds - Make sure we sign shippable builds. r=aki
https://hg.mozilla.org/integration/autoland/rev/5de8fab632bc
Shippable Builds - Add test run-on-projects to avoid scheduling shippable just yet. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/1fae9a0c3d4f
Shippable Builds - OSX tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/6dfa69851019
Shippable Builds - Linux32 Tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/8f22b80c544e
Shippable Builds - Linux64 Tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/dc7a3140939e
Shippable Builds - Win7 Tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/3e1475bc7fd5
Shippable Builds - Win10 Tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/ea10d6f34744
Shippable Builds - add shippable qr variants for osx, linux and windows. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/fa9cc5631313
Shippable Builds - Refactor SETA old->new handling a bit, to make it clearer. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/a734011ca98e
Shippable Builds - Significantly speedup SETA computation by using sets. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/2eed45e23116
Shippable Builds - Add more new/old conversions for SETA. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/f3040492fcbf
Shippable Builds - add shippable index. r=aki
https://hg.mozilla.org/integration/autoland/rev/26580c78cfb0
Shippable Builds - Consolidate raptor-chrome options up to job-defaults to reduce boilerplate spread throughout. r=jmaher
Pushed by apavel@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/046333a3b9eb
fix duplicate key on a CLOSED TREE

Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable

Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.

In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable

Attachment #9053321 - Attachment description: Bug 1352113 - amend debugger tests presets. r=ahal → Bug 1352113 - Shippable Builds - amend debugger tests presets. r=ahal
Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4a2e544fea0d
Shippable Builds - allow raptor tests to run for shippable (ends up scheduling shippable builds, due to deps. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/01699fb72384
Shippable Builds - utilize run-on-projects more for previously excluded tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/456538d78b36
Shippable Builds - Start explicitly running shippable jobs per-push. r=aki
https://hg.mozilla.org/integration/autoland/rev/6044aedac9c4
Shippable Builds - Adjust tests to align shippable sets with what was on pgo (where pgo exists) and opt. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/694eac65a72e
Shippable Builds - Switch l10n jobs to be based on shippable builds. r=aki
https://hg.mozilla.org/integration/autoland/rev/b31cbe91bdf7
Shippable Builds - Make source-test depend on shippable instead of pgo. r=aki
https://hg.mozilla.org/integration/autoland/rev/fc16a3ecfe68
Shippable Builds - Remove windows pgo entirely. r=jmaher,aki
https://hg.mozilla.org/integration/autoland/rev/dd2eddef8b43
Shippable Builds - Do not do linux32/opt build on integration. r=aki
https://hg.mozilla.org/integration/autoland/rev/686c228e3579
Shippable Builds - Do not run linux32/opt tests, we now run them for linux32-shippable/opt. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/f64dfaf86a2e
Shippable Builds - Do not do linux pgo either. r=jmaher,aki
https://hg.mozilla.org/integration/autoland/rev/7cefe92f88d5
Shippable Builds - Don't run OSX opt builds by default on integration trees. r=aki
https://hg.mozilla.org/integration/autoland/rev/b95b3f4e5243
Shippable Builds - Do not run opt-only OSX tests on integration trees. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/d7deec98601a
Shippable Builds - Do not run any macosx64/opt or macosx64-qr/opt tests, leaving only shippable. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/6b65273fab78
Shippable Builds - amend debugger tests presets. r=ahal
https://hg.mozilla.org/integration/autoland/rev/4b027c970719
Shippable Builds - make only-for-attributes accept any() instead of all() in the set. r=aki
https://hg.mozilla.org/integration/autoland/rev/51ab82722846
Shippable Builds - Support shippable where 'nightly' is used. r=aki
https://hg.mozilla.org/integration/autoland/rev/3e2b6a495e8c
Remove nightly build types replaced by shippable. r=aki
https://hg.mozilla.org/integration/autoland/rev/4b9413d05816
Shippable Builds - Update index routes for shippable. Try to be backwards compat where possible. r=aki
https://hg.mozilla.org/integration/autoland/rev/4943b23813fe
Shippable Builds - Account for new declarative artifacts work. r=mtabara
Backout by csabou@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/aea38cfc632a
Backed out 19 changesets for causing upload symbol bustages. CLOSED TREE

This will avoid needing to rebuild the shippable builds are part of nightly
graphs, while we work on nightly promotion.

Attachment #9053775 - Attachment description: Bug 1352113 - Shippable Builds - Do FULL symbol uploads only on promote phase → Bug 1352113 - Shippable Builds - Do FULL symbol uploads only on promote phase. r=aki

On Wed, March 27, 2019, 1:44 PM GMT+2, by jwood@mozilla.com.
Revisions: D22832 diff 81613 ← D22833 diff 81614 ← D22834 diff 81615 ← D23122 diff 81616 ← D23123 diff 81617 ← D23124 diff 81618 ← D23125 diff 81619 ← D23248 diff 81620 ← D23383 diff 81621 ← D23126 diff 81622 ← D23128 diff 81623 ← D23129 diff 81624 ← D23382 diff 81625 ← D23437 diff 81626 ← D23130 diff 81627 ← D24699 diff 81628 ← D24700 diff 81629 ← D24829 diff 81630 ← D24831 diff 81631 ← D24999 diff 81645
Details: We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again. (255, 'applying /tmp/tmpnLJo_c\npatching file taskcluster/ci/release-snap-repackage/kind.yml\nHunk #2 FAILED at 58\n1 out of 2 hunks FAILED -- saving rejects to file taskcluster/ci/release-snap-repackage/kind.yml.rej\nabort: patch failed to apply', '')

Flags: needinfo?(bugspam.Callek)
Pushed by jwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ece439cc2d6d
Shippable Builds - allow raptor tests to run for shippable (ends up scheduling shippable builds, due to deps. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/cc1547a13f15
Shippable Builds - utilize run-on-projects more for previously excluded tests. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/d7d2f0b6d43a
Shippable Builds - Start explicitly running shippable jobs per-push. r=aki
https://hg.mozilla.org/integration/autoland/rev/129505a33947
Shippable Builds - Adjust tests to align shippable sets with what was on pgo (where pgo exists) and opt. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/0d73807e7e2b
Shippable Builds - Switch l10n jobs to be based on shippable builds. r=aki
https://hg.mozilla.org/integration/autoland/rev/9cf5c1259a85
Shippable Builds - Make source-test depend on shippable instead of pgo. r=aki
https://hg.mozilla.org/integration/autoland/rev/28eabae429db
Shippable Builds - Remove windows pgo entirely. r=jmaher,aki
https://hg.mozilla.org/integration/autoland/rev/b19d25e352b7
Shippable Builds - Do not do linux32/opt build on integration. r=aki
https://hg.mozilla.org/integration/autoland/rev/4b333865518e
Shippable Builds - Do not run linux32/opt tests, we now run them for linux32-shippable/opt. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/c919da7ce3d6
Shippable Builds - Do not do linux pgo either. r=jmaher,aki
https://hg.mozilla.org/integration/autoland/rev/1e2f8a5f178e
Shippable Builds - Don't run OSX opt builds by default on integration trees. r=aki
https://hg.mozilla.org/integration/autoland/rev/1eae3d963d1b
Shippable Builds - Do not run opt-only OSX tests on integration trees. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/9c524a5f06ea
Shippable Builds - Do not run any macosx64/opt or macosx64-qr/opt tests, leaving only shippable. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/b91304f4d69e
Shippable Builds - amend debugger tests presets. r=ahal
https://hg.mozilla.org/integration/autoland/rev/8a1e1068d6ba
Shippable Builds - make only-for-attributes accept any() instead of all() in the set. r=aki
https://hg.mozilla.org/integration/autoland/rev/f75cdebc4e7b
Shippable Builds - Support shippable where 'nightly' is used. r=aki
https://hg.mozilla.org/integration/autoland/rev/76562c9a4269
Remove nightly build types replaced by shippable. r=aki
https://hg.mozilla.org/integration/autoland/rev/72988abda760
Shippable Builds - Update index routes for shippable. Try to be backwards compat where possible. r=aki
https://hg.mozilla.org/integration/autoland/rev/1de2be687d8f
Shippable Builds - Account for new declarative artifacts work. r=mtabara
https://hg.mozilla.org/integration/autoland/rev/dfcd5e5299f7
Shippable Builds - Don't use enable-full-crashsymbols attribute when we explicitly disable symbols for a job. r=aki
Flags: needinfo?(bugspam.Callek)
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/cc36ced7b323
Re-use builds from push graph in nightly graphs; r=Callek
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Blocks: 1540132
No longer blocks: 1540132
Depends on: 1540132
No longer depends on: 1540941
Regressions: 1540941
Blocks: 1563663
You need to log in before you can comment on or make changes to this bug.