implement shippable builds
Categories
(Release Engineering :: General, enhancement, P3)
Tracking
(firefox68 fixed)
Tracking | Status | |
---|---|---|
firefox68 | --- | fixed |
People
(Reporter: mozilla, Assigned: Callek)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Whiteboard: [releng:q12019])
Attachments
(42 files, 1 obsolete file)
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
Bug 1352113 - Shippable Builds - Refactor SETA old->new handling a bit, to make it clearer. r=jmaher
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
Reporter | ||
Comment 1•8 years ago
|
||
Comment 2•8 years ago
|
||
Updated•7 years ago
|
Updated•7 years ago
|
Updated•6 years ago
|
Comment 4•6 years ago
|
||
Assignee | ||
Updated•6 years ago
|
Assignee | ||
Comment 5•6 years ago
|
||
Assignee | ||
Comment 6•6 years ago
|
||
This commit adds a new build for OSX since there is no current PGO build type for OSX.
And calls it shippable, this mirrors that of the nightly.
Assignee | ||
Comment 7•6 years ago
|
||
Assignee | ||
Comment 8•6 years ago
|
||
Assignee | ||
Comment 9•6 years ago
|
||
Assignee | ||
Comment 10•6 years ago
|
||
Assignee | ||
Comment 11•6 years ago
|
||
Assignee | ||
Comment 12•6 years ago
|
||
Assignee | ||
Comment 13•6 years ago
|
||
Assignee | ||
Comment 14•6 years ago
|
||
Assignee | ||
Comment 15•6 years ago
|
||
Assignee | ||
Comment 16•6 years ago
|
||
Assignee | ||
Comment 17•6 years ago
|
||
Assignee | ||
Comment 18•6 years ago
|
||
Assignee | ||
Comment 19•6 years ago
|
||
Assignee | ||
Comment 20•6 years ago
|
||
Assignee | ||
Comment 21•6 years ago
|
||
This was needed since when we have job-defaults and later on a test set for
run-on-projects: by-test-platform: ...
We were ending up with both the list of by-* being extended but also
any array in that list also being extended (like default: ['a', 'b']
was getting extended to also have the new values for default)
This is not only usually wrong but very very likely not what the author wanted.
Assignee | ||
Comment 22•6 years ago
|
||
When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.
Assignee | ||
Comment 23•6 years ago
|
||
This sets all of the shippable tests to not run in the places where they would otherwise.
This patch will be effectively undone later in the patchset.
Assignee | ||
Comment 24•6 years ago
|
||
Without this the shippable builds take >20 seconds to compute the prune through new_as_old_is_high_value, locally. With this it is near instant.
Updated•6 years ago
|
Assignee | ||
Comment 25•6 years ago
|
||
This does not change the substance of what is passed down to transforms.
Assignee | ||
Comment 26•6 years ago
|
||
raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.
TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.
Assignee | ||
Comment 27•6 years ago
|
||
Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.
Assignee | ||
Comment 28•6 years ago
|
||
Assignee | ||
Comment 29•6 years ago
|
||
Assignee | ||
Comment 30•6 years ago
|
||
Assignee | ||
Comment 31•6 years ago
|
||
Assignee | ||
Comment 32•6 years ago
|
||
This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.
Assignee | ||
Comment 33•6 years ago
|
||
Assignee | ||
Comment 34•6 years ago
|
||
Assignee | ||
Comment 35•6 years ago
|
||
I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.
We are adding shippable-qr to beta because of replacing nightly too.
Autoland and inbound should have the same sets of tasks.
beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.
central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable
Assignee | ||
Comment 36•6 years ago
|
||
Assignee | ||
Comment 37•6 years ago
|
||
This is useful in order to not have to run linux64/opt on push, especially on autoland/inbound
when we need a source test. It is also required if we remove the linux64-pgo build type entirely.
Updated•6 years ago
|
Assignee | ||
Comment 38•6 years ago
|
||
Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Assignee | ||
Comment 39•6 years ago
|
||
Jmaher indicated we do not have the test capacity to incur this as a duplicated set
Updated•6 years ago
|
Updated•6 years ago
|
Assignee | ||
Updated•6 years ago
|
Comment 40•6 years ago
|
||
Comment 41•6 years ago
|
||
Comment 42•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/0fe5ffc10ff8
https://hg.mozilla.org/mozilla-central/rev/e7eeb55c71ba
https://hg.mozilla.org/mozilla-central/rev/1269ffd8a411
https://hg.mozilla.org/mozilla-central/rev/37f63cecb702
https://hg.mozilla.org/mozilla-central/rev/333ad0658d94
https://hg.mozilla.org/mozilla-central/rev/f59eb3b8d0b3
https://hg.mozilla.org/mozilla-central/rev/b764eae9ff5a
https://hg.mozilla.org/mozilla-central/rev/4a98810cec0a
https://hg.mozilla.org/mozilla-central/rev/b81435806374
https://hg.mozilla.org/mozilla-central/rev/5de8fab632bc
https://hg.mozilla.org/mozilla-central/rev/1fae9a0c3d4f
https://hg.mozilla.org/mozilla-central/rev/6dfa69851019
https://hg.mozilla.org/mozilla-central/rev/8f22b80c544e
https://hg.mozilla.org/mozilla-central/rev/dc7a3140939e
https://hg.mozilla.org/mozilla-central/rev/3e1475bc7fd5
https://hg.mozilla.org/mozilla-central/rev/ea10d6f34744
https://hg.mozilla.org/mozilla-central/rev/fa9cc5631313
https://hg.mozilla.org/mozilla-central/rev/a734011ca98e
https://hg.mozilla.org/mozilla-central/rev/2eed45e23116
https://hg.mozilla.org/mozilla-central/rev/f3040492fcbf
https://hg.mozilla.org/mozilla-central/rev/26580c78cfb0
https://hg.mozilla.org/mozilla-central/rev/046333a3b9eb
Assignee | ||
Comment 43•6 years ago
|
||
Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable
Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.
In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable
Assignee | ||
Comment 44•6 years ago
|
||
Assignee | ||
Comment 45•6 years ago
|
||
Updated•6 years ago
|
Assignee | ||
Comment 46•6 years ago
|
||
Assignee | ||
Comment 47•6 years ago
|
||
Comment 48•6 years ago
|
||
Comment 49•6 years ago
|
||
Comment 50•6 years ago
|
||
This will avoid needing to rebuild the shippable builds are part of nightly
graphs, while we work on nightly promotion.
Assignee | ||
Comment 51•6 years ago
|
||
This keeps load lower while continuing to only build full symbols on release branches, due to https://searchfox.org/mozilla-central/rev/a7315d78417179b151fef6108f2bce14786ba64d/taskcluster/taskgraph/transforms/build.py#125
Updated•6 years ago
|
Assignee | ||
Comment 52•6 years ago
|
||
Comment 53•6 years ago
|
||
On Wed, March 27, 2019, 1:44 PM GMT+2, by jwood@mozilla.com.
Revisions: D22832 diff 81613 ← D22833 diff 81614 ← D22834 diff 81615 ← D23122 diff 81616 ← D23123 diff 81617 ← D23124 diff 81618 ← D23125 diff 81619 ← D23248 diff 81620 ← D23383 diff 81621 ← D23126 diff 81622 ← D23128 diff 81623 ← D23129 diff 81624 ← D23382 diff 81625 ← D23437 diff 81626 ← D23130 diff 81627 ← D24699 diff 81628 ← D24700 diff 81629 ← D24829 diff 81630 ← D24831 diff 81631 ← D24999 diff 81645
Details: We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again. (255, 'applying /tmp/tmpnLJo_c\npatching file taskcluster/ci/release-snap-repackage/kind.yml\nHunk #2 FAILED at 58\n1 out of 2 hunks FAILED -- saving rejects to file taskcluster/ci/release-snap-repackage/kind.yml.rej\nabort: patch failed to apply', '')
Comment 54•6 years ago
|
||
Assignee | ||
Updated•6 years ago
|
Comment 55•6 years ago
|
||
Comment 56•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/ece439cc2d6d
https://hg.mozilla.org/mozilla-central/rev/cc1547a13f15
https://hg.mozilla.org/mozilla-central/rev/d7d2f0b6d43a
https://hg.mozilla.org/mozilla-central/rev/129505a33947
https://hg.mozilla.org/mozilla-central/rev/0d73807e7e2b
https://hg.mozilla.org/mozilla-central/rev/9cf5c1259a85
https://hg.mozilla.org/mozilla-central/rev/28eabae429db
https://hg.mozilla.org/mozilla-central/rev/b19d25e352b7
https://hg.mozilla.org/mozilla-central/rev/4b333865518e
https://hg.mozilla.org/mozilla-central/rev/c919da7ce3d6
https://hg.mozilla.org/mozilla-central/rev/1e2f8a5f178e
https://hg.mozilla.org/mozilla-central/rev/1eae3d963d1b
https://hg.mozilla.org/mozilla-central/rev/9c524a5f06ea
https://hg.mozilla.org/mozilla-central/rev/b91304f4d69e
https://hg.mozilla.org/mozilla-central/rev/8a1e1068d6ba
https://hg.mozilla.org/mozilla-central/rev/f75cdebc4e7b
https://hg.mozilla.org/mozilla-central/rev/76562c9a4269
https://hg.mozilla.org/mozilla-central/rev/72988abda760
https://hg.mozilla.org/mozilla-central/rev/1de2be687d8f
https://hg.mozilla.org/mozilla-central/rev/dfcd5e5299f7
https://hg.mozilla.org/mozilla-central/rev/cc36ced7b323
Assignee | ||
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Updated•3 years ago
|
Description
•