1083230 - Scope repacking CI builds instead of building from scratch

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Lawrence Mandel [:lmandel] (use needinfo)]

When building betas and releases, relman typically waits for a ci build to complete so that we know that the changes that we are picking up have resulted in a good, green build. After the ci build completes we submit a build in ship-it, which then builds again. Rebuilding in this way seems to duplicate effort and increase the overall build time from check-in until QE can test. When I first started using ship-it I had expected that a build submitted to ship-it would pick up the results of the ci build and then do the rest of the required processing such as the additional of branding, signing, and partial generation.

This request is to see if we can reduce the build time by having ship-it repackage the output of the ci build rather than building from scratch.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This is a big project, so I'm going to morph this bug to be a scoping study, which we would like to do this quarter.

Some issues to consider:
* mozconfig differences between current dep+nightly vs release (eg branding, PGO, BUILDING_RELEASE=1)
* other build time configuration using #ifdef in code (eg MOZ_UPDATE_CHANNEL)
* re-signing binaries (not possible on windows ?)
* rearranging current work
 * still need release configs/bumping/reconfigs ?
 * after-the-fact tagging of en-US
 * partial generation for en-US, balrog resubmission
 * l10n repacks
 * file locations for update verify etc

The first three points push me towards making the philosophical leap that all builds on beta/release/esr31 should use the appropriate release branding, signing certs, and update channel. In the past we've avoided that so that the download.com and neowin.net's of this world don't keep promoting 'unreleased' bits, but we should revisit that.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

We completed some discussion of this at the end of 2014, including in person in Portland and then afterwards. The result was:

No-one wants to repack, lets build what we want to ship the first time. We
should seek signoff from opsec on the signing changes (and fall back to
re-signing if necessary, osslsigncode may be able to strip sigs on windows);
ask if anyone needs to signoff on branding etc

Changes to en-US builds: release mozconfig, update channel/pretty packaging
and other environment variables, generate complete mar every time, possibly
other steps from release job too, add xulrunner sdk. Maybe do source in one
en-US build (linux/mac?), need to see how long it takes. See how the tests
go against beta signing and file naming, comparing to current dep builds
may be helpful.

Everything else is 'downstream', initially triggered by a sendchange, later
a human hitting a button on treeherder (or similar)
* new release-sanity, which grabs l10n revisions and checks 'em, all en-US
builds available, important tests pass ?
* tagging - very limited value in tagging m-b repo for each beta, but still
important for release; lazy relbranches
* source - needs to key off revision rather than tag
* generate en-US partials with funsize
* l10n repacks - keep chunking, use mozharness by preference, pass in url
to en-US builds rather than soup of info from release config, use funsize
for partials ?
* partner repacks - port over, seems wasteful to do this every day, talk to
mconnor about reduced frequency
* updates: much as now, bumping might need reworking
* update verify: will have to test less, cull off older builds
* new update test: new --> newer for rstrong, startup and post-update
action checks in marionnette
* antivirus, checksums, check_permissions: pass a dir instead of infer from
release config
* push to mirrors (auto only for beta)
* final verify
* marionette update tests
* QE signoff
* push to beta
* marionette update tests
* postrelease

Ideas discussed
* should every beta build be an RC ?  Not at first, stick with 
m-b and m-r repos to keep scope manageable, and we're not sure what extra
work we need for RC yet
* nightlies could make similar transition, eg dep builds are nightly style,
then promote by generating partials + test updates + test updates from new
builds + put on nightly channel)
* catlee would like to build from scratch where possible, to complete the
transition to a modern style

Rough roadmap for Q1, developing new system in parallel to old one still
running:
1, disposable branch based on beta
2, change CI builds to be 'release' style, see how the tests go
3, build automation to generate   (trigger with some sort of sendchange for
now)
 * l10n repacks
 * partner repacks
 * updates for en-US + locales (partial generation and balrog submission)

There are plenty more builders to add after that, and the job of gluing
things together, eg ship-it v2, with buttons/API for mirror push/beta push
etc, acting as source of truth for release history


We're planning to start working on this now, with tracking in bug 1118794.