1083277 - Create deterministic builds

Status: RESOLVED DUPLICATE of bug 885777

(Release Engineering :: Release Automation: Other, defect)

Description

[Lawrence Mandel [:lmandel] (use needinfo)]

Our current use of PGO results in non-deterministic builds. This adds some risk to the release as beta9 and RC builds are not guaranteed to produce the same bits assuming no code changes.

My understanding is that the builds are not deterministic because we generate a PGO profile and apply it for each build. An alternative approach is to generate a static PGO profile that an be applied to many builds. We can select an appropriate refresh period for the static profile. An added benefit of this approach should be a reduction in build times as we currently have to build once to generate the profile and a second time to apply it.

Some possible refresh periods for the static profile:
- create in beta9 and apply to RC
- create every two weeks on beta, weekly on aurora and central

Comment 1

[Chris AtLee [:catlee]]

I'm not sure how feasible this is, i.e. if the profiles are re-usable. cc'ed gps for some perspectives.

Some good news though is that switching to VS2013 has saved about 1.5 hours for our windows PGO builds.

Comment 2

[Gregory Szorc [:gps]]

We already have bug 885777 on file. Is this bug a duplicate?

This is a complex goal to achieve and starting by talking about refresh periods for static profiles feels a bit haphazard. We need to be very clear what the objective is. e.g. are we going after determinism and/or trust/verifiability and/or reproducibility? Are we going after all platforms, or just Linux, Windows, and/or OS X? Is the goal to make these traits part of the release distribution, or will we publish a channel that targeted at attaining these? (The proprietary bits inside Firefox like private keys make fully reproducible difficult/impossible.)

I think we need a kick-off meeting. glandium, mshal, ted, and myself have all dabbled into this before from the technical side of the core build system. As many people from that list as possible should attend.

Comment 3

[Lawrence Mandel [:lmandel] (use needinfo)]

This may have been a haphazard place to start. Let me back up. My interest is in reproducibility for all platforms for release builds. Specifically that a rebuild for whatever reason with no code change should produce the same bits if we spin it again. (A concrete example is that a release candidate build should be identical to beta9 - the last beta - if there are no additional changes to take.)

My understanding from a conversation that I had a while back with Taras is that the release builds are not deterministic because of PGO. His suggestion was that a static profile may allow us to solve this problem. That's the information that I used in filing this bug.

Comment 4

[Gregory Szorc [:gps]]

I think you may have been told a gross over-simplification of the problem. Yes, PGO is one component. But the rabbit hole goes very deep. We should talk about this offline.

Comment 5

[Ed Morley [:emorley]]

IMO this is either a dupe of bug 885777 or else the "use the same build for releasing as the one that we already built and tested previously" bug, that I can't find for now.

Comment 6

[Ed Morley [:emorley]]

(In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #3)
> This may have been a haphazard place to start. Let me back up. My interest
> is in reproducibility for all platforms for release builds. Specifically
> that a rebuild for whatever reason with no code change should produce the
> same bits if we spin it again. (A concrete example is that a release
> candidate build should be identical to beta9 - the last beta - if there are
> no additional changes to take.)

This is bug 1118794.

Comment 7

[Mike Hommey [:glandium]]

(In reply to Ed Morley [:emorley] from comment #6)
> (In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #3)
> > This may have been a haphazard place to start. Let me back up. My interest
> > is in reproducibility for all platforms for release builds. Specifically
> > that a rebuild for whatever reason with no code change should produce the
> > same bits if we spin it again. (A concrete example is that a release
> > candidate build should be identical to beta9 - the last beta - if there are
> > no additional changes to take.)
> 
> This is bug 1118794.

Is it? It's not really clear what that bug is about, but it doesn't sound it's about the same thing. In fact, back to what the present bug is about, there is no way you can make beta9 bit-identical to release, because there's at the very least the following differences:
- the channel in prefs (beta vs. release)
- the mercurial source repository url and changeset sha1 in application.ini

Now, I guess the main concern is about *compiled* bits, but then, why not make releases a repack of beta9, then, with the above changed?

Back to bug 1118794, I think it's about making e.g. a tinderbox build from mozilla-central the nightly, a tinderbox build from mozilla-aurora an aurora, etc. which doesn't involve the same constraints.

Comment 8

[Ed Morley [:emorley]]

At the least this bug is a dupe of something, or else is INCOMPLETE until the requirements are actually defined.