The default bug view has changed. See this FAQ.
Bug 885777 (fx-reproducible-build)

[meta] Deterministic, reproducible, bit-identical and/or verifiable Linux builds

NEW
Unassigned

Status

()

Core
Build Config
4 years ago
a day ago

People

(Reporter: gps, Unassigned, NeedInfo)

Tracking

(Depends on: 8 bugs, Blocks: 2 bugs, {meta})

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [tor], URL)

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

4 years ago
Currently, it's not possible for a local developer to reproduce Mozilla's official build environment. This is mostly because the operating system config and packages used on official infrastructure are not publicly available.

While there are good reasons for the tree to be developed and tested on as many OS/package/environment combinations as possible, there are also times where developers want to ensure their environment is as close to the official one as possible. e.g. when developing C++, you gain confidence that your local compiler warnings will be exactly what's encountered on the official build.

I'm opening this bug to track the ability for local developers to reproduce RelEng's build environment so produced builds are deterministic and bit-identical to what they would get on RelEng hardware (at least to a reasonable degree, such as using the same toolchain/packages). While I'd like to see us do this on all 3 of the tier 1 platforms (Windows, OS X, and Linux), I'm limiting this bug to just Linux because I feel that's the only platform where this is realistically attainable at the moment.

This effort is currently low priority. This bug is mostly for tracking purposes.
(Reporter)

Updated

4 years ago
Depends on: 768879
(Reporter)

Comment 1

4 years ago
There are some useful packages at http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/.

We *might* have enough content there to attempt automated reproduction of the official CentOS build environment...
Note that it's currently impossible to get idempotent builds even on the same machine. bug 742169 is one example.
(Reporter)

Updated

4 years ago
Depends on: 886226
(Reporter)

Comment 3

4 years ago
Created attachment 766556 [details] [diff] [review]
Add a Vagrantfile for creating the Linux64 build environment

And here is my first stab at it. Once I publish the CentOS 6.2 image I
created (will need to do it tomorrow once I have a faster internet
connection), you should be able to:

  cd config/bootstrap-linux64
  vagrant up
  # Wait a few minutes for the VM to initialize, download package
  # updates, populate mock_mozilla, etc.
  vagrant ssh
  mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/me/build --unpriv --shell /bin/bash
  ./mach build

I currently mount the cloned source directory as /builds/slave/me/build,
which is fine as the regular ssh user. However, it gets unmounted in the
mock environment. I need to do some homework.

For this to get checked into the tree, I'd like to have the Vagrant .box
file published somewhere somewhat official. Unless people counts, I may
start asking around for hosting. It should be a one-time thing,
methinks.
(Reporter)

Updated

4 years ago
Assignee: nobody → gps
(Reporter)

Comment 4

4 years ago
Comment on attachment 766556 [details] [diff] [review]
Add a Vagrantfile for creating the Linux64 build environment

And wrong bug. Derp. See bug 886226.
Attachment #766556 - Attachment is obsolete: true
(Reporter)

Updated

4 years ago
Depends on: 896023
(Reporter)

Updated

4 years ago
Assignee: gps → nobody
(Reporter)

Updated

3 years ago
Summary: [meta] Deterministic, bit-identical Linux builds → [meta] Deterministic, bit-identical and/or verifiable Linux builds
(Reporter)

Updated

3 years ago
Depends on: 935637
(Reporter)

Comment 5

3 years ago
Tor has published details of how they are doing deterministic builds. Good reading.

https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise
https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details

Comment 6

3 years ago
Created attachment 8363336 [details] [diff] [review]
Disable library timestamping

libfaketime is unable to spoof these timestamps because they use sub-millisecond precision.

Comment 7

3 years ago
Created attachment 8363337 [details] [diff] [review]
Misc reproducibility fixes (against FF24ESR)

Miscellaneous reproducibility fixes to the Firefox build system for Tor Browser.

Comment 8

3 years ago
I've attached a couple fixes we had to do to the build system (beyond using libfaketime and Gitian to provide a controlled environment, which probably gives us quite a bit in terms of build paths, hostname, package versions, etc as described in the technical details post that Gregory Szorc linked in Comment 5).

On Linux, we also had to remove the FIPS140 .chk library signatures for NSS, as these were created using a throwaway private key during the build process and are not reproducible.

For Windows and Mac, we use MinGW-W64 and a fork of 'toolchain4', respectively, to cross-compile for those platforms in Gitian in a reproducible way.
(Reporter)

Comment 9

3 years ago
Mike: Could you please open new bugs for each attachment so we can discuss each issue separately? This is intended to be a tracking bug with only high-level discussion and status updates. Feel free to ask for assistance in #build on irc.mozilla.org.
Flags: needinfo?(mikeperry)
Also rebase on mozilla-central, because i'm sure some of the things in the second patch are already fixed.

Updated

3 years ago
Whiteboard: [tor]

Updated

3 years ago
Depends on: 981558

Updated

3 years ago
Depends on: 942091

Updated

3 years ago
Depends on: 943331

Updated

3 years ago
Depends on: 982055

Updated

3 years ago
Depends on: 982075
(Reporter)

Comment 11

3 years ago
I just discovered Gentoo Prefix (https://www.gentoo.org/proj/en/gentoo-alt/prefix/). It makes it possible to install Gentoo (a Linux distro distinguished by how much it relies on compiling from source) on pretty much any UNIX-like operating system, including OS X.

It got me thinking that we could potentially use Gentoo Prefix to produce our build chroot/container archives/images. Not sure if it has any compelling advantages over Gitian (which I would assume to be the frontrunner at this stage). Just throwing the idea out there.
Duplicate of this bug: 1036178

Comment 13

3 years ago
There is also an ongoing project for Debian, if it helps. https://wiki.debian.org/ReproducibleBuilds

Comment 14

3 years ago
Did you consider creating builder with Docker?
(Reporter)

Comment 15

3 years ago
(In reply to Adam Stankiewicz from comment #14)
> Did you consider creating builder with Docker?

Efforts (not tracked by this bug) are underway to modernize Firefox's building infrastructure. This includes support for building in containers / Docker.

Comment 16

3 years ago
(In reply to Gregory Szorc [:gps] (away Sep 10 through 27) from comment #15)
> (In reply to Adam Stankiewicz from comment #14)
> > Did you consider creating builder with Docker?
> 
> Efforts (not tracked by this bug) are underway to modernize Firefox's
> building infrastructure. This includes support for building in containers /
> Docker.

@Gregory, I realize there's been a lot of noise about deterministic builds (e.g. bug 1036178).  However, I am curious, is there any status update on this?  You said "Efforts are underway", are those efforts detailed anywhere?  Can the community do anything to help move the efforts forward?
(Reporter)

Comment 17

3 years ago
catlee: Could you please provide a status update?
Flags: needinfo?(catlee)
From the infrastructure side, we've started work on building docker containers for FirefoxOS. This work should be a good basis for use for Firefox builds as well. I can't comment to any changes required by the build system.
Flags: needinfo?(catlee)
Assignee: nobody → winter2718
Depends on: 1115874
Duplicate of this bug: 1166201
Duplicate of this bug: 1083277
Summary: [meta] Deterministic, bit-identical and/or verifiable Linux builds → [meta] Deterministic, reproducible, bit-identical and/or verifiable Linux builds
Depends on: 1166243
See Also: → bug 1166208
Depends on: 1166538
Depends on: 1166547
Depends on: 1166550
Depends on: 1166554
Depends on: 1168231
Depends on: 1168316
Depends on: 1169158
Depends on: 1169174
Alias: fx-reproducible-build
Assignee: winter2718 → nobody

Updated

9 months ago
I was asked to look at the current state of this bug since it's been a while.

For Linux, we currently have nearly bit-identical builds when specifying the buildid by exporting MOZ_BUILD_DATE, except for the following:

 1) The NSS .chk files are always different (see #c8).
 2) Timestamps of the files inside the .tar.bz2 package will differ, but untarring them and using a recursive diff will reveal no differences (except for the aforementioned .chk files)
 3) PGO

For 1), it's possible we can just remove them from the built package. We already don't ship them on OSX (see the thread starting here: https://bugzilla.mozilla.org/show_bug.cgi?id=1096494#c7). It's not clear if we still want to support FIPS-mode, or if it is even possible given that we ship with NSS from our tree rather than something that's be FIPS-certified. This is probably worth looking into so we can remove that packaging code and make diffing packages easier. We might just need someone senior enough to sign off on removing it.

For 2) I'm not sure how much we care about generating two tarballs with the same hash. I think it's pretty straightforward to untar and then diff to determine equality, personally. However, it would be helpful if we had a clear idea of the end-goal for this bug to know whether this is something we should support or not.

I believe this is part of what "Disable library timestamping" patch is trying to fix, along with use of libfaketime. The "Misc reproducibility fixes" patch has already been absorbed by other bugs (bug 982075, bug 1168316, and bug 943331).

Part 3) is likely the bulk of the remaining effort, since our releases use PGO, and PGO is not currently reproducible. For this we need bug 935637, so that anyone who wants to verify a build can use the same PGO profile that we used to build a release. In addition to publishing PGO profiles, this may involve some build system work so that a PGO build can be done using an existing profile rather than first generating one.

Note that all this only applies to Linux builds - I haven't investigated other platforms yet.
(Reporter)

Comment 22

8 months ago
While we may have (mostly) deterministic builds, there is still the issue of reproducible. That requires a build environment that is reproducible. There is still a lot of work to do here.

While TaskCluster is using Docker images on Linux now, Docker images are notoriously not very reproducible (because `yum update/install` installs the latest version of packages advertised on servers and that can change over time). We /could/ publish the Docker images Mozilla uses (they are probably already public for all I know). However, paranoid people will want to reproduce those independently. It's turtles all the way down of course. The question is how far do we want to go.

That's why we need an end-goal for this bug :)

Comment 23

8 months ago
I started a discussion of what a good end-goal could be https://groups.google.com/forum/#!topic/mozilla.dev.platform/dFCNZC_pnq4
(Reporter)

Updated

8 months ago
Depends on: 1288610
(Reporter)

Updated

8 months ago
Depends on: 1289812
Blocks: 1166208
(Reporter)

Updated

2 months ago
Depends on: 1330608

Updated

a month ago
Depends on: 1341674

Updated

18 days ago
Blocks: 1325617
You need to log in before you can comment on or make changes to this bug.