Open Bug 1621157 Opened 5 years ago Updated 2 years ago

Build system backends should be reproducible and we should have automated testing to ensure this

Categories

(Firefox Build System :: General, task)

Product:
Firefox Build System
Component:
General
Type:
task

Tracking

(Not tracked)

People

(Reporter: rstewart, Unassigned)

References

Details

One would expect that running ./mach configure/./mach build-backend multiple times without changing any files in tree would always produce the same build backend files, but we haven't generally bothered to maintain this environment. I've tried to fix the majority of the major issues in bug 1617748 but 1) I may have missed some stuff and 2) there's no guarantee things won't regress in the future.

We should have something more robust in place to detect and prevent regressions in reproducibility in the future. Anything from additional unit tests to further integration/end-to-end tests that specifically check diffs between subsequent runs of build-backend can work here.

See Also: → 1620429

Would it make sense to include 'export' as well? That would have helped avoid issues with sccache misses due to python3 re-ordering some header files, for example. Though it wouldn't catch all generated files, since some are done during the compile/misc tier.

Every generated file/export/etc. SHOULD be reproducible AFAIK, and it makes sense to include generated files under the purview of this bug where it makes sense.

I'm not really sure what this automated testing would look like right now, so if we come up with a design that works for that stuff as well as build backends, that would be great news :)

I think it could be similar to the reproducible-linux32 job, which depends on two build jobs and compares the tarballs. Only instead of comparing the output of 'mach package', we'd need to generate a separate artifact after build-backend (or export, to do it all at once) that bundles up the objdir. The equivalent of the reproducible-linux32 task would then be diffing the two objdirs instead of two packages. Testing locally, tarring the objdir after export is less than 4 seconds and makes a 26MB artifact, so it could probably be added to an existing build. Then the diff just needs to ignore some files:

diff -x ".pyc" -x last_log.json -x build_resources.json -x buildid.h -x config.log -x application.ini -x application.ini.h -x ".pp" -x "*.stub" -r objdir-1 objdir-2

Some of those only differ because of the buildid - if the reproducible-linux32 jobs already enforce the same buildids, then they don't need to be skipped.

That artifact exists in most part: target.generated-files. The reproducible-linux32 job even checks it... but only when the firefox package differs.

I think it would be reasonable to split the check of firefox package and generated files in two different tasks, and run both all the time.

(In reply to Mike Hommey [:glandium] from comment #4)

That artifact exists in most part: target.generated-files. The reproducible-linux32 job even checks it... but only when the firefox package differs.

Oh, awesome. Is there any reason we couldn't stuff the output from build-backend into that artifact as well?

(In reply to Michael Shal [:mshal] from comment #6)

(In reply to Mike Hommey [:glandium] from comment #4)

That artifact exists in most part: target.generated-files. The reproducible-linux32 job even checks it... but only when the firefox package differs.

Oh, awesome. Is there any reason we couldn't stuff the output from build-backend into that artifact as well?

It would be good to be sure it doesn't bother whatever else than the diff tasks consumes those artifacts. Worst case scenario, we create a separate artifact with the build system files only. BTW, we have config.status already, whichever we end up doing, it could stop being its own separate artifact.

The generated-files tarball comes from bug 1259832 and is consumed for crash reports.

See Also: → 1621198, 1621452
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.