1577212 - Create a Linux build job that compares two builds to detect reproducible regressions

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Tom Ritter [:tjr]]

We recently made Linux builds reproducible (https://glandium.org/blog/?p=3923) - it would be neat to detect regressions to this by comparing them. We could do a fuzzy comparison using a script that ignores expected, innocuous differences like signatures and timestamps.

Comment 1

[Mike Hommey [:glandium]]

Duplicate the linux64-shippable/opt task, and edit taskcluster/ci/diffoscope/kind.yml to add a comparison job.

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1577212 - Add diffoscope jobs ensuring reproducibility of the linux shippable builds.

Comment 3

[Mike Hommey [:glandium]]

The patch is trivial. The problem is it's not landable: reproducibility is already broken.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:glandium, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1577212 - Add diffoscope jobs ensuring reproducibility of the linux32 shippable builds.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/358e2c496639
Add diffoscope jobs ensuring reproducibility of the linux32 shippable builds. r=mshal

Comment 7

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/358e2c496639

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

why is this on linux32 and not linux64?

Comment 9

[Mike Hommey [:glandium]]

See the dependencies on this bug.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:rstewart, maybe it's time to close this bug?

Comment 11

[Ricky Stewart]

Is there progress being made on the dependencies? Wondering if we should close as WONTFIX or just leave this open indefinitely.

Comment 12

[Tom Ritter [:tjr]]

Yes, we are still pursuing this.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:mhentges, maybe it's time to close this bug?

Comment 14

[Mitchell Hentges [:mhentges] 🦀]

Based on the previous response by :tjr, this is still valuable.

Comment 15

[(Away)]

Hm, I thought we had this already for ages. Why the leave-open? Is that to cover the fact that we only do 32-bit and still want 64-bit?

Comment 16

[Mike Hommey [:glandium]]

yes

Comment 17

[(Away)]

It looks like the blocker for 64-bit builds is wasm sandboxing (at least with clang 12, which has been really good wrt intermittents on Linux32). The ​lucet_trap_table symbols appear in nondeterministic order: https://firefoxci.taskcluster-artifacts.net/NnHtHnWFT4e3XWBD9pMgqQ/0/public/diff.html

Can I interest you in digging into that? Or maybe as baby steps we can zap those libraries with pre-diff-commands?

Comment 18

[Tom Ritter [:tjr]]

Tagging the rlbox folks in case they think the lucet thing is easy or could give a pointer.

Comment 19

[Georg Koppen]

(In reply to Tom Ritter [:tjr] (ni? for response to sec-[advisories/bounties/ratings/cves]) from comment #18)

Tagging the rlbox folks in case they think the lucet thing is easy or could give a pointer.

I supposed https://bugzilla.mozilla.org/show_bug.cgi?id=1612035 is the one for this particular issue, no?

Comment 20

[Mike Hommey [:glandium]]

(In reply to :dmajor from comment #17)

It looks like the blocker for 64-bit builds is wasm sandboxing (at least with clang 12, which has been really good wrt intermittents on Linux32). The ​lucet_trap_table symbols appear in nondeterministic order: https://firefoxci.taskcluster-artifacts.net/NnHtHnWFT4e3XWBD9pMgqQ/0/public/diff.html

Can I interest you in digging into that? Or maybe as baby steps we can zap those libraries with pre-diff-commands?

The blocker(s) are in the bug dependencies and are not related to wasm sandboxing (although that's also another problem). They also were 64-bits only. They may or may not have been fixed in clang 11 or 12, but that would be something to test.

Comment 21

[(Away)]

(In reply to Mike Hommey [:glandium] from comment #20)

The blocker(s) are in the bug dependencies and are not related to wasm sandboxing (although that's also another problem). They also were 64-bits only. They may or may not have been fixed in clang 11 or 12, but that would be something to test.

Bug 1596350 comment 2 says it's 32-bit only.
I don't see an explicit mention of bitness in bug 1596283, but it's not reproducing with clang 11 for me.

Let's start running tasks for 64-bit builds, minus lib*wasm.so. Spinoff bug incoming.

Comment 22

[Shravan Narayan]

Re lucet reproducible builds --- unfortunately, I don't have too much info in this. I know that the C->Wasm compilation via clang is reproducible, so the problem is definitely inside lucet. Fwiw, I believe the codegen of lucet is first party code, but the object file output is based on third part rust components. I think the only info I have is that the first step is to probably figure out if this is due to first party Lucet code vs 3rd party object file outputs.

Comment 23

[(Away)]

The main (only?) difference I see in this diff is that the lucet_trap_table_guest_func_... are not in consistent order. Those symbols appear to be written out by write_trap_tables but it's not clear to me which is the origin of the problem: is write_trap_tables at fault for not doing some sorting right there and then, or should some earlier piece of the pipeline be on the hook for feeding in the data in stable order? This is getting outside my familiarity, Shravan any chance you might have any ideas here?

Comment 24

[Shravan Narayan]

We will be addressing this soon with https://bugzilla.mozilla.org/show_bug.cgi?id=1720828

Comment 25

[Mike Hommey [:glandium]]

We can actually close this bug, because both the linux and linux64 jobs exist and are tier 2 right now.