1426324 - Toolchain indexes should vary depending on the docker-image they were built on

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Mike Hommey [:glandium]]

Example of problem this has just caused me:
- Made changes to build-gcc script *and* switched the gcc build task to a docker image based on Debian.
- This resulted in a toolchain slighly different from what would have been produced on Centos (for instance, built on Debian, we have a gcc/lib32 path, but that path doesn't exist when built on Centos, it's gcc/lib instead)
- I did a subsequent push with the same build-gcc script changes, but *without* the switch to a docker image based on Debian.
- The build tasks still got the GCC toolchain with the gcc/lib32 path.

The problem is that by the time we compute a hash for the index during transformations, we don't actually have access to the docker image to possibly add its hash to the equation.

Dustin, do you have thoughts?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Maybe just pick new names for these new docker images?

Comment 2

[Mike Hommey [:glandium]]

(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> Maybe just pick new names for these new docker images?

And add the docker image name to the hash? That seems footgun-y but less so than the current status quo, so I can get behind this.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, I'm not sure this case is worth a lot of heroics..

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1426324 - Make toolchain cache indexes vary depending on the docker image name.

Review commit: https://reviewboard.mozilla.org/r/209018/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/209018/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8938257 [details]
Bug 1426324 - Make toolchain cache indexes vary depending on the docker image name.

https://reviewboard.mozilla.org/r/209018/#review214888

::: taskcluster/taskgraph/transforms/job/toolchain.py:97
(Diff revision 1)
> +    # the index path as well. Ideally, the content of the docker image itself
> +    # should have an influence, but at the moment, we can't get that
> +    # information here. So use the docker image name as a proxy. Not a lot of
> +    # changes to docker images actually have an impact on the resulting
> +    # toolchain artifact, so we'll just rely on such important changes to be
> +    # accompanied with a docker image name change.

I like this reasoning :)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f41ca59052be
Make toolchain cache indexes vary depending on the docker image name. r=dustin

Comment 7

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/f41ca59052be

Comment 8

[Mike Hommey [:glandium]]

Had to back this out temporarily to work around bug 1421100 not being deterministic.

Comment 9

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c1e20de19b6f
Backout changeset f41ca59052be in an attempt to unbust win32 builds. a=bustage on a CLOSED TREE

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/841178c05781
Make toolchain cache indexes vary depending on the docker image name. r=dustin

Comment 11

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/841178c05781