Closed Bug 1459980 Opened 2 years ago Closed Last year

google-play-strings image is non-deterministic

Categories

(Firefox Build System :: Task Configuration, task)

3 Branch
task
Not set

Tracking

(firefox61 fixed, firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox61 --- fixed
firefox62 --- fixed

People

(Reporter: gps, Assigned: jlorenzo)

References

Details

Attachments

(1 file)

The google-play-strings Docker image is non-deterministic because it does a `git clone https://github.com/mozilla-releng/mozapkpublisher` and implicitly uses the commit on the master branch.

This means that as the master branch changes, the Docker image may or may not build due to shifting requirements.

This bit us in bug 1459737. We changed a file that triggered rebuilding google-play-strings and the image failed to build because something in Python requirements land started requiring libfreetype, libxml2, libxslt, libpng, and pkg-config since the last time the image was built.

I worked around this by adding the missing requirements to the Dockerfile. But this is only a workaround: if mozapkpublisher changes requirements, we could break the next time we build google-play-strings.

The fix is pin the checked out commit SHA-1 in the google-play-strings Dockerfile so behavior is deterministic over time.

This means that any upstream changes to mozapkpublisher that impact tasks using google-play-strings may require bumping the pinned commit in the google-play-strings Dockerfile.
Flags: needinfo?(jlorenzo)
Thank you for calling this out! I'm sorry for the trouble it cause. mozapkpublisher uses tags which we can rely on. I just made a simple fix to get this out.

Speaking of which, I see the image pulls a lot of unnecessary dependencies (this docker image is just made to make some HTTPS requests and write a file). I'll have a closer look at it.
Assignee: nobody → jlorenzo
Flags: needinfo?(jlorenzo)
Comment on attachment 8976229 [details]
Bug 1459980 - google-play-strings: pin mozapkpublisher to 0.7.2

https://reviewboard.mozilla.org/r/244426/#review250432

::: taskcluster/docker/google-play-strings/Dockerfile:24
(Diff revision 1)
>      python3-dev \
>      python3-setuptools
>  
>  WORKDIR /builds/worker/
> -RUN git clone https://github.com/mozilla-releng/mozapkpublisher
> +# Change "--branch $tag" to point to a newer tag
> +RUN git clone --branch 0.7.2 --depth=1 https://github.com/mozilla-releng/mozapkpublisher

Can this use a specific commit instead? Branch names (and tags) are not deterministic over time. If someone updated the 0.7.2 branch/tag, results could change.

Introducing a SHA-1 collision on the full Git commit much harder :)
Blocks: 1459181
Comment on attachment 8976229 [details]
Bug 1459980 - google-play-strings: pin mozapkpublisher to 0.7.2

https://reviewboard.mozilla.org/r/244426/#review250720

::: taskcluster/docker/google-play-strings/Dockerfile:24
(Diff revision 1)
>      python3-dev \
>      python3-setuptools
>  
>  WORKDIR /builds/worker/
> -RUN git clone https://github.com/mozilla-releng/mozapkpublisher
> +# Change "--branch $tag" to point to a newer tag
> +RUN git clone --branch 0.7.2 --depth=1 https://github.com/mozilla-releng/mozapkpublisher

Okay. In this project we use tags as immutable. I understand an error may come up. I replaced it with its sha.
Comment on attachment 8976229 [details]
Bug 1459980 - google-play-strings: pin mozapkpublisher to 0.7.2

https://reviewboard.mozilla.org/r/244426/#review251126
Attachment #8976229 - Flags: review?(gps) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/940433abb9cf
google-play-strings: pin mozapkpublisher to 0.7.2 r=gps
https://hg.mozilla.org/mozilla-central/rev/940433abb9cf
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Landed on beta at https://hg.mozilla.org/releases/mozilla-beta/rev/c05faf09f5e0502cdba95c591b2a195ccd5d0676. This unblocks bug 1459181 for the next staging release.
Blocks: 1464445
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.