1692400 - updatebot docker image fails to build: error: Can not find Rust compiler

Status: RESOLVED FIXED

(Developer Infrastructure :: Mach Vendor & Updatebot, defect)

Description

[Mike Hommey [:glandium]]

The docker image is rebuilt when one of its ancestors is rebuilt, and when that happens, it fails with:

    copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.7/cryptography
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-3.7/_padding.c'
    creating build/temp.linux-x86_64-3.7
    generating cffi module 'build/temp.linux-x86_64-3.7/_openssl.c'
    running build_rust
    
        =============================DEBUG ASSISTANCE=============================
        If you are seeing a compilation error please try the following steps to
        successfully install cryptography:
        1) Upgrade to the latest pip and try again. This will fix errors for most
           users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
        2) Read https://cryptography.io/en/latest/installation.html for specific
           instructions for your platform.
        3) Check our frequently asked questions for more information:
           https://cryptography.io/en/latest/faq.html
        4) Ensure you have a recent Rust toolchain installed:
           https://cryptography.io/en/latest/installation.html#rust
        5) If you are experiencing issues with Rust for *this release only* you may
           set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
        =============================DEBUG ASSISTANCE=============================
    
    error: Can not find Rust compiler

This is due to the recent change in the python cryptography module, that now has a dependency on rust. It wouldn't be a problem if they provided wheels for python 3.7, but they don't.

Comment 1

[Mike Hommey [:glandium]]

This raises the problem that what's installed in the docker image depends on when the docker image is built, which is also not great. Ideally, we'd have versions frozen, and checked with sha256s. In this instance, that would also allow to pin to a version that doesn't require rust.

Comment 2

[Mike Hommey [:glandium]]

(Upgrading pip would be another workaround)

Comment 3

[Tom Ritter [:tjr]]

I think what's happening is that python-requests is pulling in python-cryptography. Updatebot itself does pin versions in poetry.

What do you recommend about this situation? It seems like this is specific instance of a generic problem we have with our Dockerfiles and just happened to manifest here vs in some other Dockerfile?

From my limited understanding, apt supports doing something like apt-get install foo=<version> which would pin by version (but not SHA.) But I don't think we do that anywhere else in-tree. I'm not sure how to pin apt packages by SHA.

I'm happy enough with a workaround if this is something that's going to need to be tackled eventually for all our Dockerfiles...

Comment 4

[Mike Hommey [:glandium]]

(Note that one reason we use Debian in docker images is that apt packages don't change much, this is not a case of apt going rogue ; Also note that in the case of this particular image, it's not based on the snapshot-based Debian images, but that's not really a concern, although ideally, this would need fixing, but there's no rush)

I realize I had forgotten to link to the full log:
https://treeherder.mozilla.org/logviewer?job_id=329742341&repo=try&lineNumber=3189

This all starts from python3 -m pip install poetry... ironic.

Comment 5

[Tom Ritter [:tjr]]

Attachment: Bug 1692400: Update pip before using it r?glandium

This enables us to get a wheel for python cryptography instead of
building it ourselves, which breaks because we don't have a rust
compiler.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c8dd84d00ce2
Update pip before using it r=glandium

Comment 7

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/c8dd84d00ce2

Comment 8

[Tom Ritter [:tjr]]

Re-assigning bugs to the new Mach Vendor and Updatebot Component. Guid for this change: ade7b229-ec85-41f8-bafe-9d4451bfeed1

Comment 9

[Tom Ritter [:tjr]]

A mass updatebot change inadvertently added a security group to bugs. Removing it. Guid for this change: 09d63db4-11a1-4820-a398-d718f112f885