1690215 - Building backend container fails

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ionuț Goldan [:igoldan]]

Since yesterday I noticed that running docker-compose up --build fails with the following error:

[...]
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    markupsafe==1.1.1 from https://files.pythonhosted.org/packages/c2/37/2e4def8ce3739a258998215df907f5815ecd1af71e62147f5eea2d12d4e8/MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl#sha256=a6a744282b7718a2a62d2ed9d993cad6f5f585605ad352c11de459f4108df0a1 (from -r requirements/common.txt (line 252)):
        Expected sha256 00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473
        Expected     or 
[...]
        Expected     or e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be
             Got        a6a744282b7718a2a62d2ed9d993cad6f5f585605ad352c11de459f4108df0a1

The command '/bin/sh -c pip install --no-cache-dir --disable-pip-version-check --require-hashes -r requirements/common.txt' returned a non-zero code: 1
ERROR: Service 'backend' failed to build
ERROR: InvocationError for command /usr/local/bin/docker-compose build (exited with code 1)
docker run-test-post: commands[0] | docker-compose down
Removing network project_default
WARNING: Network project_default not found.
___________________________________ summary ____________________________________
ERROR:   docker: commands failed


Exited with code exit status 1

CircleCI received exit code 1

More precisely, the backend container is the one experiencing this issue.
Because of this, we cannot run our backend tests, test any local changes or review someone else' s PR.

We're experiencing this not only with latest master, but also when checking out various (older) revisions.

Note: in case there are issues reproducing this, make sure you have a clean environment first, by running docker system prune --all --volumes --force.

Comment 1

[Ionuț Goldan [:igoldan]]

Sarah, could you look over this issue?
Thanks in advance!

Comment 2

[Ionuț Goldan [:igoldan]]

To the looks of it, the markupsafe==1.1.1 dependency seems to have changed its content, as our old hashes don't match its upstream ones anymore.

Comment 3

[Ionuț Goldan [:igoldan]]

Following part of Kyle' s instructions from this requirements README.md, I've only updated the hashes for markupsafe package.
This seems to fix the issue, but I don't know if it's the ideal solution. Still, I'm attaching the patch in case it is.

Comment 4

[Ionuț Goldan [:igoldan]]

Attachment: update-markupsafe-hashes.diff

Comment 5

[Ionuț Goldan [:igoldan]]

Looking over the markupsafe' s wheels, :davehunt noticed some of them have been updated recently.
Specifically, starting from Jan 29, 2021 (matching the time we started experiencing this).

Previously, we've been using the MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl wheel (matching our current hashes).
Now we're using the new MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl wheel, which has different hashes.

Looking more into it, Dave noticed 2 recent markupsafe PRs which may have impacted this & are just 4 days old:

• https://github.com/pallets/markupsafe/pull/173#issuecomment-770110247
• https://github.com/pallets/markupsafe/pull/163/files

Comment 6

[Sarah Clements [:sclements]]

I have a pr to update the hashes but I'm hitting an issue (btw, it'd be a big help to open a pr for a fix when these types of issues occur): https://github.com/mozilla/treeherder/pull/7007

I'll resume working on this when I'm back online.

Comment 7

[Ionuț Goldan [:igoldan]]

Alex Irimovici submitted PR 7008 for this, which contains the minimal changes.

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Merged as https://github.com/mozilla/treeherder/commit/c3ca563822ac5643efac217d491e991d87cdc542