1748737 - ./mach mochitest fails with "Not uninstalling attrs at .third_party/python/attrs, outside environment ./obj-x86_64-pc-linux-gnu/_virtualenvs/common"

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Andreas Pehrson [:pehrsons]]

Attachment: mach_mochitest.txt

I'm on a pretty recent m-c (hg) checkout (8bc2581b2c7b; Jan 3) on Ubuntu 20.04 and running just ./mach mochitest fails due to an outdated dependency on attrs.

I'll attach the log but the interesting bits are:

Collecting attrs>=19.2.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
(...)
Installing collected packages: attrs
Attempting uninstall: attrs
Found existing installation: attrs 19.1.0
Not uninstalling attrs at /home/pehrsons/dev/m-c-3/third_party/python/attrs, outside environment /home/pehrsons/dev/m-c-3/obj-x86_64-pc-linux-gnu/_virtualenvs/common
Can't uninstall 'attrs'. No files were found to uninstall.
Successfully installed attrs-21.4.0
twisted 21.7.0 has requirement attrs>=19.2.0, but you have attrs 19.1.0.
automat 20.2.0 has requirement attrs>=19.2.0, but you have attrs 19.1.0.
Error running mach:

So it fetches the correct attrs and tries to install it, but finds the attrs we have vendored into the tree which is too old and barfs.

Comment 1

[Andrew Halberstadt [:ahal]]

This is stemming from the installation here:
https://searchfox.org/mozilla-central/rev/6eab79ab46c362b0191c4dad919f4cea2d6efb40/testing/mochitest/mach_commands.py#411

There were significant changes to our dependency handling recently, and I'd guess that we now prioritize the vendored version over the one we install here. Maybe Mitch can confirm what's going on.

Hopefully we can upgrade the vendored attrs to at least 19.2.0 without too much trouble to fix this.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

There were significant changes to our dependency handling recently, and I'd guess that we now prioritize the vendored version over the one we install here. Maybe Mitch can confirm what's going on.

That's correct, the vendored attrs may have been imported by the time mochitest runs, so we prioritize it.
I'd recommend either bumping attrs to 19.2.0, or downgrading twisted/automat.
(from light local testing a few months back, bumping attrs to 19.2.0 was pretty low-risk. I'll pull the patches up and assign them here).

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1748737: Bump attrs to be compatible with mochitest

A bunch of modern packages (pytest, twisted, automat) all need attrs==19.2.0 (or newer).
We could bump attrs all the way to the modern 21.4.0 version, but I'd like to defer that upgrade risk, since there's a
lot of backwards-incompatible changes and deprecations. So, lightly bump it to 19.2.0.

Mitchell Hentges <[e-mail]> HG: branch 'default' HG: changed third_party/python/requirements.in HG: changed
third_party/python/requirements.txt

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1748737: [WILL-SQUASH] Attr version bump vendoring

Depends on D135178

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/65842984903e
Bump attrs to be compatible with new pytest r=ahal

Comment 6

[Iulian Moraru]

Backed out for causing multiple py3 failures.

Push with failures

Failure log for mach
Failure log for mbu
Failure log for tg

Backout link

Comment 7

[Phabricator Automation]

Comment on attachment 9257746 [details]
Bug 1748737: Bump attrs to be compatible with mochitest

Revision D135178 was moved to bug 1732795. Setting attachment 9257746 [details] to obsolete.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

This is at the top of my stack tomorrow, I'm still working on this :)

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

There were issues around pluggy being too old, and now that's been bumped (and dependencies added), pytest is complaining about a bunch of behaviours we're doing that are deprecated. More work is needed tomorrow.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1748737: Update deprecated pytest code

pytest is about to be upgraded, so this patch resolves a bunch of the
issues currently blocking the upgrade.
Most changes done in this patch are described in the "deprecations" page
on the pytest docs:
https://docs.pytest.org/en/latest/deprecations.html#using-class-in-custom-collectors

One undocumented change is that defining a pytest argument as the same name as an
in-scope, imported fixture function, will no longer use that imported function.

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

Attachment: WIP: Bug 1748737: [WILL-SQUASH] Attr version bump vendoring

Depends on D135178

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Hopefully we can upgrade the vendored attrs to at least 19.2.0 without too much trouble to fix this.

Hah, this needed a bunch of pytest changes, as well as some docker Python requirements fiddling. A non-trivial amount of trouble, but should be ready now :)

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/36254094fb78
Update deprecated pytest code r=webdriver-reviewers,ahal,whimboo
https://hg.mozilla.org/integration/autoland/rev/984572e6da8a
Bump attrs to be compatible with mochitest r=ahal

Comment 15

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/36254094fb78
https://hg.mozilla.org/mozilla-central/rev/984572e6da8a