1857470 - Build fails on Python 3.12; the vendored pip 23.0.1 uses obsolete pkgutil.ImpImporter

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2024-10-06)]

Updating pip should fix the issue.

Comment 1

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2024-10-06)]

Attachment: Bug 1857470 - Part 1: Vendor pip 23.2.1 r=#firefox-build-system-reviewers

Comment 2

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2024-10-06)]

Attachment: Bug 1857470 - Part 2: Use pip 23.2.1 wheel in mozharness r=#firefox-build-system-reviewers

Depends on D190284

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2d95b8f16d39
Part 1: Vendor pip 23.2.1 r=firefox-build-system-reviewers,sergesanspaille
https://hg.mozilla.org/integration/autoland/rev/2f6d9a971e18
Part 2: Use pip 23.2.1 wheel in mozharness r=firefox-build-system-reviewers,sergesanspaille

Comment 4

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/2d95b8f16d39
https://hg.mozilla.org/mozilla-central/rev/2f6d9a971e18

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3289f416aab7
downgrade pip from 23.2.1 to 23.0.1 to revert performance regression (bug 1858663)
https://hg.mozilla.org/integration/autoland/rev/d87f65ca5a6f
switch back to pip 23.0.1 in mozharness because of performance regression (bug 1858663)

Comment 6

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2024-10-06)]

How can we confirm this has no issue when landing this again?

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

One can run this query after the tasks completed and the change in CPU time spent will be obvious for the area with the higher values.
For testing, needinfo sheriffs@ and it can be added to a central-as-beta simulation and it can be compared manually afterwards with this query - works also between any two Try pushes with the same set of tasks,

Comment 8

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/3289f416aab7
https://hg.mozilla.org/mozilla-central/rev/d87f65ca5a6f

Comment 10

[Intermittent Failures Robot]

1 failures in 4224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1857470&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 11

[Andrew Halberstadt [:ahal]]

Pasting a comment from bug 1868690 over here for more visibility:

I'm not surprised that adding pyproject.toml didn't fix things. We still need to build the wheels regardless.
From pip install --help:

Use PEP 517 for building source distributions (use --no-use-pep517 to force legacy behaviour).

So our options are to either:

• Pass --no-use-pep517 into our pip install command (unsure if this will be removed in the future or not)
• Have the build tasks build the wheels as part of the package-tests step

Given this is blocking Python 3.12 support, I proposed we do 1 and file a follow-up for 2.

Lmk if you need any help with this!

Comment 12

[Mike Hommey [:glandium]]

Given this is blocking Python 3.12 support

Is it, though? With bug 1869690 a bootstrap + a full build can go through. I don't know about other commands, I haven't tried, but I don't think pip is blocking anything at the moment, really.

Comment 13

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2024-10-06)]

I can confirm that this doesn't block build on Py3.12 anymore.

Comment 14

[Andrew Halberstadt [:ahal]]

Ah, I see. It looks like we directly edited the vendored pip though, so this is still pretty high priority as that will get blown away next time we need to update Python packages.

Aside: I wonder if it's worth supporting a .patch mechanism in ./mach vendor like updatebot has.

Comment 15

[Jim Blandy :jimb]

A similar failure seems to have been re-introduced: Bug 1876121

Comment 16

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

(In reply to Jim Blandy :jimb from comment #15)

A similar failure seems to have been re-introduced: Bug 1876121

As :ahal mentioned in the previous comment, the hack to fix it got blown away by an unrelated dependency update. I don't mind re-approving the hack, but what's the timeline on the actual fix? I'm not keen on doing this again and again long term. We could add it to the excluded packages list and the hack won't get blown away again, but that's sounds like a recipe for this being forgotten about indefinitely, so I'm not keen on this either, but it might be the lesser of two evils.

Comment 17

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2024-10-06)]

Perhaps we can do the option 1 of your this previous comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1868690#c3

This is not on my radar right now, perhaps some build team member can take a look?

Edit: Oh wait you are not Andrew, modified the comment

Comment 18

[Emilio Cobos Álvarez (:emilio)]

Build seems to work on python 3.12 here at least... Was this fixed elsewhere?

Comment 19

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

D200106 and D199369 combined are the hacks that are enabling Python 3.12 for now. I haven't gotten around to figuring out a proper solution yet. This hack will probably stick around for a while longer.

Comment 20

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

I think D201231 resolved the blocker, and we should just be able to upgrade pip/setuptools now. I'll get a patch up.

Comment 21

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1857470 - Vendor `pip` at version `24.0` r?#build

Comment 22

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1857470 - Vendor `pip-tools` at version `7.4.1` r?#build

While nothing explicitly prompted updating this, letting pip and
pip-tools diverage too greatly in version release dates seems like
a bad idea, especially with the various deprecations in
pip and python3.12.

Comment 23

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1857470 - Use pip `24.0`, setuptools `70.0.0` wheels and add wheel `0.43.0` in mozharness r?#build

Comment 24

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1857470 - Vendor `wheel` at version `0.43.0` r?#build

We need to use wheel in mozharness to use --no-use-pep517 (so we can
use a modern pip). We don't explicitly need to upgrade wheel, but it
seems sensible to upgrade it to a similar baseline as both pip and
setuptools (which we also need are are upgrading with this stack).

Comment 25

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1857470 - Vendor `setuptools` at version `70.0.0` r?#build

Same rationale as vendoring of a recent version of wheel. We need it
in mozharness for --no-use-pep517 so that we can use a recent pip,
so we'll update to a similar baseline to minimize issues.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/734408c526ab
Vendor `pip` at version `24.0` r=mach-reviewers,ahal
https://hg.mozilla.org/integration/autoland/rev/be779cd3d824
Vendor `pip-tools` at version `7.4.1` r=mach-reviewers,ahal
https://hg.mozilla.org/integration/autoland/rev/25aa13608af7
Vendor `wheel` at version `0.43.0` r=firefox-build-system-reviewers,mach-reviewers,sergesanspaille,ahal
https://hg.mozilla.org/integration/autoland/rev/8eed3936a087
Vendor `setuptools` at version `70.0.0` r=firefox-build-system-reviewers,mach-reviewers,sergesanspaille,ahal
https://hg.mozilla.org/integration/autoland/rev/6a6614edc627
Use pip `24.0`, setuptools `70.0.0` wheels and add wheel `0.43.0` in mozharness r=mach-reviewers,ahal

Comment 27

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/734408c526ab
https://hg.mozilla.org/mozilla-central/rev/be779cd3d824
https://hg.mozilla.org/mozilla-central/rev/25aa13608af7
https://hg.mozilla.org/mozilla-central/rev/8eed3936a087
https://hg.mozilla.org/mozilla-central/rev/6a6614edc627