Closed Bug 1654457 Opened 4 years ago Closed 4 years ago

Upgrade vendored virtualenv

Categories

(Firefox Build System :: General, task)

task

Tracking

(firefox-esr78 fixed, firefox83 fixed)

RESOLVED FIXED
83 Branch
Tracking Status
firefox-esr78 --- fixed
firefox83 --- fixed

People

(Reporter: glandium, Assigned: hamzah18051)

References

(Blocks 1 open bug)

Details

Attachments

(2 files, 1 obsolete file)

I was trying to reproduce an automation test run on a mac under catalina, and it failed running virtualenv with an error that looks like the ones mentioned in https://github.com/pypa/virtualenv/issues/1561, which suggests it's fixed in later versions of virtualenv.

Flags: needinfo?(ahal)

It failed on Mojave too, but not on High Sierra.

Yes, upgrading would be nice. For context, there are two copies of virtualenv in-tree. One used by mozharness, and then the typical one under third_party/python. I'm not sure which copy of virtualenv you were using, but they are both below version 20. Bug 1408051 will get rid of the mozharness copy and should be landing soon (whether it sticks is another question).

So this should block on bug 1408051, though if it ends up being backed out due to virtualenv related issues, we could reverse the dependencies and fix this one first.

I won't have time to do this myself for awhile. I'll see if Hamzah is interested in taking this one after 1408051 is wrapped up.

p.s moving to Firefox Build System :: General as the virtualenv dependency is much broader than just testing.

Depends on: 1408051
Flags: needinfo?(ahal)
Product: Testing → Firefox Build System
Assignee: nobody → hamzah18051
Status: NEW → ASSIGNED
Attachment #9165485 - Attachment description: Bug 1654457 - Update virtualenv to latest release version 20.0.27 → Bug 1654457 - Update virtualenv to latest release version 20.0.28
Attachment #9165485 - Attachment description: Bug 1654457 - Update virtualenv to latest release version 20.0.28 → Bug 1654457 - Update virtualenv to latest release version
See Also: → 1660183

Just checking in because bug 1660183 is making this more important: what's the status of this work. The review suggests that we're blocked on removing pipenv (bug 1659542, itself blocked on bug 1659539), but then we can proceed as planned with this. Is that correct?

Flags: needinfo?(hamzah18051)

Yeah, you are correct, pipenv is blocking this.
Since virtualenv20 was one of the biggest refactors after virtualenv16 and was refactored into a full-fledged module. In venv16 there used to be virtualenv.py which used to be executed everytime a virtualenv was to be created, but in venv20 it was broken down into several different files as earlier it was difficult to maintain, because of that we could no longer call virtualenv.py and instead had to work on modifying the PYTHONPATH
So jezdez suggested that we use the venv zipapp file which is just a zip file that contains the total content of the virtualenv packages and script https://phabricator.services.mozilla.com/D84577#2640136.
So I think once we are done removing pipenv we can proceed with upgrading the virtualenv.

Flags: needinfo?(hamzah18051)

Okay, pipenv is gone, this work can continue.

Depends on: 1665675

Newer versions of pip output something like:
certifi @ file:///builds/worker/workspace/build/tests/tools/wpt_third_party/certifi
rather than
certifi==2018.4.16

and the package_versions method doesn't know how to handle that.

Both old and new versions of pip, however, output the same thing for
pip list --format freeze, which is equivalent to the old
pip freeze --all output (which means we'll now list more packages,
essentially pip, setuptools and wheels).

Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/9dfb511b5dc3
Use `pip list --format freeze` instead of `pip freeze`. r=ahal
https://hg.mozilla.org/integration/autoland/rev/91f4934e4063
Update virtualenv to 20.0.31. r=firefox-build-system-reviewers,mhentges,rstewart
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 83 Branch
Regressions: 1669807
Regressions: 1669934
Regressions: 1670167
No longer regressions: 1670167
Regressions: 1670039
Regressions: 1670784
Regressions: 1670788

Hi, may I know that whether this patch will be applied to ESR78 branch? If so, when will it happen? Thanks!

Sure, I'll nominate this patch and its dependencies for ESR uplift.
Heads up that there's a lot of dependencies (see the regressions list!), so this might not land smoothly. Let's find out :)

Comment on attachment 9179169 [details]
Bug 1654457 - Update virtualenv to 20.0.31.

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Common platforms (e.g.: macOS Mojave) are unable to run some ./mach commands.
  • User impact if declined: Harder for downstreams to perform builds on various platforms.
  • Fix Landed on Version: 83
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Is a build patch, failures should be found in CI.
    Application users are not affected.
  • String or UUID changes made by this patch:
Attachment #9179169 - Flags: approval-mozilla-esr78?
Attachment #9179170 - Flags: approval-mozilla-esr78?

@Mitchell, great. I am planning to upgrade our browser version from esr68 to esr 78 due to Mac Big Sur release, so I have to rebuild my code from scratch.

Here is another bug I have ever encountered when I try to build the source from scratch:
https://bugzilla.mozilla.org/show_bug.cgi?id=1674162
Since the code change is not that large, I merged it by hand in my local codebase, not sure whether it can be merged as well, that'll be good for other ESR users.

Attachment #9165485 - Attachment is obsolete: true

There's various conflicts along the way trying to graft this and dependencies to ESR78. Most of them are pretty trivial to rebase around, but I hit conflicts in bug 1670167 I wasn't sure how to deal with and feel it'd be better for someone who knows the build system better to handle them. Can you please try to create a rebased set of patches for uplift? If it makes it easier, we can graft them directly from a Try push rather than attaching various patches throughout all the different bugs.

Flags: needinfo?(mhentges)

I've got a stack of rebased patches on try here (tip is id 829aaf14f1c5).
I'm running local Mac, Linux and Windows builds right now, I'll report the status in an hour or two.

A couple notes:

  • I had to take some liberties with that last patch, just slorping the if PY2 bit and losing the rest.
  • For my local macOS dev, I have a newer SDK, so I locally grafted this patch. I didn't include it in my try stack because theoretically it shouldn't be needed, since downstream can use an older SDK.
  • There's issues around ./mach bootstrap and Big Sur that will be solved by this bug. However, this is workaroundable by not bootstrapping from the ESR78 checkout.
  • There's a lot of uplifted patches here. If this is an excessive amount, please let me know! I'm not sure the rules around this :)
Flags: needinfo?(mhentges)

Seems to be happy on my machines.
Should be good to go :RyanVM, let me know how it goes :)

Flags: needinfo?(ryanvm)

https://hg.mozilla.org/try/rev/829aaf14f1c5 has a .orig file removal in it, can we clean up the patch stack to avoid introducing it in the first place?

Flags: needinfo?(ryanvm) → needinfo?(mhentges)

Heh, thought I had properly hg absorb'd the removal of that .orig file, but I suppose not.
Thanks for the good eye Ryan!

Second try's the charm: a59f8db313ccb1a73b43a159b0d230da5da71388 should be a clean stack.

Flags: needinfo?(mhentges) → needinfo?(ryanvm)

CI is angry.
Update: I've made some progress, but CI still wants to write to /usr/lib/python2.7/sitecustomize.py. More work is needed.

Flags: needinfo?(ryanvm) → needinfo?(mhentges)
Flags: needinfo?(mhentges)
Attachment #9179170 - Flags: approval-mozilla-esr78?

Update: I set up the stack anew (tip is 80667ddf82858373813b073cf650db60b0e642fc), and CI seems happy.
I've got successful local builds in Linux and Mac, and I'm expecting Windows to pass as well (running it locally overnight tonight).

The uplift stack I have here is the minimal stack so that this bug's patch is moved to ESR. If you're building locally with Big Sur, you may run into other issues fixed by other tickets:

  • You may need 1686168 if you're on Mac and using system (XCode's) python instead of brew's python.
  • You may need 1669471 so you don't see a bunch of DeprecationWarnings for the imp module.
  • You may need all the patches in 1680802 if you're on Big Sur.

Comment on attachment 9179169 [details]
Bug 1654457 - Update virtualenv to 20.0.31.

Needed to better support running mach on newer macOS releases. Approved for 78.8esr.

Attachment #9179169 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+

Great to see it. I also found there was another fix which is also required by esr78:
https://bugzilla.mozilla.org/show_bug.cgi?id=1683929

Flags: needinfo?(mhentges)
Flags: needinfo?(mhentges)

Comment on attachment 9179170 [details]
Bug 1654457 - Use pip list --format freeze instead of pip freeze.

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Needed to be compatible with newer version of pip used after https://phabricator.services.mozilla.com/D83181
  • User impact if declined: None
  • Fix Landed on Version: 83
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): It's a build-only patch
  • String or UUID changes made by this patch:
Attachment #9179170 - Flags: approval-mozilla-esr78?
Attachment #9179170 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
Attachment #9165485 - Attachment is obsolete: false
Attachment #9165485 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: