1679640 - Upgrade Python to 3.7.9

Status: RESOLVED INVALID

(Developer Infrastructure :: Source Documentation, enhancement)

Description

[Zac Svoboda :zacnomore]

In order to upgrade sphinx-js we need to be on at least 3.7, this resolves that issue.

Comment 1

[Akshat [:baka]]

I will like to do this, This will also help me get a good idea of the project. How should I begin....change configuration to py3 and fix errors as they show up...? Please help me start-up with this enhancement.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

Dunno, maybe champion knows more

Comment 3

[Shivam Singhal [ :championshuttler ]]

Yeah that would be correct approach I guess.

Comment 4

[Akshat [:baka]]

(In reply to Shivam Singhal [ :championshuttler ] from comment #3)

Yeah that would be correct approach I guess.

Where can I change the configuration to py3?

Comment 5

[Akshat [:baka]]

I was trying to look for how to upgrade to python3 and according to these docs, I have to what I mentioned earlier...but in mach file moztreedocs is not explicitly mentioned to run with python 2 and when I checked in my obj-x86_64-pc-linux-gnu/_virtualenvs/docs/lib it has python3.8 installed which is the virtualenv for doc command, I think, as mentioned here. Does that mean this bug is obsolete and we have already migrated to > = python3.7.9? Help me close this bug : )

Comment 6

[Akshat [:baka]]

Since I found documentation to migrate mach commands I am cancelling the requested information tag from :championshuttler.

Comment 7

[Amine]

Can I get more info on this? I've been trying to upgrade sphinx-js to version 3.1+ but in order for this to happen Python needs to be upgraded to version 3.7+. Right now it seems like the Python being used is python 3.6. How would I go about upgrading it and pushing it to the patch server?

Comment 8

[Andrew Halberstadt [:ahal]]

The doc tasks use the lint docker image:
https://searchfox.org/mozilla-central/source/taskcluster/docker/lint/Dockerfile

We just use whatever system Python that comes bundled with the distro (Ubuntu 18.04 in this case). We could try upgrading it to the latest LTS release instead (Ubuntu 20.04), though lots of things use this image, so it could break unrelated tasks. It might be worth a shot though..

You'd need to request level 1 commit access to access try server:
https://www.mozilla.org/en-US/about/governance/policies/commit/

I can vouch for you.

Comment 9

[Amine]

Ok, thanks for the information, I have requested for level 1 access here: https://bugzilla.mozilla.org/show_bug.cgi?id=1706231

When it comes to upgrading Python, is it not possible to upgrade the version in the venv and patch it without upgrading the distro?

Comment 10

[Andrew Halberstadt [:ahal]]

Yes, it would be possible. We could alternatively update system-setup.sh to install Python via some other means (like pyenv for instance).

It would be great to do this, though that is a bit more work. If that's an approach you'd like to try, then I'd be happy to assist, but simply updating Ubuntu (which we should probably do anyway) might be easier. Unless it causes other tasks to fail that is :)

Comment 11

[Amine]

Ok, I'll try updating the distro first, when it comes to updating it, do I modify the docker image you linked earlier or do we have to use a different image altogether?

Comment 12

[Andrew Halberstadt [:ahal]]

Yes, you can modify the Dockerfile I linked (change the Ubuntu line to 20.04).

Then you can test out that it works with this command:

./mach try fuzzy --query "^source-test"

This will run every task whose label starts with the string "source-test" (which are typically the tasks that use that image). It will run a lot of extra stuff on top of the doc tasks, but at least we'll be able to see if any of them get broken as a result of the upgrade. You can read this page for more info on mach try fuzzy if you like.

Comment 13

[Amine]

Is there a good way to test specific things locally? Specifically I want to run the docker image and system-setup.sh locally. I can do so remotely but it takes very long (usually hours) to finish. I'm doing a pyenv setup on system-setup.sh to install Python 3.7.9. Is there a way to test that locally?

Comment 14

[Andrew Halberstadt [:ahal]]

Yes! If you have docker installed and set up properly (can run docker run hello-world without sudo), then you can use:

$ ./mach taskcluster-build-image lint

to build the image locally. This will save the image as taskcluster/lint, you should see it in docker images. Then you can use it like any other docker container (first create and start the container, then run docker exec -it <container> bash)

Comment 15

[Amine]

In order for me to test mach doc in the docker container for instance, would I have to copy over the mozilla-central folder first? Running the container alone doesn't really do much since there's not much on it to begin with.

I've tried running docker cp to copy over the files from my host machine to the docker container and it just causes my bash shell to hang indefinitetly. Not sure why that is.

Comment 16

[Andrew Halberstadt [:ahal]]

Ah, yeah getting that set up properly will likely be hard. I was thinking you could use the local docker image just to test that pytest was available and working properly.

If you want to test the doc generation itself, I think you can just run ./mach doc with the appropriate version of Python locally and it should be the same thing.

You can also request level 1 commit access (if you don't have it already) and test your changes on the try server. If you end up filing a bug, let me know and I can vouch for you. Once you have that set up, you can test your changes by running ./mach try fuzzy --query "'doc-generate"

Comment 17

[Andrew Halberstadt [:ahal]]

Clearing stale needinfos.

Comment 18

[Andrew Halberstadt [:ahal]]

We're about to drop support for 3.7.