Mozharness support for Python3 binaries

RESOLVED FIXED in Firefox 55

Status

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: armenzg, Assigned: armenzg)

Tracking

Version 3
mozilla55
Points:
---

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(3 attachments)

If we figure it out we won't need to update every Talos platform we support.
The assumption is that the support for mitmproxy would not be *just* for Windows 7/10.
We can probably use the embeddable download:
https://www.python.org/downloads/release/python-353/
https://www.python.org/ftp/python/3.5.3/python-3.5.3-embed-amd64.zip 1264131c4c2f3f935f34c455bceedee1

I have uploaded it to tooltool:
[
  {
    "size": 6913264,
    "visibility": "public",
    "digest": "4966e18f078cff2fd04fea661712305705328de1f47e61a1af36688eea053d7ca3d8d475683ab1101186611b0eb22b17b9f1e1aa67907853fe4d09914845c799",
    "algorithm": "sha512",
    "filename": "python-3.5.3-embed-amd64.zip"
  }
]

In the case of Linux and Mac we might need to build them from the source files before uploading them to tooltool.
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #1)

>     "filename": "python-3.5.3-embed-amd64.zip"

Aren't the Win7 machines in CI 32 bit?
(In reply to Robert Wood [:rwood] from comment #2)
> (In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #1)
> 
> >     "filename": "python-3.5.3-embed-amd64.zip"
> 
> Aren't the Win7 machines in CI 32 bit?

Yes. I was trying on my Win10 machine.

In any case, the embeddable version is not going to be good:
> Third-party packages should be installed by the application installer alongside the embedded distribution. Using pip to 
> manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it
> may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of
> the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates
> to users.

I'm close to say that we should install Python 3.5 at the host level.
I'm going to try one more thing.
For some reason I can't get the silent installation to work.
We shouldn't be doing this with tooltool.
We are going to be adding too much overhead for little gain.
I will be reporting on the other bug.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
FWIW, the Python we use in MozillaBuild is simply installed from the msi, and then packaged with the rest of the MozillaBuild installer:
https://hg.mozilla.org/mozilla-build/file/tip/packageit.py#l97

You ought to be able to do something similar to create a Python install that you could then zip up and put in tooltool. Note that a few lines after running the installer we install pip etc, so that you can use pip from the resulting Python environment.
Yeah, I create an administrative install point from the installer MSI during packaging to avoid polluting the host OS. From a tooltool standpoint, you might do something like that locally than package it all up into a zip file that can then be downloaded and installed from tooltool.

Also, should I be thinking about adding Python3 to MozillaBuild?
I will give this one more try, however, I don't know of a way where tooltool would not be extracting such archive for every talos run, thus, adding more overhead.

RyanVM: if you could add Python 3.6 it would be great.

I will give 3.6 a try since it has various improvements over 3.5:
http://www.infoworld.com/article/3149782/application-development/python-36-is-packed-with-goodness.html
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Investigate using tooltool for Python 3.5 requirement → Investigate using tooltool for Python 3.6 requirement
See Also: → bug 1361732
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Attachment #8864628 - Flags: review?(rwood)
aki, rwood, I'm going to change the zip files to have a consistent directory structure ("Python3.6.1/Python3.6.1.amd64" versus just "python3.6").
Otherwise the patch works.
I will need to test the code on try for the two smaller patches (the big one does not affect automation atm).
Summary: Investigate using tooltool for Python 3.6 requirement → Mozharness support for Python3 binaries
See Also: → bug 1360338

Comment 12

2 years ago
mozreview-review
Comment on attachment 8864626 [details]
Bug 1361462 - Add tooltool_cache for Windows Talos jobs

https://reviewboard.mozilla.org/r/136298/#review139376
Attachment #8864626 - Flags: review?(aki) → review+

Comment 13

2 years ago
mozreview-review
Comment on attachment 8864627 [details]
Bug 1361462 - Mozharness' tooltool should create directories if missing

https://reviewboard.mozilla.org/r/136300/#review139378
Attachment #8864627 - Flags: review?(aki) → review+

Comment 14

2 years ago
mozreview-review
Comment on attachment 8864628 [details]
Bug 1361462 - Python3 support for Mozharness

https://reviewboard.mozilla.org/r/136302/#review139380

I was worried about not seeing which manifest we use in the logs, but `tooltool_fetch` looks like it uses `self.run_command` which logs, so we should be good.
Attachment #8864628 - Flags: review?(aki) → review+
Comment hidden (mozreview-request)
Great! Thank you Aki!

I've pushed a new set of zip files with consistent directory name:
https://reviewboard.mozilla.org/r/136302/diff/1-2/

I won't ask for review since it's the zip files + expected directory name after extraction.

I will be testing on try.

Once rwood gives the r+ I will land this.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
The first try run showed an issue with the creation of directories as the value can be None.
I've adjusted that line and pushed again.

Interdiff:
https://reviewboard.mozilla.org/r/136296/diff/2-3/

If rwood has no objections to the patch and the try push works I will land it.
Comment on attachment 8864628 [details]
Bug 1361462 - Python3 support for Mozharness

Looks awesome thanks Armen
Attachment #8864628 - Flags: review?(rwood) → review+

Comment 22

2 years ago
mozreview-review
Comment on attachment 8864628 [details]
Bug 1361462 - Python3 support for Mozharness

https://reviewboard.mozilla.org/r/136302/#review139652

Comment 23

2 years ago
hg error in cmd: hg push -r tip ssh://hg.mozilla.org/integration/autoland: pushing to ssh://hg.mozilla.org/integration/autoland
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 3 changesets with 6 changes to 5 files
remote: 
remote: 
remote: ************************** ERROR ****************************
remote: Rev 9f35b31a0dcf needs "Bug N" or "No bug" in the commit message.
remote: Armen Zambrano G. <armenzg@mozilla.com>
remote: Mozharness' tooltool should create directories if missing r=aki
remote: 
remote: MozReview-Commit-ID: 5HjD7smfCZx
remote: *************************************************************
remote: 
remote: 
remote: transaction abort!
remote: rollback completed
remote: pretxnchangegroup.c_commitmessage hook failed
abort: push failed on remote
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 26

2 years ago
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/400dfd6e6f04
Add tooltool_cache for Windows Talos jobs r=aki
https://hg.mozilla.org/integration/autoland/rev/fa46b44d5629
Mozharness' tooltool should create directories if missing r=aki
https://hg.mozilla.org/integration/autoland/rev/ae723b6d1271
Python3 support for Mozharness r=aki,rwood
See Also: → bug 1358306

Comment 27

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/400dfd6e6f04
https://hg.mozilla.org/mozilla-central/rev/fa46b44d5629
https://hg.mozilla.org/mozilla-central/rev/ae723b6d1271
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.