Closed Bug 1361462 Opened 8 years ago Closed 8 years ago

Mozharness support for Python3 binaries

Categories

(Testing :: General, enhancement)

Product:
Testing
Component:
General
Version:
Version 3
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox55 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla55
Tracking Flags:
Tracking Status
firefox55 --- fixed

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(3 files)

Bug 1361462 - Add tooltool_cache for Windows Talos jobs
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1361462 - Mozharness' tooltool should create directories if missing
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1361462 - Python3 support for Mozharness
59 bytes, text/x-review-board-request
mozilla
: review+
rwood
: review+
Details
If we figure it out we won't need to update every Talos platform we support. The assumption is that the support for mitmproxy would not be *just* for Windows 7/10.
We can probably use the embeddable download: https://www.python.org/downloads/release/python-353/ https://www.python.org/ftp/python/3.5.3/python-3.5.3-embed-amd64.zip 1264131c4c2f3f935f34c455bceedee1 I have uploaded it to tooltool: [ { "size": 6913264, "visibility": "public", "digest": "4966e18f078cff2fd04fea661712305705328de1f47e61a1af36688eea053d7ca3d8d475683ab1101186611b0eb22b17b9f1e1aa67907853fe4d09914845c799", "algorithm": "sha512", "filename": "python-3.5.3-embed-amd64.zip" } ] In the case of Linux and Mac we might need to build them from the source files before uploading them to tooltool.
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #1) > "filename": "python-3.5.3-embed-amd64.zip" Aren't the Win7 machines in CI 32 bit?
(In reply to Robert Wood [:rwood] from comment #2) > (In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #1) > > > "filename": "python-3.5.3-embed-amd64.zip" > > Aren't the Win7 machines in CI 32 bit? Yes. I was trying on my Win10 machine. In any case, the embeddable version is not going to be good: > Third-party packages should be installed by the application installer alongside the embedded distribution. Using pip to > manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it > may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of > the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates > to users. I'm close to say that we should install Python 3.5 at the host level. I'm going to try one more thing.
For some reason I can't get the silent installation to work. We shouldn't be doing this with tooltool. We are going to be adding too much overhead for little gain. I will be reporting on the other bug.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
FWIW, the Python we use in MozillaBuild is simply installed from the msi, and then packaged with the rest of the MozillaBuild installer: https://hg.mozilla.org/mozilla-build/file/tip/packageit.py#l97 You ought to be able to do something similar to create a Python install that you could then zip up and put in tooltool. Note that a few lines after running the installer we install pip etc, so that you can use pip from the resulting Python environment.
Yeah, I create an administrative install point from the installer MSI during packaging to avoid polluting the host OS. From a tooltool standpoint, you might do something like that locally than package it all up into a zip file that can then be downloaded and installed from tooltool. Also, should I be thinking about adding Python3 to MozillaBuild?
I will give this one more try, however, I don't know of a way where tooltool would not be extracting such archive for every talos run, thus, adding more overhead. RyanVM: if you could add Python 3.6 it would be great. I will give 3.6 a try since it has various improvements over 3.5: http://www.infoworld.com/article/3149782/application-development/python-36-is-packed-with-goodness.html
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Investigate using tooltool for Python 3.5 requirement → Investigate using tooltool for Python 3.6 requirement
See Also: → 1361732
Attachment #8864628 - Flags: review?(rwood)
aki, rwood, I'm going to change the zip files to have a consistent directory structure ("Python3.6.1/Python3.6.1.amd64" versus just "python3.6"). Otherwise the patch works. I will need to test the code on try for the two smaller patches (the big one does not affect automation atm).
Summary: Investigate using tooltool for Python 3.6 requirement → Mozharness support for Python3 binaries
See Also: → 1360338
Comment on attachment 8864626 [details] Bug 1361462 - Add tooltool_cache for Windows Talos jobs https://reviewboard.mozilla.org/r/136298/#review139376
Attachment #8864626 - Flags: review?(aki) → review+
Comment on attachment 8864627 [details] Bug 1361462 - Mozharness' tooltool should create directories if missing https://reviewboard.mozilla.org/r/136300/#review139378
Attachment #8864627 - Flags: review?(aki) → review+
Comment on attachment 8864628 [details] Bug 1361462 - Python3 support for Mozharness https://reviewboard.mozilla.org/r/136302/#review139380 I was worried about not seeing which manifest we use in the logs, but `tooltool_fetch` looks like it uses `self.run_command` which logs, so we should be good.
Attachment #8864628 - Flags: review?(aki) → review+
Great! Thank you Aki! I've pushed a new set of zip files with consistent directory name: https://reviewboard.mozilla.org/r/136302/diff/1-2/ I won't ask for review since it's the zip files + expected directory name after extraction. I will be testing on try. Once rwood gives the r+ I will land this.
The first try run showed an issue with the creation of directories as the value can be None. I've adjusted that line and pushed again. Interdiff: https://reviewboard.mozilla.org/r/136296/diff/2-3/ If rwood has no objections to the patch and the try push works I will land it.
Comment on attachment 8864628 [details] Bug 1361462 - Python3 support for Mozharness Looks awesome thanks Armen
Attachment #8864628 - Flags: review?(rwood) → review+
Comment on attachment 8864628 [details] Bug 1361462 - Python3 support for Mozharness https://reviewboard.mozilla.org/r/136302/#review139652
hg error in cmd: hg push -r tip ssh://hg.mozilla.org/integration/autoland: pushing to ssh://hg.mozilla.org/integration/autoland searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 3 changesets with 6 changes to 5 files remote: remote: remote: ************************** ERROR **************************** remote: Rev 9f35b31a0dcf needs "Bug N" or "No bug" in the commit message. remote: Armen Zambrano G. <armenzg@mozilla.com> remote: Mozharness' tooltool should create directories if missing r=aki remote: remote: MozReview-Commit-ID: 5HjD7smfCZx remote: ************************************************************* remote: remote: remote: transaction abort! remote: rollback completed remote: pretxnchangegroup.c_commitmessage hook failed abort: push failed on remote
Pushed by armenzg@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/400dfd6e6f04 Add tooltool_cache for Windows Talos jobs r=aki https://hg.mozilla.org/integration/autoland/rev/fa46b44d5629 Mozharness' tooltool should create directories if missing r=aki https://hg.mozilla.org/integration/autoland/rev/ae723b6d1271 Python3 support for Mozharness r=aki,rwood
See Also: → 1358306
https://hg.mozilla.org/mozilla-central/rev/400dfd6e6f04 https://hg.mozilla.org/mozilla-central/rev/fa46b44d5629 https://hg.mozilla.org/mozilla-central/rev/ae723b6d1271
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: