Closed Bug 1361462 Opened 7 years ago Closed 7 years ago

Mozharness support for Python3 binaries

Categories

(Testing :: General, enhancement)

Product:
Testing
Component:
General
Version:
Version 3
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox55 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla55
Tracking Flags:
Tracking Status
firefox55 --- fixed

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(3 files)

Bug 1361462 - Add tooltool_cache for Windows Talos jobs
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1361462 - Mozharness' tooltool should create directories if missing
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1361462 - Python3 support for Mozharness
59 bytes, text/x-review-board-request
mozilla
: review+
rwood
: review+
Details 
If we figure it out we won't need to update every Talos platform we support.
The assumption is that the support for mitmproxy would not be *just* for Windows 7/10.
We can probably use the embeddable download:
https://www.python.org/downloads/release/python-353/
https://www.python.org/ftp/python/3.5.3/python-3.5.3-embed-amd64.zip 1264131c4c2f3f935f34c455bceedee1

I have uploaded it to tooltool:
[
  {
    "size": 6913264,
    "visibility": "public",
    "digest": "4966e18f078cff2fd04fea661712305705328de1f47e61a1af36688eea053d7ca3d8d475683ab1101186611b0eb22b17b9f1e1aa67907853fe4d09914845c799",
    "algorithm": "sha512",
    "filename": "python-3.5.3-embed-amd64.zip"
  }
]

In the case of Linux and Mac we might need to build them from the source files before uploading them to tooltool.
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #1)

>     "filename": "python-3.5.3-embed-amd64.zip"

Aren't the Win7 machines in CI 32 bit?
(In reply to Robert Wood [:rwood] from comment #2)
> (In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #1)
> 
> >     "filename": "python-3.5.3-embed-amd64.zip"
> 
> Aren't the Win7 machines in CI 32 bit?

Yes. I was trying on my Win10 machine.

In any case, the embeddable version is not going to be good:
> Third-party packages should be installed by the application installer alongside the embedded distribution. Using pip to 
> manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it
> may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of
> the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates
> to users.

I'm close to say that we should install Python 3.5 at the host level.
I'm going to try one more thing.
For some reason I can't get the silent installation to work.
We shouldn't be doing this with tooltool.
We are going to be adding too much overhead for little gain.
I will be reporting on the other bug.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
FWIW, the Python we use in MozillaBuild is simply installed from the msi, and then packaged with the rest of the MozillaBuild installer:
https://hg.mozilla.org/mozilla-build/file/tip/packageit.py#l97

You ought to be able to do something similar to create a Python install that you could then zip up and put in tooltool. Note that a few lines after running the installer we install pip etc, so that you can use pip from the resulting Python environment.
Yeah, I create an administrative install point from the installer MSI during packaging to avoid polluting the host OS. From a tooltool standpoint, you might do something like that locally than package it all up into a zip file that can then be downloaded and installed from tooltool.

Also, should I be thinking about adding Python3 to MozillaBuild?
I will give this one more try, however, I don't know of a way where tooltool would not be extracting such archive for every talos run, thus, adding more overhead.

RyanVM: if you could add Python 3.6 it would be great.

I will give 3.6 a try since it has various improvements over 3.5:
http://www.infoworld.com/article/3149782/application-development/python-36-is-packed-with-goodness.html
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Investigate using tooltool for Python 3.5 requirement → Investigate using tooltool for Python 3.6 requirement
See Also: → 1361732
Attachment #8864628 - Flags: review?(rwood)
aki, rwood, I'm going to change the zip files to have a consistent directory structure ("Python3.6.1/Python3.6.1.amd64" versus just "python3.6").
Otherwise the patch works.
I will need to test the code on try for the two smaller patches (the big one does not affect automation atm).
Summary: Investigate using tooltool for Python 3.6 requirement → Mozharness support for Python3 binaries
See Also: → 1360338
Comment on attachment 8864626 [details]
Bug 1361462 - Add tooltool_cache for Windows Talos jobs

https://reviewboard.mozilla.org/r/136298/#review139376
Attachment #8864626 - Flags: review?(aki) → review+
Comment on attachment 8864627 [details]
Bug 1361462 - Mozharness' tooltool should create directories if missing

https://reviewboard.mozilla.org/r/136300/#review139378
Attachment #8864627 - Flags: review?(aki) → review+
Comment on attachment 8864628 [details]
Bug 1361462 - Python3 support for Mozharness

https://reviewboard.mozilla.org/r/136302/#review139380

I was worried about not seeing which manifest we use in the logs, but `tooltool_fetch` looks like it uses `self.run_command` which logs, so we should be good.
Attachment #8864628 - Flags: review?(aki) → review+
Great! Thank you Aki!

I've pushed a new set of zip files with consistent directory name:
https://reviewboard.mozilla.org/r/136302/diff/1-2/

I won't ask for review since it's the zip files + expected directory name after extraction.

I will be testing on try.

Once rwood gives the r+ I will land this.
The first try run showed an issue with the creation of directories as the value can be None.
I've adjusted that line and pushed again.

Interdiff:
https://reviewboard.mozilla.org/r/136296/diff/2-3/

If rwood has no objections to the patch and the try push works I will land it.
Comment on attachment 8864628 [details]
Bug 1361462 - Python3 support for Mozharness

Looks awesome thanks Armen
Attachment #8864628 - Flags: review?(rwood) → review+
Comment on attachment 8864628 [details]
Bug 1361462 - Python3 support for Mozharness

https://reviewboard.mozilla.org/r/136302/#review139652
hg error in cmd: hg push -r tip ssh://hg.mozilla.org/integration/autoland: pushing to ssh://hg.mozilla.org/integration/autoland
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 3 changesets with 6 changes to 5 files
remote: 
remote: 
remote: ************************** ERROR ****************************
remote: Rev 9f35b31a0dcf needs "Bug N" or "No bug" in the commit message.
remote: Armen Zambrano G. <armenzg@mozilla.com>
remote: Mozharness' tooltool should create directories if missing r=aki
remote: 
remote: MozReview-Commit-ID: 5HjD7smfCZx
remote: *************************************************************
remote: 
remote: 
remote: transaction abort!
remote: rollback completed
remote: pretxnchangegroup.c_commitmessage hook failed
abort: push failed on remote
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/400dfd6e6f04
Add tooltool_cache for Windows Talos jobs r=aki
https://hg.mozilla.org/integration/autoland/rev/fa46b44d5629
Mozharness' tooltool should create directories if missing r=aki
https://hg.mozilla.org/integration/autoland/rev/ae723b6d1271
Python3 support for Mozharness r=aki,rwood
See Also: → 1358306
https://hg.mozilla.org/mozilla-central/rev/400dfd6e6f04
https://hg.mozilla.org/mozilla-central/rev/fa46b44d5629
https://hg.mozilla.org/mozilla-central/rev/ae723b6d1271
Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: