Upgrade talos to 3.8 for perftest
Categories
(Testing :: mozperftest, enhancement, P4)
Tracking
(Not tracked)
People
(Reporter: tarek, Assigned: tarek)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
We want to make sure perftest works with python 3.5 .
Unless we can move talos generic workers to 3.6, Joel?
|
Assignee | |
Updated•4 years ago
|
|
||
Comment 1•4 years ago
|
||
talos (and any other test suite) is launched from mozharness via python <testharness>.py <args>, to run with python3, we just need to do python3 <testharness>.py <args>.
There is a quirk here that you need to get python3 virtualenv setup properly (mozharness does this). If you run ./mach talos ... and it only uses python3, then all that is left is fixing mozharness to use python3. In the past (Q3 2017) when we setup tp6 to run in talos it required python3 and mozharness called talos with python3.
|
|
Assignee | |
Comment 2•4 years ago
•
|
||
yeah so, the new issue is that our virtualenv wrappers in mozbuild (we use in perftest) now requires Python 3.6 !
[task 2020-06-25T00:50:39.060Z] One of the following Python versions are required:
[task 2020-06-25T00:50:39.060Z] * Python 2.7.3 or greater
[task 2020-06-25T00:50:39.060Z] * Python 3.6.0 or greater
[task 2020-06-25T00:50:39.060Z] You are running Python 3.5.2.
[task 2020-06-25T00:50:39.060Z] Run |mach bootstrap| to ensure your system is up to date.
[task 2020-06-25T00:50:39.060Z]
[task 2020-06-25T00:50:39.060Z] If you still receive this error, your shell environment is likely detecting
[task 2020-06-25T00:50:39.060Z] another Python version. Ensure a modern Python can be found in the paths
[task 2020-06-25T00:50:39.060Z] defined by the $PATH environment variable and try again.
[task 2020-06-25T00:50:39.106Z] Traceback (most recent call last):
[task 2020-06-25T00:50:39.106Z] File "python/mozperftest/mozperftest/runner.py", line 123, in <module>
[task 2020-06-25T00:50:39.106Z] sys.exit(main())
[task 2020-06-25T00:50:39.106Z] File "python/mozperftest/mozperftest/runner.py", line 119, in main
[task 2020-06-25T00:50:39.106Z] run_tests(mach_cmd, **dict(args._get_kwargs()))
[task 2020-06-25T00:50:39.106Z] File "python/mozperftest/mozperftest/runner.py", line 86, in run_tests
[task 2020-06-25T00:50:39.106Z] e.run(metadata)
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozperftest/mozperftest/environment.py", line 91, in run
[task 2020-06-25T00:50:39.106Z] with self.layers[SYSTEM] as syslayer, self.layers[TEST] as testlayer:
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozperftest/mozperftest/layers.py", line 137, in __enter__
[task 2020-06-25T00:50:39.106Z] self.setup()
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozperftest/mozperftest/layers.py", line 147, in setup
[task 2020-06-25T00:50:39.106Z] layer.setup()
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozperftest/mozperftest/test/browsertime/runner.py", line 129, in setup
[task 2020-06-25T00:50:39.106Z] super(BrowsertimeRunner, self).setup()
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozperftest/mozperftest/test/noderunner.py", line 29, in setup
[task 2020-06-25T00:50:39.106Z] self.mach_cmd._activate_virtualenv()
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 840, in _activate_virtualenv
[task 2020-06-25T00:50:39.106Z] self.virtualenv_manager.ensure()
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 186, in ensure
[task 2020-06-25T00:50:39.106Z] return self.build(python)
[task 2020-06-25T00:50:39.106Z] File "/home/cltbld/tasks/task_1593044261/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 509, in build
[task 2020-06-25T00:50:39.106Z] raise Exception('Error populating virtualenv.')
[task 2020-06-25T00:50:39.106Z] Exception: Error populating virtualenv.
I can bypass it for the time being but I think we'll need to move talos to 3.6/3.7 eventually. Do you know who I should contact?
|
|
||
Comment 3•4 years ago
|
||
can you give me the OS and maybe machine names or workerType? I don't know if this is at bitbar, our datacenter, or aws, likewise linux, windows, or osx.
|
|
Assignee | |
Comment 4•4 years ago
|
||
sorry! It's our linux ubuntu boxes @ datacenter
worker-type: t-linux-talos
...
platform: linux64-shippable/opt
|
|
Assignee | |
Updated•4 years ago
|
|
|
||
Comment 5•4 years ago
|
||
:dhouse, would you or :dividehex be the person to talk to about installing/upgrading a newer version of python 3 on the linux moonshots? I think 3.6 or higher would be good, currently it is 3.5.2? Luckily only perf jobs run on these machines, and we would need to test this upgraded version (unless it is installed side by side) on talos/raptor/browsertime.
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)
:dhouse, would you or :dividehex be the person to talk to about installing/upgrading a newer version of python 3 on the linux moonshots? I think 3.6 or higher would be good, currently it is 3.5.2? Luckily only perf jobs run on these machines, and we would need to test this upgraded version (unless it is installed side by side) on talos/raptor/browsertime.
Upgrading python3 on the linux moonshots will not be a problem on the system side.
3.5.1-3 is the latest for Ubuntu 16.04 (we're upgrading to 18+ this year) but we can use the packages from https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa or building from the src archives to get up to 3.6+ (the ppa has up to 3.9, and I built and installed 3.8.3 on one of the moonshots without trouble).
Do you want a pool of 10 machines set up with 3.8.3 (latest stable) to run tests against? I'll plan to set up a pool of 10 machines, and then run the talos+raptor+browsertime against them.
|
|
||
Comment 7•4 years ago
|
||
thanks :dhouse, I forgot these were on 16.04- all in due time.
|
|
||
Comment 8•4 years ago
|
||
10 machines is just fine
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #8)
10 machines is just fine
okay great.
I copied the packages for python3.8.3 and set them up on a test pool: releng-hardware/gecko-t-linux-talos-python38
I pushed with '^test-linux64' to try: https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=TOOtyeQqQ7SmGMFD40QczQ.0&revision=aa5379687585e7523af577e70fa93a3e108b2b38 (I do not know if that covers all of the tests)
I attached the PR to apply this upgrade to the production pool. If the tests look good, I'll merge and apply the upgrade.
|
|
||
Comment 10•4 years ago
|
||
:tarek, can you push and verify perftest runs ok on these machines?
./mach try fuzzy --worker-suffix t-linux-talos=-python38
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 12•4 years ago
|
||
Almost there, I am bumping into one issue that should be fixed by installing python3-distutils
Dave, would you mind installing that lib? thanks
|
|
Assignee | |
Comment 13•4 years ago
|
||
we also need libjpeg and zlib
python-imaging would also be great but optional
|
||
Comment 14•4 years ago
|
||
(In reply to Tarek Ziadé (:tarek) from comment #13)
we also need
libjpegandzlib
python-imagingwould also be great but optional
libjpeg, zlib, and python3-distutils seem standard to just install in the base system. I'll find the packages and get them copied into our custom repo.
for python-imaging, can the task use tooltool or pip install a wheel?
|
|
||
Comment 15•4 years ago
|
||
I put python3-distutils on the -python38 workertype
It passes that failed test now: https://firefox-ci-tc.services.mozilla.com/tasks/cFCXlwQiSGib1FvyUPbALA/runs/1
I'll look at and set up libjpeg and zlib later.
|
|
Assignee | |
Comment 16•4 years ago
|
||
Awesome, thanks!
|
|
Assignee | |
Comment 17•4 years ago
|
||
(In reply to Dave House [:dhouse] from comment #14)
for python-imaging, can the task use tooltool or pip install a wheel?
Sure yeah, no problem
|
|
||
Comment 18•4 years ago
|
||
For zlib and libjpeg, what are you needing?
the machines already have:
zlib (1:1.2.8.dfsg-2ubuntu4)
libjpeg9 (1:9b-1ubuntu1)
|
|
Assignee | |
Comment 19•4 years ago
|
||
oh weird, I asked for them because when I installed Pillow, it complained that they were not present (Pillow is also provided by python-imaging)
Maybe I need to tweak the include headers paths. Will try again, thanks
|
|
Assignee | |
Comment 20•4 years ago
|
||
All good for windows (we can look at Pillow later) https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0c5165dcfacabdf5aa12af1c3805f498279a4ba2
|
|
Assignee | |
Updated•4 years ago
|
|
|
||
Comment 21•4 years ago
|
||
The python 3.8.3 upgrade is applied across all linux hardware workers now.
[root@t-linux64-ms-140 ~]# python3.8 --version
Python 3.8.3
[root@t-linux64-ms-140 ~]# python3 --version
Python 3.5.2
[root@t-linux64-ms-140 ~]# which python3.8
/usr/bin/python3.8
(python3 is still the system's latest and python3.8 is the new 3.8.3)
|
|
Assignee | |
Comment 22•4 years ago
|
||
Thanks!
|
|
Assignee | |
Updated•4 years ago
|
Description
•