Closed Bug 1212502 Opened 9 years ago Closed 5 years ago

Prepare in mozinfo for platform.linux_distribution() going away in python 3.7

Categories

(Testing :: Mozbase, defect)

Product:
Testing
Component:
Mozbase
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox72 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla72
Tracking Flags:
Tracking Status
firefox72 --- fixed

People

(Reporter: aceman, Assigned: egao)

References

Details

Attachments

(3 files)

Bug 1212502 - Switch mozinfo to using the 'distro' package to get linux distribution info
47 bytes, text/x-phabricator-request
Details | Review
Bug 1212502 - revert _pslinux.py file to initially vendored state
47 bytes, text/x-phabricator-request
Details | Review
Bug 1212502 - bump mozinfo version up to 1.2.0 in preparation for upload to pypi
47 bytes, text/x-phabricator-request
Details | Review 
The function linux_distribution() in the platform.py of Python seems to be going away in Python 3.7

See:
https://docs.python.org/3/library/platform.html#platform.linux_distribution and https://docs.python.org/3/whatsnew/3.5.html?highlight=linux_distribution

Also, in python 2.7 the function is returning empty distro/version on distros that have the /etc directory not world readable (only executable/enterable). See bug 1117543. Yes, this should be fixed in platform.py (for 2.7) but it seems they no longer accept fixes for this code (see final comments of https://bugs.python.org/issue1322).

So I propose to prepare to find other solution for finding this info if mozinfo and tests really need to know the distro version being run.
The timetable for the build system and Firefox automation supporting Python 3 is likely measured in years.
As per the python documentation, platform.linux_distribution() will be removed in python 3.8.

A pip package 'distro' has been named as an alternative: https://stackoverflow.com/questions/49554443/platform-linux-distribution-deprecated-what-are-the-alternatives

Looks like this is a drop-in replacement for the platform.linux_distribution() call.
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ca117d13ca06
Switch mozinfo to using the 'distro' package to get linux distribution info r=ahal

:ahal - I can take over this if you need someone to work on this.

Thanks, I'd appreciate that!

Flags: needinfo?(kwierso) → needinfo?(egao)

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1590843 to have distro==1.4.0 uploaded to our internal pypi server at pypi.pub.build.mozilla.org.

Depends on: 1590843
Flags: needinfo?(egao)
Assignee: kwierso → egao

I didn't get a lot of time to test things, but it seemed like the distro package returned actual information for more distributions than platform.linux_distribution did. Hopefully that doesn't mess up test expectations or anything.

Hey Wes, I guess we didn't really ask.. If you'd still like to work on this please feel free!

I'm not in a position to continue working on this, just commenting on something I observed while I was.

(In reply to Wes Kocher (:KWierso) from comment #9)

I didn't get a lot of time to test things, but it seemed like the distro package returned actual information for more distributions than platform.linux_distribution did. Hopefully that doesn't mess up test expectations or anything.

I'm fairly confident that distro will be a suitable drop-in replacement. It fixes some of the bugs that exist for platform.linux_distribution() on non-Linux platforms as well, so that's a plus.

I'll adjust test harnesses and/or expectations that mess up when the switch to distro is made.

p.s. :KWierso - thanks for getting this started.

Thanks to :jlund for uploading distro==1.4.0 to our internal pypi mirror.

I've submitted a new try run to see if this would resolve the issues seen on macosx, linux and windows.

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=a12b00c0d6c2cb9482aeadb8568be9edcea1bf8c

So it looks like uploading distro to the internal pypi mirror wasn't the solution:

linux64/opt mochitest-e10s-3:

[task 2019-10-29T22:18:18.103Z] + /builds/worker/bin/run-mozharness
[task 2019-10-29T22:18:18.108Z] Running: python2.7 /builds/worker/workspace/mozharness/scripts/desktop_unittest.py  --config-file /builds/worker/workspace/mozharness/configs/unittests/linux_unittest.py --config-file /builds/worker/workspace/mozharness/configs/remove_executables.py  --mochitest-suite=mochitest-plain-chunked --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --total-chunk=5 --this-chunk=3 --download-symbols=ondemand 
[task 2019-10-29T22:18:18.280Z] Traceback (most recent call last):
[task 2019-10-29T22:18:18.281Z]   File "/builds/worker/workspace/mozharness/scripts/desktop_unittest.py", line 31, in <module>
[task 2019-10-29T22:18:18.281Z]     from mozharness.base.script import PreScriptAction
[task 2019-10-29T22:18:18.282Z]   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 57, in <module>
[task 2019-10-29T22:18:18.282Z]     import mozinfo
[task 2019-10-29T22:18:18.283Z]   File "/builds/worker/workspace/mozharness/mozinfo/__init__.py", line 57, in <module>
[task 2019-10-29T22:18:18.284Z]     from . import mozinfo
[task 2019-10-29T22:18:18.285Z]   File "/builds/worker/workspace/mozharness/mozinfo/mozinfo.py", line 18, in <module>
[task 2019-10-29T22:18:18.286Z]     import distro as distribution
[task 2019-10-29T22:18:18.286Z] ImportError: No module named distro

windows10-64:

[task 2019-10-29T21:51:36.635Z] executing ['c:/mozilla-build/python/python.exe', './build/src/testing/mozharness/scripts/fx_desktop_build.py', '--config', 'builds/releng_base_firefox.py', '--config', 'builds/taskcluster_base_windows.py', '--config', 'builds/taskcluster_base_win64.py', '--branch', 'try', '--work-dir', 'z:\\task_1572385108\\build', '--get-secrets', '--build', '--append-env-variables-from-configs']
[task 2019-10-29T21:51:36.810Z] Traceback (most recent call last):
[task 2019-10-29T21:51:36.810Z]   File "./build/src/testing/mozharness/scripts/fx_desktop_build.py", line 23, in <module>
[task 2019-10-29T21:51:36.810Z]     import mozharness.base.script as script
[task 2019-10-29T21:51:36.810Z]   File "Z:\task_1572385108\build\src\testing\mozharness\mozharness\base\script.py", line 57, in <module>
[task 2019-10-29T21:51:36.810Z]     import mozinfo
[task 2019-10-29T21:51:36.810Z]   File "Z:\task_1572385108\build\src\testing\mozbase\mozinfo\mozinfo\__init__.py", line 57, in <module>
[task 2019-10-29T21:51:36.810Z]     from . import mozinfo
[task 2019-10-29T21:51:36.810Z]   File "Z:\task_1572385108\build\src\testing\mozbase\mozinfo\mozinfo\mozinfo.py", line 18, in <module>
[task 2019-10-29T21:51:36.810Z]     import distro as distribution
[task 2019-10-29T21:51:36.810Z] ImportError: No module named distro
[fetches 2019-10-29T21:51:36.813Z] removing Z:/task_1572385108/fetches

Figured out the solution, it was hiding in plain sight the whole time - distro package has issues with Windows and (I did not observe but) MacOSX.

Workaround was to import distro in the Linux-specific if/else in mozinfo.py. This would permit Windows and MacOSX builds/tests to continue running as intended.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d2d9fe01fc33
Switch mozinfo to using the 'distro' package to get linux distribution info r=ahal,KWierso

https://hg.mozilla.org/mozilla-central/rev/d2d9fe01fc33

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox72: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla72

This seems to have re-introduced Bug 1518762.

Flags: needinfo?(egao)

(In reply to Ted Campbell [:tcampbell] from comment #18)

This seems to have re-introduced Bug 1518762.

:tcampbell - could you please clarify? Is there a treeherder push where I can see some failure logs, or is this observed locally?

Flags: needinfo?(egao) → needinfo?(tcampbell)

This happens in local builds on machines with linux kernel 4.18+. The very first line of output when I run |./mach build|.

It should be possible to just apply the same patch from Bug 1518762 again. According to that, this is fixed in upstream psutil 5.5.0 but that is more recent than what we vendor.

Flags: needinfo?(tcampbell)

Thanks for info. I'll try to set up a local virutal machine and try it out.

No need to test it out. Just revert the change to psutil your patch introduced.

This happens because we manually modified psutil awhile back and now anytime anyone runs ./mach vendor those manual changes get undone. I should have caught this in the review, sorry. We should also make sure there's a bug on file to prevent this from happening again (ideally by updating our vendored copy of psutil).

I've put up a patch to manually revert the changes introduced in the patch to the file third_party/python/psutil/psutil/_pslinux.py.

I'll also put in the tracker a bug to update the vendored copy of psutil to prevent this from happening in the future.

See Also: → 1508320
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bc351b05e0d0
revert _pslinux.py file to initially vendored state r=ahal
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

https://hg.mozilla.org/mozilla-central/rev/bc351b05e0d0

Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
status-firefox72: affected → fixed
Resolution: --- → FIXED
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b611a31d62cc
bump mozinfo version up to 1.2.0 in preparation for upload to pypi r=gbrown
Regressions: 1595147
Blocks: 1598161
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: