1425102 - (mozregression-py3) Port mozregression to python3

Status: RESOLVED FIXED

(Testing :: mozregression, enhancement)

Description

[William Lachance (:wlach)]

I think we should consider updating mozregression to work with python3. 

Reasons:

1. Allows us to move to a more modern, supported, tested build environment (because of various python2 flakiness, currently travis only runs on the legacy precise environment which will be going away soon).
2. Might be able to use python-adb with a pip-installable crypto package, eliminating the need for mozdevice and a system-installable adb.
3. Opens the door to using pip-installable PyQt5, simplifying the development of the GUI.

The main blocker are the current mozbase packages we use only support python2:

https://github.com/mozilla/mozregression/blob/4cbd5577fbd5ee7ae3e4b16eeeea9a6e09d85ca2/setup.py#L30

So we'll probably need to make some headway on bug 1093212 first. We might be able to skip working on mozdevice if we do (2) above first.

Comment 1

[Vedant Chakravadhanula(:vedantc98)]

Hey, actually I'd already started working on bug 1093212 through bug 1388018 (porting mozfile).
I was hoping to move on to bug 1388019 (porting mozlog) but my review request for bug 1388018 is still pending.

So I'd be willing to take up anything related to this bug or bug 1093212.
Thanks!

Comment 2

[William Lachance (:wlach)]

(In reply to Vedant Chakravadhanula(:vedantc98) from comment #1)
> Hey, actually I'd already started working on bug 1093212 through bug 1388018
> (porting mozfile).
> I was hoping to move on to bug 1388019 (porting mozlog) but my review
> request for bug 1388018 is still pending.
> 
> So I'd be willing to take up anything related to this bug or bug 1093212.
> Thanks!

That would be great, I would feel free to start porting the other mozbase packages I link to in comment 0 (hint: mozprofile is the easiest, mozprocess is likely to be the hardest). Just file a new bug for each one, make it depend on bug 1093212 and CC me on the issue:

https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Mozbase

I will double check with people on who is best to review these, but I guess you can choose me for now.

Comment 3

[Vedant Chakravadhanula(:vedantc98)]

Thanks for the advice, William.

I've filed bug 1425399 for porting mozprofile and added you as mentor.
I'll add the other mozbase components once I'm done with that. 
Thanks!

Comment 4

[Daniel Holbert [:dholbert]]

FWIW many pip commands (with python 2.7) now spit out this warning:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

So if possible, it seems like we should consider migrating to python3 by then (or before too long after that). Note that the deprecation date is ~11 months away.

Comment 5

[William Lachance (:wlach)]

(In reply to Daniel Holbert [:dholbert] from comment #4)

So if possible, it seems like we should consider migrating to python3 by then (or before too long after that). Note that the deprecation date is ~11 months away.

Yup, I am hoping this will happen this year. It looks like the mozbase migration to python 3 is finally starting to pick up steam, which is the only blocking issue.

Comment 6

[William Lachance (:wlach)]

Attachment: GitHub Pull Request

Starting to work on this again, piece by piece. Building up the CI infrastructure to test python2 and python3 in parallel. Ran modernize on the command-line version + tests, currently working through the set of test failures. Porting the GUI will come last.

Comment 7

[William Lachance (:wlach)]

Update: I have run modernize on the codebase and am running some basic linting/ast checks to make sure that we don't inadvertently regress basic python 3 compatibility while this work is in flight.

https://github.com/mozilla/mozregression/pull/543

I think the command-line version actually does run on python3 now, though I'm out of time to actually test this thoroughly and add CI. My plan here goes something like this:

1. Release mozregression 3.0, which supports both python 2.7 and python 3 (not sure what minimum version of python 3 we'll require... tbd).
2. Port mozregression-gui to python 3.0, might also involve a port to pyqt5.

Probably shortly after (2) I will stop supporting python 2.7.

Comment 8

[William Lachance (:wlach)]

mozregression command-line now works with python3.5 and up (we have unit tests to prove it): https://github.com/mozilla/mozregression/pull/545

Will probably release a new version shortly. Still need to port the GUI

Comment 9

[61.1p57]

we have unit tests to prove it
I'm afraid that's not the case, python3-linux job is using Python 2.7.6... https://travis-ci.org/mozilla/mozregression/jobs/562715426#L676

FWIW I got 28 failed, 395 passed, 60 warnings when tested against Python 3.7.3

Comment 10

[William Lachance (:wlach)]

(In reply to 61.1p57 from comment #9)

we have unit tests to prove it
I'm afraid that's not the case, python3-linux job is using Python 2.7.6... https://travis-ci.org/mozilla/mozregression/jobs/562715426#L676

FWIW I got 28 failed, 395 passed, 60 warnings when tested against Python 3.7.3

Sigh, you are right. Can't believe I missed that...

I don't think the issues are all that difficult to fix, I'll try to work through them over the next couple weeks as time allows. Anyone else should feel free to submit PRs (especially next week, when I'll be out).

Comment 11

[William Lachance (:wlach)]

Ok, finally this is working! Thanks to gloomy ghost for helping out.