Closed Bug 1093212 (mozbase-py3) Opened 10 years ago Closed 4 years ago

Support python 3 in mozbase.

Categories

(Testing :: Mozbase, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Ms2ger, Unassigned)

References

(Blocks 2 open bugs)

Details

Attachments

(1 obsolete file)

#ateam seemed generally supportive of supporting python 3 alongside python 2.7, to avoid forcing new consumers to stick to python 2.
Attached patch WIP (obsolete) — Splinter Review
This allows us to at least run most mozlog tests under python 3; a bunch fail because of the expected byte/string issues.

Actual patches should probably end up in dependent bugs.
Python 2.7 will be EOL in 2020 (https://www.python.org/dev/peps/pep-0373/#update), and by not supporting Python 3 we are preventing other packages from adding mozbase packages as dependencies if they want to support Python 3. Should we turn this into a tracking bug, and raise blocking bugs for each mozbase package?

We should probably start with at least running the mozbase tests against Python 3, so that once we fix this, we avoid regressions in the future.
CC'ing Gregory to this bug given that he knows best about the build system and what we could do for Python 3 support.
(In reply to Henrik Skupin (:whimboo) from comment #3)
> CC'ing Gregory to this bug given that he knows best about the build system
> and what we could do for Python 3 support.

It might help to also do it.
While I personally like a lot of the advancements in Python 3, there is really not yet a compelling reason for us to prioritize supporting Python 3 in the build system. Python 2.7 works just fine and spending the time to support Python 3 for little to no end-user benefit is difficult to justify when we could be spending time improving the build system in other ways.

That being said, most of the Python code I've written in the past few years strives to be compatible with 2.7 and 3.5+. I encourage others to take this approach so an eventual Python 3 port is easier. Also, I encourage others to not care about anything less than Python 3.5, as IMO that is the first Python 3 release that is reasonable to target for writing code that works on both 2.7 and 3.5 (a lot of that is due to 3.5 reintroducing % formatting on bytes types).

Finally, the build system relies on so many other components that it will likely be the last thing to be ported to Python 3. So I think it is fine for projects like mozbase to cross the Python 3 bridge earlier.
Also what I have seen when working on mozbuild changes, we are using __future__ imports a lot which already helps with a lot of those cross-version incompatibilities. Maybe we should get started with those to ensure everything is still tested by unittests.
I've been watching the Mercurial project make pretty good work porting from Python 2.6/2.7 to dual support with 3.5+ and think their process is worth following:

1) Adopt `from __future__ import absolute_import`
2) Make every file AST parse cleanly in Python 3 (does `ast.parse()` not raise)
3) Make every file import cleanly in Python (importlib.import_module() doesn't raise)
4) Start modifying code to actually run properly on Python 3. This involves using six (or rolling your own), using b'' literals when appropriate, etc.

#4 is definitely the time consuming part. But you can generally divide and conquer once all the modules import cleanly. Once 1-3 happened with Mercurial, the patches for #4 started flowing very rapidly.
The last time I looked into this I stumbled upon modernize: http://python-modernize.readthedocs.io/en/latest/index.html

It's basically a layer on top of 2to3 that rewrites your code to use six instead of just rewriting it to be Python 3-compatible. This seems like it might be a viable approach to tackling the problem.

Now that we have the mozbase tests running in a standalone job (bug 1003417) we should be able to stand up another job that runs the same tests in Python 3. We could add a mozinfo key for Python 3, and skip/fail tests as necessary to get a green subset, and then enable more as we do the work to fix them.
Of the 200 most widely used Python packages, only 13 have not been converted to Python 3, according to: https://python3wos.appspot.com/

Nine of those are moz packages. That means, given the proportions, that almost all the most important python packages that have not been converted to Python 3 are moz packages.

The other non-converts are httplib2 (renamed to http.client in Python 3), supervisor, carbon, graphite web and..., no that's it.

It makes Mozilla look luddite.

It makes Mozilla look like a stubborn old man holed up in a shanty shack on a little grassy roundabout around which a whole city and its busy roads are whirring on regardless. Hurrumph!

That's just how it looks.

Five years ago it was already obvious that anyone starting afresh should start with Python 3. They continue to do so: building around Mozilla?
The following Moz packages are all in the top 500 most downloaded PyPI packages of all time.  If they can not be ported to Python 3 then their setup.py trove classifiers should be updated to add 'Programming Language :: Python :: 2 :: Only'.  Similar changes should be made to their README docs which currently often say 'Python 2.7 or later' which is ambiguous because Python 3.6.1 is later than 2.7 but is not supported.  These simple updates would save a lot of end user aggravation.


* manifestparser
* marionette_client
* marionette_driver
* marionette-transport
* mozcrash
* mozdevice
* mozfile
* mozhttpd
* mozinfo
* mozlog
* moznetwork
* mozprocess
* mozprofile
* mozrunner
* mozsystemmonitor
* moztest
* mozversion

PyPI Trove Classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers
PyPI Top 500: https://python3wos.mybluemix.net
@Mark: thanks for raising this up.

New projects are to use Python 3.

From my understanding those moz packages are actually not that used. I believe the high stats was because during a while some of the automation would grab python packages directly from pypi instead of internal alternatives, thus, inflating the metrics. I think we fixed that a while back.

If there was a way to reset the metrics that would be super! From looking at the table I'm at least happy to see that all those packages are down to below 100 most used. I remember the day when they showed up very very high on the able.

@cclaus: that seems like a good suggestion. I will pass it along.
I agree that there's no current significant benefit to the build system itself supporting Python 3. However for mozbase specifically, lack of Python 3 support will soon have an impact, since it's consumed by others. 

For example, Treeherder uses Django whose next major release (2.0, due December 2017) drops support for Python 2. Treeherder also uses mozlog (during the ingestion of structured logs from test runs), so if by the end of the year mozlog doesn't support Python 3, it will prevent Treeherder from being able to use the latest version of Django (the complexities of Treeherder making itself Python 3 compatible aside).
Armen - I'm sorry to hear usage is low after all. Let's then dismiss popularity from our reckonings. Their use will decline further anyway, as long as they are not updated. I meant only to say high popularity increased the justification for converting these tools to python3. You imply low popularity suggests no need to upgrade the tools. Yet this argument would not hold.

If the tools are intended to be used, then their utility depends on their being updated. It seems then Mozilla intends to let the tools die, though your reply also implies alternatives might be developed. In either case, clear statements up front in *user documentation* are essential. The documentation I saw was actually good, but still neglected to give these most crucial details: it won't work on python3 and Mozilla either intends to let them die a natural death or to replace them at some likely time in the future. If Mozilla has a non-resuscitate order on the moz tools, it should put a non-resuscitate notice on them. It's like a central bank making it easy for people to buy bonds, but issuing bonds in such a way that, say, only babyboomers can buy them; and then not telling anybody about the provisos, so half the next generation waste time trying to learn about bonds and trying to buy bonds they can't buy; while the bank issues only vague and enigmatic explanations when people try to find out why they can't. In a market of misleading information everyone trundles around bumping into things, getting lost, retracing their steps, studying maps, stepping out and getting lost again, in circles and snarl-ups. Investors, as they say, lose confidence in the future. The economy evaporates as any fuel does in a traffic jam.
Mark: if you're signing up to do the work then that's great! If not, writing long rants on bugzilla is not going to change things.
(In reply to Mark from comment #13)
> Armen - I'm sorry to hear usage is low after all. Let's then dismiss
> popularity from our reckonings. Their use will decline further anyway, as
> long as they are not updated. I meant only to say high popularity increased
> the justification for converting these tools to python3. You imply low
> popularity suggests no need to upgrade the tools. Yet this argument would
> not hold.

Hey Mark, mozbase is almost entirely used internally by us to build and test Firefox (although there might be external users of the marionette packages). It's an essential part of our internal toolchain, it won't be going away any time soon. For our own reasons (some of which are outlined above), I think it is pretty much inevitable that we will make mozbase work with python 3 *at some point*, perhaps even sooner than later -- it really just depends on finding motivated person(s) to do the work.

As always, the people best qualified/placed to do this are tasked with other stuff which is higher priority, but there's nothing stopping someone with a bit of determination from making this happen. As RMS once said when asked when asked if there was a release date for some FSF package: "No. It will be done sooner if you help." :)
For the record, I don’t encourage non-Mozilla projects to rely on the Marionette Python packages.  If you can, please use geckodriver (https://github.com/mozilla/geckodriver) and a W3C WebDriver-conforming client binding such as Selenium (https://github.com/seleniumhq/selenium) wdclient (https://github.com/w3c/wpt-tools/tree/master/webdriver).
Any progress on adding 'Programming Language :: Python :: 2 :: Only' to the PyPI trove classifiers of the the above modules?
I like gps' suggested approach from comment 7. The only thing I'd add is that getting tox to work with the python unittests so we can run them on multiple versions should be a prerequisite for this. Until there's a way to do this, the python 3 support would likely regress quickly.
Looking into this, I'm not sure that tox is viable, however I'd be happy to be proven wrong. It looks to me like we'd need mach to support Python 3 before we'd be able to run the mozbase unit tests in continuous integration, which would be essential for preventing any of the mozbase packages from regressing after we introduce support for Python 3.
We could re-write |mach python-test| so it computes the necessary files, then passes them into a 'tox' subprocess. But I guess that's a bigger change than necessary. Running |mach python-test| with python 3 would be good enough, and if we need to add python 3 support to mach and parts of mozbuild along the way, then that's gravy.
Shouldn't we run those tests with Python 2 and 3 while we are in a transition period? We cannot change all of our tools using Mozbase at once, so I see trouble upcoming if we don't run with Python 2.
(In reply to Henrik Skupin (:whimboo) from comment #21)
> Shouldn't we run those tests with Python 2 and 3 while we are in a
> transition period? We cannot change all of our tools using Mozbase at once,
> so I see trouble upcoming if we don't run with Python 2.

I'd expect it would happen gradually, and we should continue to support and test against Python 2.
(In reply to Henrik Skupin (:whimboo) from comment #21)
> Shouldn't we run those tests with Python 2 and 3 while we are in a
> transition period? We cannot change all of our tools using Mozbase at once,
> so I see trouble upcoming if we don't run with Python 2.

Yes, I think we're proposing that instead of hooking up tox, we could just have two separate tasks. One for py2 and one for py3.
We're certainly not going to switch all of our existing CI over to Python 3 in one fell swoop! I think getting a minimal `mach python-test` job stood up and running with Python 3 would be a good start, even if it doesn't run any actual mozharness tests yet (maybe just a simple no-op test). If we get that running in CI that should ensure that we don't regress it. Then people can work on fixing individual mozharness modules+tests and enabling them.
I've raised bug 1378422 for adding the trove classifiers to include 'Programming Language :: Python :: 2 :: Only'.
Depends on: 1388019
Depends on: 1397849
Depends on: 1388016
Attachment #8516081 - Attachment is obsolete: true
As we close out 2017 there are only 11 Python3WoS packages that have not achieved Python 3 compatibility and 9 of those are MOZ packages: http://python3wos.appspot.com

The other two (httplib2 and supervisor) are already compatible in their repos so they will be green once they push their next releases.

I know that the Python3WoS selection criteria is out of date but still... 80+% does not look good.
(In reply to cclauss from comment #26)
> As we close out 2017 there are only 11 Python3WoS packages that have not
> achieved Python 3 compatibility and 9 of those are MOZ packages:
> http://python3wos.appspot.com

As you can see by the dependencies listed on this bug there is actually work happening for the mozbase modules. Two have already been fixed, and for some others we have patches to review. Updated versions will be pushed to PyPI when we are ready, which maybe could happen already in January - big thanks to Vedant's work here!
Many thanks for everyone's hard work in the dep bugs!

I noticed that for mozlog (which now supports Python 3 and has the Python 3 trove classifer added), the wheel is Python 2 only, rather than being a universal wheel:
https://pypi.python.org/pypi/mozlog

I'm not sure how the releases are created/automated, but ideally a `setup.cfg` could be created, with:
[bdist_wheel]
universal=1

...or failing that, `--universal` be added to whatever script runs the `setup.py bdist_wheel` step.
Thanks for the tip! Maybe we can add that as part of bug 1428362. I accidentally broke the mozlog py3 support by adding the mozterm dependency. Tbh, I'd maybe recommend we stop work on converting things until we have the unittests running under py3 (bug 1388013).
Depends on: 1428362
(In reply to Andrew Halberstadt [:ahal] from comment #29)
> Thanks for the tip! Maybe we can add that as part of bug 1428362. I
> accidentally broke the mozlog py3 support by adding the mozterm dependency.
> Tbh, I'd maybe recommend we stop work on converting things until we have the
> unittests running under py3 (bug 1388013).

Oh, I missed this comment and just filed all the other bugs. Those are not marked as mentored at least, so we don't expose them yet.
No longer blocks: 1466211
Depends on: 1466211
Blocks: 1520463
Blocks: py3
Alias: mozbase-py3
Type: defect → enhancement
See Also: → 1520458
Depends on: 1621226
QA Contact: hskupin
Depends on: 1642662
Depends on: 1642672

Looks like all dependencies are closed out. Let's resolve this and treat any new Python 3 issue that we run into in mozbase just like a normal regression.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: