1330474 - Switch Treeherder from Python 2.7 to Python 3.6

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

The upcoming Django 1.11 release (April 2017) will be the last that supports Python 2.7. Django 2.0 (due December 2017) will only support Python 3.5+.

See:
https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django

Now that we're no longer using Datasource (which was Python 2 only) virtually all (if not all) of our dependencies are Python 3 compatible.

This will still be some work (and no rush until nearer the end of this year), but it's now actually within the realm of possibility.

Comment 1

[Ed Morley [:emorley]]

As of today Django master is Python 3.x only.

(Django 1.11a1 was released yesterday, the final release will be in ~2 months from the stable/1.11.x branch).

Comment 2

[Rupanshu Verma]

Hi
I would really like to contribute in open source and specially in my domain i.e python.A little help would be highly appreciated.
Thanks!

Comment 3

[Ed Morley [:emorley]]

Hi Rupanshu - thank you for your interest in contributing!

This hadn't been marked as a "good first bug" since it's probably a bit more of a tricky one, however if you still want to take a look, the best place to start is making sure all of the Python packages used by Treeherder are Python 3 compatible:
https://pyup.io/account/repos/github/mozilla/treeherder/
(Click the "Python 3" button to show the compatibility column)

Currently the following aren't marked as compatible with Python 3:
* mozlog
  -> bug 1388019 filed for this
* futures
  -> though this won't be needed for Python 3.
* functools32
  -> though this won't be needed for Python 3.
* coreapi
  -> I've previously filed https://github.com/core-api/python-client/issues/130
* coreschema
  -> I've previously filedhttps://github.com/core-api/python-coreschema/issues/6
* itypes
  -> I've previously filed https://github.com/tomchristie/itypes/issues/6
* sphinxcontrib-httpdomain
  -> though this only really runs on Read The Docs, so isn't urgent.
  -> perhaps still file an issue here: https://bitbucket.org/birkenfeld/sphinx-contrib ?

For the ones that aren't needed on Python 3, use the special requirements file notation to limit them to Python2:
https://www.python.org/dev/peps/pep-0508/#environment-markers

For the rest, I'd:
1) Check if their tests are already running against Python 3.
2) If so, just update the trove classifiers in setup.py to make it clear that they do support Python 3.
3) If not:
  * Make their tests run against Python 3 (for things on GitHub this will typically mean adding python 3 to .travis.yml; for mozlog this will mean depending on bug 1388013 and bug 1385381)
  * Fix any test failures
  * Add the trove classifiers the same as above

Once all of that is complete, next steps will likely be running some of the Python 3 linters against the Treeherder codebase to see what needs changing there.

Comment 4

[Ed Morley [:emorley]]

coreschema and core-api just needed trove classifier updates, so I've opened:
https://github.com/core-api/python-client/pull/144
https://github.com/core-api/python-coreschema/pull/9

itypes didn't actually run its tests on Travis yet, so I've added a Travis config and the trove classifiers (since the tests already passed on Python 3):
https://github.com/tomchristie/itypes/pull/8

Comment 5

[Ed Morley [:emorley]]

For inspiration as to an initial lint check we could try, see bug 1390968.

Comment 6

[Ed Morley [:emorley]]

Not that it blocks the work here, but I've opened an issue against Pyup.io to see if they could fix the "PY3: X" notations appearing against the packages that are not intended for use with Python 3 (and for which we've already added the conditional syntax to the requirements file), to avoid confusion:
https://github.com/pyupio/pyup/issues/247

Comment 7

[Ed Morley [:emorley]]

Some useful resources:
https://eev.ee/blog/2016/07/31/python-faq-how-do-i-port-to-python-3/
http://python-future.org/

Comment 8

[Ed Morley [:emorley]]

Django 1.11 is EOL on April 2020:
https://www.djangoproject.com/download/#supported-versions

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #7)
> Some useful resources:

Also:
https://docs.djangoproject.com/en/1.11/topics/python3/

Comment 10

[Ed Morley [:emorley]]

There are now three bugs directly or indirectly blocked by this conversion work. Since it's only just over a year before Django 1.11 is EOL, I think we need to prioritise this soon - since we'll need to not only switch to Python 3, but also upgrade through 3 breaking change versions of Django to be on a supported release in that timeframe.

On the plus side, I believe this is now very close. I have a few WIP branches locally for some of the remaining issues, though hit some quirks wrt log parsing and line numbers not matching when using requests' decode_unicode=True in streaming mode.

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4619

Comment 12

[Ed Morley [:emorley]]

Comment on attachment 9043034 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4619

(Wrong bug number, meant to use bug 1527020)

Comment 13

[Ed Morley [:emorley]]

Ah so it turns out that Celery doesn't yet support Python 3.7 (fix on celery master, not yet released: https://github.com/celery/celery/issues/4500), so we'll need to use Python 3.6 instead for now. Once they support Python 3.7, and we've had a chance to figure out the celery upgrade issue in bug 1337717, we can then upgrade.

This incompatibility didn't show up during tests likely due to bug 1349362. It could be seen via Papertrail in the form of:

app/worker_store_pulse_jobs.4:   File "/app/.heroku/python/lib/python3.7/site-packages/celery/concurrency/prefork.py", line 20, in <module> 
app/worker_store_pulse_jobs.4:     from celery.concurrency.base import BasePool 
app/worker_store_pulse_jobs.4:   File "/app/.heroku/python/lib/python3.7/site-packages/celery/concurrency/base.py", line 21, in <module> 
app/worker_store_pulse_jobs.4:     from celery.utils import timer2 
app/worker_store_pulse_jobs.4:   File "/app/.heroku/python/lib/python3.7/site-packages/celery/utils/timer2.py", line 19 
app/worker_store_pulse_jobs.4:     from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger 
app/worker_store_pulse_jobs.4:                    ^ 
app/worker_store_pulse_jobs.4: SyntaxError: invalid syntax

Comment 14

[Ed Morley [:emorley]]

A Python 3 compatible version of treeherder-client will be released in bug 1529191.

Comment 15

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4668

Comment 16

[Ed Morley [:emorley]]

Switch to Python 3 landed in:
https://github.com/mozilla/treeherder/commit/ff2993d2fb562676ab933e5dd3e5ddb4604f8d65

Comment 17

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4675

Comment 18

[Ed Morley [:emorley]]

The switch to Python 3 was deployed to production at 08:12am UTC Thursday 21st.

No issues seen so far. Only notable change I've seen is a ~20% reduction in memory usage on certain worker dyno types, which is a welcome additional benefit :-)

Once this has been deployed for a bit longer, we'll remove Travis testing against Python 2, at which point this bug can be closed, and PRs opened for the bugs this one is marked as blocking.

Comment 19

[Ed Morley [:emorley]]

Everything has been looking great on prod using Python 3 since last week.

Testing of Python 2 (for fallback purposes) removed in:
https://github.com/mozilla/treeherder/commit/a0d1f5fccfeb0a6cd6593d5017e4b70a8a191446

We're now fine to move on with the bugs that required we drop Python 2 support.