1362323 - Upgrade to python 3.6

Status: RESOLVED FIXED

(Webtools :: Pontoon, enhancement, P3)

Description

[Emin Mastizada]

After 3 years python 2.x will be discontinued. Also new features will not come to python 2.x anymore. Also python 3.x consumes less memory in string variables (optimized for Unicode) and works a bit faster.

# Statements in wiki[1]:

List of features that will not be backported to python 2.x:
* strings are Unicode by default
* clean Unicode/bytes separation
* exception chaining
* function annotations
* syntax for keyword-only arguments
* extended tuple unpacking
* non-local variable declarations

Heroku supports python-3.6.1 runtime. Some operating systems (Linux distributions) are already using python 3.x as default.

# What needs to be done:
1. Checking all dependencies for python 3.6 support.
2. Updating documentation (for example specifying python version for virtualenv as MacOS and some Linux distros are currently using python 2.x as default).
3. Checking and updating code to support it.

I'll be happy to work on it, but as it is a major change, it needs to be discussed. Mostly about what can be side effects and etc.

[1] https://wiki.python.org/moin/Python2orPython3

Comment 1

[Matjaz Horvat [:mathjazz]]

Thanks for filing, Emin!

I don't have enough insight right now, so I'm removing the NI.

Comment 2

[Axel Hecht [:Pike]]

As of dependencies, compare-locales isn't py3 yet, and I don't expect to invest in an update on that until we can use python3 in the firefox build system. translate-toolkit seems to be fine with py3.

Also, django 2 will not support python 2 (oh the irony).

Comment 3

[Osmose [:osmose, :mkelly]]

I don't have any major insight except "Yes, do this yesterday."

For dependencies like compare-locales, I'd recommend using six[1] to make them both Python 2 and Python 3 compatible. You can use tox[2] to help run your test suite under multiple versions of Python easily, as well.

[1] https://pythonhosted.org/six/
[2] https://tox.readthedocs.io/en/latest/

Comment 4

[Emin Mastizada]

Wiki to track updates: https://wiki.mozilla.org/L10n:Pontoon/Python_3
Pad: https://public.etherpad-mozilla.org/p/pontoon.python3

Comment 5

[Emin Mastizada]

depends on bug 1386193

Comment 6

[Matjaz Horvat [:mathjazz]]

An interesting read on how Socorro got migrated to Python 3:
https://bluesock.org/~willkg/blog/mozilla/socorro_python3.html

A note from Pike:
"Splitting up requirements into requirements and constraints sounds nice"

Comment 7

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/pontoon/pull/1360

Comment 8

[[DEACTIVATED] Adrian Gaudebert]

adngdb merged PR #1360: "Bug 1362323 - Fix tests for the administration app." in 7f1336f.

Comment 9

[Jarek Śmiejczak [:jotes]]

:mathjazz

Hey,
Sorry, I got stuck for a month because of things related to my family and shrinking time resources (AWS exams and so on).
I saw a memo on the Pontoon Call about OKRs for Q4.
Unfortunately, the whole migration is blocked by the failing unit-tests. There's a lot of them.
One approach that comes to my mind group them by type (or per file?) and create a separate bug per group (to keep PR as small as possible).
Is that okay with you? I'll definitely need some help with fixes.

In order to help you make a decision, I've prepared a small analysis of the current state of things:
There are 21 types of failures/errors that occur during a test-run on Python 3.
Failures/errors sorted by number of occurrences:

• 118 times - TypeError: object of type 'map' has no len()
• 53 times - django.db.utils.DataError: invalid regular expression: quantifier operand invalid
• 14 times - TypeError: can only concatenate list (not "dict_values") to list
• 11 times - django.http.request.RawPostDataException: You cannot access body after reading from request's data stream
• 7 times - TypeError: 'dict_values' object is not subscriptable
• 5 times - TypeError: a bytes-like object is required, not 'str'
• 4 times - AttributeError: 'bytes' object has no attribute 'traceback'
• 3 times - TypeError: 'filter' object is not subscriptable
• 3 times - TypeError: '>=' not supported between instances of 'MagicMock' and 'int'
• 3 times - assert b'Bad Request: Request must be AJAX' == 'Bad Request: Request must be AJAX' + where b'Bad Request: Request must be AJAX' = <HttpResponseBadRequest status_code=400, "text/html; charset=utf-8">.content
• 2 times - TypeError: 'odict_values' object is not subscriptable
• 2 times - KeyError: 'has_next'
• 1 times - AssertionError: DoesNotExist not raised
• 1 times - TypeError: argument of type 'ResourceQuerySet' is not iterable
• 1 times - AssertionError: assert 'p' == 'cl' - p + cl
• 1 times - AssertionError: assert odict_keys(['...a', 'search']) == ['tag', 'type'...ta', 'search'] Use -v to get the full diff
• 1 times - ValueError: must have exactly one of create/read/write/append mode
• 1 times - TypeError: 'odict_items' object is not subscriptable
• 1 times - AssertionError: assert '^(?s:.)\Z' == '^.$' - ^(?s:.)\Z + ^.$
• 1 times - pontoon.base.models.DoesNotExist: Translation matching query does not exist.
• 1 times - TypeError: sequence item 0: expected str instance, bytes found

Detailed analysis of them per file: https://gist.github.com/jotes/49c64b360ec29ee26efc6455a11fb853
There's a chance that some bigger problems/exceptions are hidden, but I don't think thats a blocker right now.

Comment 10

[Matjaz Horvat [:mathjazz]]

:jotes, thanks for the analysis - this is very helpful.

I've filed bug 1585251 to track the unit test work. Later of we can file dependencies if needed, or simply reference that bug in commits.

Comment 11

[[DEACTIVATED] Adrian Gaudebert]

Thanks to the great work of jotes, then contributions by Pike, Matjaz and myself, this is now done. :-) Thanks everyone!