Closed Bug 1362323 Opened 4 years ago Closed 1 year ago

Upgrade to python 3.6

Categories

(Webtools :: Pontoon, enhancement, P3)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emin, Assigned: jotes)

References

Details

Attachments

(1 file)

After 3 years python 2.x will be discontinued. Also new features will not come to python 2.x anymore. Also python 3.x consumes less memory in string variables (optimized for Unicode) and works a bit faster.

# Statements in wiki[1]:

List of features that will not be backported to python 2.x:
* strings are Unicode by default
* clean Unicode/bytes separation
* exception chaining
* function annotations
* syntax for keyword-only arguments
* extended tuple unpacking
* non-local variable declarations

Heroku supports python-3.6.1 runtime. Some operating systems (Linux distributions) are already using python 3.x as default.

# What needs to be done:
1. Checking all dependencies for python 3.6 support.
2. Updating documentation (for example specifying python version for virtualenv as MacOS and some Linux distros are currently using python 2.x as default).
3. Checking and updating code to support it.

I'll be happy to work on it, but as it is a major change, it needs to be discussed. Mostly about what can be side effects and etc.

[1] https://wiki.python.org/moin/Python2orPython3
Flags: needinfo?(mkelly)
Flags: needinfo?(m)
Thanks for filing, Emin!

I don't have enough insight right now, so I'm removing the NI.
Flags: needinfo?(m)
As of dependencies, compare-locales isn't py3 yet, and I don't expect to invest in an update on that until we can use python3 in the firefox build system. translate-toolkit seems to be fine with py3.

Also, django 2 will not support python 2 (oh the irony).
I don't have any major insight except "Yes, do this yesterday."

For dependencies like compare-locales, I'd recommend using six[1] to make them both Python 2 and Python 3 compatible. You can use tox[2] to help run your test suite under multiple versions of Python easily, as well.

[1] https://pythonhosted.org/six/
[2] https://tox.readthedocs.io/en/latest/
Flags: needinfo?(mkelly)
Priority: -- → P3
depends on bug 1386193
Depends on: 1386193
Depends on: 1388313
An interesting read on how Socorro got migrated to Python 3:
https://bluesock.org/~willkg/blog/mozilla/socorro_python3.html

A note from Pike:
"Splitting up requirements into requirements and constraints sounds nice"
Assignee: nobody → poke
Status: NEW → ASSIGNED
Summary: Updating environment to python 3.6 → Upgrade to python 3.6
Depends on: 1536670
Depends on: 1537613
Depends on: 1541252
Depends on: 1541260
Depends on: 1542084

:mathjazz

Hey,
Sorry, I got stuck for a month because of things related to my family and shrinking time resources (AWS exams and so on).
I saw a memo on the Pontoon Call about OKRs for Q4.
Unfortunately, the whole migration is blocked by the failing unit-tests. There's a lot of them.
One approach that comes to my mind group them by type (or per file?) and create a separate bug per group (to keep PR as small as possible).
Is that okay with you? I'll definitely need some help with fixes.

In order to help you make a decision, I've prepared a small analysis of the current state of things:
There are 21 types of failures/errors that occur during a test-run on Python 3.
Failures/errors sorted by number of occurrences:

  • 118 times - TypeError: object of type 'map' has no len()
  • 53 times - django.db.utils.DataError: invalid regular expression: quantifier operand invalid
  • 14 times - TypeError: can only concatenate list (not "dict_values") to list
  • 11 times - django.http.request.RawPostDataException: You cannot access body after reading from request's data stream
  • 7 times - TypeError: 'dict_values' object is not subscriptable
  • 5 times - TypeError: a bytes-like object is required, not 'str'
  • 4 times - AttributeError: 'bytes' object has no attribute 'traceback'
  • 3 times - TypeError: 'filter' object is not subscriptable
  • 3 times - TypeError: '>=' not supported between instances of 'MagicMock' and 'int'
  • 3 times - assert b'Bad Request: Request must be AJAX' == 'Bad Request: Request must be AJAX' + where b'Bad Request: Request must be AJAX' = <HttpResponseBadRequest status_code=400, "text/html; charset=utf-8">.content
  • 2 times - TypeError: 'odict_values' object is not subscriptable
  • 2 times - KeyError: 'has_next'
  • 1 times - AssertionError: DoesNotExist not raised
  • 1 times - TypeError: argument of type 'ResourceQuerySet' is not iterable
  • 1 times - AssertionError: assert 'p' == 'cl' - p + cl
  • 1 times - AssertionError: assert odict_keys(['...a', 'search']) == ['tag', 'type'...ta', 'search'] Use -v to get the full diff
  • 1 times - ValueError: must have exactly one of create/read/write/append mode
  • 1 times - TypeError: 'odict_items' object is not subscriptable
  • 1 times - AssertionError: assert '^(?s:.)\Z' == '^.$' - ^(?s:.)\Z + ^.$
  • 1 times - pontoon.base.models.DoesNotExist: Translation matching query does not exist.
  • 1 times - TypeError: sequence item 0: expected str instance, bytes found

Detailed analysis of them per file: https://gist.github.com/jotes/49c64b360ec29ee26efc6455a11fb853
There's a chance that some bigger problems/exceptions are hidden, but I don't think thats a blocker right now.

Flags: needinfo?(m)

:jotes, thanks for the analysis - this is very helpful.

I've filed bug 1585251 to track the unit test work. Later of we can file dependencies if needed, or simply reference that bug in commits.

Depends on: 1585251
Flags: needinfo?(m)

Thanks to the great work of jotes, then contributions by Pike, Matjaz and myself, this is now done. :-) Thanks everyone!

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.