1451518 - Make Django check pass under Python 3

Status: RESOLVED FIXED

(Tree Management :: Treeherder, enhancement, P2)

Description

[Ed Morley [:emorley]]

Bug 1428045 added flake8 to the Python 3 Travis sub-job, and made enough fixes to make it pass.

Next on the bug 1330474 journey is adding the django system check step, and making that pass. This check covers a fair amount of the Django code, and will ensure that we don't regress on the basics not covered by flake8 (such as the Python 3 module renames and similar).

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3400

Comment 2

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/e4b0f5378826b2576244f9b21ab621da1ff01d1e
Bug 1451518 - Add more 'from __future__ import print_function'

Generated by running:
`futurize -w -n -f print_with_import .`

...and then cleaning up import style using `isort -y`.

See:
https://github.com/PythonCharmers/python-future/blob/39a066ed8c29a0b3a32adac2ffd407119fe9ea6d/src/libfuturize/fixes/fix_print_with_import.py

https://github.com/mozilla/treeherder/commit/20ca827cc0224cb616e543a8ca3495ffb2a970e1
Bug 1451518 - Use absolute instead of relative imports

Relative imports found by running:
`futurize -w -n -f absolute_import .`

...however then hand-edited to convert to absolute, and the newly added
`from __future__ import absolute_import` removed (since we mostly
don't use relative imports, and PEP8 prefers absolute).

https://github.com/mozilla/treeherder/commit/c5a2e80b737b85ff55e2f855816e4a1557773b96
Bug 1451518 - Use next(obj) instead of obj.next()

Fixed via:
`futurize -w -n -f next_call .`

https://github.com/mozilla/treeherder/commit/46ced2c0310df6bddbbdba33d0548432a091a988
Bug 1451518 - Use more idiomatic type comparisons

Fixed via:
`futurize -w -n -f idioms .`

See:
https://docs.python.org/2/library/2to3.html#2to3fixer-idioms

https://github.com/mozilla/treeherder/commit/f67e75560356e55a7192b5ff9f193d4636890dc5
Bug 1451518 - Make all classes subclass object

Fixed via:
`futurize -w -n -f newstyle .`

...and then the unnecessary `from builtins import object` removed,
since none of our classes define `next`:
https://stackoverflow.com/q/36517648

https://github.com/mozilla/treeherder/commit/6dd2793602aaa168c4535454d898b2aa48f902a6
Bug 1451518 - Tweak six.moves.input usage

* Switches from the deprecated Django `six` shim to importing directly.
* No longer imports `input` into the global namespace, to stop `futurize`
  from falsely thinking it needs converting to use `future.builtins`.

https://github.com/mozilla/treeherder/commit/ee44d0086f148ea6a66472d8bf5fcbee408965f1
Bug 1451518 - Replace urlparse with six.moves.urlib.parse

https://pythonhosted.org/six/#module-six.moves
https://pythonhosted.org/six/#module-six.moves.urllib.parse

https://github.com/mozilla/treeherder/commit/2e8f09655f2f9c72965416f9cc1c6f1de67a5263
Bug 1451518 - Replace HTMLParser with six.moves.html_parser

https://pythonhosted.org/six/#module-six.moves

https://github.com/mozilla/treeherder/commit/b08c03132afb99ff5f10769b3131e87541fdefbb
Bug 1451518 - Fix SyntaxError in etl/text.py under Python 3

Even though the raw unicode (`ur`) string is in the Python 2 branch
of the conditional, the whole file was parsed, causing a `SyntaxError`
when running Django check under Python 3:

```
...
  File "/home/vagrant/treeherder/treeherder/etl/text.py", line 13
    filter_re = re.compile(ur"([\U00010000-\U0010FFFF])", re.U)  # noqa: E999
                                                       ^
SyntaxError: invalid syntax
```

The `ur` syntax isn't supported under Python 3 (since it was removed
by https://bugs.python.org/issue15096), however we can use the
workaround suggested here:
https://stackoverflow.com/a/33027349

https://github.com/mozilla/treeherder/commit/dd4c0dbec44d5aec287b1e78b388c7f2e2df9a7d
Bug 1451518 - Run Django check under Python 3 on Travis

This increases coverage over just running flake8, and provides a
stepping-stone prior to working on making the pytest tests pass too.