1471625 - [mozinfo] Add support for Python 3

Status: RESOLVED FIXED

(Testing :: Mozbase, enhancement, P3)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

+++ This bug was initially created as a clone of Bug #1426330 +++

We need to add Python 3 support without dropping support for legacy Python. Whilst some aspects were taken care of in bug 1426330 we are now able to run the unit tests against Python 3, and many of these are failing.

To work on this bug you will need to install and configure Mercurial, which will enable you to download the Firefox source code. It will also be used to commit your changes locally and prepare a patch for review. See the installation guide at https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/installing.html for help getting Mercurial on your system. Once installed, there are some extensions we recommend installing, details of these can be found here: https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/extensions.html

To clone the Firefox source code we recommend using the unified repository. Details of this and how to create a bookmark for your work can be found here: https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/unifiedrepo.html

As this bug relates to Python 3, you will need to have this installed on your system. Our continuous integration is currently using Python 3.5, so for best results we recommend using the same version locally. There are a number of ways to install Python, and they vary depending on your environment. We suggest reading over the guides at http://docs.python-guide.org/en/latest/starting/installation/#python-3-installation-guides to find the best method for you. Note that we also need to maintain support for Python 2, so you'll also need to have Python 2.7 installed.

Whilst we're moving towards adding support for Python 3, we've disabled any tests that fail against this version. This means that in order to work on this bug you will need to enable the tests. This can be done by removing "skip-if = python == 3" from the manifest file in `testing/mozbase/mozinfo/tests/manifest.ini`

To run the tests against Python 3, execute the following command:

```
mach python-test --python=3.5 testing/mozbase/mozinfo
```

Work through the test failures, and update the tests and source code for the package to simultaneously support Python 2.7 and Python 3.5. You can always check that your changes have not inadvertently broken support for Python 2 by using `--python=2.7` on the command line. 

If you're new to Python 3, the guide at https://docs.python.org/3/howto/pyporting.html may help you to get started with understanding the differences. To assist with supporting both Python 2 and 3, we have vendored the "six" package. You can read more about this and how to use it at https://pythonhosted.org/six/

Note that this package depends on others that may not yet support Python 3. If these dependencies are preventing tests from passing, then we can block this bug on getting support in those packages. If there are circular dependencies then we may need to coordinate landing a sequence of patches between bugs,

Once the package supports Python 3 and the tests pass, we will also need to prepare for a new release. Bump the version number in `testing/mozbase/mozinfo/setup.py` to "1.0.0", and update the classifiers so they reflect the versions of Python that we now support.

Finally, to prepare a universal distribution, create a `testing/mozbase/mozinfo/setup.cfg` file with the following contents:

```
[bdist_wheel]
universal=1
```

When your patch(es) are ready, you will need to push them for review. We are use MozReview for this, and instructions for how to prepare your machine and how to structure your commit messages can be found in our guide: https://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview-user.html

There may be some review issues to address, but once your contribution has been approved and subsequently landed, we will release the new version of the package!

Comment 1

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: Bug 1471625 - Resolve Python 3.5 compat issues in mozinfo;

Review commit: https://reviewboard.mozilla.org/r/254460/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/254460/

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8989369 [details]
Bug 1471625 - Resolve Python 3.5 compat issues in mozinfo;

https://reviewboard.mozilla.org/r/254460/#review261258

Excellent! Could you also take care of bumping the version number, updating the classifiers, and marking this package as universal? Details can be found in the bug comments.

Comment 3

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: Bug 1471625 - Create setup.cfg for universal wheels in mozinfo;

Review commit: https://reviewboard.mozilla.org/r/254488/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/254488/

Comment 4

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: Bug 1471625 - Bump mozinfo to v1.0.0 and update classifiers;

Review commit: https://reviewboard.mozilla.org/r/254490/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/254490/

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8989394 [details]
Bug 1471625 - Create setup.cfg for universal wheels in mozinfo;

https://reviewboard.mozilla.org/r/254488/#review261272

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8989395 [details]
Bug 1471625 - Bump mozinfo to v1.0.0 and update classifiers;

https://reviewboard.mozilla.org/r/254490/#review261274

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/70d0097e8c03
Resolve Python 3.5 compat issues in mozinfo;r=davehunt
https://hg.mozilla.org/integration/autoland/rev/217570468fde
Create setup.cfg for universal wheels in mozinfo;r=davehunt
https://hg.mozilla.org/integration/autoland/rev/7d0fb7a6b9fa
Bump mozinfo to v1.0.0 and update classifiers;r=davehunt

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We'll need one further patch once we land support for py3 to mozfile, which will update the dependencies. Once this has landed we'll be able to release new versions of mozinfo, mozfile, and mozhttpd.

Comment 9

[Cosmin Sabou [:CosminS]]

Backed out for linting failure on mozinfo/tests/test.py

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=186192037&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&fromchange=7d0fb7a6b9fac5ac3490baaabb35f55fdf4daf44&tochange=e29214662562389d45a7dd4b00c290e5e4c7e759&filter-searchStr=Linting%20opt%20source-test-mozlint-py-flake8%20(f8)

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=186192037&repo=autoland&lineNumber=270

Backout link: https://hg.mozilla.org/integration/autoland/rev/e29214662562389d45a7dd4b00c290e5e4c7e759

[task 2018-07-03T13:48:19.419Z] building 'psutil._psutil_posix' extension
[task 2018-07-03T13:48:19.419Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-07-03T13:48:19.420Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-07-03T13:48:19.420Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2018-07-03T13:48:19.420Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2018-07-03T13:48:19.420Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2018-07-03T13:48:19.420Z] 
[task 2018-07-03T13:48:19.420Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-07-03T13:49:01.534Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/mozbase/mozinfo/tests/test.py:27:1 | expected 2 blank lines, found 1 (E302)

Comment 10

[Cosmin Sabou [:CosminS]]

Please also take a look over these failures https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&filter-resultStatus=pending&filter-resultStatus=running&selectedJob=186197173&filter-searchStr=awsy

https://treeherder.mozilla.org/logviewer.html#?job_id=186197173&repo=autoland&lineNumber=550

07:12:40     INFO -      dists = ws.resolve(reqs, Environment())
07:12:40     INFO -    File "/Users/cltbld/tasks/task_1530627033/build/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 834, in resolve
07:12:40     INFO -      raise VersionConflict(dist, req).with_context(dependent_req)
07:12:40     INFO -  pkg_resources.ContextualVersionConflict: (mozinfo 1.0.0 (/Users/cltbld/tasks/task_1530627033/build/venv/lib/python2.7/site-packages), Requirement.parse('mozinfo<1,>=0.7'), set(['mozrunner']))
07:12:40    ERROR - Return code: 1
07:12:40    ERROR - No checks run.
07:12:40    ERROR - No suite end message was emitted by this harness.
07:12:40     INFO - AWSY exited with return code 1: FAILURE
07:12:40    ERROR - # TBPL FAILURE #
07:12:40  WARNING - setting return code to 2
07:12:40     INFO - Running post-action listener: _package_coverage_data
07:12:40     INFO - Running post-action listener: _resource_record_post_action
07:12:40     INFO - [mozharness: 2018-07-03 14:12:40.882955Z] Finished run-tests step (success)
07:12:40     INFO - Running post-run listener: _resource_record_post_run
07:12:40     INFO - Total resource usage - Wall time: 26s; CPU: 51.0%; Read bytes: 765952; Write bytes: 234846208; Read time: 17; Write time: 416
07:12:40     INFO - TinderboxPrint: CPU usage<br/>51.0%
07:12:40     INFO - TinderboxPrint: I/O read bytes / time<br/>765,952 / 17
07:12:40     INFO - TinderboxPrint: I/O write bytes / time<br/>234,846,208 / 416
07:12:40     INFO - TinderboxPrint: CPU idle<br/>50.9 (49.0%)
07:12:40     INFO - TinderboxPrint: CPU system<br/>22.4 (21.5%)
07:12:40     INFO - TinderboxPrint: CPU user<br/>30.7 (29.5%)
07:12:40     INFO - TinderboxPrint: Swap in / out<br/>230,260,736 / 0
07:12:40     INFO - install - Wall time: 22s; CPU: 53.0%; Read bytes: 228612096; Write bytes: 222951424; Read time: 16980; Write time: 387
07:12:40     INFO - run-tests - Wall time: 4s; CPU: 36.0%; Read bytes: 0; Write bytes: 1728512; Read time: 0; Write time: 3
07:12:40     INFO - Running post-run listener: copy_logs_to_upload_dir
07:12:40     INFO - Copying logs to upload dir...
07:12:40     INFO - mkdir: /Users/cltbld/tasks/task_1530627033/build/upload/logs

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:raphael the linter issue should be straightforward, but please let me know if you have any questions. The other issues are related to the dependencies from other packages. In particular we would need to update mozinstall [1], mozrunner [2], and tps [3].

[1] https://searchfox.org/mozilla-central/rev/97d488a17a848ce3bebbfc83dc916cf20b88451c/testing/mozbase/mozinstall/setup.py#19 
[2] https://searchfox.org/mozilla-central/rev/97d488a17a848ce3bebbfc83dc916cf20b88451c/testing/mozbase/mozrunner/setup.py#17
[3] https://searchfox.org/mozilla-central/rev/97d488a17a848ce3bebbfc83dc916cf20b88451c/testing/tps/setup.py#14

Comment 12

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: Bug 1471625 - Resolve Python 3.5 compat issues in mozinfo;

Review commit: https://reviewboard.mozilla.org/r/254942/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/254942/

Comment 13

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 8989933 [details]
Bug 1471625 - Resolve Python 3.5 compat issues in mozinfo;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/254942/diff/1-2/

Comment 14

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 8989394 [details]
Bug 1471625 - Create setup.cfg for universal wheels in mozinfo;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/254488/diff/1-2/

Comment 15

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 8989395 [details]
Bug 1471625 - Bump mozinfo to v1.0.0 and update classifiers;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/254490/diff/1-2/

Comment 16

[Raphael Aurich [:raphael] UTC+02:00]

Comment on attachment 8989395 [details]
Bug 1471625 - Bump mozinfo to v1.0.0 and update classifiers;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/254490/diff/2-3/

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8989933 [details]
Bug 1471625 - Resolve Python 3.5 compat issues in mozinfo;

https://reviewboard.mozilla.org/r/254942/#review261836

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e3a5a5a094d5
Resolve Python 3.5 compat issues in mozinfo;r=davehunt
https://hg.mozilla.org/integration/autoland/rev/dc681722b537
Create setup.cfg for universal wheels in mozinfo;r=davehunt
https://hg.mozilla.org/integration/autoland/rev/ff2f24e2bab5
Bump mozinfo to v1.0.0 and update classifiers;r=davehunt

Comment 19

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/e3a5a5a094d5
https://hg.mozilla.org/mozilla-central/rev/dc681722b537
https://hg.mozilla.org/mozilla-central/rev/ff2f24e2bab5

Comment 20

[Raphael Aurich [:raphael] UTC+02:00]

Thanks for the review!

Comment 21

[(not currently active) Ted Mielczarek]

(In reply to Dave Hunt [:davehunt] ⌚️UTC+1 (PTO until 22-October-2018) from comment #8)
> We'll need one further patch once we land support for py3 to mozfile, which
> will update the dependencies. Once this has landed we'll be able to release
> new versions of mozinfo, mozfile, and mozhttpd.

With bug 1471622 fixed, can this be closed now?

Comment 22

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Yes, I believe we can release version 1.0.0 of mozinfo and then resolve this bug. I can take care of this when I return from PTO on Monday if nobody gets to it before me.

Comment 23

[Raphael Aurich [:raphael] UTC+02:00]

I'm happy to push a release to PyPI tomorrow. Do we have any documentation on releasing mozbase Python packages to PyPI? Otherwise I would build an sdist and upload that with twine.

Comment 24

[Dave Hunt [:davehunt] [he/him] ⌚BST]

$ cd testing/mozbase/mozinfo
$ python setup.py sdist bdist_wheel
$ twine upload dist/mozinfo-1.0.0*

https://pypi.org/project/mozinfo/1.0.0/