1428718 - [moztest] Add support for Python 3

Status: RESOLVED FIXED

(Testing :: Mozbase, enhancement, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Without dropping support for legacy Python, we need to add support for Python 3 to moztest.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

To work on this bug you will need to install and configure Mercurial, which will enable you to download the Firefox source code. It will also be used to commit your changes locally and prepare a patch for review. See the installation guide at https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/installing.html for help getting Mercurial on your system. Once installed, there are some extensions we recommend installing, details of these can be found here: https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/extensions.html

To clone the Firefox source code we recommend using the unified repository. Details of this and how to create a bookmark for your work can be found here: https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/unifiedrepo.html

As this bug relates to Python 3, you will need to have this installed on your system. Our continuous integration is currently using Python 3.5, so for best results we recommend using the same version locally. There are a number of ways to install Python, and they vary depending on your environment. We suggest reading over the guides at http://docs.python-guide.org/en/latest/starting/installation/#python-3-installation-guides to find the best method for you. Note that we also need to maintain support for Python 2, so you'll also need to have Python 2.7 installed.

Whilst we're moving towards adding support for Python 3, we've disabled any tests that fail against this version. This means that in order to work on this bug you will need to enable the tests. This can be done by removing "skip-if = python == 3" from the manifest file in `testing/mozbase/moztest/tests/manifest.ini`

To run the tests against Python 3, execute the following command:

```
mach python-test --python=3.5 testing/mozbase/moztest
```

Work through the test failures, and update the tests and source code for the package to simultaneously support Python 2.7 and Python 3.5. You can always check that your changes have not inadvertently broken support for Python 2 by using `--python=2.7` on the command line. 

If you're new to Python 3, the guide at https://docs.python.org/3/howto/pyporting.html may help you to get started with understanding the differences. To assist with supporting both Python 2 and 3, we have vendored the "six" package. You can read more about this and how to use it at https://pythonhosted.org/six/

Note that this package depends on others that may not yet support Python 3. If these dependencies are preventing tests from passing, then we can block this bug on getting support in those packages. If there are circular dependencies then we may need to coordinate landing a sequence of patches between bugs.

Once the package supports Python 3 and the tests pass, we will also need to prepare for a new release. Bump the version number in `testing/mozbase/moztest/setup.py` to "1.0.0", and update the classifiers so they reflect the versions of Python that we now support.

Finally, to prepare a universal distribution, create a `testing/mozbase/moztest/setup.cfg` file with the following contents:

```
[bdist_wheel]
universal=1
```

When your patch(es) are ready, you will need to push them for review. We are use MozReview for this, and instructions for how to prepare your machine and how to structure your commit messages can be found in our guide: https://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview-user.html

There may be some review issues to address, but once your contribution has been approved and subsequently landed, we will release the new version of the package!

Comment 2

[clairebitterness]

Hello, I would like to work on this issue

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to clairebitterness from comment #2)

Hello, I would like to work on this issue

Sure, feel free to get started. Once you uploaded the first patch, the bug will be assigned to you.

Comment 4

[Pooja Gadige (:pgadige__)(:pooja_)]

Since a patch has not yet been uploaded, I wonder if I can also work on this one? If not, I'll appreciate if you could please suggest a good-first-issue for me.

Thanks!

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Pooja Gadige (:pgadige__)(:pooja) from comment #4)

Since a patch has not yet been uploaded, I wonder if I can also work on this one?

Yes, as stated in comment 3 we will assign this once a patch has been submitted.

Comment 6

[Vrinda Singhal]

Attachment: [moztest] Added support for Python 3

Without dropping support for legacy Python, need to add support for Python 3 to moztest.

Comment 7

[Vrinda Singhal]

Hello @davehunt,
I'm an Outreachy Applicant and started working on this bug, I have uploaded a partial patch, kindly review and let me know if I'm working in the right direction. Also, I couldn't figure out the fix to the following error, could use some help here.
I changed-

• import __builtin__ to import builtins [1]
• b: getattr(__builtin__, b) to b: getattr(__builtins__, b) [2]

Error- https://pastebin.com/raw/XWgkkURV

[1] https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/configure/__init__.py#7
[2] https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/configure/__init__.py#283

Comment 8

[Vrinda Singhal]

(In reply to Vrinda Singhal from comment #7)

Hello @davehunt,
I'm an Outreachy Applicant and started working on this bug, I have uploaded a partial patch, kindly review and let me know if I'm working in the right direction. Also, I couldn't figure out the fix to the following error, could use some help here.
I changed-

• import __builtin__ to import builtins [1]
• b: getattr(__builtin__, b) to b: getattr(__builtins__, b) [2]

Error- https://pastebin.com/raw/XWgkkURV

[1] https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/configure/__init__.py#7
[2] https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/configure/__init__.py#283

Hi @ahal,
Could you please help me out here? Thanks!

Comment 9

[Andrew Halberstadt [:ahal]]

Hi Vrinda,

Really sorry, it looks like Dave and I both missed your comments here. For future reference you need to use the needinfo feature to get someone's attention, using @ doesn't cause any notifications or anything (and it is very easy to miss bug comments in our e-mail).

I know it's been awhile and the Outreachy term is long gone, but is there any chance you are still interested in working on this? If I don't hear back within a few days I'll assume the answer is no :)

Comment 10

[Andrew Halberstadt [:ahal]]

Going with no.

Comment 11

[TEJASWI]

I am new to Mozilla and i have recently started contributing to it. I have the entire firefox environment set. I would like to work on this bug while using some mentoring. Please do let me know if i can start working on this..!

Comment 12

[Shivam Singhal [ :championshuttler ]]

Hi Tejaswi , Thanks for your interest, I guess bug 1428711 will be a good start for you. Let us know if you need any help.

Comment 13

[TEJASWI]

Hi shivam , thanks for giving me a chance for " bug 1428711 " . i am going to start it and if i need any help then i ping you a message .

Comment 14

[TEJASWI]

Hi Shivam , please assign bug 1428711 to so that I start work on it

Comment 15

[Shivam Singhal [ :championshuttler ]]

Tejaswi , we assign the bug once you raised a patch for it.

Comment 16

[Edwin Takahashi (:egao | infrequent contributor)]

:ahal - given that there is nothing using moztest in-tree at this point, and the mozbase documentation at this link doesn't even mention moztest, is it worth keeping this in-tree?

EDIT: I spoke too soon, there are a couple of places it's being used outside of the moztest module itself.

For whoever that works on this (if no one does, I'll just go ahead and make the change to strike this one off the list) tests all pass on python2/3 currently, and a good starting point might the patch that's been left up.

Comment 17

[Edwin Takahashi (:egao | infrequent contributor)]

Removing unnecessary ni from :ahal.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I can see quiet a good amount of usage for this package:
https://searchfox.org/mozilla-central/search?q=from+moztest&case=false&regexp=false&path=

Comment 19

[Andrew Halberstadt [:ahal]]

I think the initial intent of moztest was to provide a generic test framework (that uses all the various components of mozbase), that other test frameworks could inherit from. But that never ended up happening.

Instead it is basically just a utility module now.

Comment 20

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1428718 - remove moztest/moztest/output directory

Comment 21

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1428718 - make moztest syntax python3 compatible

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a4fb397fa5c
remove moztest/moztest/output directory r=gbrown,mozbase

Comment 23

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/8a4fb397fa5c

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Only half of the patch set landed yet.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/17386ce6dd6f
make moztest syntax python3 compatible r=gbrown,mozbase

Comment 26

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/17386ce6dd6f

Comment 27

[Edwin Takahashi (:egao | infrequent contributor)]

Here's the current state of things, to reduce confusion for whoever that reads this in the future (like myself after a few weeks):

Bug 1599261 is blocking the enabling of test_resolve.py. This is because both the test and resolve.py imports many modules from mozbuild which is not yet python3 compatible at the time of writing.

Once the imported mozbuild modules have python3 compatibility, test_resolve.py can be worked on and fixed for python3.

After that, bump either the bugfix or minor version up, release to pypi and this bug can be closed.

Comment 28

[Geoff Brown [:gbrown]]

Attachment: Bug 1428718 - [moztest] Support python 3; r=

I noticed one more moztest issue in bug 1638993: sorting an array of dict is
poorly defined on py2 and not tolerated on py3; adding the lambda clarifies
the behavior. With this resolved, I think we can bump the moztest version and
close this bug.

Comment 29

[Geoff Brown [:gbrown]]

ni to remind myself to release to pypi.

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e5f3c99e509
[moztest] Support python 3; r=egao

Comment 31

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/2e5f3c99e509

Comment 32

[Geoff Brown [:gbrown]]

https://pypi.org/project/moztest/1.1.0/