Closed Bug 1532427 Opened 2 years ago Closed 2 months ago

Support python 3 in mozharness

Categories

(Release Engineering :: Applications: MozharnessCore, enhancement)

enhancement
Not set
normal

Tracking

(firefox73 fixed)

RESOLVED FIXED
Tracking Status
firefox73 --- fixed

People

(Reporter: davehunt, Assigned: gvelchuru)

References

(Depends on 2 open bugs, Blocks 2 open bugs)

Details

(Keywords: leave-open)

User Story

In order for automation built on mozharness to support Python 3, we need to add support to mozharness itself. A good first step would be to get the mozharness unit tests passing in Python 3.

To work on this bug you will need to install and configure Mercurial, which will enable you to download the Firefox source code. It will also be used to commit your changes locally and prepare a patch for review. See the [installation guide](https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/installing.html) for help getting Mercurial on your system. Once installed, there are some extensions we recommend installing, details of these can be found [here](https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/extensions.html).

To clone the Firefox source code we recommend using the unified repository. Details of this and how to create a bookmark for your work can be found [here](https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/unifiedrepo.html).

As this bug relates to Python 3, you will need to have this installed on your system. Our continuous integration is currently using Python 3.5, so for best results we recommend using the same version locally. There are a number of ways to install Python, and they vary depending on your environment. We suggest reading over [these guides](http://docs.python-guide.org/en/latest/starting/installation/#python-3-installation-guides) to find the best method for you. Note that we also need to maintain support for Python 2, so you'll also need to have Python 2.7 installed.

To run the tests against Python 3, add a `py35-hg4.3` environment to `testing/mozharness/tox.ini` and then run `tox -e py35-hg4.3` from `testing/mozharness`.

Work through the test failures, and update the tests and source code for the package to simultaneuously support Python 2.7 and Python 3.5. You can always check that your changes have not inadvertantly broken support for Python 2 by using `-e py27-hg4.3` on the command line. 

If you're new to Python 3, [this guide](https://docs.python.org/3/howto/pyporting.html) may help you to get started with understanding the differences. To assist with supporting both Python 2 and 3, we have vendored the "six" package. You can read more about this and how to use it [here](https://pythonhosted.org/six/).

Once the package supports Python 3 and the tests pass, we will also need to bump the version number in `testing/mozharness/mozharness/__ini__.py`, and update the classifiers in `testing/mozharness/setup.py` so they reflect the versions of Python that we now support.

When your patch(es) are ready, you will need to push them for review. We are use Phabricator for this, and instructions for how to prepare your machine and how to structure your commit messages can be found [in our guide](https://wiki.mozilla.org/Phabricator).

Attachments

(8 files, 10 obsolete files)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
No description provided.
No longer blocks: 1520463
Blocks: 1520463

Can I work on this issue? @davehunt. Also I wanted to know what needs to be done of my patch for Bug 1520463?

(In reply to Utkarsh from comment #1)

Can I work on this issue? @davehunt. Also I wanted to know what needs to be done of my patch for Bug 1520463?

Yes, and working on this bug will unblock bug 1520463. Please use the "Need more information from" feature below the comment box when you need to get someone's attention for feedback. Thanks!

When I try to run 'tox -e py27-hg4.3' I am getting this error

No file to run: 'C:\mozilla-unified\testing\mozharness\.tox\py27-hg4.3\Scripts\nosetests'
ERROR: InvocationError for command 'C:\\mozilla-unified\\testing\\mozharness\\.tox\\py27-hg4.3\\Scripts\\coverage.EXE run -- 
source configs,mozharness,scripts --branch C:\\mozilla-unified\\testing\\mozharness\\.tox\\py27-hg4.3\\Scripts\\nosetests -v - 
-with-xunit --rednose --force-color' (exited with code 1)

I get the same error with 'tox -e py35-hg4.3'

Flags: needinfo?(dave.hunt)

Perhaps try reducing that test command to see what's introducing the error. Do you have anything else in your console that might suggest the problem?

Flags: needinfo?(dave.hunt)

Currently working on this!

Attached file Bug 1532427 - stage 1 pass (obsolete) —

Depends on D35092

Attached file Bug 1532427 - fixing merkle (obsolete) —

Depends on D35093

Attached file Bug 1532427 - no clue (obsolete) —

Depends on D35094

Attached file Bug 1532427 - progress (obsolete) —

Depends on D35095

Attached file Bug 1532427 - --amend (obsolete) —

Depends on D35096

Attached file Bug 1532427 - fix keys sorting (obsolete) —

Depends on D35097

Attached file Bug 1532427 - mercurial errors now (obsolete) —

Depends on D35098

Attached file Bug 1532427 - 25 errors (obsolete) —

Depends on D35099

Attached file Bug 1532427 - 24 (obsolete) —

Depends on D35100

Hi, I posted some patches and I was able to get it down to 24 bugs. The remaining bugs are related to mercurial, so if possible I would like some help/pointers because I can't wrap my head around them.

Flags: needinfo?(dave.hunt)

It may be worth landing all non-mercurial patches once they're ready, and fix mercurial as a followup.

Gautham: Thanks for the patches. I've looked over a few and have some concerns over the approach. Could you look over my comments and respond in general? What approach are you using? Are you using a tool to add support for Python 3? What are te 24 bugs related to mercurial that you are referring to?

Could you also revise your commit messages to provide context on the files that are being modified? Messages such as "progress" and "no clue" will not be helpful when looking at the commit log.

Flags: needinfo?(dave.hunt)

Work to support Python 3 has been gaining speed, and mozharness is blocking several harnesses from running against Python 3. It's possible that we may decide to pick this up soon, but I'd like to give Gautham some time to address the previous comments.

Flags: needinfo?(gvelchuru)
Attachment #9072304 - Attachment is obsolete: true
Attachment #9072303 - Attachment is obsolete: true
Attachment #9072301 - Attachment is obsolete: true
Attachment #9072300 - Attachment is obsolete: true
Attachment #9072299 - Attachment is obsolete: true
Attachment #9072298 - Attachment is obsolete: true
Attachment #9072297 - Attachment is obsolete: true

Hey Dave,

I have largely been using the futurize tool to add support for python3. However, when running tox with the py3.5 environment, I still get 24 bugs relating some of to the mercurial tests (all other tests pass). I will certainly cleanup my commits and address the comments.

Flags: needinfo?(gvelchuru)
Attachment #9072296 - Attachment is obsolete: true
Attachment #9072294 - Attachment is obsolete: true
Attachment #9072849 - Attachment description: Bug 153247 - Fix some python3 errors in mozharness, still 24 errors remaining when running tox → Bug 1532427 - Fix some python3 errors in mozharness, still 24 errors remaining when running tox
Attachment #9072849 - Attachment description: Bug 1532427 - Fix some python3 errors in mozharness, still 24 errors remaining when running tox → Bug 1532427 - Fix some python3 errors in mozharness, still 3 errors remaining when running tox
Attachment #9072849 - Attachment description: Bug 1532427 - Fix some python3 errors in mozharness, still 3 errors remaining when running tox → Bug 1532427 - Fix some python3 errors in mozharness, still failing mercurial tests

Comment on attachment 9072849 [details]
Bug 1532427 - Fix some python3 errors in mozharness, still failing mercurial tests

Hi,

I fixed a few more of the tests! I dug into the errors some more -- it looks like Mercurial doesn't support unicode strings for the configuration variables in python3. I tried passing bytestrings in, but it seemed to make no difference. I wasn't exactly sure what the problem is, so maybe someone with more Mercurial experience can take a look? Here's a sample error:

======================================================================
23) ERROR: test_update_revision (test_base_vcs_mercurial.TestHg)
----------------------------------------------------------------------
   Traceback (most recent call last):
    test/test_base_vcs_mercurial.py line 158 in test_update_revision
      rev = m.clone(self.repodir, self.wc, update_dest=False)
    mozharness/base/vcs/mercurial.py line 229 in clone
      raise VCSException("Unable to clone %s to %s!" % (repo, dest))
   VCSException: Unable to clone /tmp/tmpy30hbja8/repo to /tmp/tmpy30hbja8/wc!

   -------------------- >> begin captured stdout << ---------------------
   Running command: ['bash', '/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/test/helper_files/init_hgrepo.sh', '/tmp/tmpy30hbja8/repo']
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   Return code: 0
   Getting output from command: ['hg', '--config', b'ui.merge=internal:merge', 'log', '-R', '/tmp/tmpy30hbja8/repo', '--template', '{node}\n']
   Reading from file tmpfile_stdout
   Output received:
    6aded2bc7aba4632d4e75783769685819e3d3002
    95dfe38c19a6b392f1de692c9ccb4189be4810a6
    c6ab9f10ec719dfd7638b7d5711af99381395e9f
   Cloning /tmp/tmpy30hbja8/repo to /tmp/tmpy30hbja8/wc.
   Running command: ['hg', '--config', b'ui.merge=internal:merge', 'clone', '-U', '/tmp/tmpy30hbja8/repo', '/tmp/tmpy30hbja8/wc']
    ** https://mercurial-scm.org/wiki/BugTracker
    ** Python 3.5.7 (default, Jun  9 2019, 21:27:40) [GCC 8.3.0]
    ** Mercurial Distributed SCM (version 4.3.1)
    ** Extensions loaded: graphlog, strip, mq, rebase, share, convert, transplant, pager, purge
    Traceback (most recent call last):
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/bin/hg", line 45, in <module>
        mercurial.dispatch.run()
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 81, in run
        status = (dispatch(req) or 0) & 255
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 162, in dispatch
        ret = _runcatch(req)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 302, in _runcatch
        return _callcatch(ui, _runcatchfunc)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 310, in _callcatch
        return scmutil.callcatch(ui, func)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/scmutil.py", line 150, in callcatch
        return func()
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 292, in _runcatchfunc
        return _dispatch(req)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 896, in _dispatch
        cmdpats, cmdoptions)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 658, in runcommand
        ret = _runcommand(ui, options, cmd, d)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/extensions.py", line 322, in closure
        return func(*(args + a), **kw)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/hgext/pager.py", line 69, in pagecmd
        return orig(ui, options, cmd, cmdfunc)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 904, in _runcommand
        return cmdfunc()
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 893, in <lambda>
        d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/util.py", line 1077, in check
        return func(*args, **kwargs)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/extensions.py", line 322, in closure
        return func(*(args + a), **kw)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/util.py", line 1077, in check
        return func(*args, **kwargs)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/hgext/share.py", line 140, in clone
        return orig(ui, source, *args, **opts)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/util.py", line 1077, in check
        return func(*args, **kwargs)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/commands.py", line 1410, in clone
        shareopts=opts.get('shareopts'))
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/hg.py", line 651, in clone
        destrepo.ui.setconfig('paths', 'default', defaulturl, 'clone')
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/ui.py", line 431, in setconfig
        cfg.set(section, name, value, source)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/config.py", line 75, in set
        'config values may not be unicode strings on Python 3')
    AssertionError: b'config values may not be unicode strings on Python 3'

   --------------------- >> end captured stdout << ----------------------
Flags: needinfo?(dave.hunt)
Attachment #9072849 - Flags: review+

It looks like :egao has been looking at this bug and its dependencies. I'm not involved in this area much these days, so rerouting the needinfo/review. I would highlight though that this bug doesn't have an assignee. If you're still working on it Gautham then we should assign it to you, although your last update was 5 months ago, and Python 2 will reach end of life within the next few months.

Flags: needinfo?(dave.hunt) → needinfo?(egao)

I can definitely pick up work on it, so that we can make the deadline. I'm just blocked on the mercurial issues.

:gvelchuru - yes, I have done some work in the related bugs 1595982 and 1597616, broken up by subdirectory.

I would prefer if the patch was smaller and more self-contained such that it is easier to review and keep in mind the context. For example, changes to manifestparser.py will conflict with the changes made by :championshuttler in bug 1428705.

Changes to mozharness/base/ would conflict with the now-landed patches that I have authored, so they should not be included in D35332.
As a side note, I added a test environment for python3.5 in tox which would conflict with your patch here.

To reiterate, my preference would be to have the patch broken up into smaller chunks as well as having the patch be rebased to work off the current or reasonably updated mozilla-central. Take one file or subdirectory and work in small chunks at a time, solving issues that running tox -e py35-hg4.3 would throw up for the file you're working on, and repeat.

For me that workflow looks like:

  1. identify a file/subdirectory to tackle (that does not overlap with existing work in progress by another engineer)
  2. fix the issues
  3. put up patch
  4. land
  5. once landed, pull updated mozilla-central, repeat from step 1.

This way multiple engineers can tackle the same issue and work in parallel.

Flags: needinfo?(egao)
Assignee: nobody → gvelchuru
Attachment #9072849 - Attachment is obsolete: true

How do I land the patch so I can get to work on the other fixes?

Flags: needinfo?(egao)

Open the link above to go to your patch. I've approved proposed patch.

I'm not sure if you have landing permission, so let's find out:

  1. in your patch, click on 'view in lando' on the right sidebar
  2. attempt to land

If the landing button is grey then you likely don't have landing permission, so I'll have to land it on your behalf. We can try doing it, but my concern is that your patch would have 'bitrotted' by now and landing will fail. Let's try it anyway.

Flags: needinfo?(egao)

Looks like I cannot land it

Thanks for checking. I've landed the patch for you, let's see if everything goes well.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f8c76d71f5b
Fix tox errors in mozharness/mozharness/mozilla when running python 3.5 r=egao
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED

Reopening because work is not complete; this should have a leave-open keyword until all work is done.

Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---

The new tox.ini isn't building for me, I get the following error. Am I missing something?

https://paste.gg/p/anonymous/e97bc96dd9374f0c91c7456d250bf31e

Flags: needinfo?(egao)
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/acfb7cdad469
add mozharness python3 task as tier 3 r=jmaher

(In reply to Gautham Velchuru from comment #36)

The new tox.ini isn't building for me, I get the following error. Am I missing something?

https://paste.gg/p/anonymous/e97bc96dd9374f0c91c7456d250bf31e

    collect2: error: ld returned 1 exit status
	
    error: command 'gcc' failed with exit status 1

This is your issue - I assume you're on some sort of Linux distribution (eg. ubuntu, debian, fedora).

If so, you'll need follow the instructions here:
https://stackoverflow.com/questions/11094718/error-command-gcc-failed-with-exit-status-1-while-installing-eventlet

Flags: needinfo?(egao)
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/febbd4aba93f
Fix mozharness/test/test_mozilla_merkle when running tox with python 3.5 r=egao

Looks like all tests pass now, should I mark as resolved?

Flags: needinfo?(egao)

(In reply to Gautham Velchuru from comment #42)

Looks like all tests pass now, should I mark as resolved?

Not quite - though the tests now pass, there are some other works still needed in files within mozharness that aren't directly checked by the tests.

Feel free to unassign yourself from this bug though, if you are interested in something else.

Attachment #9118231 - Attachment description: Bug 1532427 - sanitize docs/conf.py of python2 specific code → Bug 1532427 - sanitize mozharness/docs/conf.py of python2 specific code
Attachment #9118231 - Attachment description: Bug 1532427 - sanitize mozharness/docs/conf.py of python2 specific code → Bug 1532427 - miscellaneous small fixes of mozharness files
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/06e14c2602ca
miscellaneous small fixes of mozharness files r=aki

Clearing ni

Flags: needinfo?(egao)
Depends on: 1672181
Depends on: 1679434
No longer blocks: 1599102

I'm going to resolve this because there is no possible way for us to hit every code path in mozharness to verify it works under Python 3. Instead, let's leave the mozharness conversion up to ad-hoc fixes as we run into them.

Status: REOPENED → RESOLVED
Closed: 1 year ago2 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.