1532427 - Support python 3 in mozharness

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, enhancement)

Comment 1

[Utkarsh]

Can I work on this issue? @davehunt. Also I wanted to know what needs to be done of my patch for Bug 1520463?

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Utkarsh from comment #1)

Can I work on this issue? @davehunt. Also I wanted to know what needs to be done of my patch for Bug 1520463?

Yes, and working on this bug will unblock bug 1520463. Please use the "Need more information from" feature below the comment box when you need to get someone's attention for feedback. Thanks!

Comment 3

[Utkarsh]

When I try to run 'tox -e py27-hg4.3' I am getting this error

No file to run: 'C:\mozilla-unified\testing\mozharness\.tox\py27-hg4.3\Scripts\nosetests'
ERROR: InvocationError for command 'C:\\mozilla-unified\\testing\\mozharness\\.tox\\py27-hg4.3\\Scripts\\coverage.EXE run -- 
source configs,mozharness,scripts --branch C:\\mozilla-unified\\testing\\mozharness\\.tox\\py27-hg4.3\\Scripts\\nosetests -v - 
-with-xunit --rednose --force-color' (exited with code 1)

I get the same error with 'tox -e py35-hg4.3'

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Perhaps try reducing that test command to see what's introducing the error. Do you have anything else in your console that might suggest the problem?

Comment 5

[Gautham Velchuru]

Currently working on this!

Comment 6

[Gautham Velchuru]

Attachment: Bug 1532427 - stage 1 pass

Comment 7

[Gautham Velchuru]

Attachment: Bug 1532427 - fix script.py

Depends on D35092

Comment 8

[Gautham Velchuru]

Attachment: Bug 1532427 - fixing merkle

Depends on D35093

Comment 9

[Gautham Velchuru]

Attachment: Bug 1532427 - no clue

Depends on D35094

Comment 10

[Gautham Velchuru]

Attachment: Bug 1532427 - progress

Depends on D35095

Comment 11

[Gautham Velchuru]

Attachment: Bug 1532427 - --amend

Depends on D35096

Comment 12

[Gautham Velchuru]

Attachment: Bug 1532427 - fix keys sorting

Depends on D35097

Comment 13

[Gautham Velchuru]

Attachment: Bug 1532427 - mercurial errors now

Depends on D35098

Comment 14

[Gautham Velchuru]

Attachment: Bug 1532427 - 25 errors

Depends on D35099

Comment 15

[Gautham Velchuru]

Attachment: Bug 1532427 - 24

Depends on D35100

Comment 16

[Gautham Velchuru]

Hi, I posted some patches and I was able to get it down to 24 bugs. The remaining bugs are related to mercurial, so if possible I would like some help/pointers because I can't wrap my head around them.

Comment 17

[Aki Sasaki (not active)]

It may be worth landing all non-mercurial patches once they're ready, and fix mercurial as a followup.

Comment 18

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Gautham: Thanks for the patches. I've looked over a few and have some concerns over the approach. Could you look over my comments and respond in general? What approach are you using? Are you using a tool to add support for Python 3? What are te 24 bugs related to mercurial that you are referring to?

Could you also revise your commit messages to provide context on the files that are being modified? Messages such as "progress" and "no clue" will not be helpful when looking at the commit log.

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Work to support Python 3 has been gaining speed, and mozharness is blocking several harnesses from running against Python 3. It's possible that we may decide to pick this up soon, but I'd like to give Gautham some time to address the previous comments.

Comment 20

[Gautham Velchuru]

Hey Dave,

I have largely been using the futurize tool to add support for python3. However, when running tox with the py3.5 environment, I still get 24 bugs relating some of to the mercurial tests (all other tests pass). I will certainly cleanup my commits and address the comments.

Comment 21

[Gautham Velchuru]

Attachment: Bug 1532427 - Fix some python3 errors in mozharness, still failing mercurial tests

Comment 22

[Gautham Velchuru]

Comment on attachment 9072849 [details]
Bug 1532427 - Fix some python3 errors in mozharness, still failing mercurial tests

Hi,

I fixed a few more of the tests! I dug into the errors some more -- it looks like Mercurial doesn't support unicode strings for the configuration variables in python3. I tried passing bytestrings in, but it seemed to make no difference. I wasn't exactly sure what the problem is, so maybe someone with more Mercurial experience can take a look? Here's a sample error:

======================================================================
23) ERROR: test_update_revision (test_base_vcs_mercurial.TestHg)
----------------------------------------------------------------------
   Traceback (most recent call last):
    test/test_base_vcs_mercurial.py line 158 in test_update_revision
      rev = m.clone(self.repodir, self.wc, update_dest=False)
    mozharness/base/vcs/mercurial.py line 229 in clone
      raise VCSException("Unable to clone %s to %s!" % (repo, dest))
   VCSException: Unable to clone /tmp/tmpy30hbja8/repo to /tmp/tmpy30hbja8/wc!

   -------------------- >> begin captured stdout << ---------------------
   Running command: ['bash', '/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/test/helper_files/init_hgrepo.sh', '/tmp/tmpy30hbja8/repo']
    1 files updated, 0 files merged, 0 files removed, 0 files unresolved
   Return code: 0
   Getting output from command: ['hg', '--config', b'ui.merge=internal:merge', 'log', '-R', '/tmp/tmpy30hbja8/repo', '--template', '{node}\n']
   Reading from file tmpfile_stdout
   Output received:
    6aded2bc7aba4632d4e75783769685819e3d3002
    95dfe38c19a6b392f1de692c9ccb4189be4810a6
    c6ab9f10ec719dfd7638b7d5711af99381395e9f
   Cloning /tmp/tmpy30hbja8/repo to /tmp/tmpy30hbja8/wc.
   Running command: ['hg', '--config', b'ui.merge=internal:merge', 'clone', '-U', '/tmp/tmpy30hbja8/repo', '/tmp/tmpy30hbja8/wc']
    ** https://mercurial-scm.org/wiki/BugTracker
    ** Python 3.5.7 (default, Jun  9 2019, 21:27:40) [GCC 8.3.0]
    ** Mercurial Distributed SCM (version 4.3.1)
    ** Extensions loaded: graphlog, strip, mq, rebase, share, convert, transplant, pager, purge
    Traceback (most recent call last):
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/bin/hg", line 45, in <module>
        mercurial.dispatch.run()
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 81, in run
        status = (dispatch(req) or 0) & 255
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 162, in dispatch
        ret = _runcatch(req)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 302, in _runcatch
        return _callcatch(ui, _runcatchfunc)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 310, in _callcatch
        return scmutil.callcatch(ui, func)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/scmutil.py", line 150, in callcatch
        return func()
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 292, in _runcatchfunc
        return _dispatch(req)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 896, in _dispatch
        cmdpats, cmdoptions)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 658, in runcommand
        ret = _runcommand(ui, options, cmd, d)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/extensions.py", line 322, in closure
        return func(*(args + a), **kw)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/hgext/pager.py", line 69, in pagecmd
        return orig(ui, options, cmd, cmdfunc)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 904, in _runcommand
        return cmdfunc()
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/dispatch.py", line 893, in <lambda>
        d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/util.py", line 1077, in check
        return func(*args, **kwargs)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/extensions.py", line 322, in closure
        return func(*(args + a), **kw)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/util.py", line 1077, in check
        return func(*args, **kwargs)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/hgext/share.py", line 140, in clone
        return orig(ui, source, *args, **opts)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/util.py", line 1077, in check
        return func(*args, **kwargs)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/commands.py", line 1410, in clone
        shareopts=opts.get('shareopts'))
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/hg.py", line 651, in clone
        destrepo.ui.setconfig('paths', 'default', defaulturl, 'clone')
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/ui.py", line 431, in setconfig
        cfg.set(section, name, value, source)
      File "/mnt/data1/gdrive/open_source/mozilla_unified/testing/mozharness/.tox/py35-hg4.3/lib/python3.5/site-packages/mercurial/config.py", line 75, in set
        'config values may not be unicode strings on Python 3')
    AssertionError: b'config values may not be unicode strings on Python 3'

   --------------------- >> end captured stdout << ----------------------

Comment 23

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It looks like :egao has been looking at this bug and its dependencies. I'm not involved in this area much these days, so rerouting the needinfo/review. I would highlight though that this bug doesn't have an assignee. If you're still working on it Gautham then we should assign it to you, although your last update was 5 months ago, and Python 2 will reach end of life within the next few months.

Comment 24

[Gautham Velchuru]

I can definitely pick up work on it, so that we can make the deadline. I'm just blocked on the mercurial issues.

Comment 25

[Edwin Takahashi (:egao | infrequent contributor)]

:gvelchuru - yes, I have done some work in the related bugs 1595982 and 1597616, broken up by subdirectory.

I would prefer if the patch was smaller and more self-contained such that it is easier to review and keep in mind the context. For example, changes to manifestparser.py will conflict with the changes made by :championshuttler in bug 1428705.

Changes to mozharness/base/ would conflict with the now-landed patches that I have authored, so they should not be included in D35332.
As a side note, I added a test environment for python3.5 in tox which would conflict with your patch here.

To reiterate, my preference would be to have the patch broken up into smaller chunks as well as having the patch be rebased to work off the current or reasonably updated mozilla-central. Take one file or subdirectory and work in small chunks at a time, solving issues that running tox -e py35-hg4.3 would throw up for the file you're working on, and repeat.

For me that workflow looks like:

1. identify a file/subdirectory to tackle (that does not overlap with existing work in progress by another engineer)
2. fix the issues
3. put up patch
4. land
5. once landed, pull updated mozilla-central, repeat from step 1.

This way multiple engineers can tackle the same issue and work in parallel.

Comment 26

[Gautham Velchuru]

Attachment: Bug 1532427 - Fix tox errors in mozharness/mozharness/mozilla when running python 3.5

Comment 27

[Gautham Velchuru]

How do I land the patch so I can get to work on the other fixes?

Comment 28

[Edwin Takahashi (:egao | infrequent contributor)]

Open the link above to go to your patch. I've approved proposed patch.

I'm not sure if you have landing permission, so let's find out:

1. in your patch, click on 'view in lando' on the right sidebar
2. attempt to land

If the landing button is grey then you likely don't have landing permission, so I'll have to land it on your behalf. We can try doing it, but my concern is that your patch would have 'bitrotted' by now and landing will fail. Let's try it anyway.

Comment 29

[Gautham Velchuru]

Looks like I cannot land it

Comment 30

[Edwin Takahashi (:egao | infrequent contributor)]

Thanks for checking. I've landed the patch for you, let's see if everything goes well.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f8c76d71f5b
Fix tox errors in mozharness/mozharness/mozilla when running python 3.5 r=egao

Comment 32

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6f8c76d71f5b

Comment 33

[Edwin Takahashi (:egao | infrequent contributor)]

Reopening because work is not complete; this should have a leave-open keyword until all work is done.

Comment 34

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1532427 - add mozharness python3 task as tier 3

Comment 35

[Gautham Velchuru]

Attachment: Bug 1532427 - Fix mozharness/test/test_mozilla_merkle when running tox with python 3.5 r=egao

Comment 36

[Gautham Velchuru]

The new tox.ini isn't building for me, I get the following error. Am I missing something?

https://paste.gg/p/anonymous/e97bc96dd9374f0c91c7456d250bf31e

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/acfb7cdad469
add mozharness python3 task as tier 3 r=jmaher

Comment 38

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Gautham Velchuru from comment #36)

The new tox.ini isn't building for me, I get the following error. Am I missing something?

https://paste.gg/p/anonymous/e97bc96dd9374f0c91c7456d250bf31e

    collect2: error: ld returned 1 exit status
	
    error: command 'gcc' failed with exit status 1

This is your issue - I assume you're on some sort of Linux distribution (eg. ubuntu, debian, fedora).

If so, you'll need follow the instructions here:
https://stackoverflow.com/questions/11094718/error-command-gcc-failed-with-exit-status-1-while-installing-eventlet

Comment 39

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/acfb7cdad469

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/febbd4aba93f
Fix mozharness/test/test_mozilla_merkle when running tox with python 3.5 r=egao

Comment 41

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/febbd4aba93f

Comment 42

[Gautham Velchuru]

Looks like all tests pass now, should I mark as resolved?

Comment 43

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Gautham Velchuru from comment #42)

Looks like all tests pass now, should I mark as resolved?

Not quite - though the tests now pass, there are some other works still needed in files within mozharness that aren't directly checked by the tests.

Feel free to unassign yourself from this bug though, if you are interested in something else.

Comment 44

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1532427 - miscellaneous small fixes of mozharness files

Comment 45

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/06e14c2602ca
miscellaneous small fixes of mozharness files r=aki

Comment 46

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/06e14c2602ca

Comment 47

[Edwin Takahashi (:egao | infrequent contributor)]

Clearing ni

Comment 48

[Hamzah Akhtar]

Attachment: Bug 1532427 - make mozharness::scripts python3 compatible

Comment 49

[Hamzah Akhtar]

Attachment: Bug 1532427 - make mozharness::test python3 compatible

Comment 50

[Hamzah Akhtar]

Attachment: Bug 1532427 - make mozharness::external_tools and mozharness::examples python3 compatible

Comment 51

[Andrew Halberstadt [:ahal]]

I'm going to resolve this because there is no possible way for us to hit every code path in mozharness to verify it works under Python 3. Instead, let's leave the mozharness conversion up to ad-hoc fixes as we run into them.