Closed Bug 1550848 Opened 5 years ago Closed 5 years ago

OSX Mojave (10.14) - __ needs to be installed on the mojave machines

Categories

(Infrastructure & Operations :: RelOps: Hardware, task)

Product:
Infrastructure & Operations
Component:
RelOps: Hardware
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jmaher, Assigned: dhouse)

References

Details

Attachments

(1 file)

GitHub Pull Request
60 bytes, text/x-github-pull-request
Details | Review

in some of our test jobs we require python 3.6:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c8e77564df62b704ebfb9f502a13dd7275978590&searchStr=python%2Cosx

the tasks are trying to run:
Executing command 1: '/tools/python36/bin/python3.6' run-task '--gecko-checkout=./checkouts/gecko' -- bash -cx 'cd $GECKO_PATH && ./mach python-test --python /tools/python36/bin/python3.6 --subsuite mozbase'

and get this error:
fork/exec /tools/python36/bin/python3.6: no such file or directory

would it be possible that the python path is moved? for python2 we use:
/usr/local/bin/python2

Flags: needinfo?(jwatkins)

On one of the workers, this is what I'm seeing:

[dhouse@t-mojave-r7-467.test.releng.mdc1.mozilla.com ~]$ ls -la /tools
total 0
drwxr-xr-x   6 root  wheel  192 Apr  9 10:53 .
drwxr-xr-x  30 root  wheel  960 Apr  9 10:50 ..
lrwxr-xr-x   1 root  wheel   22 Apr  9 10:53 python -> /usr/local/bin/python2
lrwxr-xr-x   1 root  wheel   22 Apr  9 10:53 python2 -> /usr/local/bin/python2
lrwxr-xr-x   1 root  wheel   22 Apr  9 10:53 python3 -> /usr/local/bin/python3
lrwxr-xr-x   1 root  wheel   26 Apr  9 10:53 tooltool.py -> /usr/local/bin/tooltool.py
[dhouse@t-mojave-r7-467.test.releng.mdc1.mozilla.com ~]$ /tools/python3 --version
Python 3.7.3

we're linking the python3 binary in to /tools/python3: https://github.com/mozilla-platform-ops/ronin_puppet/blob/5a2bd738b844c372bb691edef994b1ca62c57288/modules/roles_profiles/manifests/profiles/gecko_t_osx_1014_generic_worker.pp#L77

in puppetagain, we have a link for python37:
https://github.com/mozilla-releng/build-puppet/blob/28fa5998bdacae681f7fc82a868faa335d8a69b2/modules/packages/manifests/mozilla/python3.pp#L34
it looks like the python36 path is being created by the install package:
https://github.com/mozilla-releng/build-puppet/blob/b705bae33f559192c37997f3a30c2c0a8796657c/modules/packages/manifests/mozilla/python3-dmg.sh#L17

Thanks :dhouse, I was able to use /tools/python3 successfully.

I also had to hack our references for 'hg', to just be 'hg' instead of a full path.

When doing that, I run into a missing hg extension 'robustcheckout':

[vcs 2019-05-12T11:20:46.990Z] executing ['hg', 'robustcheckout', '--sharebase', '/Users/cltbld/tasks/task_1557612785/checkouts/hg-shared', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '6d3c9724ab80e0cbdfac22f41d11a43bb2dad752', 'https://hg.mozilla.org/try', '/Users/cltbld/tasks/task_1557612785/checkouts/gecko']
[vcs 2019-05-12T11:20:47.281Z] *** failed to import extension robustcheckout from /usr/local/lib/hgext/robustcheckout.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/robustcheckout.py'
[vcs 2019-05-12T11:20:47.282Z] Traceback (most recent call last):
[vcs 2019-05-12T11:20:47.282Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/extensions.py", line 258, in loadall
[vcs 2019-05-12T11:20:47.282Z] load(ui, name, path, loadingtime)
[vcs 2019-05-12T11:20:47.282Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/extensions.py", line 180, in load
[vcs 2019-05-12T11:20:47.282Z] mod = _importext(name, path, bind(_reportimporterror, ui))
[vcs 2019-05-12T11:20:47.282Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/extensions.py", line 108, in _importext
[vcs 2019-05-12T11:20:47.282Z] mod = loadpath(path, 'hgext.%s' % name)
[vcs 2019-05-12T11:20:47.282Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/extensions.py", line 89, in loadpath
[vcs 2019-05-12T11:20:47.282Z] return imp.load_source(module_name, path)
[vcs 2019-05-12T11:20:47.282Z] IOError: [Errno 2] No such file or directory: '/usr/local/lib/hgext/robustcheckout.py'
[vcs 2019-05-12T11:20:47.315Z] Traceback (most recent call last):
[vcs 2019-05-12T11:20:47.315Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/scmutil.py", line 165, in callcatch
[vcs 2019-05-12T11:20:47.315Z] return func()
[vcs 2019-05-12T11:20:47.315Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/dispatch.py", line 367, in _runcatchfunc
[vcs 2019-05-12T11:20:47.315Z] return _dispatch(req)
[vcs 2019-05-12T11:20:47.315Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/dispatch.py", line 880, in _dispatch
[vcs 2019-05-12T11:20:47.315Z] cmd, func, args, options, cmdoptions = _parse(lui, args)
[vcs 2019-05-12T11:20:47.315Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/dispatch.py", line 687, in _parse
[vcs 2019-05-12T11:20:47.315Z] ui.configbool("ui", "strict"))
[vcs 2019-05-12T11:20:47.315Z] File "/usr/local/Cellar/mercurial/4.9.1/lib/python2.7/site-packages/mercurial/cmdutil.py", line 740, in findcmd
[vcs 2019-05-12T11:20:47.315Z] raise error.UnknownCommand(cmd, allcmds)
[vcs 2019-05-12T11:20:47.315Z] UnknownCommand: ('robustcheckout', ['qheader', 'debugsuccessorssets', 'help', 'tip', 'qgoto', 'config', 'showconfig', 'debugconfig', 'debugdate', 'debuglocks', 'qrefresh', 'qprev', 'debugdata', 'forget', 'debugwhyunstable', 'annotate', 'blame', 'addremove', 'init', 'debugrename', 'qtop', 'debugapplystreamclonebundle', 'debugwireproto', 'rollback', 'debugmergestate', 'pull', 'qseries', 'debugformat', 'debugrevspec', 'debugwalk', 'status', 'st', 'qdelete', 'qremove', 'qrm', 'resolve', 'qdiff', 'incoming', 'in', 'debugdag', 'bisect', 'unshare', 'rebase', 'revert', 'manifest', 'merge', 'debugpushkey', 'debuguiprompt', 'update', 'up', 'checkout', 'co', 'locate', 'log', 'history', 'debugnamecomplete', 'qinit', 'qapplied', 'debugindexstats', 'export', 'debugfileset', 'debuggetbundle', 'debugpeer', 'outgoing', 'out', 'debugupgraderepo', 'version', 'debugrebuilddirstate', 'debugrebuildstate', 'debugancestor', 'commit', 'ci', 'tags', 'debugcolor', 'qcommit', 'qci', 'debugpathcomplete', 'purge', 'clean', 'debugcheckstate', 'debugpvec', 'qpush', 'qselect', 'diff', 'summary', 'sum', 'qsave', 'qqueue', 'debugdeltachain', 'backout', 'qimport', 'rename', 'move', 'mv', 'debugcapabilities', 'debugsparse', 'qguard', 'strip', 'debugssl', 'identify', 'id', 'qclone', 'qnew', 'heads', 'debugsetparents', 'add', 'debugignore', 'debugobsolete', 'debugupdatecaches', 'parents', 'qunapplied', 'qpop', 'files', 'import', 'patch', 'debugrebuildfncache', 'serve', 'debugdiscovery', 'debugwireargs', 'push', 'debugpickmergetool', 'branch', 'unbundle', 'qnext', 'debugcomplete', 'grep', 'debugserve', 'cat', 'qfinish', 'debugrevlog', 'qrename', 'qmv', 'debugcommands', 'root', 'debugsub', 'debuglabelcomplete', 'debugmanifestfulltextcache', 'qrestore', 'share', 'tag', 'debugtemplate', 'recover', 'archive', 'graft', 'paths', 'debuguigetpass', 'verify', 'debugextensions', 'debugfsinfo', 'debuginstall', 'copy', 'cp', 'clone', 'bundle', 'debugknown', 'bookmarks', 'bookmark', 'debugbuilddag', 'debugindex', 'phase', 'remove', 'rm', 'branches', 'debugdownload', 'debugindexdot', 'debugdirstate', 'debugstate', 'debugcreatestreamclonebundle', 'debugrevlogindex', 'debugbundle', 'qfold'])
[vcs 2019-05-12T11:20:47.571Z] hg: unknown command 'robustcheckout'
[vcs 2019-05-12T11:20:47.571Z] (did you mean checkout?)

Is there any other tips for installing robustcheckout, or using a different path, version of hg?

is there something I should do differently for finding the path to hg

(In reply to Dave House [:dhouse] from comment #3)

I see robustcheckout being set up in roninpuppet and added to the hgrc for the user:
https://github.com/mozilla-platform-ops/ronin_puppet/blob/d4b09f1e8f1858437741f0cfbe9bac29abca1429/modules/mercurial/templates/hgrc.erb#L19
But the path is the same as the failed (not found) path:
'/usr/local/lib/hgext'
https://github.com/mozilla-platform-ops/ronin_puppet/blob/d4b09f1e8f1858437741f0cfbe9bac29abca1429/modules/mercurial/manifests/settings.pp#L7

Looking on one of the workers, there is no hqext directory. Maybe it is not applying through the mercurial module:

[dhouse@t-mojave-r7-467.test.releng.mdc1.mozilla.com ~]$ hg --version
*** failed to import extension robustcheckout from /usr/local/lib/hgext/robustcheckout.py: [Errno 2] No such file or directory: '/usr/local/lib/hgext/robustcheckout.py'
Attached file GitHub Pull Request —

This works for me when I tested:

[dhouse@t-mojave-r7-467.test.releng.mdc1.mozilla.com ~]$ sudo su - cltbld
[cltbld@t-mojave-r7-467.test.releng.mdc1.mozilla.com ~]$ hg robustcheckout --sharebase /Users/cltbld/tasks/task_1557612785/checkouts/hg-shared --purge --upstream https://hg.mozilla.org/mozilla-unified --revision 6d3c9724ab80e0cbdfac22f41d11a43bb2dad752 https://hg.mozilla.org/try /Users/cltbld/tasks/task_1557612785/checkouts/gecko
(using Mercurial 4.9.1)
ensuring https://hg.mozilla.org/try@6d3c9724ab80e0cbdfac22f41d11a43bb2dad752 is available at /Users/cltbld/tasks/task_1557612785/checkouts/gecko
(cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
(sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/b83d8a064f1694627e66f2dd3a683b66c350b3b3.zstd-max.hg
adding changesets
changesets [============>                                                                                                          ]  60625/529117 55s
Assignee: nobody → dhouse
Summary: OSX Mojave (10.14) - python 3.6 needs to be installed on the mojave machines → OSX Mojave (10.14) - __ needs to be installed on the mojave machines
Blocks: 1530418
Blocks: 1541951

I've run puppet with the robustcheckout branch on the test pool:

$ for hn in t-mojave-r7-{467..467}.test.releng.mdc1.mozilla.com; do ssh -o LogLevel=ERROR -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o ConnectTimeout=180 -o UserKnownHostsFile=/dev/null $hn 'sudo bash -c "rm -rf /usr/local/lib/hgext; export PATH=/usr/local/bin:$PATH:/opt/puppetlabs/bin; cd /etc/puppet/environments/production/code/; git log -1; cat .git/HEAD; git remote -vv; git remote rename origin upstream; git remote add origin https://github.com/davehouse/ronin_puppet.git; python3 --version; which pip3; export PUPPET_REPO=https://github.com/davehouse/ronin_puppet.git; export PUPPET_BRANCH=bug1550848_worker-python-deps; /usr/local/bin/run-puppet.sh"; ls -la /usr/local/lib/hgext; /usr/local/bin/hg --version'; done

some did not have the repo locally and so I've re-bootstrapped them:

$ for hn in t-mojave-r7-{466,468,469,470}.test.releng.mdc1.mozilla.com; do ssh -o LogLevel=ERROR -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o ConnectTimeout=180 -o UserKnownHostsFile=/dev/null $hn 'sudo bash -c "export PATH=/usr/local/bin:$PATH:/opt/puppetlabs/bin; curl -LO https://raw.githubusercontent.com/mozilla-platform-ops/ronin_puppet/master/provisioners/macos/bootstrap_mojave.sh; chmod u+x bootstrap_mojave.sh; sed -i tmp \"s/mozilla-platform-ops\/ronin_puppet\/archive\/master/davehouse\/ronin_puppet\/archive\/bug1550848_worker-python-deps/\" ./bootstrap_mojave.sh; grep github bootstrap_mojave.sh; ./bootstrap_mojave.sh "'; done

this solution works for the hgext, now I have new problems:
[task 2019-05-13T22:45:14.092Z] Successfully built virtualenv-clone
[task 2019-05-13T22:45:14.092Z] Installing collected packages: virtualenv-clone
[task 2019-05-13T22:45:14.118Z] Successfully installed virtualenv-clone-0.3.0
[task 2019-05-13T22:45:14.945Z] Traceback (most recent call last):
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/bin/pipenv", line 11, in <module>
[task 2019-05-13T22:45:14.945Z] load_entry_point('pipenv==2018.5.18', 'console_scripts', 'pipenv')()
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in call
[task 2019-05-13T22:45:14.945Z] return self.main(*args, **kwargs)
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
[task 2019-05-13T22:45:14.945Z] rv = self.invoke(ctx)
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1043, in invoke
[task 2019-05-13T22:45:14.945Z] return Command.invoke(self, ctx)
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
[task 2019-05-13T22:45:14.945Z] return ctx.invoke(self.callback, **ctx.params)
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
[task 2019-05-13T22:45:14.945Z] return callback(*args, **kwargs)
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
[task 2019-05-13T22:45:14.945Z] return f(get_current_context(), *args, **kwargs)
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/cli.py", line 270, in cli
[task 2019-05-13T22:45:14.945Z] three=three, python=python, warn=True, site_packages=site_packages
[task 2019-05-13T22:45:14.945Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/core.py", line 620, in ensure_project
[task 2019-05-13T22:45:14.946Z] three=three, python=python, site_packages=site_packages
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/core.py", line 553, in ensure_virtualenv
[task 2019-05-13T22:45:14.946Z] if not project.virtualenv_exists:
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/project.py", line 225, in virtualenv_exists
[task 2019-05-13T22:45:14.946Z] if self.pipfile_exists and os.path.exists(self.virtualenv_location):
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/project.py", line 315, in virtualenv_location
[task 2019-05-13T22:45:14.946Z] loc = self._get_virtualenv_location(self.virtualenv_name)
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/project.py", line 238, in _get_virtualenv_location
[task 2019-05-13T22:45:14.946Z] from .patched.pew.pew import get_workon_home
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/build/mach_bootstrap.py", line 393, in call
[task 2019-05-13T22:45:14.946Z] module = self._original_import(name, globals, locals, fromlist, level)
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/patched/pew/init.py", line 3, in <module>
[task 2019-05-13T22:45:14.946Z] from . import pew
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/build/mach_bootstrap.py", line 393, in call
[task 2019-05-13T22:45:14.946Z] module = self._original_import(name, globals, locals, fromlist, level)
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/patched/pew/pew.py", line 44, in <module>
[task 2019-05-13T22:45:14.946Z] from pew._utils import (check_call, invoke, expandpath, own, env_bin_dir,
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/build/mach_bootstrap.py", line 393, in call
[task 2019-05-13T22:45:14.946Z] module = self._original_import(name, globals, locals, fromlist, level)
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/patched/pew/init.py", line 3, in <module>
[task 2019-05-13T22:45:14.946Z] from . import pew
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/build/mach_bootstrap.py", line 393, in call
[task 2019-05-13T22:45:14.946Z] module = self._original_import(name, globals, locals, fromlist, level)
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/patched/pew/pew.py", line 44, in <module>
[task 2019-05-13T22:45:14.946Z] from pew._utils import (check_call, invoke, expandpath, own, env_bin_dir,
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/build/mach_bootstrap.py", line 393, in call
[task 2019-05-13T22:45:14.946Z] module = self._original_import(name, globals, locals, fromlist, level)
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/site-packages/pipenv/patched/pew/_utils.py", line 25, in <module>
[task 2019-05-13T22:45:14.946Z] encoding = locale.getlocale()[1] or 'ascii'
[task 2019-05-13T22:45:14.946Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/locale.py", line 562, in getlocale
[task 2019-05-13T22:45:14.947Z] return _parse_localename(localename)
[task 2019-05-13T22:45:14.947Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/locale.py", line 475, in _parse_localename
[task 2019-05-13T22:45:14.947Z] raise ValueError, 'unknown locale: %s' % localename
[task 2019-05-13T22:45:14.947Z] ValueError: unknown locale: UTF-8
[task 2019-05-13T22:45:14.967Z] Error running mach:
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] ['python-test', '--python', '/tools/python3', '--subsuite', 'mozbase']
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] The error occurred in code that was called by the mach command. This is either
[task 2019-05-13T22:45:14.967Z] a bug in the called code itself or in the way that mach is calling it.
[task 2019-05-13T22:45:14.967Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2019-05-13T22:45:14.967Z] isn't, please use |./mach busted file| to report it. If |./mach busted| is
[task 2019-05-13T22:45:14.967Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] If filing a bug, please include the full output of mach, including this error
[task 2019-05-13T22:45:14.967Z] message.
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] The details of the failure are as follows:
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] CalledProcessError: Command '[u'/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/bin/pipenv', u'--python', '/tools/python3']' returned non-zero exit status 1
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/python/mach_commands.py", line 99, in python_test
[task 2019-05-13T22:45:14.967Z] return self.run_python_tests(*args, **kwargs)
[task 2019-05-13T22:45:14.967Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/python/mach_commands.py", line 113, in run_python_tests
[task 2019-05-13T22:45:14.967Z] self.activate_pipenv(pipfile=None, populate=True, python=python)
[task 2019-05-13T22:45:14.967Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 846, in activate_pipenv
[task 2019-05-13T22:45:14.967Z] self.virtualenv_manager.activate_pipenv(pipfile, populate, python)
[task 2019-05-13T22:45:14.967Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 627, in activate_pipenv
[task 2019-05-13T22:45:14.967Z] self.virtualenv_root = ensure_venv()
[task 2019-05-13T22:45:14.967Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 608, in ensure_venv
[task 2019-05-13T22:45:14.967Z] env=env)
[task 2019-05-13T22:45:14.967Z] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
[task 2019-05-13T22:45:14.967Z] raise CalledProcessError(retcode, cmd)
[taskcluster 2019-05-13T22:45:15.015Z] Exit Code: 1

you can see the raw log here:
https://taskcluster-artifacts.net/YA9erQ-1THCe4lJnOSSmEw/0/public/logs/live_backing.log

I suspect this is a quirk between python2 and python3.

:ahal, do you know how the py3 mozbase tests (['python-test', '--python', '/tools/python3', '--subsuite', 'mozbase']) work in detail? possibly something in this call stack that you see as fixable?

Flags: needinfo?(ahal)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #8)

[task 2019-05-13T22:45:14.947Z] File "/Users/cltbld/tasks/task_1557777119/checkouts/gecko/obj-x86_64-apple-darwin18.0.0/_virtualenvs/init/lib/python2.7/locale.py", line 475, in _parse_localename
[task 2019-05-13T22:45:14.947Z] raise ValueError, 'unknown locale: %s' % localename
[task 2019-05-13T22:45:14.947Z] ValueError: unknown locale: UTF-8
[task 2019-05-13T22:45:14.967Z] Error running mach:
[task 2019-05-13T22:45:14.967Z]
[task 2019-05-13T22:45:14.967Z] ['python-test', '--python', '/tools/python3', '--subsuite', 'mozbase']

Is this "unknown locale: UTF-8" the cause of the failure?

I don't know the test setup, but maybe we can set the locale for the user like:
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
re: some suggestions searching for this failure: https://www.google.com/search?q=raise+ValueError%2C+%27unknown+locale%3A+%25s%27+%25+ValueError%3A+unknown+locale%3A+UTF-8 (looks like various python packages have encountered this with MacOS el capitan + )

ok, I got things working using:
command_prefix = 'cd $GECKO_PATH && LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 ./mach '

I also need to use:
run['python-version'] = '/usr/local/bin/python3'

here is the try run:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=246347549&revision=a86266650e629f2384a1e979ebf0df8c7fc3415f

and now there are 4 test failures:
0:41.11 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_process_timeout_no_kill TEST-UNEXPECTED-FAIL
0:43.42 testing/mozbase/mozprocess/tests/test_wait.py::ProcTestWait::test_timeout TEST-UNEXPECTED-FAIL
0:48.96 testing/mozbase/mozprocess/tests/test_kill.py::ProcTestKill::test_process_kill_broad_delayed TEST-UNEXPECTED-FAIL
0:48.96 testing/mozbase/mozprocess/tests/test_kill.py::ProcTestKill::test_process_kill_deep_wait TEST-UNEXPECTED-FAIL

but this is good because tests are running.

So I think we are good, I just need to make my changes more generic and get a bug on file for fixing the failing tests on python 3 for osx 10.14.

Flags: needinfo?(jwatkins)
Flags: needinfo?(ahal)

we could probably close this bug, :dhouse, feel free to do so if you don't have more work to do.

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #11)

we could probably close this bug, :dhouse, feel free to do so if you don't have more work to do.

Thanks! I'll close it out when I get my PR merged (the workers are pinned to the PR'd branch for now; once merged, I'll un-pin them to run off master/production).

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: