1668097 - CalledProcessError running 'mach mochitest' during pip install of websocketprocessbridge_requirements.txt

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

gbrown@exmozpad:~/src$ ./mach mochitest --this-chunk 1 --total-chunks 100
Collecting six
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting vcversioner==2.16.0.0
  Using cached https://files.pythonhosted.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcversioner-2.16.0.0.tar.gz
Collecting twisted==10.2.0
  Using cached https://files.pythonhosted.org/packages/f9/80/50b40d787ee26af3062eb83b9a57fa3bdb5e0417f6a3047fffdbd09de6d9/Twisted-10.2.0.tar.bz2
    ERROR: Command errored out with exit status 1:
     command: /home/gbrown/objdirs/x86_64/_virtualenvs/init_py3/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-aedazlfg/twisted/setup.py'"'"'; __file__='"'"'/tmp/pip-install-aedazlfg/twisted/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-aedazlfg/twisted/pip-egg-info
         cwd: /tmp/pip-install-aedazlfg/twisted/
    Complete output (12 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-aedazlfg/twisted/setup.py", line 87
        print """You seem to be running a very old version of setuptools.
    This version of setuptools has a bug parsing dependencies, so automatic
    dependency resolution is disabled.
    """
              ^
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.
    This version of setuptools has a bug parsing dependencies, so automatic
    dependency resolution is disabled.
    """)?
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Error running mach:

    ['mochitest', '--this-chunk', '1', '--total-chunks', '100']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file mochitest| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/home/gbrown/objdirs/x86_64/_virtualenvs/init_py3/bin/pip', 'install', '--requirement', '/home/gbrown/src/testing/tools/websocketprocessbridge/websocketprocessbridge_requirements.txt']' returned non-zero exit status 1.

  File "/home/gbrown/src/testing/mochitest/mach_commands.py", line 377, in run_mochitest_general
    self.virtualenv_manager.install_pip_requirements(req, require_hashes=False)
  File "/home/gbrown/src/python/mozbuild/mozbuild/virtualenv.py", line 609, in install_pip_requirements
    return self._run_pip(args)
  File "/home/gbrown/src/python/mozbuild/mozbuild/virtualenv.py", line 620, in _run_pip
    subprocess.check_call([pip] + args, stderr=subprocess.STDOUT, cwd=self.topsrcdir,
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Geoff Brown [:gbrown]]

If I use the latest version of twisted that installs fine, but then I hit a similar issue for ipaddr:

Collecting ipaddr==2.1.11
  Downloading https://files.pythonhosted.org/packages/08/80/7539938aca4901864b7767a23eb6861fac18ef5219b60257fc938dae3568/ipaddr-2.1.11.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /home/gbrown/objdirs/x86_64/_virtualenvs/init_py3/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-46jxgxn7/ipaddr/setup.py'"'"'; __file__='"'"'/tmp/pip-install-46jxgxn7/ipaddr/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-46jxgxn7/ipaddr/pip-egg-info
         cwd: /tmp/pip-install-46jxgxn7/ipaddr/
    Complete output (10 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-46jxgxn7/ipaddr/setup.py", line 19, in <module>
        import ipaddr
      File "/home/gbrown/src/build/mach_bootstrap.py", line 504, in __call__
        module = self._original_import(name, globals, locals, fromlist, level)
      File "/tmp/pip-install-46jxgxn7/ipaddr/ipaddr.py", line 1450
        ip_int = 0L
                  ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Comment 2

[Geoff Brown [:gbrown]]

btw, I see this when running in an android build context; I'm not sure about applicability to desktop. I do not have trouble with 'mach mochitest <directory>', nor 'mach mochitest <test>'.

Comment 3

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2caf1abc214385c331e30fa6d6f9c2658b32f1dd

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #3)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2caf1abc214385c331e30fa6d6f9c2658b32f1dd

I changed:

-twisted==10.2.0
+twisted

and that worked fine locally but failed on Windows in the try run.

Updating to another specific version fails in CI, since it's not on pypi.pub.build.mozilla.org:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=85ecb45fede3f38d232b275faf6a768dcd3e76dc

Suggestions? Is there an easy way to keep twisted==10.2.0 for python 2 (or in CI?) but use a more recent version for python 3 (or via mach)?

Comment 5

[Geoff Brown [:gbrown]]

I guess I could use a separate requirements.txt for mach:
https://searchfox.org/mozilla-central/rev/dfd9c0f72f9765bd4a187444e0c1e19e8834a506/testing/mochitest/mach_commands.py#375

Comment 6

[Geoff Brown [:gbrown]]

Yeah, I'll go with that approach.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: Bug 1668097 - Update websocketprocessbridge dependencies for py3; r=

Under python 3, use a separate requirements file to install newer versions of twisted and ipaddr;
keep the original requirements under python 2 as the latest versions are problematic in our
Windows CI environment.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1638974

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4f08d9360dd7
Update websocketprocessbridge dependencies for py3; r=bc

Comment 11

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/4f08d9360dd7

Comment 12

[Byron Campen [:bwc]]

Looks like there's some sort of escaping problem, probably in txws:

python2

0000   02 00 00 00 45 00 00 d9 00 00 40 00 40 06 00 00   ....E.....@.@...
0010   7f 00 00 01 7f 00 00 01 1f ff c1 d1 bc a0 1c 85   ................
0020   02 e9 31 aa 80 18 18 e2 fe cd 00 00 01 01 08 0a   ..1.............
0030   70 80 9f 98 70 80 9e 3a 81 7e 00 a1 5b 7b 22 75   p...p..:.~..[{"u
0040   72 6c 73 22 3a 5b 22 73 74 75 6e 3a 6c 6f 63 61   rls":["stun:loca
0050   6c 68 6f 73 74 22 2c 20 22 73 74 75 6e 3a 6c 6f   lhost", "stun:lo
0060   63 61 6c 68 6f 73 74 3f 74 72 61 6e 73 70 6f 72   calhost?transpor
0070   74 3d 74 63 70 22 5d 7d 2c 20 7b 22 75 73 65 72   t=tcp"]}, {"user
0080   6e 61 6d 65 22 3a 22 66 6f 6f 22 2c 22 63 72 65   name":"foo","cre
0090   64 65 6e 74 69 61 6c 22 3a 22 62 61 72 22 2c 22   dential":"bar","
00a0   75 72 6c 73 22 3a 20 5b 22 74 75 72 6e 3a 6c 6f   urls": ["turn:lo
00b0   63 61 6c 68 6f 73 74 22 2c 20 22 74 75 72 6e 3a   calhost", "turn:
00c0   6c 6f 63 61 6c 68 6f 73 74 3f 74 72 61 6e 73 70   localhost?transp
00d0   6f 72 74 3d 74 63 70 22 20 5d 20 7d 5d            ort=tcp" ] }]

The websocket frame is 81 7e 00 a1 here.

python 3:

0000   02 00 00 00 45 00 00 e2 00 00 40 00 40 06 00 00   ....E.....@.@...
0010   7f 00 00 01 7f 00 00 01 1f ff fe fd 95 31 36 4b   .............16K
0020   8b 53 69 fa 80 18 18 e2 fe d6 00 00 01 01 08 0a   .Si.............
0030   70 70 f5 bc 70 70 f4 43 81 7e 62 27 5c 78 30 30   pp..pp.C.~b'\x00
0040   5c 78 61 31 27 5b 7b 22 75 72 6c 73 22 3a 5b 22   \xa1'[{"urls":["
0050   73 74 75 6e 3a 6c 6f 63 61 6c 68 6f 73 74 22 2c   stun:localhost",
0060   20 22 73 74 75 6e 3a 6c 6f 63 61 6c 68 6f 73 74    "stun:localhost
0070   3f 74 72 61 6e 73 70 6f 72 74 3d 74 63 70 22 5d   ?transport=tcp"]
0080   7d 2c 20 7b 22 75 73 65 72 6e 61 6d 65 22 3a 22   }, {"username":"
0090   66 6f 6f 22 2c 22 63 72 65 64 65 6e 74 69 61 6c   foo","credential
00a0   22 3a 22 62 61 72 22 2c 22 75 72 6c 73 22 3a 20   ":"bar","urls": 
00b0   5b 22 74 75 72 6e 3a 6c 6f 63 61 6c 68 6f 73 74   ["turn:localhost
00c0   22 2c 20 22 74 75 72 6e 3a 6c 6f 63 61 6c 68 6f   ", "turn:localho
00d0   73 74 3f 74 72 61 6e 73 70 6f 72 74 3d 74 63 70   st?transport=tcp
00e0   22 20 5d 20 7d 5d                                 " ] }]

Where we would expect 81 7e 00 a1 as before, we see instead a larger number of bytes beginning with 81 7e, followed by bytes that encode the string b'\x00\xa1'.