Closed Bug 1473648 Opened 6 years ago Closed 6 years ago

[wpt-sync] Sync PR 11806 - [WIP] Prototype HTTP/2.0 Server

Categories

(Testing :: web-platform-tests, enhancement, P4)

Product:
Testing
Component:
web-platform-tests
Type:
enhancement

Tracking

(firefox63 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla63
Tracking Flags:
Tracking Status
firefox63 --- fixed

People

(Reporter: mozilla.org, Unassigned)

References

(
URL
)

Details

(Whiteboard: [wptsync downstream]) 

Sync web-platform-tests PR 11806 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/11806
Details from upstream follow.

David Heiberg <dheiberg@mozilla.com> wrote:
>  [WIP] Prototype HTTP/2.0 Server
>  
>  Hey, so this is a prototype/proof of concept of how to integrate HTTP/2.0 into the current wptserve infrastructure.
>  
>  I'm attempting to use as much of the existing infrastructure as possible, as it gives me functionality for free (such as handling TCP connections and threading client connections), and also means I don't have to re-invent the wheel and risk making mistakes in the various bits of infrastructure I might have to redo. 
>  
>  Currently, this prototype is able to serve directories, handle multiple h2 connections simultaneously, as well as h2 connections alongside h1 requests. Obviously it is going to have many bugs, but please point out anything glaring that doesn't have a note in the comments. 
>  
>  So the general idea here is to launch a H2 server on a port, 9000 currently, using the existing `WebTestServer` in `server.py` by wrapping the socket with an ssl context that sets up the ALPN protocol to negotiate H2. Because of this, the server has to be run on python 2.7.15+, as it requires OpenSSL 1.0.2+. When the server receives the first request/data from the client, it will enter the `handle_one_request` method in `WebTestRequestHandler`, and in here I simply move to a H2 handler: `handle_h2`. 
>  
>  In `handle_one_request`, I have moved the logic that handles the parsed request to another method: `finish_handling`. This is because the H2 Handler will be reusing that logic, so by placing this in a separate method, both H1 and H2 handlers are able to use that. 
>  
>  The `handle_h2` method is fairly straightforward, it initiates the HTTP/2.0 connection, and then enters a loop that continuously reads directly from the socket and parses the H2 frames. This changes the way that the original handler works, as it now handles the entire connection from start to finish, instead of a single request at a time in a one and done fashion. 
>  
>  The `Request` object did not currently need to be changed much, just adding a `h2_stream_id` attribute. On the Response side, I created a `H2Response` object which mostly just changes which writer methods are called. The `write_content` method in here is a bit hacky, uses a `try/except` to check for the last iteration of the generator. This is because you need to know what the last item from the generator is, as you need to flag the end of the stream. Please let me know if you know of a better way!  The `H2ResponseWriter` handles the construction of the H2 frames and sending them via the socket. 
>  
>  Please let me know what you think. This currently feels like the right approach to implementing this solution, but would love some extra eyes!
>  
>  P.S. The large additions and file changes are due to adding the dependencies (h2, hyperframe, hpack, enum and certifi) in the third_party folder. To view the actual changes that implement the H2 server, view the changes made by the first commit. Also sorry for mini WOT.
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
The PR was not expected to affect any tests, but the try push wasn't a success. Check the try results for infrastructure issues
The PR was not expected to affect any tests, but the try push wasn't a success. Check the try results for infrastructure issues
The PR was not expected to affect any tests, but the try push wasn't a success. Check the try results for infrastructure issues
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ae56525d8d67
[wpt PR 11806] - [WIP] Prototype HTTP/2.0 Server, a=testonly
Backed out changeset ae56525d8d67 (bug 1473648) for breaking l10n locale tests

Log:

15:04:15  WARNING - Return code: 1
15:04:15     INFO - retry: Failed, sleeping 300 seconds before retrying
15:09:15     INFO - retry: Calling run_command with args: [['/Users/cltbld/tasks/task_1532886900/build/venv/bin/pip', 'install', '--no-deps', '--timeout', '120', '-r', '/Users/cltbld/tasks/task_1532886900/build/tests/config/firefox_ui_requirements.txt', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x102cf7b90>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x102e7e6c0>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x7faf81c69900>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x102eac030>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x102b8f998>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x102d222a0>, 'level': 'critical'}], 'cwd': '/Users/cltbld/tasks/task_1532886900/build/tests/config', 'env': {'MOZ_AUTOMATION': '1', 'VERSIONER_PYTHON_PREFER_32_BIT': 'no', 'LOGNAME': 'cltbld', 'USER': 'cltbld', 'HOME': '/Users/cltbld', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin', 'NO_EM_RESTART': '1', 'IDLEIZER_DISABLE_SHUTDOWN': 'true', 'LANG': 'en_US.UTF-8', 'XPCOM_DEBUG_BREAK': 'warn', 'SHELL': '/bin/bash', 'MOZ_NODE_PATH': '/usr/local/bin/node', 'VERSIONER_PYTHON_VERSION': '2.7', 'SHLVL': '1', 'XPC_FLAGS': '0x0', '_': '/usr/local/bin/generic-worker', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/mozilla-inbound', 'NO_FAIL_ON_TEST_ERRORS': '1', 'SCCACHE_DISABLE': '1', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.8guS9e6gzl/Listeners', 'MOZ_NO_REMOTE': '1', 'MOZ_HIDE_RESULTS_TABLE': '1', 'GECKO_HEAD_REV': 'f8d046d2b0ae2cbaca669f6177141f33a383fd51', 'XPC_SERVICE_NAME': '0', 'Apple_PubSub_Socket_Render': '/private/tmp/com.apple.launchd.2gfafDcetb/Render', 'LC_ALL': 'en_US.UTF-8', 'TMPDIR': '/var/folders/v2/hjzssk755h35_566nx3ml0xh00000w/T/', 'TASK_ID': 'OWtC459JQYKNsRpahqo9Og', '__CF_USER_TEXT_ENCODING': '0x1C:0:0', 'PWD': '/Users/cltbld'}}, attempt #5
15:09:15     INFO - Running command: ['/Users/cltbld/tasks/task_1532886900/build/venv/bin/pip', 'install', '--no-deps', '--timeout', '120', '-r', '/Users/cltbld/tasks/task_1532886900/build/tests/config/firefox_ui_requirements.txt', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub'] in /Users/cltbld/tasks/task_1532886900/build/tests/config
15:09:15     INFO - Copy/paste: /Users/cltbld/tasks/task_1532886900/build/venv/bin/pip install --no-deps --timeout 120 -r /Users/cltbld/tasks/task_1532886900/build/tests/config/firefox_ui_requirements.txt --no-index --find-links https://pypi.pub.build.mozilla.org/pub
15:09:15     INFO - Using env: (same as previous command)
15:09:15     INFO -  Ignoring indexes: https://pypi.python.org/simple
15:09:15     INFO -  Invalid requirement: '../tools/third_party/certifi'
15:09:15     INFO -  It looks like a path. Does it exist ?
15:09:15  WARNING - Return code: 1
15:09:15    FATAL - Could not install python package: /Users/cltbld/tasks/task_1532886900/build/venv/bin/pip install --no-deps --timeout 120 -r /Users/cltbld/tasks/task_1532886900/build/tests/config/firefox_ui_requirements.txt --no-index --find-links https://pypi.pub.build.mozilla.org/pub failed after 5 tries!
15:09:15    FATAL - Running post_fatal callback...
15:09:15    FATAL - Exiting -1
15:09:15     INFO - Running post-action listener: _resource_record_post_action
15:09:15     INFO - Running post-action listener: _start_resource_monitoring
15:09:15     INFO - Starting resource monitoring.
15:09:15     INFO - [mozharness: 2018-07-29 22:09:15.946063Z] Finished create-virtualenv step (failed)
15:09:15     INFO - Running post-run listener: _resource_record_post_run

https://hg.mozilla.org/integration/mozilla-inbound/rev/57b6d0d734f4ef46e5c197264ccc3746c0fad329
Flags: needinfo?(james)
dheiberg is taking a look at this. It seems like something in the packages doesn't work in the anticipated way, although it's a little mystifying that the Mn tests work and these l10n ones don't.
Flags: needinfo?(james) → needinfo?(dheiberg)
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/mozilla-inbound/rev/dbb14978acf8
[wpt PR 11806] - [WIP] Prototype HTTP/2.0 Server, a=testonly
Backed out 2 changesets (bug 1473648, bug 1476661) for mochitest failure on testing/mochitest/tests/python/test_basic_mochitest_plain.py

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=191353825&repo=mozilla-inbound&lineNumber=75081

=========================== 1 passed in 0.33 seconds ===========================
[task 2018-08-01T12:42:49.879Z]  0:28.29 /builds/worker/checkouts/gecko/testing/mochitest/tests/python/test_basic_mochitest_plain.py
[task 2018-08-01T12:42:49.880Z]  0:28.29 ============================= test session starts ==============================
[task 2018-08-01T12:42:49.880Z]  0:28.29 platform linux2 -- Python 2.7.12, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-QUm8IANb-2/bin/python
[task 2018-08-01T12:42:49.880Z]  0:28.29 rootdir: /builds/worker/checkouts/gecko, inifile: /builds/worker/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2018-08-01T12:42:49.880Z]  0:28.29 collecting ... collected 6 items
[task 2018-08-01T12:42:49.880Z]  0:28.29 
[task 2018-08-01T12:42:49.880Z]  0:28.29 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_pass TEST-UNEXPECTED-FAIL
[task 2018-08-01T12:42:49.881Z]  0:28.29 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_fail TEST-UNEXPECTED-FAIL
[task 2018-08-01T12:42:49.881Z]  0:28.29 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_crash TEST-UNEXPECTED-FAIL
[task 2018-08-01T12:42:49.881Z]  0:28.30 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_asan SKIPPED
[task 2018-08-01T12:42:49.881Z]  0:28.30 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_assertion TEST-UNEXPECTED-FAIL
[task 2018-08-01T12:42:49.881Z]  0:28.30 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_leak TEST-UNEXPECTED-FAIL
[task 2018-08-01T12:42:49.881Z]  0:28.30 
[task 2018-08-01T12:42:49.882Z]  0:28.30 =================================== FAILURES ===================================
[task 2018-08-01T12:42:49.882Z]  0:28.30 _______________________________ test_output_pass _______________________________

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=dbb14978acf80db27c38f00fe95214d90bae6af2&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified

Backout:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3b82435477e60bc187b5a840f2444064c4f3f121
Flags: needinfo?(james)
ahal: help

This patch, together with the one in bug 1476661 somehow conspire to break the mochitest harness (self)tests. I can't reproduce because both locally with a clean central checkout and on an interactive task with a clone of central I'm seeing

New python executable in /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
Also creating executable in /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
Installing setuptools, pip, wheel...done.
running build_ext
building 'psutil._psutil_linux' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/psutil
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/psutil
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
building 'psutil._psutil_posix' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
 
Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
Processing ./third_party/python/certifi
Installing collected packages: certifi
  Running setup.py install for certifi ... done
Successfully installed certifi-2018.4.16
Processing ./third_party/python/pipenv
Installing collected packages: pipenv
  Running setup.py install for pipenv ... done
Successfully installed pipenv-2018.5.18
Processing ./third_party/python/six
Installing collected packages: six
  Running setup.py install for six ... done
Successfully installed six-1.10.0
Processing ./third_party/python/virtualenv
Installing collected packages: virtualenv
  Running setup.py install for virtualenv ... done
Successfully installed virtualenv-15.2.0
Processing ./third_party/python/virtualenv-clone
Building wheels for collected packages: virtualenv-clone
  Running setup.py bdist_wheel for virtualenv-clone ... done
  Stored in directory: /home/james/.cache/pip/wheels/d0/41/2d/d807ea04cc2a2d4085a058f386162e4a7ffb6c9b7b0cfb7d16
Successfully built virtualenv-clone
Installing collected packages: virtualenv-clone
Successfully installed virtualenv-clone-0.3.0
Creating a virtualenv for this project…
Using /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2 (2.7.15) to create virtualenv…
⠋Running virtualenv with interpreter /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2
Using real prefix '/usr'
New python executable in /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-_Id6dpOc-2/bin/python2
Also creating executable in /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-_Id6dpOc-2/bin/python
Installing setuptools, pip, wheel...done.
 
Virtualenv location: /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-_Id6dpOc-2
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
 
Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
cat: backend.TestManifestBackend.in: No such file or directory
Build configuration changed. Regenerating backend.
No build detected, test metadata may be incomplete.
No handlers could be found for logger "mozbuild.frontend.reader"
 1:03.13 /home/james/develop/gecko/testing/mochitest/tests/python/test_build_profile.py
 1:03.13 ============================= test session starts ==============================
 1:03.13 platform linux2 -- Python 2.7.15rc1, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-_Id6dpOc-2/bin/python
 1:03.13 rootdir: /home/james/develop/gecko, inifile: /home/james/develop/gecko/config/mozunit/mozunit/pytest.ini
 1:03.13 collecting ... collected 1 item
 1:03.13
 1:03.13 testing/mochitest/tests/python/test_build_profile.py::test_common_prefs_are_all_set ERROR
 1:03.13
 1:03.13 ==================================== ERRORS ====================================
 1:03.13 _______________ ERROR at setup of test_common_prefs_are_all_set ________________
 1:03.13
 1:03.13 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_common_prefs_are_all_set'>>
 1:03.13 setup_test_harness = <function inner at 0x7fe4863028c0>
 1:03.13
 1:03.13     @pytest.fixture(autouse=True)  # noqa: F811
 1:03.13     def skip_using_mozinfo(request, setup_test_harness):
 1:03.13         """Gives tests the ability to skip based on values from mozinfo.
 1:03.14     
 1:03.14         Example:
 1:03.14             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:03.14             def test_foo():
 1:03.14                 pass
 1:03.14         """
 1:03.14     
 1:03.14         setup_test_harness(*setup_args)
 1:03.14         runtests = pytest.importorskip('runtests')
 1:03.14 >       runtests.update_mozinfo()
 1:03.14 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:03.14
 1:03.14 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:03.14 =========================== 1 error in 0.40 seconds ============================
 1:03.14 Setting retcode to 1 from /home/james/develop/gecko/testing/mochitest/tests/python/test_build_profile.py
 1:03.14 /home/james/develop/gecko/testing/mochitest/tests/python/test_get_active_tests.py
 1:03.14 ============================= test session starts ==============================
 1:03.14 platform linux2 -- Python 2.7.15rc1, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-_Id6dpOc-2/bin/python
 1:03.14 rootdir: /home/james/develop/gecko, inifile: /home/james/develop/gecko/config/mozunit/mozunit/pytest.ini
 1:03.14 collecting ... collected 1 item
 1:03.14
 1:03.14 testing/mochitest/tests/python/test_get_active_tests.py::test_prefs_validation ERROR
 1:03.14
 1:03.14 ==================================== ERRORS ====================================
 1:03.14 ___________________ ERROR at setup of test_prefs_validation ____________________
 1:03.14
 1:03.14 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_prefs_validation'>>
 1:03.14 setup_test_harness = <function inner at 0x7fc51ec36a28>
 1:03.14
 1:03.14     @pytest.fixture(autouse=True)  # noqa: F811
 1:03.14     def skip_using_mozinfo(request, setup_test_harness):
 1:03.14         """Gives tests the ability to skip based on values from mozinfo.
 1:03.14     
 1:03.14         Example:
 1:03.14             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:03.14             def test_foo():
 1:03.14                 pass
 1:03.14         """
 1:03.14     
 1:03.14         setup_test_harness(*setup_args)
 1:03.15         runtests = pytest.importorskip('runtests')
 1:03.15 >       runtests.update_mozinfo()
 1:03.15 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:03.15
 1:03.15 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:03.15 =========================== 1 error in 0.39 seconds ============================
 1:05.73 /home/james/develop/gecko/testing/mochitest/tests/python/test_basic_mochitest_plain.py
 1:05.73 ============================= test session starts ==============================
 1:05.73 platform linux2 -- Python 2.7.15rc1, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /home/james/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-_Id6dpOc-2/bin/python
 1:05.73 rootdir: /home/james/develop/gecko, inifile: /home/james/develop/gecko/config/mozunit/mozunit/pytest.ini
 1:05.73 collecting ... collected 6 items
 1:05.73
 1:05.73 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_pass ERROR
 1:05.73 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_fail ERROR
 1:05.73 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_crash ERROR
 1:05.74 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_asan ERROR
 1:05.74 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_assertion ERROR
 1:05.74 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_leak ERROR
 1:05.74
 1:05.74 ==================================== ERRORS ====================================
 1:05.74 ______________________ ERROR at setup of test_output_pass ______________________
 1:05.74
 1:05.74 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_pass'>>
 1:05.74 setup_test_harness = <function inner at 0x7f1a8365ade8>
 1:05.74
 1:05.74     @pytest.fixture(autouse=True)  # noqa: F811
 1:05.74     def skip_using_mozinfo(request, setup_test_harness):
 1:05.74         """Gives tests the ability to skip based on values from mozinfo.
 1:05.74     
 1:05.74         Example:
 1:05.74             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:05.74             def test_foo():
 1:05.74                 pass
 1:05.74         """
 1:05.74     
 1:05.74         setup_test_harness(*setup_args)
 1:05.74         runtests = pytest.importorskip('runtests')
 1:05.74 >       runtests.update_mozinfo()
 1:05.74 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:05.74
 1:05.74 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:05.74 ______________________ ERROR at setup of test_output_fail ______________________
 1:05.74
 1:05.74 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_fail'>>
 1:05.74 setup_test_harness = <function inner at 0x7f1a8365ade8>
 1:05.74
 1:05.74     @pytest.fixture(autouse=True)  # noqa: F811
 1:05.74     def skip_using_mozinfo(request, setup_test_harness):
 1:05.74         """Gives tests the ability to skip based on values from mozinfo.
 1:05.74     
 1:05.74         Example:
 1:05.74             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:05.74             def test_foo():
 1:05.74                 pass
 1:05.74         """
 1:05.74     
 1:05.75         setup_test_harness(*setup_args)
 1:05.75         runtests = pytest.importorskip('runtests')
 1:05.75 >       runtests.update_mozinfo()
 1:05.75 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:05.75
 1:05.75 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:05.75 _____________________ ERROR at setup of test_output_crash ______________________
 1:05.75
 1:05.75 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_crash'>>
 1:05.75 setup_test_harness = <function inner at 0x7f1a8365ade8>
 1:05.75
 1:05.75     @pytest.fixture(autouse=True)  # noqa: F811
 1:05.75     def skip_using_mozinfo(request, setup_test_harness):
 1:05.75         """Gives tests the ability to skip based on values from mozinfo.
 1:05.75     
 1:05.75         Example:
 1:05.75             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:05.75             def test_foo():
 1:05.75                 pass
 1:05.75         """
 1:05.75     
 1:05.75         setup_test_harness(*setup_args)
 1:05.75         runtests = pytest.importorskip('runtests')
 1:05.75 >       runtests.update_mozinfo()
 1:05.75 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:05.75
 1:05.75 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:05.75 ______________________ ERROR at setup of test_output_asan ______________________
 1:05.75
 1:05.75 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_asan'>>
 1:05.75 setup_test_harness = <function inner at 0x7f1a8365ade8>
 1:05.75
 1:05.75     @pytest.fixture(autouse=True)  # noqa: F811
 1:05.75     def skip_using_mozinfo(request, setup_test_harness):
 1:05.75         """Gives tests the ability to skip based on values from mozinfo.
 1:05.75     
 1:05.75         Example:
 1:05.76             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:05.76             def test_foo():
 1:05.76                 pass
 1:05.76         """
 1:05.76     
 1:05.76         setup_test_harness(*setup_args)
 1:05.76         runtests = pytest.importorskip('runtests')
 1:05.76 >       runtests.update_mozinfo()
 1:05.76 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:05.76
 1:05.76 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:05.76 ___________________ ERROR at setup of test_output_assertion ____________________
 1:05.76
 1:05.76 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_assertion'>>
 1:05.76 setup_test_harness = <function inner at 0x7f1a8365ade8>
 1:05.76
 1:05.76     @pytest.fixture(autouse=True)  # noqa: F811
 1:05.76     def skip_using_mozinfo(request, setup_test_harness):
 1:05.76         """Gives tests the ability to skip based on values from mozinfo.
 1:05.76     
 1:05.76         Example:
 1:05.76             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:05.76             def test_foo():
 1:05.76                 pass
 1:05.76         """
 1:05.76     
 1:05.76         setup_test_harness(*setup_args)
 1:05.76         runtests = pytest.importorskip('runtests')
 1:05.76 >       runtests.update_mozinfo()
 1:05.77 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:05.77
 1:05.77 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:05.77 ______________________ ERROR at setup of test_output_leak ______________________
 1:05.77
 1:05.77 request = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_leak'>>
 1:05.77 setup_test_harness = <function inner at 0x7f1a8365ade8>
 1:05.77
 1:05.77     @pytest.fixture(autouse=True)  # noqa: F811
 1:05.77     def skip_using_mozinfo(request, setup_test_harness):
 1:05.77         """Gives tests the ability to skip based on values from mozinfo.
 1:05.77     
 1:05.77         Example:
 1:05.77             @pytest.mark.skip_mozinfo("!e10s || os == 'linux'")
 1:05.77             def test_foo():
 1:05.77                 pass
 1:05.77         """
 1:05.77     
 1:05.77         setup_test_harness(*setup_args)
 1:05.77         runtests = pytest.importorskip('runtests')
 1:05.77 >       runtests.update_mozinfo()
 1:05.77 E       AttributeError: 'module' object has no attribute 'update_mozinfo'
 1:05.77
 1:05.77 testing/mochitest/tests/python/conftest.py:104: AttributeError
 1:05.77 =========================== 6 error in 0.60 seconds ============================
 1:05.77 Return code from mach python-test: 1
james@ginny:~/develop/gecko$ git log --oneline -n1 HEAD
5629338b1a1b (HEAD -> master, mozilla/central, refs/cinnabar/tip, refs/cinnabar/refs/heads/bookmarks/central) Merge inbound to mozilla-central.  a=merge 

i.e. all the tests error (it seems to be loading runtests.py from the webidl parser rather than from mochitest). For unknown reasons this works for ato on linux, and evidently works in CI. So I don't know how to fix either the original issue or the blocking issue that the tests don't run at all for me.
Flags: needinfo?(james) → needinfo?(ahal)
Is it possible testing/web-platform/runtests.py is being added to the PYTHONPATH and the mochitest selftests are picking that up instead of testing/mochitest/runtests.py? The patch in bug 1476661 seems suspicious because it's modifying build/virtualenv_packages.txt.

If you can reproduce locally, try adding a:
print(runtests.__file__)

here:
https://searchfox.org/mozilla-central/source/testing/mochitest/tests/python/conftest.py#103
Flags: needinfo?(ahal)
Ah, sorry I didn't explain well. The error I see comes with an unmodified mozilla-central. I can't debug the problems with the other change becuase I can't get these tests to work at all. (update: it seemed to work when I first tried again, and then failed as described after a clobber)

Debugging using print seems to not work because pytest is swallowing that (for some reason passing -s in mozunit doesn't seem to help). But with -l I get:

 0:21.58 request    = <SubRequest 'skip_using_mozinfo' for <Function 'test_output_leak'>>
 0:21.58 runtests   = <module 'runtests' from '/home/james/develop/gecko/dom/bindings/parser/runtests.py'>
 0:21.58 setup_test_harness = <function inner at 0x7f7edd0338c0>
Flags: needinfo?(ahal)
Ah, I see what you mean now (I can reproduce this too). Without a build, these tests should be skipped but for some reason they aren't. I'll file a bug to make sure they are skipped properly when there is no build, but in the meantime you should be able to work around it by having a build (just an artifact build will suffice).
Flags: needinfo?(ahal)
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/mozilla-inbound/rev/62e63235190b
[wpt PR 11806] - [WIP] Prototype HTTP/2.0 Server, a=testonly
https://hg.mozilla.org/mozilla-central/rev/62e63235190b
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox63: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Flags: needinfo?(dheiberg)
David, with this patch it's no longer possible to install and run wptserve outside of the mach command because the h2 package is not referenced as dependency in wptserve's setup.py. Can you please fix that? Thanks.
Flags: needinfo?(dheiberg)
(In reply to Henrik Skupin (:whimboo) from comment #18)
> David, with this patch it's no longer possible to install and run wptserve
> outside of the mach command because the h2 package is not referenced as
> dependency in wptserve's setup.py. Can you please fix that? Thanks.

Looking into this now
This has been landed in the upstream WPT, will update when the sync bot has merged it
This is fixed as of https://bugzilla.mozilla.org/show_bug.cgi?id=1484042
Flags: needinfo?(dheiberg)
You need to log in before you can comment on or make changes to this bug.