Closed Bug 1691731 Opened 4 years ago Closed 4 years ago

Perma taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests] TEST-UNEXPECTED-FAIL

Categories

(Testing :: web-platform-tests, defect)

Product:
Testing
Component:
web-platform-tests
Type:
defect

Tracking

(firefox-esr78 unaffected, firefox85 unaffected, firefox86 unaffected, firefox87 fixed)

Status:
RESOLVED FIXED
Milestone:
87 Branch
Tracking Flags:
Tracking Status
firefox-esr78 --- unaffected
firefox85 --- unaffected
firefox86 --- unaffected
firefox87 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: marco)

References

(Regression)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Bug 1691731 - Only run taskgraph tests with Python 3. r=aki
48 bytes, text/x-phabricator-request
Details | Review

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=329362174&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/U3-RJCxBRm-NyHY977AcrA/runs/0/artifacts/public/logs/live_backing.log

0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_mock_chunk_manifests[20-xpcshell] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests-reftest] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-xpcshell] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-mochitest-plain] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-mochitest-devtools-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-mochitest-browser-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-mochitest-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-web-platform-tests] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-web-platform-tests-reftest] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-xpcshell] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-mochitest-plain] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-mochitest-devtools-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-mochitest-browser-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform1-mochitest-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-web-platform-tests] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-web-platform-tests-reftest] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-xpcshell] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-mochitest-plain] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-mochitest-devtools-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-mochitest-browser-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform2-mochitest-chrome] PASSED
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform3-web-platform-tests] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform3-web-platform-tests-reftest] TEST-UNEXPECTED-FAIL
[task 2021-02-09T14:51:38.644Z]  0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform3-xpcshell] PASSED
<...>
=================================== FAILURES ===================================
[task 2021-02-09T14:51:38.652Z]  0:58.72 _______________ test_get_manifests[platform0-web-platform-tests] _______________
[task 2021-02-09T14:51:38.652Z]  0:58.72 
[task 2021-02-09T14:51:38.652Z]  0:58.72 suite = 'web-platform-tests', platform = ('mac', 'x86_64')
[task 2021-02-09T14:51:38.653Z]  0:58.72 mock_mozinfo = <function inner at 0x7f68e13e3c50>
[task 2021-02-09T14:51:38.653Z]  0:58.72 
[task 2021-02-09T14:51:38.653Z]  0:58.72     @pytest.mark.parametrize(
[task 2021-02-09T14:51:38.653Z]  0:58.72         "suite",
[task 2021-02-09T14:51:38.653Z]  0:58.72         [
[task 2021-02-09T14:51:38.653Z]  0:58.72             "web-platform-tests",
[task 2021-02-09T14:51:38.653Z]  0:58.72             "web-platform-tests-reftest",
[task 2021-02-09T14:51:38.653Z]  0:58.72             "xpcshell",
[task 2021-02-09T14:51:38.653Z]  0:58.72             "mochitest-plain",
[task 2021-02-09T14:51:38.654Z]  0:58.72             "mochitest-devtools-chrome",
[task 2021-02-09T14:51:38.654Z]  0:58.72             "mochitest-browser-chrome",
[task 2021-02-09T14:51:38.654Z]  0:58.72             "mochitest-chrome",
[task 2021-02-09T14:51:38.654Z]  0:58.72         ],
[task 2021-02-09T14:51:38.654Z]  0:58.72     )
[task 2021-02-09T14:51:38.654Z]  0:58.72     @pytest.mark.parametrize(
[task 2021-02-09T14:51:38.655Z]  0:58.72         "platform",
[task 2021-02-09T14:51:38.655Z]  0:58.72         [
[task 2021-02-09T14:51:38.655Z]  0:58.72             ("mac", "x86_64"),
[task 2021-02-09T14:51:38.655Z]  0:58.72             ("win", "x86_64"),
[task 2021-02-09T14:51:38.655Z]  0:58.72             ("win", "x86"),
[task 2021-02-09T14:51:38.655Z]  0:58.72             ("win", "aarch64"),
[task 2021-02-09T14:51:38.655Z]  0:58.72             ("linux", "x86_64"),
[task 2021-02-09T14:51:38.656Z]  0:58.72             ("linux", "x86"),
[task 2021-02-09T14:51:38.656Z]  0:58.72         ],
[task 2021-02-09T14:51:38.656Z]  0:58.72     )
[task 2021-02-09T14:51:38.656Z]  0:58.72     def test_get_manifests(suite, platform, mock_mozinfo):
[task 2021-02-09T14:51:38.656Z]  0:58.72         """Tests the DefaultLoader class' ability to load manifests."""
[task 2021-02-09T14:51:38.656Z]  0:58.72         mozinfo = mock_mozinfo(*platform)
         loader = chunking.DefaultLoader([])
[task 2021-02-09T14:51:38.657Z]  0:58.72 >       manifests = loader.get_manifests(suite, frozenset(mozinfo.items()))
[task 2021-02-09T14:51:38.657Z]  0:58.72 
[task 2021-02-09T14:51:38.657Z]  0:58.72 taskcluster/taskgraph/test/test_util_chunking.py:335:
[task 2021-02-09T14:51:38.657Z]  0:58.72 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2021-02-09T14:51:38.657Z]  0:58.72 python/mozbuild/mozbuild/util.py:1049: in method_call
[task 2021-02-09T14:51:38.657Z]  0:58.72     cache[args] = self.func(instance, *args)
[task 2021-02-09T14:51:38.658Z]  0:58.72 taskcluster/taskgraph/util/chunking.py:199: in get_manifests
[task 2021-02-09T14:51:38.658Z]  0:58.72     tests = self.get_tests(suite)
[task 2021-02-09T14:51:38.658Z]  0:58.72 python/mozbuild/mozbuild/util.py:1049: in method_call
[task 2021-02-09T14:51:38.658Z]  0:58.72     cache[args] = self.func(instance, *args)
[task 2021-02-09T14:51:38.658Z]  0:58.72 taskcluster/taskgraph/util/chunking.py:190: in get_tests
[task 2021-02-09T14:51:38.658Z]  0:58.72     "subsuite", "undefined"
[task 2021-02-09T14:51:38.658Z]  0:58.72 testing/mozbase/moztest/moztest/resolve.py:943: in resolve_tests
[task 2021-02-09T14:51:38.659Z]  0:58.72     for test in self._resolve(**kwargs):
[task 2021-02-09T14:51:38.659Z]  0:58.72 testing/mozbase/moztest/moztest/resolve.py:690: in _resolve
[task 2021-02-09T14:51:38.659Z]  0:58.72     self.add_wpt_manifest_data()
[task 2021-02-09T14:51:38.659Z]  0:58.72 testing/mozbase/moztest/moztest/resolve.py:836: in add_wpt_manifest_data
[task 2021-02-09T14:51:38.659Z]  0:58.72     logger=logger,
[task 2021-02-09T14:51:38.659Z]  0:58.73 testing/web-platform/manifestupdate.py:100: in run
[task 2021-02-09T14:51:38.660Z]  0:58.73     do_delayed_imports(src_wpt_dir)
[task 2021-02-09T14:51:38.660Z]  0:58.73 testing/web-platform/manifestupdate.py:31: in do_delayed_imports
[task 2021-02-09T14:51:38.660Z]  0:58.73     import manifest
[task 2021-02-09T14:51:38.660Z]  0:58.73 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2021-02-09T14:51:38.660Z]  0:58.73 
[task 2021-02-09T14:51:38.660Z]  0:58.73 >   from . import item, manifest, sourcefile, update  # noqa: F401
[task 2021-02-09T14:51:38.661Z]  0:58.73 E     File "/builds/worker/checkouts/gecko/testing/web-platform/tests/tools/manifest/item.py", line 44
[task 2021-02-09T14:51:38.661Z]  0:58.73 E       class ManifestItem(metaclass=ManifestItemMeta):
[task 2021-02-09T14:51:38.661Z]  0:58.73 E                                   ^
[task 2021-02-09T14:51:38.661Z]  0:58.73 E   SyntaxError: invalid syntax
[task 2021-02-09T14:51:38.661Z]  0:58.73 
[task 2021-02-09T14:51:38.661Z]  0:58.73 testing/web-platform/tests/tools/manifest/__init__.py:1: SyntaxError
[task 2021-02-09T14:51:38.661Z]  0:58.73 ___________ test_get_manifests[platform0-web-platform-tests-reftest] ___________
[task 2021-02-09T14:51:38.662Z]  0:58.73 
[task 2021-02-09T14:51:38.662Z]  0:58.73 suite = 'web-platform-tests-reftest', platform = ('mac', 'x86_64')
[task 2021-02-09T14:51:38.662Z]  0:58.73 mock_mozinfo = <function inner at 0x7f68e13e3c50>
[task 2021-02-09T14:51:38.662Z]  0:58.73 
[task 2021-02-09T14:51:38.662Z]  0:58.73     @pytest.mark.parametrize(
[task 2021-02-09T14:51:38.662Z]  0:58.73         "suite",
        [
[task 2021-02-09T14:51:38.663Z]  0:58.73             "web-platform-tests",
[task 2021-02-09T14:51:38.663Z]  0:58.73             "web-platform-tests-reftest",
[task 2021-02-09T14:51:38.663Z]  0:58.73             "xpcshell",
[task 2021-02-09T14:51:38.663Z]  0:58.73             "mochitest-plain",
[task 2021-02-09T14:51:38.663Z]  0:58.73             "mochitest-devtools-chrome",
[task 2021-02-09T14:51:38.664Z]  0:58.73             "mochitest-browser-chrome",
[task 2021-02-09T14:51:38.664Z]  0:58.73             "mochitest-chrome",
[task 2021-02-09T14:51:38.664Z]  0:58.73         ],
[task 2021-02-09T14:51:38.664Z]  0:58.73     )
[task 2021-02-09T14:51:38.664Z]  0:58.73     @pytest.mark.parametrize(
[task 2021-02-09T14:51:38.664Z]  0:58.73         "platform",
[task 2021-02-09T14:51:38.665Z]  0:58.73         [
[task 2021-02-09T14:51:38.665Z]  0:58.73             ("mac", "x86_64"),
[task 2021-02-09T14:51:38.665Z]  0:58.73             ("win", "x86_64"),
[task 2021-02-09T14:51:38.665Z]  0:58.73             ("win", "x86"),
[task 2021-02-09T14:51:38.665Z]  0:58.73             ("win", "aarch64"),
[task 2021-02-09T14:51:38.665Z]  0:58.73             ("linux", "x86_64"),
[task 2021-02-09T14:51:38.665Z]  0:58.73             ("linux", "x86"),
[task 2021-02-09T14:51:38.665Z]  0:58.73         ],
[task 2021-02-09T14:51:38.666Z]  0:58.73     )
[task 2021-02-09T14:51:38.666Z]  0:58.73     def test_get_manifests(suite, platform, mock_mozinfo):
[task 2021-02-09T14:51:38.666Z]  0:58.73         """Tests the DefaultLoader class' ability to load manifests."""
[task 2021-02-09T14:51:38.666Z]  0:58.73         mozinfo = mock_mozinfo(*platform)
[task 2021-02-09T14:51:38.666Z]  0:58.73 
[task 2021-02-09T14:51:38.666Z]  0:58.73         loader = chunking.DefaultLoader([])
[task 2021-02-09T14:51:38.666Z]  0:58.73 >       manifests = loader.get_manifests(suite, frozenset(mozinfo.items()))
[task 2021-02-09T14:51:38.666Z]  0:58.73 
[task 2021-02-09T14:51:38.666Z]  0:58.73 taskcluster/taskgraph/test/test_util_chunking.py:335:
[task 2021-02-09T14:51:38.666Z]  0:58.73 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2021-02-09T14:51:38.666Z]  0:58.73 python/mozbuild/mozbuild/util.py:1049: in method_call
[task 2021-02-09T14:51:38.666Z]  0:58.73     cache[args] = self.func(instance, *args)
[task 2021-02-09T14:51:38.666Z]  0:58.73 taskcluster/taskgraph/util/chunking.py:199: in get_manifests
[task 2021-02-09T14:51:38.667Z]  0:58.73     tests = self.get_tests(suite)
[task 2021-02-09T14:51:38.667Z]  0:58.73 python/mozbuild/mozbuild/util.py:1049: in method_call
[task 2021-02-09T14:51:38.667Z]  0:58.73     cache[args] = self.func(instance, *args)
[task 2021-02-09T14:51:38.667Z]  0:58.73 taskcluster/taskgraph/util/chunking.py:190: in get_tests
[task 2021-02-09T14:51:38.667Z]  0:58.73     "subsuite", "undefined"
[task 2021-02-09T14:51:38.667Z]  0:58.73 testing/mozbase/moztest/moztest/resolve.py:943: in resolve_tests
[task 2021-02-09T14:51:38.667Z]  0:58.73     for test in self._resolve(**kwargs):
testing/mozbase/moztest/moztest/resolve.py:690: in _resolve
[task 2021-02-09T14:51:38.667Z]  0:58.73     self.add_wpt_manifest_data()
[task 2021-02-09T14:51:38.667Z]  0:58.73 testing/mozbase/moztest/moztest/resolve.py:836: in add_wpt_manifest_data
[task 2021-02-09T14:51:38.667Z]  0:58.73     logger=logger,
[task 2021-02-09T14:51:38.667Z]  0:58.73 testing/web-platform/manifestupdate.py:100: in run
[task 2021-02-09T14:51:38.667Z]  0:58.73     do_delayed_imports(src_wpt_dir)
[task 2021-02-09T14:51:38.668Z]  0:58.73 testing/web-platform/manifestupdate.py:31: in do_delayed_imports
[task 2021-02-09T14:51:38.668Z]  0:58.73     import manifest
[task 2021-02-09T14:51:38.668Z]  0:58.73 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2021-02-09T14:51:38.668Z]  0:58.73 
[task 2021-02-09T14:51:38.668Z]  0:58.73 >   from . import item, manifest, sourcefile, update  # noqa: F401
[task 2021-02-09T14:51:38.668Z]  0:58.73 E     File "/builds/worker/checkouts/gecko/testing/web-platform/tests/tools/manifest/item.py", line 44
[task 2021-02-09T14:51:38.669Z]  0:58.73 E       class ManifestItem(metaclass=ManifestItemMeta):
[task 2021-02-09T14:51:38.669Z]  0:58.73 E                                   ^
[task 2021-02-09T14:51:38.669Z]  0:58.73 E   SyntaxError: invalid syntax
[task 2021-02-09T14:51:38.669Z]  0:58.73 
[task 2021-02-09T14:51:38.669Z]  0:58.73 testing/web-platform/tests/tools/manifest/__init__.py:1: SyntaxError
[task 2021-02-09T14:51:38.670Z]  0:58.73 _______________ test_get_manifests[platform1-web-platform-tests] _______________
[task 2021-02-09T14:51:38.670Z]  0:58.73 
[task 2021-02-09T14:51:38.670Z]  0:58.73 suite = 'web-platform-tests', platform = ('win', 'x86_64')
[task 2021-02-09T14:51:38.670Z]  0:58.73 mock_mozinfo = <function inner at 0x7f68e13e3c50>
[task 2021-02-09T14:51:38.670Z]  0:58.73 
[task 2021-02-09T14:51:38.671Z]  0:58.73     @pytest.mark.parametrize(
[task 2021-02-09T14:51:38.671Z]  0:58.73         "suite",
[task 2021-02-09T14:51:38.671Z]  0:58.73         [
[task 2021-02-09T14:51:38.671Z]  0:58.73             "web-platform-tests",
[task 2021-02-09T14:51:38.671Z]  0:58.73             "web-platform-tests-reftest",
[task 2021-02-09T14:51:38.671Z]  0:58.73             "xpcshell",
[task 2021-02-09T14:51:38.671Z]  0:58.73             "mochitest-plain",
[task 2021-02-09T14:51:38.672Z]  0:58.73             "mochitest-devtools-chrome",
[task 2021-02-09T14:51:38.672Z]  0:58.73             "mochitest-browser-chrome",
[task 2021-02-09T14:51:38.672Z]  0:58.73             "mochitest-chrome",
[task 2021-02-09T14:51:38.672Z]  0:58.73         ],
[task 2021-02-09T14:51:38.673Z]  0:58.73     )
[task 2021-02-09T14:51:38.673Z]  0:58.73     @pytest.mark.parametrize(
[task 2021-02-09T14:51:38.673Z]  0:58.73         "platform",
Severity: S4 → --
status-firefox85: --- → unaffected
status-firefox86: --- → unaffected
status-firefox87: --- → affected
status-firefox-esr78: --- → unaffected
Component: Task Configuration → web-platform-tests
Priority: P5 → --
Product: Firefox Build System → Testing
Regressed by: 1690784
Summary: Intermittent 0:58.71 taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests] TEST-UNEXPECTED-FAIL → Perma taskcluster/taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests] TEST-UNEXPECTED-FAIL

I think this is happening because the tests are trying to run with py2 and wpt no longer supports py2. I can see a few options:

  • Change the task definition to only run these tests with Py3.
  • Change the parameterization to exclude wpt tests in the case where we are running py2
  • Early return from the test with the combination py2 + wpt.

I don't own this feature though, so I probably shouldn't make the decision (for my money, the options above are ranked best to worst, but idk if we have other constraints that mean parts of this code has to work with py2).

Flags: needinfo?(james) → needinfo?(mcastelluccio)

(In reply to James Graham [:jgraham] from comment #2)

I think this is happening because the tests are trying to run with py2 and wpt no longer supports py2. I can see a few options:

  • Change the task definition to only run these tests with Py3.
  • Change the parameterization to exclude wpt tests in the case where we are running py2
  • Early return from the test with the combination py2 + wpt.

I don't own this feature though, so I probably shouldn't make the decision (for my money, the options above are ranked best to worst, but idk if we have other constraints that mean parts of this code has to work with py2).

I would prefer the first option too, though I don't know if there are some taskgraph tests that would still be useful to run under Python 2.

Flags: needinfo?(mcastelluccio)
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Pushed by mcastelluccio@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/83324af06417 Only run taskgraph tests with Python 3. r=aki DONTBUILD
Regressions: 1692421

https://hg.mozilla.org/mozilla-central/rev/83324af06417

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox87: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 87 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: