Closed Bug 1505474 Opened 1 year ago Closed 1 year ago

jstests.py still using the in-tree manifest

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox65 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

The wpt manifest is a large JSON blob which causes problems. So in bug 1493674 we stopped using the in-tree manifest for tests run with wpt run and instead moved to a model where the manifest is downloaded from taskcluster into the objdir and updated when the tests are run. As a result the in-tree manifest is no longer updated and should be removed. However that's blocked on the use of the manifest by jstests.py, which was missed in the original patch.

Ideally the js tests would also gain the ability to download a recent manifest from taskcluster and update it on demand. However it's unclear where that download should go; afaict jstests.py doesn't know about the usual mozbuild objdir.
Flags: needinfo?(Ms2ger)
Blocks: 1333433
web-platform-tests gets its test list from a manifest file that's
generated from the content of the tests. Unfortunately generating this
manifest is slow, so it's unreasonable to create it from scratch for
every test run. Until recently the generated manifest was kept
in-tree, which was suboptimal in a few ways:

* The manifest tended to get out of sync with the actual source
* The large json file caused problems for tooling including source
  control and the review frontends.

We previously switched `mach wpt` to download a manifest on demand and
apply an incremental update. However this work missed the usage in
jstests.py. This continued to use the increasing outdated in-tree
manifest, which causes a number of problems

* It doesn't have an up-to-date list of tests
* It blocks removing that file
* It blocks landing various optimisations to make updating the
  manifest faster.

This patch fixes jstests.py to use a downloaded manifest. Unlike the
tests run through a mach frontend jstests.py doesn't know where the
objdir is, so it's hard to work out where to download the
manifest. This patch adopts a heuristic approach; if the path to the
jsshell looks like <root>/dist/bin and <root>/_tests exists, we assume
it's a gecko-like objdir and use <root>/_tests/web-platform/ for the
manifest; otherwise we just put it into the system tempdir.

Because the manifest has to be updated on startup, this patch causes a
startup time regression, but this will be considerably reduced by the
work in Bug 1497898 for which this is a prerequisite.
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/32aa5695f2ca
Use downloaded manifest for wpt in jsshell, r=bbouvier,Ms2ger
https://hg.mozilla.org/mozilla-central/rev/32aa5695f2ca
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
Depends on: 1509774
Flags: needinfo?(Ms2ger)
You need to log in before you can comment on or make changes to this bug.