Closed Bug 1558363 Opened 5 years ago Closed 5 years ago

MANIFEST.json doesn't update when adding new web-platform tests

Categories

(Testing :: web-platform-tests, defect, P1)

Product:
Testing
Component:
web-platform-tests
Version:
Version 3
Type:
defect

Tracking

(firefox69 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla69
Tracking Flags:
Tracking Status
firefox69 --- fixed

People

(Reporter: cmarlow, Assigned: gsnedders)

References

Details

Attachments

(1 file)

Bug 1558363 - Fix WPT manifest updates when within a git tree
47 bytes, text/x-phabricator-request
Details | Review

When adding new web-platform tests, tests can fail unexpectedly because the MANIFEST.json file located in obj-XXX/_tests/web-platform/meta/MANIFEST.json doesn't get automatically updated when running the ./mach wpt /path/ command. A work around is to delete this file, run the mach command, and then it will be redownloaded for you.

I'm running into this issue as well.

A specific example to reproduce this problem.

  1. In this test file testing/web-platform/tests/css/css-multicol/multicol-span-all-010.html, I change the reference file from <link rel="match" href="multicol-span-all-010-ref.html"> to <link rel="match" href="multicol-span-all-009-ref.html">
  2. Running ./mach wpt testing/web-platform/tests/css/css-multicol/multicol-span-all-010.html

Expected result:
The manifest should be updated, and the reftest should compare multicol-span-all-010.html and multicol-span-all-009-ref.html.

Actual result:
The reftest compares multicol-span-all-010.html and multicol-span-all-010-ref.html.

0:34.90 pid:29074 1560276317416 Marionette INFO Testing http://web-platform.test:8000/css/css-multicol/multicol-span-all-010.html == http://web-platform.test:8000/css/css-multicol/multicol-span-all-010-ref.html

Running ./mach wpt-manifest-update --rebuild yields the following error.

$ ./mach wpt-manifest-update --rebuild
 0:00.00 WARNING The wpt manifest is now automatically updated, so running this command is usually unnecessary
Error running mach:

    ['wpt-manifest-update', '--rebuild']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AssertionError

  File "/home/tlin/Projects/gecko/testing/web-platform/mach_commands.py", line 330, in wpt_manifest_update
    return 0 if wpt_runner.update_manifest(logger, **params) else 1
  File "/home/tlin/Projects/gecko/testing/web-platform/mach_commands_base.py", line 48, in update_manifest
    **kwargs)
  File "/home/tlin/Projects/gecko/testing/web-platform/manifestupdate.py", line 121, in run
    cache_root=kwargs["cache_root"])
  File "/home/tlin/Projects/gecko/testing/web-platform/manifestupdate.py", line 190, in load_and_update
    cache_root=this_cache_root)
  File "/home/tlin/Projects/gecko/testing/web-platform/tests/tools/manifest/manifest.py", line 499, in load_and_update
    changed = manifest.update(tree)
  File "/home/tlin/Projects/gecko/testing/web-platform/tests/tools/manifest/manifest.py", line 290, in update
    assert rel_path in path_hash

Hi James,

I feel this bug might bite layout team members who develop web platform reftests in a subtle way. They might think the problem is in the content of their reftests, not some bugs in the reftest framework. Could you take a look at this?

Flags: needinfo?(james)

I'm now getting an error when trying to run any test. I've tried deleting the MANIFESTS.json file, but it gives the same error after downloading it. I've also manually downloaded the file an moved it into obj-XXX/_tests/web-platform/meta/MANIFEST.json, but I still get the following error.

 0:00.01 INFO Skipping manifest download because existing file is recent
Error running mach:

    ['wpt', 'testing/web-platform/tests/css/css-text-decor/text-underline-offset-001.html']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ValueError: No JSON object could be decoded

  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/mach_commands.py", line 296, in run_wpt
    return self.run_web_platform_tests(**params)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/mach_commands.py", line 288, in run_web_platform_tests
    return wpt_runner.run(logger, **params)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/mach_commands_base.py", line 28, in run
    self.update_manifest(logger)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/mach_commands_base.py", line 48, in update_manifest
    **kwargs)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/manifestupdate.py", line 121, in run
    cache_root=kwargs["cache_root"])
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/manifestupdate.py", line 190, in load_and_update
    cache_root=this_cache_root)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests/tools/manifest/manifest.py", line 498, in load_and_update
    working_copy, rebuild)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests/tools/manifest/vcs.py", line 34, in get_tree
    rebuild=rebuild)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests/tools/manifest/vcs.py", line 88, in __init__
    self.mtime_cache = MtimeCache(cache_path, root, manifest_path, rebuild)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests/tools/manifest/vcs.py", line 156, in __init__
    super(MtimeCache, self).__init__(cache_root, tests_root, rebuild=False)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests/tools/manifest/vcs.py", line 126, in __init__
    self.data = self.load(rebuild)
  File "/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests/tools/manifest/vcs.py", line 139, in load
    data = json.load(f)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")```

I'm now getting an error when trying to run any test.

There's another manifest in obj-XXX/_tests/web-platform/mozilla/meta/ which could be corrupt. It's basically safe to delete all of obj-XXX/_tests/web-platform/ since we don't actually use the things in there except when packaging tests.

I'm going to investigate the overall bug tomorrow, we may have a broken revision from upstream, or there may be a bug in some of the refactoring work that's gone on.

Priority: P3 → P1

(In reply to Ting-Yu Lin [:TYLin] (UTC-7) from comment #2)

Running ./mach wpt-manifest-update --rebuild yields the following error.

This was fixed upstream in https://github.com/web-platform-tests/wpt/pull/17072, if I'm not mistaken.

This has been broken since 97f4d5cc6aa151, when we started using git
to (quickly) get hashes of unchanged files. Unfortunately, the logic
for finding changed files only worked when the test root was the root
of the git repository.

Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/b975b4c1dfae Fix WPT manifest updates when within a git tree r=jgraham

I tried it out just now and I'm still having the same issue. I'm using the latest mozilla-central repo. I've tried doing a ./mach clobber and rebuilding, running the ./mach wpt-manifest-update --rebuild, and also manually deleting the two MANIFEST.json files and trying to run a test. They all come with a python error of ValueError: No JSON object could be decoded

That seems like a different issue. Can you work out which file it's trying to read when you do that (if nothing else then strace or similar should help, but you can also edit testing/web-platform/mach_commands.py and add something like

try:
    # Run the entrypoint into wpt-manifest-update
except Exception:
    import pdb
    pdb.post_mortem()

to get dropped into a debugger from which point it should be possible to figure out which file is being read.

Flags: needinfo?(james)

So far I've traced it to the load_and_update function in testing/web-platform/manifestupdate.py. When I print out the paths variable in the for loop, I get the following output:
{'metadata_path': '/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/meta', 'manifest_path': '/Users/cmarlow/src/builds/mozilla-central/obj-x86_64-apple-darwin18.6.0/_tests/web-platform/meta/MANIFEST.json', 'tests_path': '/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests', 'manifest_rel_path': 'meta/MANIFEST.json'}

The manifest path variable is
/Users/cmarlow/src/builds/mozilla-central/obj-x86_64-apple-darwin18.6.0/_tests/web-platform/meta/MANIFEST.json
I'm not super familiar with mach but it seems like that could be where the issue is coming from.

The exception is being raised here: https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/manifest/vcs.py#139
The cache_root variable passed into the CacheFile object is
/Users/cmarlow/.mozbuild/cache/wpt/c857ad80ec8add117d4598dda72cdce3585826ae40404b28d04a5fcf86657b84/meta
The tests_root variable is
/Users/cmarlow/src/builds/mozilla-central/testing/web-platform/tests
The cached json being loaded on that line is
/Users/cmarlow/.mozbuild/cache/wpt/c857ad80ec8add117d4598dda72cdce3585826ae40404b28d04a5fcf86657b84/meta/mtime.json

OK. So I forgot about that file also being json. Sorry.

If you delete that file nothing bad should happen, and we can make it robust to failure there.

Thanks for your help, I've got it working now

Blocks: 1558948

https://hg.mozilla.org/mozilla-central/rev/b975b4c1dfae

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox69: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Assignee: nobody → geoffers+mozilla
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/17785 for changes under testing/web-platform/tests
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/mozilla-inbound/rev/e25261038370 [wpt PR 17785] - [Gecko Bug 1558363] Fix WPT manifest updates when within a git tree, a=testonly
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: