Open Bug 1657305 Opened 9 months ago Updated 6 months ago

TypeError: keys must be str, int, float, bool or None, not bytes when running "mach taskgraph optimized"

Categories

(Testing :: web-platform-tests, defect)

Default
defect

Tracking

(Not tracked)

People

(Reporter: marco, Unassigned)

References

(Blocks 1 open bug)

Details

Traceback (most recent call last):
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/mach_commands.py", line 352, in show_taskgraph
    tg = getattr(tgg, graph_attr)
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/generator.py", line 192, in optimized_task_graph
    return self._run_until('optimized_task_graph')
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/generator.py", line 375, in _run_until
    k, v = next(self._run)
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/generator.py", line 278, in _run
    new_tasks = kind.load_tasks(
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/generator.py", line 72, in load_tasks
    tasks = [Task(self.name,
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/generator.py", line 72, in <listcomp>
    tasks = [Task(self.name,
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1971, in check_run_task_caches
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1889, in check_task_dependencies
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1877, in check_task_identifiers
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1860, in chain_of_trust
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1694, in build_task
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1685, in try_task_config_routes
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1675, in try_task_config_chemspill_prio
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1665, in try_task_config_env
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1630, in add_index_routes
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1435, in validate
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1412, in task_name_from_label
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1372, in set_defaults
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/task.py", line 1351, in set_implementation
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/job/__init__.py", line 311, in make_task_description
    for job in jobs:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/job/__init__.py", line 201, in use_fetches
    for job in order_tasks(config, jobs):
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/cached_tasks.py", line 22, in order_tasks
    pending = deque(tasks)
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/job/__init__.py", line 156, in set_label
    for job in jobs:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/job/__init__.py", line 140, in set_implementation
    for job in jobs:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/job/__init__.py", line 120, in rewrite_when_to_optimization
    for job in jobs:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/base.py", line 79, in __call__
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1765, in make_job_description
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1731, in set_schedules_components
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1676, in set_worker_type
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1666, in set_test_type
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1657, in set_tag
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1645, in set_profile
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1635, in set_retry_exit_status
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1624, in set_schedules_for_webrender_android
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1606, in enable_webrender
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1591, in allow_software_gl_layers
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1544, in split_chunks
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1494, in resolve_dynamic_chunks
    for task in tasks:
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/transforms/tests.py", line 1470, in set_test_manifests
    task['test-manifests'] = loader.get_manifests(
  File "/home/marco/Documenti/FD/mozilla-unified/python/mozbuild/mozbuild/util.py", line 1018, in method_call
    cache[args] = self.func(instance, *args)
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/util/chunking.py", line 191, in get_manifests
    tests = self.get_tests(suite)
  File "/home/marco/Documenti/FD/mozilla-unified/python/mozbuild/mozbuild/util.py", line 1018, in method_call
    cache[args] = self.func(instance, *args)
  File "/home/marco/Documenti/FD/mozilla-unified/taskcluster/taskgraph/util/chunking.py", line 182, in get_tests
    return list(resolver.resolve_tests(
  File "/home/marco/Documenti/FD/mozilla-unified/testing/mozbase/moztest/moztest/resolve.py", line 856, in resolve_tests
    for test in self._resolve(**kwargs):
  File "/home/marco/Documenti/FD/mozilla-unified/testing/mozbase/moztest/moztest/resolve.py", line 621, in _resolve
    self.add_wpt_manifest_data()
  File "/home/marco/Documenti/FD/mozilla-unified/testing/mozbase/moztest/moztest/resolve.py", line 752, in add_wpt_manifest_data
    manifests = manifestupdate.run(self.topsrcdir, self.topobjdir, rebuild=False,
  File "/home/marco/Documenti/FD/mozilla-unified/testing/web-platform/manifestupdate.py", line 120, in run
    manifests = load_and_update(logger, src_wpt_dir, test_paths,
  File "/home/marco/Documenti/FD/mozilla-unified/testing/web-platform/manifestupdate.py", line 186, in load_and_update
    m = manifest.manifest.load_and_update(paths["tests_path"],
  File "/home/marco/Documenti/FD/mozilla-unified/testing/web-platform/tests/tools/manifest/manifest.py", line 398, in load_and_update
    return _load_and_update(ensure_text(tests_root),
  File "/home/marco/Documenti/FD/mozilla-unified/testing/web-platform/tests/tools/manifest/manifest.py", line 466, in _load_and_update
    tree.dump_caches()
  File "/home/marco/Documenti/FD/mozilla-unified/testing/web-platform/tests/tools/manifest/vcs.py", line 130, in dump_caches
    cache.dump()
  File "/home/marco/Documenti/FD/mozilla-unified/testing/web-platform/tests/tools/manifest/vcs.py", line 153, in dump
    json.dump(self.data, f, indent=1)
  File "/usr/lib/python3.8/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.8/json/encoder.py", line 376, in _iterencode_dict
    raise TypeError(f'keys must be str, int, float, bool or None, '
TypeError: keys must be str, int, float, bool or None, not bytes

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

This is working for me; I wonder if it's the same problem as https://bugzilla.mozilla.org/show_bug.cgi?id=1652988#c9 which can be fixed by deleting the cache in ~/.mozbuild/cache/wpt.

Severity: -- → S3
Flags: needinfo?(james)

I'm getting this same error when running mach mochitest. I'm guessing that blowing away the cache will work -- but I debugged through this, and it seems wrong:

vcs.py is loading a gitignore.json file via gitignore.py: https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/gitignore/gitignore.py#242,270

gitignore.py intentionally uses bytes for the paths, so the cache ends up with bytes-valued keys. testing/web-platform/tests/tools/manifest/manifest.py has _load_and_update that reads in the cache and writes it back out using json.dump: https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/manifest/vcs.py#151

json.dump does not take kindly to keys being bytes, and issues the error here.

So it seems like anything that goes through this code is going to end up hitting this problem? Blowing away the cache files might fix it temporarily (I didn't trace through that path), but there's a mismatch between using bytes for these paths and json.dump to write them out.

I would just submit a patch, except that filenames always scare me. It seems generally easiest to UTF-8 decode them to strings, but on some filesystems it is possible to have non-UTF8 filenames, so decoding isn't guaranteed to be right. And for this chunk of code, I don't know whether that matters or not.

You need to log in before you can comment on or make changes to this bug.