1704620 - "./mach artifact toolchain --bootstrap --from-build linux64-node" fails with input/output error on Linux for me

Status: RESOLVED INVALID

(Firefox Build System :: General, defect)

Description

[Markus Stange [:mstange]]

I encountered this during ./mach bootstrap.

mstange@mstange-Ubuntu:~/code/mozilla$ ./mach artifact toolchain --bootstrap --from-build linux64-node
Error running mach:

    ['artifact', 'toolchain', '--bootstrap', '--from-build', 'linux64-node']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 5] Input/output error

  File "/home/mstange/code/mozilla/python/mozbuild/mozbuild/artifact_commands.py", line 388, in artifact_toolchain
    tasks = toolchain_task_definitions()
  File "/home/mstange/code/mozilla/python/mozbuild/mozbuild/toolchains.py", line 19, in toolchain_task_definitions
    toolchains = load_tasks_for_kind(params, "toolchain", root_dir=root_dir)
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/generator.py", line 490, in load_tasks_for_kind
    for task in tgg.full_task_set
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/generator.py", line 164, in full_task_set
    return self._run_until("full_task_set")
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/generator.py", line 435, in _run_until
    k, v = next(self._run)
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/generator.py", line 309, in _run
    new_tasks = kind.load_tasks(
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/generator.py", line 79, in load_tasks
    tasks = [
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/generator.py", line 79, in <listcomp>
    tasks = [
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 2112, in check_run_task_caches
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 2020, in check_task_dependencies
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 2006, in check_task_identifiers
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1987, in chain_of_trust
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1778, in build_task
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1769, in try_task_config_routes
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1759, in try_task_config_chemspill_prio
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1749, in try_task_config_env
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1711, in add_index_routes
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1506, in validate
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1483, in task_name_from_label
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1438, in set_defaults
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/task.py", line 1417, in set_implementation
    for task in tasks:
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/cached_tasks.py", line 61, in cache_task
    for task in order_tasks(config, tasks):
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/cached_tasks.py", line 19, in order_tasks
    pending = deque(tasks)
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/job/__init__.py", line 398, in make_task_description
    configure_taskdesc_for_run(
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/job/__init__.py", line 472, in configure_taskdesc_for_run
    func(config, job, taskdesc)
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/job/toolchain.py", line 156, in docker_worker_toolchain
    "digest-data": get_digest_data(config, run, taskdesc),
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/transforms/job/toolchain.py", line 80, in get_digest_data
    data = [hash_paths(GECKO, files)]
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/util/hash.py", line 53, in hash_paths
    hash_path(mozpath.abspath(mozpath.join(base_path, path))),
  File "/home/mstange/code/mozilla/python/mozbuild/mozbuild/util.py", line 1040, in __call__
    self[args] = self.func(*args)
  File "/home/mstange/code/mozilla/taskcluster/taskgraph/util/hash.py", line 21, in hash_path
    return hashlib.sha256(fh.read()).hexdigest()
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Is something broken with my python environment?

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Hmm, that's bizarre. I'm guessing that the fh.read() call is failing?
To further investigate, I'd recommend:

1. Checking which part of that line is failing: fh.read()? hashlib.sha256()? .hexdigest()?
2. If it is indeed an issue with fh.read(), then determine which file is failing to be read. Then, attempt to read it manually (cat, etc)

Let me know if that helps :)

Comment 2

[Markus Stange [:mstange]]

It's the fh.read call, on the file toolkit/crashreporter/google-breakpad/src/third_party/libdisasm/ia32_modrm.c. And you were right, cat also displays an input/output error for that file.

Comment 3

[Markus Stange [:mstange]]

I deleted the parent directory and then ran hg revert on it, and now everything's fine. Weird!

Comment 4

[Markus Stange [:mstange]]

Thanks for the help! I should have checked a bit more myself, but I inititally interpreted the call stack upside down. Hah!

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Nice work! I'd recommend running an fsck/chkdsk/whatever on that partition. If you're seeing input/output errors with cat, then there's something spicy going on. In the worst circumstance, your disk might be failing. Watch out for that :)