1844349 - Broken artifact build - RuntimeError: Could not detect Taskcluster instance, set $TASKCLUSTER_ROOT_URL

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect)

Description

[Nicolas Chevobbe [:nchevobbe]]

I'm getting this error when running mach build for artifact build on latest central:

Error running mach:
 0:07.56     mach --log-no-times artifact install
 0:07.56 The error occurred in code that was called by the mach command. This is either
 0:07.56 a bug in the called code itself or in the way that mach is calling it.
 0:07.56 You can invoke ``./mach busted`` to check if this issue is already on file. If it
 0:07.56 isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
 0:07.56 misbehaving, you can also inspect the dependencies of bug 1543241.
 0:07.56 If filing a bug, please include the full output of mach, including this error
 0:07.56 message.
 0:07.56 The details of the failure are as follows:
 0:07.56 RuntimeError: Could not detect Taskcluster instance, set $TASKCLUSTER_ROOT_URL
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifact_commands.py", line 188, in artifact_install
 0:07.56     return artifacts.install_from(source, distdir or command_context.distdir)
 0:07.56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1656, in install_from
 0:07.56     return self.install_from_recent(distdir)
 0:07.56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1552, in install_from_recent
 0:07.56     return self._install_from_hg_pushheads(hg_pushheads, distdir)
 0:07.56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1533, in _install_from_hg_pushheads
 0:07.56     urls = self.find_pushhead_artifacts(
 0:07.56            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1389, in find_pushhead_artifacts
 0:07.56     taskId, artifacts = task_cache.artifacts(
 0:07.56                         ^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 903, in wrapper
 0:07.56     result = method(self, *args, **kwargs)
 0:07.56              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/python/mozbuild/mozbuild/artifacts.py", line 1075, in artifacts
 0:07.56     taskId = find_task_id(namespace)
 0:07.56              ^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 197, in find_task_id
 0:07.56     response = _do_request(get_index_url(index_path, use_proxy))
 0:07.56                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 191, in get_index_url
 0:07.56     index_tmpl = liburls.api(get_root_url(use_proxy), "index", "v1", "task{}/{}")
 0:07.56                              ^^^^^^^^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/third_party/python/taskcluster_taskgraph/taskgraph/util/memoize.py", line 25, in __call__
 0:07.56     self[args] = self.func(*args)
 0:07.56                  ^^^^^^^^^^^^^^^^
 0:07.56   File "/Users/nchevobbe/Projects/mozilla-central/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 67, in get_root_url
 0:07.56     raise RuntimeError(
 0:07.56 Sentry event ID: b755b72cc3ac458fa876a31d743d7ebd
 0:07.66 Sentry is attempting to send 0 pending error messages
 0:07.66 Waiting up to 2 seconds
 0:07.66 Press Ctrl-C to quit
 0:08.08 make[3]: *** [recurse_artifact] Error 1
 0:08.08 make[2]: *** [artifact] Error 2
 0:08.08 make[1]: *** [default] Error 2
 0:08.08 make: *** [build] Error 2
 0:08.10 W 0 compiler warnings present.

Issue seems to start with Bug 1695312

Comment 1

[:Gijs (he/him)]

Backing out only bug 1695312 doesn't appear to be sufficient, I have to back out bug 1808732 as well to make things work, otherwise I get errors like this:

Traceback (most recent call last):
  File "$srcdir/./mach", line 125, in <module>
    main(sys.argv[1:])
  File "$srcdir/./mach", line 117, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "$srcdir/./mach", line 37, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "$srcdir/./mach", line 23, in load_mach
    return mach_initialize.initialize(dir_path)
  File "$srcdir/build/mach_initialize.py", line 534, in initialize
    driver.load_commands_from_spec(MACH_COMMANDS, topsrcdir, missing_ok=missing_ok)
  File "$srcdir/python/mach/mach/main.py", line 294, in load_commands_from_spec
    self.load_commands_from_file(topsrcdir / path)
  File "$srcdir/python/mach/mach/main.py", line 276, in load_commands_from_file
    imp.load_source(module_name, str(path))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 711, in _load
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "$srcdir/python/mozbuild/mozbuild/vendor/mach_commands.py", line 10, in <module>
    from mozbuild.vendor.moz_yaml import MozYamlVerifyError, load_moz_yaml
  File "$srcdir/python/mozbuild/mozbuild/vendor/moz_yaml.py", line 14, in <module>
    import voluptuous
ModuleNotFoundError: No module named 'voluptuous'

Comment 2

[:Gijs (he/him)]

:ahal, are you able to help here / have any idea what's going on?

If I back out both bug 1808732 and bug 1695312, the issue appears to go away for me.

Comment 3

[Andrew Halberstadt [:ahal]]

The voluptuous error is a result of bug 1808732 for sure. Adding that package to python/sites/build.txt should make the problem go away, though I wouldn't be surprised if that unlocked another missing package. I'm unsure about the initial error though. It's probably best to backout both patches at this point as I see other bugs being filed as regressions too.

Comment 4

[:Gijs (he/him)]

Both patches backed out of central so central builds for me now - let's close this out.

Comment 5

[Nick Alexander :nalexander [he/him]]

I am on:

$ git rev-parse HEAD
29534558705c235c0ec8950f863f32544669801c
$ git cinnabar git2hg 29534558705c235c0ec8950f863f32544669801c
898b3ffbdca34afc5fd0304f07f769e3d243265e

which corresponds to https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=898b3ffbdca34afc5fd0304f07f769e3d243265e and has the backouts. But this is still failing for me locally:

 0:09.93 Error running mach:
 0:09.93     mach --log-no-times artifact install
 0:09.93 The error occurred in code that was called by the mach command. This is either
 0:09.93 a bug in the called code itself or in the way that mach is calling it.
 0:09.93 You can invoke ``./mach busted`` to check if this issue is already on file. If it
 0:09.93 isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
 0:09.93 misbehaving, you can also inspect the dependencies of bug 1543241.
 0:09.93 If filing a bug, please include the full output of mach, including this error
 0:09.93 message.
 0:09.93 The details of the failure are as follows:
 0:09.93 requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/P0KyLRifSuKADmwjNJ9E-w/artifacts
 0:09.93   File "/Users/nalexander/Mozilla/gecko.git/python/mozbuild/mozbuild/artifact_commands.py", line 188, in artifact_install
 0:09.93     return artifacts.install_from(source, distdir or command_context.distdir)
 0:09.93            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.93   File "/Users/nalexander/Mozilla/gecko.git/python/mozbuild/mozbuild/artifacts.py", line 1654, in install_from
 0:09.93     return self.install_from_task(os.environ[var], distdir)
 0:09.93            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.93   File "/Users/nalexander/Mozilla/gecko.git/python/mozbuild/mozbuild/artifacts.py", line 1612, in install_from_task
 0:09.93     artifacts = list_artifacts(taskId)
 0:09.93                 ^^^^^^^^^^^^^^^^^^^^^^
 0:09.93   File "/Users/nalexander/Mozilla/gecko.git/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 171, in list_artifacts
 0:09.93     response = _do_request(get_artifact_url(task_id, "", use_proxy).rstrip("/"))
 0:09.93                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:09.93   File "/Users/nalexander/Mozilla/gecko.git/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 124, in _do_request
 0:09.93     response.raise_for_status()
 0:09.93   File "/Users/nalexander/Mozilla/gecko.git/third_party/python/requests/requests/models.py", line 943, in raise_for_status
 0:09.93     raise HTTPError(http_error_msg, response=self)
 0:09.93 Sentry event ID: 7ad3ffc6c58d4f0d8bd626b4a11ce45b
 0:10.04 Sentry is attempting to send 0 pending error messages
 0:10.04 Waiting up to 2 seconds
 0:10.04 Press Ctrl-C to quit
 0:10.09 gmake[3]: *** [Makefile:122: recurse_artifact] Error 1
 0:10.09 gmake[2]: *** [/Users/nalexander/Mozilla/gecko.git/config/recurse.mk:34: artifact] Error 2
 0:10.09 gmake[1]: *** [/Users/nalexander/Mozilla/gecko.git/config/rules.mk:361: default] Error 2
 0:10.09 gmake: *** [client.mk:60: build] Error 2
 0:10.12 W 0 compiler warnings present.

Any thoughts?

Comment 6

[Julien Cristau [:jcristau] (back August 18)]

Do you maybe have MOZ_ARTIFACT_TASK set in your environment? (guessing from https://searchfox.org/mozilla-central/rev/8d43262674d6c6d469b821cca579b1240ebb42a5/python/mozbuild/mozbuild/artifacts.py#1654)

Comment 7

[Nick Alexander :nalexander [he/him]]

(In reply to Julien Cristau [:jcristau] from comment #6)

Do you maybe have MOZ_ARTIFACT_TASK set in your environment? (guessing from https://searchfox.org/mozilla-central/rev/8d43262674d6c6d469b821cca579b1240ebb42a5/python/mozbuild/mozbuild/artifacts.py#1654)

Ding ding ding! Well spotted. Thanks, Julien!