Unable to download and install clang-tidy when running mach clang-format

RESOLVED FIXED in Firefox -esr60

Status

defect
P1
normal
RESOLVED FIXED
6 months ago
3 months ago

People

(Reporter: sfink, Assigned: andi)

Tracking

unspecified
mozilla66

Firefox Tracking Flags

(firefox-esr60 fixed, firefox66 fixed)

Details

Attachments

(1 attachment)

Reporter

Description

6 months ago
% mach clang-format
0:02.52 Could not find artifacts for a toolchain build named `linux64-clang-tidy`. Local commits and other changes in your checkout may cause this error. Try updating to a fresh checkout of mozilla-central to use artifact builds.
% ./mach artifact toolchain --from-build linux64-clang-tidy
0:02.57 Could not find artifacts for a toolchain build named `linux64-clang-tidy`. Local commits and other changes in your checkout may cause this error. Try updating to a fresh checkout of mozilla-central to use artifact builds.

This *does* exist:

https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.static-analysis.linux64-clang-tidy/artifacts/public/build/clang-tidy.tar.xz

Sadly, mach clang-format still insists on trying and failing to download its own.

The task it is getting is

Task(kind=u'toolchain', label=u'toolchain-linux64-clang-tidy', attributes={u'always_target': False, u'kind': u'toolchain', u'toolchain-artifact': 'public/build/clang-tidy.tar.xz', u'shipping_phase': None, u'run_on_projects': ['trunk', 'try'], u'shipping_product': None, u'cached_task': {u'type': u'toolchains.v3', u'name': u'linux64-clang-tidy', u'digest': '093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82'}, u'job_try_name': 'linux64-clang-tidy'}, task={u'scopes': [u'secrets:get:project/taskcluster/gecko/hgfingerprint', u'docker-worker:cache:level-3-checkouts-sparse-v3-4223580987c776a8c0d7'], u'tags': {u'os': u'linux', u'createdForUser': u'nobody@mozilla.com', u'worker-implementation': u'docker-worker', u'kind': u'toolchain', u'label': u'toolchain-linux64-clang-tidy'}, u'expires': {u'relative-datestamp': u'1 year'}, u'deadline': {u'relative-datestamp': u'1 day'}, u'payload': {u'onExitStatus': {u'retry': [72], u'purgeCaches': [72]}, u'maxRunTime': 3600, u'image': {u'path': u'public/image.tar.zst', u'type': u'task-image', u'taskId': {u'task-reference': u'<docker-image>'}}, u'cache': {u'level-3-checkouts-sparse-v3-4223580987c776a8c0d7': u'/builds/worker/checkouts'}, u'artifacts': {u'public/build': {u'path': u'/builds/worker/artifacts/', u'expires': {u'relative-datestamp': u'1 year'}, u'type': u'directory'}}, u'command': [u'/builds/worker/bin/run-task', u'--vcs-checkout=/builds/worker/workspace/build/src', u'--sparse-profile=build/sparse-profiles/toolchain-build', u'--', u'bash', u'-c', u'cd /builds/worker && workspace/build/src/taskcluster/scripts/misc/build-clang-tidy-linux.sh'], u'env': {u'SCCACHE_DISABLE': u'1', u'MOZ_BUILD_DATE': '20181205085718', u'TASKCLUSTER_CACHES': u'/builds/worker/checkouts', u'HG_STORE_PATH': u'/builds/worker/checkouts/hg-store', u'GECKO_HEAD_REV': '743fa529c58094f9ee904b92bc3bd26c6a21f346', u'MOZ_TOOLCHAINS': {u'task-reference': u'public/build/gcc.tar.xz@<toolchain-linux64-gcc-4.9>'}, u'GECKO_PATH': u'/builds/worker/workspace/build/src', u'MOZ_SCM_LEVEL': u'3', u'TASKCLUSTER_VOLUMES': u'/builds/worker/checkouts;/builds/worker/tooltool-cache;/builds/worker/workspace', u'GECKO_BASE_REPOSITORY': u'https://hg.mozilla.org/mozilla-unified', u'TOOLS_DISABLE': u'true', u'GECKO_HEAD_REPOSITORY': u'https://hg.mozilla.org/mozilla-central', u'MOZ_FETCHES_DIR': u'/builds/worker/workspace/build', u'MOZ_AUTOMATION': u'1', u'TASKCLUSTER_PROXY_URL': u'http://taskcluster/'}, u'features': {u'taskclusterProxy': True, u'chainOfTrust': True}}, u'provisionerId': u'aws-provisioner-v1', u'workerType': u'gecko-3-b-linux-large', u'created': {u'relative-datestamp': u'0 seconds'}, u'extra': {u'index': {u'rank': 1544029038}, u'chainOfTrust': {u'inputs': {u'docker-image': {u'task-reference': u'<docker-image>'}}}, u'treeherder': {u'jobKind': 'build', u'groupSymbol': 'TL', u'collection': {u'opt': True}, u'machine': {u'platform': u'toolchains'}, u'groupName': 'Toolchain builds for Linux 64-bits', u'tier': 1, u'symbol': 'clang-tidy'}, u'treeherder-platform': 'toolchains/opt', u'parent': u''}, u'priority': u'medium', u'routes': [u'index.gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82', u'index.gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.latest', u'index.gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.pushdate.2018.12.05.20181205165718', u'index.gecko.v2.mozilla-central.latest.static-analysis.linux64-clang-tidy', u'index.gecko.v2.mozilla-central.pushdate.2018.12.05.20181205165718.static-analysis.linux64-clang-tidy', u'index.gecko.v2.mozilla-central.pushlog-id.0.static-analysis.linux64-clang-tidy', u'index.gecko.v2.mozilla-central.revision.743fa529c58094f9ee904b92bc3bd26c6a21f346.static-analysis.linux64-clang-tidy', u'index.gecko.v2.trunk.revision.743fa529c58094f9ee904b92bc3bd26c6a21f346.static-analysis.linux64-clang-tidy', u'tc-treeherder.v2.mozilla-central.743fa529c58094f9ee904b92bc3bd26c6a21f346.0'], u'metadata': {u'owner': u'nobody@mozilla.com', u'source': u'https://hg.mozilla.org/mozilla-central/file/743fa529c58094f9ee904b92bc3bd26c6a21f346//home/sfink/src/mozilla/taskcluster/ci/toolchain', u'description': u'Clang-tidy build ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=743fa529c58094f9ee904b92bc3bd26c6a21f346))', u'name': u'toolchain-linux64-clang-tidy'}}, task_id=None, optimization={u'index-search': [u'gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82']}, dependencies={u'docker-image': u'build-docker-image-toolchain-build', u'toolchain-linux64-gcc-4.9': u'toolchain-linux64-gcc-4.9'}, release_artifacts=frozenset([]))

in particular, task.optimization is

{u'index-search': [u'gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82']}

and the URL https://index.taskcluster.net/v1/task/gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82 is a 404.

This happens when I am on the inbound tip, currently 49a3abc14837, and I have no lingering changes under taskcluster/.
Reporter

Comment 1

6 months ago
Ok, I finally tracked this down by dumping out everything used to generate the hashes and diffing it with a try push.

--- /tmp/th.txt 2018-12-11 11:02:58.413098677 -0800
+++ /tmp/mine.txt       2018-12-11 11:03:31.051698301 -0800
@@ -15,7 +15,9 @@
 HASH_PATH: u'2dcd472b4b755cf36820e8bc7f195453a599a14a5a788d232e548654903ff571 build/clang-plugin/CustomAttributes.inc\n'
 HASH_PATH: u'2ff0bb54175e664a25955d8da903c1cae84be95a5bde51f4fe1ee17e841e5670 build/clang-plugin/CustomMatchers.h\n'
 HASH_PATH: u'12d17a02166eae4203316bdbcb7923d21b360e3f13346fcbf780638679c57c38 build/clang-plugin/CustomTypeAnnotation.cpp\n'
+HASH_PATH: u'a58ba5c5ab469894e5c0c40935cd7262b312a537840cf92520520f2391f49fcd build/clang-plugin/CustomTypeAnnotation.cpp~\n'
 HASH_PATH: u'a01532a684766b81f89b2fabb716e416ebd141c1a8576cd6ad77b78023ddc6e3 build/clang-plugin/CustomTypeAnnotation.h\n'
+HASH_PATH: u'2f56609d9c37691fde4aba4b60fa7bb7f4ad93d26b1fe62140940fd2ba59be47 build/clang-plugin/CustomTypeAnnotation.h~\n'
 HASH_PATH: u'ad52dfbe49196f075a306aac1830c828c9be4872c5f00ef2068bd191ddd2687a build/clang-plugin/DanglingOnTemporaryChecker.cpp\n'
 HASH_PATH: u'8b03d0ea6eb8671e3bbb46eb48181b6a422999dd27011fce30dd9d25f58a6e4a build/clang-plugin/DanglingOnTemporaryChecker.h\n'
 HASH_PATH: u'bd1a5898eb1ef65f3f4247d665d2d7e85ebc0aba6f7862b76c3088b69a48ece3 build/clang-plugin/DiagnosticsMatcher.cpp\n'

So the problem is some temporary editor files I had lying around. I did many runs of |hg purge -p| and related commands to check for stuff like this, but I'm guessing that purge is ignoring it due to .hgignore.

This basically means that if you ever edit any files in build/clang-plugin/, you'll never be able to run mach clang-tidy again no matter what version you update to. This seems suboptimal.

This comes from the file pattern 'build/clang-plugin/**' from job.run.resources. I don't know the code well enough to suggest the right spot to fix this, but I do think it should be fixed; it's currently a major footgun that is blocking me from rebasing anything (because the format-source extension is trying to do mach clang-format which is failing.)
Assignee

Updated

6 months ago
Assignee: nobody → bpostelnicu
Assignee

Comment 2

6 months ago
this is a problem that actually hinders all of our artifacts unfortunately, we're building a hash, and part of that hash are the files from 'build/clang-plugin/**' and based on that hash we try to match on the artifacts server a package that matches the hash.
We could change the path to be more explicit about e.g. file extensions.
Reporter

Comment 4

6 months ago
I guess nobody else is running into this because once you get clang-tidy downloaded, mach clang-format via hg format-source will reuse the binary? (And of course, few people would have changes in that directory.)

Looks like FileFinder has an ignore parameter. Maybe toolchains.py could send in the contents of .hgignore if it exists? Or maybe the ignores should be given in the task's yml or something.

Alternatively, we could grab the clang-tidy.tar.xz via the taskcluster index.
Duplicate of this bug: 1514015
I'm experiencing the same problem when trying to run |mach clang-format| and |mach bootstrap|. I have some 'pyc' files in 'build/' dir but nothing under 'build/clang-plugin/':

> $ hg status
> $ hg status -i | egrep "^I build/"
> I build/appini_header.pyc
> I build/buildconfig.pyc
> I build/clang-plugin/ThirdPartyPaths.pyc
> I build/gen_automation.pyc
> I build/gen_symverscript.pyc
> I build/mach_bootstrap.pyc
> I build/valgrind/mach_commands.pyc
> I build/variables.pyc
> I build/clang-plugin/ThirdPartyPaths.pyc

This would cause the digest for the clang-tidy to change.
(In reply to Eric Rahm [:erahm] from comment #6)
> but nothing under 'build/clang-plugin/':
(...)
> > I build/clang-plugin/ThirdPartyPaths.pyc

hum...
As for a fix, see comment 3.
Priority: -- → P1

Comment 13

5 months ago
Pushed by bpostelnicu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/266dce70b10e
For building clang-tidy don't use temporary files when composing the hashes. r=froydnj,firefox-build-system-reviewers

Yeah, thanks!

Comment 15

5 months ago
bugherder
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
You need to log in before you can comment on or make changes to this bug.