1512253 - Unable to download and install clang-tidy when running mach clang-format

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, defect, P1)

Description

[Steve Fink [:sfink] [:s:]]

% mach clang-format
0:02.52 Could not find artifacts for a toolchain build named `linux64-clang-tidy`. Local commits and other changes in your checkout may cause this error. Try updating to a fresh checkout of mozilla-central to use artifact builds.
% ./mach artifact toolchain --from-build linux64-clang-tidy
0:02.57 Could not find artifacts for a toolchain build named `linux64-clang-tidy`. Local commits and other changes in your checkout may cause this error. Try updating to a fresh checkout of mozilla-central to use artifact builds.

This *does* exist:

https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.static-analysis.linux64-clang-tidy/artifacts/public/build/clang-tidy.tar.xz

Sadly, mach clang-format still insists on trying and failing to download its own.

The task it is getting is

Task(kind=u'toolchain', label=u'toolchain-linux64-clang-tidy', attributes={u'always_target': False, u'kind': u'toolchain', u'toolchain-artifact': 'public/build/clang-tidy.tar.xz', u'shipping_phase': None, u'run_on_projects': ['trunk', 'try'], u'shipping_product': None, u'cached_task': {u'type': u'toolchains.v3', u'name': u'linux64-clang-tidy', u'digest': '093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82'}, u'job_try_name': 'linux64-clang-tidy'}, task={u'scopes': [u'secrets:get:project/taskcluster/gecko/hgfingerprint', u'docker-worker:cache:level-3-checkouts-sparse-v3-4223580987c776a8c0d7'], u'tags': {u'os': u'linux', u'createdForUser': u'[e-mail]', u'worker-implementation': u'docker-worker', u'kind': u'toolchain', u'label': u'toolchain-linux64-clang-tidy'}, u'expires': {u'relative-datestamp': u'1 year'}, u'deadline': {u'relative-datestamp': u'1 day'}, u'payload': {u'onExitStatus': {u'retry': [72], u'purgeCaches': [72]}, u'maxRunTime': 3600, u'image': {u'path': u'public/image.tar.zst', u'type': u'task-image', u'taskId': {u'task-reference': u'<docker-image>'}}, u'cache': {u'level-3-checkouts-sparse-v3-4223580987c776a8c0d7': u'/builds/worker/checkouts'}, u'artifacts': {u'public/build': {u'path': u'/builds/worker/artifacts/', u'expires': {u'relative-datestamp': u'1 year'}, u'type': u'directory'}}, u'command': [u'/builds/worker/bin/run-task', u'--vcs-checkout=/builds/worker/workspace/build/src', u'--sparse-profile=build/sparse-profiles/toolchain-build', u'--', u'bash', u'-c', u'cd /builds/worker && workspace/build/src/taskcluster/scripts/misc/build-clang-tidy-linux.sh'], u'env': {u'SCCACHE_DISABLE': u'1', u'MOZ_BUILD_DATE': '20181205085718', u'TASKCLUSTER_CACHES': u'/builds/worker/checkouts', u'HG_STORE_PATH': u'/builds/worker/checkouts/hg-store', u'GECKO_HEAD_REV': '743fa529c58094f9ee904b92bc3bd26c6a21f346', u'MOZ_TOOLCHAINS': {u'task-reference': u'public/build/gcc.tar.xz@<toolchain-linux64-gcc-4.9>'}, u'GECKO_PATH': u'/builds/worker/workspace/build/src', u'MOZ_SCM_LEVEL': u'3', u'TASKCLUSTER_VOLUMES': u'/builds/worker/checkouts;/builds/worker/tooltool-cache;/builds/worker/workspace', u'GECKO_BASE_REPOSITORY': u'https://hg.mozilla.org/mozilla-unified', u'TOOLS_DISABLE': u'true', u'GECKO_HEAD_REPOSITORY': u'https://hg.mozilla.org/mozilla-central', u'MOZ_FETCHES_DIR': u'/builds/worker/workspace/build', u'MOZ_AUTOMATION': u'1', u'TASKCLUSTER_PROXY_URL': u'http://taskcluster/'}, u'features': {u'taskclusterProxy': True, u'chainOfTrust': True}}, u'provisionerId': u'aws-provisioner-v1', u'workerType': u'gecko-3-b-linux-large', u'created': {u'relative-datestamp': u'0 seconds'}, u'extra': {u'index': {u'rank': 1544029038}, u'chainOfTrust': {u'inputs': {u'docker-image': {u'task-reference': u'<docker-image>'}}}, u'treeherder': {u'jobKind': 'build', u'groupSymbol': 'TL', u'collection': {u'opt': True}, u'machine': {u'platform': u'toolchains'}, u'groupName': 'Toolchain builds for Linux 64-bits', u'tier': 1, u'symbol': 'clang-tidy'}, u'treeherder-platform': 'toolchains/opt', u'parent': u''}, u'priority': u'medium', u'routes': [u'index.gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82', u'index.gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.latest', u'index.gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.pushdate.2018.12.05.20181205165718', u'index.gecko.v2.mozilla-central.latest.static-analysis.linux64-clang-tidy', u'index.gecko.v2.mozilla-central.pushdate.2018.12.05.20181205165718.static-analysis.linux64-clang-tidy', u'index.gecko.v2.mozilla-central.pushlog-id.0.static-analysis.linux64-clang-tidy', u'index.gecko.v2.mozilla-central.revision.743fa529c58094f9ee904b92bc3bd26c6a21f346.static-analysis.linux64-clang-tidy', u'index.gecko.v2.trunk.revision.743fa529c58094f9ee904b92bc3bd26c6a21f346.static-analysis.linux64-clang-tidy', u'tc-treeherder.v2.mozilla-central.743fa529c58094f9ee904b92bc3bd26c6a21f346.0'], u'metadata': {u'owner': u'[e-mail]', u'source': u'https://hg.mozilla.org/mozilla-central/file/743fa529c58094f9ee904b92bc3bd26c6a21f346//home/sfink/src/mozilla/taskcluster/ci/toolchain', u'description': u'Clang-tidy build ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=743fa529c58094f9ee904b92bc3bd26c6a21f346))', u'name': u'toolchain-linux64-clang-tidy'}}, task_id=None, optimization={u'index-search': [u'gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82']}, dependencies={u'docker-image': u'build-docker-image-toolchain-build', u'toolchain-linux64-gcc-4.9': u'toolchain-linux64-gcc-4.9'}, release_artifacts=frozenset([]))

in particular, task.optimization is

{u'index-search': [u'gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82']}

and the URL https://index.taskcluster.net/v1/task/gecko.cache.level-3.toolchains.v3.linux64-clang-tidy.hash.093df8b7f620073ec5e71f67b187de7f81cee11c883cda31143f2f4435e0dc82 is a 404.

This happens when I am on the inbound tip, currently 49a3abc14837, and I have no lingering changes under taskcluster/.

Comment 1

[Steve Fink [:sfink] [:s:]]

Ok, I finally tracked this down by dumping out everything used to generate the hashes and diffing it with a try push.

--- /tmp/th.txt 2018-12-11 11:02:58.413098677 -0800
+++ /tmp/mine.txt       2018-12-11 11:03:31.051698301 -0800
@@ -15,7 +15,9 @@
 HASH_PATH: u'2dcd472b4b755cf36820e8bc7f195453a599a14a5a788d232e548654903ff571 build/clang-plugin/CustomAttributes.inc\n'
 HASH_PATH: u'2ff0bb54175e664a25955d8da903c1cae84be95a5bde51f4fe1ee17e841e5670 build/clang-plugin/CustomMatchers.h\n'
 HASH_PATH: u'12d17a02166eae4203316bdbcb7923d21b360e3f13346fcbf780638679c57c38 build/clang-plugin/CustomTypeAnnotation.cpp\n'
+HASH_PATH: u'a58ba5c5ab469894e5c0c40935cd7262b312a537840cf92520520f2391f49fcd build/clang-plugin/CustomTypeAnnotation.cpp~\n'
 HASH_PATH: u'a01532a684766b81f89b2fabb716e416ebd141c1a8576cd6ad77b78023ddc6e3 build/clang-plugin/CustomTypeAnnotation.h\n'
+HASH_PATH: u'2f56609d9c37691fde4aba4b60fa7bb7f4ad93d26b1fe62140940fd2ba59be47 build/clang-plugin/CustomTypeAnnotation.h~\n'
 HASH_PATH: u'ad52dfbe49196f075a306aac1830c828c9be4872c5f00ef2068bd191ddd2687a build/clang-plugin/DanglingOnTemporaryChecker.cpp\n'
 HASH_PATH: u'8b03d0ea6eb8671e3bbb46eb48181b6a422999dd27011fce30dd9d25f58a6e4a build/clang-plugin/DanglingOnTemporaryChecker.h\n'
 HASH_PATH: u'bd1a5898eb1ef65f3f4247d665d2d7e85ebc0aba6f7862b76c3088b69a48ece3 build/clang-plugin/DiagnosticsMatcher.cpp\n'

So the problem is some temporary editor files I had lying around. I did many runs of |hg purge -p| and related commands to check for stuff like this, but I'm guessing that purge is ignoring it due to .hgignore.

This basically means that if you ever edit any files in build/clang-plugin/, you'll never be able to run mach clang-tidy again no matter what version you update to. This seems suboptimal.

This comes from the file pattern 'build/clang-plugin/**' from job.run.resources. I don't know the code well enough to suggest the right spot to fix this, but I do think it should be fixed; it's currently a major footgun that is blocking me from rebasing anything (because the format-source extension is trying to do mach clang-format which is failing.)

Comment 2

[Andi [:andi]]

this is a problem that actually hinders all of our artifacts unfortunately, we're building a hash, and part of that hash are the files from 'build/clang-plugin/**' and based on that hash we try to match on the artifacts server a package that matches the hash.

Comment 3

[Mike Hommey [:glandium]]

We could change the path to be more explicit about e.g. file extensions.

Comment 4

[Steve Fink [:sfink] [:s:]]

I guess nobody else is running into this because once you get clang-tidy downloaded, mach clang-format via hg format-source will reuse the binary? (And of course, few people would have changes in that directory.)

Looks like FileFinder has an ignore parameter. Maybe toolchains.py could send in the contents of .hgignore if it exists? Or maybe the ignores should be given in the task's yml or something.

Alternatively, we could grab the clang-tidy.tar.xz via the taskcluster index.

Comment 6

[Eric Rahm [:erahm]]

I'm experiencing the same problem when trying to run |mach clang-format| and |mach bootstrap|. I have some 'pyc' files in 'build/' dir but nothing under 'build/clang-plugin/':

> $ hg status
> $ hg status -i | egrep "^I build/"
> I build/appini_header.pyc
> I build/buildconfig.pyc
> I build/clang-plugin/ThirdPartyPaths.pyc
> I build/gen_automation.pyc
> I build/gen_symverscript.pyc
> I build/mach_bootstrap.pyc
> I build/valgrind/mach_commands.pyc
> I build/variables.pyc

Comment 7

[Tom Prince [:tomprince]]

> I build/clang-plugin/ThirdPartyPaths.pyc

This would cause the digest for the clang-tidy to change.

Comment 8

[Mike Hommey [:glandium]]

(In reply to Eric Rahm [:erahm] from comment #6)
> but nothing under 'build/clang-plugin/':
(...)
> > I build/clang-plugin/ThirdPartyPaths.pyc

hum...

Comment 9

[Mike Hommey [:glandium]]

As for a fix, see comment 3.

Comment 10

[Andi [:andi]]

Let's make sure we don't regress first: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38fa07e58b318495bf4c707f6ec0afb472b961d8

Comment 11

[Andi [:andi]]

Also for macOS with some modifications: https://treeherder.mozilla.org/#/jobs?repo=try&revision=39f144dd9ec20a707fa0f161418e5f6610a36954

Comment 12

[Andi [:andi]]

Attachment: Bug 1512253 - For building clang-tidy don't use temporary files when composing the hashes. r?froydnj

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/266dce70b10e
For building clang-tidy don't use temporary files when composing the hashes. r=froydnj,firefox-build-system-reviewers

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Yeah, thanks!

Comment 15

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/266dce70b10e

Comment 16

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/caa6172eac8e