1403997 - Toolchain Dependencies are not downloaded if a Tooltool manifest is omitted

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

While trying to remove the last item from the tooltool manifest (rust) I discovered that if I completely removed the tooltool manifest - none of the toolchain dependencies was downloaded. 

Example: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d4523d179228b607c70a7aedfc4df48790e0e4c6&selectedJob=133839927



If I created an empty tooltool manifest and added it back, the toolchain dependencies were downloaded.

Example: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0bbf4e751d6a99255f056332f37d71e1d1017286&selectedJob=133849319
(This build is broken, but because I didn't know the correct rust dep to use at time-of-writing this bug. You can see in the log however that gcc is present and the toolchain deps were downloaded - search for 'Downloading... 0.0')

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1403997 - Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured.

Bug 1382564 made the `mach artifact toolchain` invocations from
mozharness use the MOZ_TOOLCHAINS environment variable when it's set by
taskcluster through the decision task, so that toolchain dependencies
from the task graph are used, but the mozharness code is still skipping
mach artifact toolchain when MOZ_TOOLCHAINS is set but there is no
tooltool manifest set. Most jobs today still have a tooltool manifest
set, but jobs shouldn't need a dummy tooltool manifest to use toolchain
dependencies automatically.

Review commit: https://reviewboard.mozilla.org/r/184836/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/184836/

Comment 2

[Mike Hommey [:glandium]]

Comment on attachment 8913494 [details]
Bug 1403997 - Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured.

Duh, I forgot gps is away.

Comment 3

[Michael Shal [:mshal]]

Comment on attachment 8913494 [details]
Bug 1403997 - Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured.

https://reviewboard.mozilla.org/r/184836/#review190676

It's sad how many different tooltool fetching functions we have.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/610baed314d5
Still run `mach artifact toolchain` when no tooltool manifest is set but MOZ_TOOLCHAINS is. r=mshal

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/610baed314d5

Comment 6

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

This patch has a bug in it:

>  [mozharness: 2017-10-03 17:15:05.246924Z] Finished build step (failed)
>  Uncaught exception: Traceback (most recent call last):
>    File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 2059, in run
>      self.run_action(action)
>    File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 1997, in run_action
>      self._possibly_run_method("preflight_%s" % method_name)
>    File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 1938, in _possibly_run_method
>      return getattr(self, method_name)()
>    File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/mozilla/building/buildbase.py", line 1604, in preflight_build
>      self._run_tooltool()
>    File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/mozilla/building/buildbase.py", line 1126, in _run_tooltool
>      manifest_src)
>    File "/usr/lib/python2.7/posixpath.py", line 68, in join
>      if b.startswith('/'):
>  AttributeError: 'NoneType' object has no attribute 'startswith'

Try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c7b99a3887e189bc4ddbc497ac0b8ac4fdb4dfb&selectedJob=134680030

Try run without removing tooltool (still errors, but it compiles): https://treeherder.mozilla.org/#/jobs?repo=try&revision=bb495264c5f69ed02a94c4725fc27777a17064cd&selectedJob=134679973

Comment 7

[Mike Hommey [:glandium]]

Comment on attachment 8913494 [details]
Bug 1403997 - Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/184836/diff/1-2/

Comment 8

[Mike Hommey [:glandium]]

Comment on attachment 8913494 [details]
Bug 1403997 - Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured.

This is separate patch on top of the one that already landed, but mozreview idn't really great for those situations.

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8913494 [details]
Bug 1403997 - Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured.

https://reviewboard.mozilla.org/r/184836/#review191526

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e0ba108cf6d9
Don't pass tooltool flags to `mach artifact toolchain` when no tooltool manifest is configured. r=mshal

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/e0ba108cf6d9