Closed Bug 1471905 Opened Last year Closed 11 months ago

Include toolchain dependency hashes in downstream toolchain hashes.

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(firefox-esr60 fixed, firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox-esr60 --- fixed
firefox65 --- fixed

People

(Reporter: CosminS, Assigned: tomprince)

References

Details

Attachments

(3 files, 1 obsolete file)

All trees (except try and try-comm-central) are closed because of this issue:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&selectedJob=185381850

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=185376540&repo=autoland&lineNumber=1134

[task 2018-06-28T14:28:08.536Z] 14:28:08     INFO -  configure: error: installation or configuration problem: C compiler cannot create executables.
[task 2018-06-28T14:28:08.538Z] 14:28:08     INFO -  DEBUG: <truncated - see config.log for full output>
[task 2018-06-28T14:28:08.538Z] 14:28:08     INFO -  DEBUG: configure:854: checking target system type
[task 2018-06-28T14:28:08.538Z] 14:28:08     INFO -  DEBUG: configure:872: checking build system type
[task 2018-06-28T14:28:08.538Z] 14:28:08     INFO -  DEBUG: configure:1293: checking for ranlib
[task 2018-06-28T14:28:08.538Z] 14:28:08     INFO -  DEBUG: configure:1328: checking for ar
[task 2018-06-28T14:28:08.538Z] 14:28:08     INFO -  DEBUG: configure:1368: checking for gcc
[task 2018-06-28T14:28:08.539Z] 14:28:08     INFO -  DEBUG: configure:1481: checking whether the C compiler (/builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin11 -B /builds/worker/workspace/build/src/cctools/bin -isysroot /builds/worker/workspace/build/src/MacOSX10.11.sdk -std=gnu99  -Wl,-syslibroot,/builds/worker/workspace/build/src/MacOSX10.11.sdk -Wl,-dead_strip) works
[task 2018-06-28T14:28:08.539Z] 14:28:08     INFO -  DEBUG: configure:1497: /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin11 -B /builds/worker/workspace/build/src/cctools/bin -isysroot /builds/worker/workspace/build/src/MacOSX10.11.sdk -std=gnu99 -o conftest   -Wl,-syslibroot,/builds/worker/workspace/build/src/MacOSX10.11.sdk -Wl,-dead_strip conftest.c  1>&5
[task 2018-06-28T14:28:08.539Z] 14:28:08     INFO -  DEBUG: configure:1494:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
[task 2018-06-28T14:28:08.539Z] 14:28:08     INFO -  DEBUG: main(){return(0);}
[task 2018-06-28T14:28:08.540Z] 14:28:08     INFO -  DEBUG: ^
[task 2018-06-28T14:28:08.540Z] 14:28:08     INFO -  DEBUG: 1 warning generated.
[task 2018-06-28T14:28:08.540Z] 14:28:08     INFO -  DEBUG: /builds/worker/workspace/build/src/cctools/bin/x86_64-apple-darwin11-ld: /builds/worker/workspace/build/src/clang/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /builds/worker/workspace/build/src/cctools/bin/x86_64-apple-darwin11-ld)
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG: clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG: configure: failed program was:
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG:
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG: #line 1492 "configure"
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG: #include "confdefs.h"
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG:
[task 2018-06-28T14:28:08.541Z] 14:28:08     INFO -  DEBUG: main(){return(0);}
[task 2018-06-28T14:28:08.542Z] 14:28:08     INFO -  DEBUG: configure: error: installation or configuration problem: C compiler cannot create executables.
[task 2018-06-28T14:28:08.542Z] 14:28:08     INFO -  ERROR: old-configure failed
[task 2018-06-28T14:28:08.569Z] 14:28:08     INFO -  *** Fix above errors and then restart with\
[task 2018-06-28T14:28:08.570Z] 14:28:08     INFO -                 "/usr/bin/make -f client.mk build"
[task 2018-06-28T14:28:08.570Z] 14:28:08     INFO -  client.mk:123: recipe for target 'configure' failed
[task 2018-06-28T14:28:08.570Z] 14:28:08     INFO -  make: *** [configure] Error 1
[task 2018-06-28T14:28:08.650Z] 14:28:08    ERROR - Return code: 2
[task 2018-06-28T14:28:08.650Z] 14:28:08  WARNING - setting return code to 2
[task 2018-06-28T14:28:08.651Z] 14:28:08    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-06-28T14:28:08.651Z] 14:28:08    FATAL - Running post_fatal callback...
[task 2018-06-28T14:28:08.651Z] 14:28:08    FATAL - Exiting -1
[task 2018-06-28T14:28:08.651Z] 14:28:08     INFO - [mozharness: 2018-06-28 14:28:08.651538Z] Finished build step (failed)
[task 2018-06-28T14:28:08.651Z] 14:28:08     INFO - Running post-run listener: _summarize
[task 2018-06-28T14:28:08.651Z] 14:28:08    ERROR - # TBPL FAILURE #
[task 2018-06-28T14:28:08.651Z] 14:28:08     INFO - [mozharness: 2018-06-28 14:28:08.651893Z] FxDesktopBuild summary:
[task 2018-06-28T14:28:08.652Z] 14:28:08    ERROR - # TBPL FAILURE #
Severity: normal → blocker
It looks like this is being caused by cctools being built with a newer version of gcc than we are installing. Bug 1451104 landed a newer version of gcc, which caused a new version of cctools to be built (https://tools.taskcluster.net/groups/D2dEtlRqQgaicMHpWSxcJA/tasks/XuqwjvQ1RkSNICs1wm8KGw/details) but apparently it got the same hash, so when that got backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/b59bec39784a7bc0808836029a555c0286e74043 other tasks were still picking it up.
It looks like `toolchain-linux64-cctools-port` bakes in that it depends on `toolchain-linux64-clang-6`, and I'm guessing  Bug 1451104 made that implicitly depend on a newer version of libstdc++.
I'm going to get rid of the bad indexed cctools toolchain. But we should fix the dependency information in the index to track the version of toolchains that are getting used.
Component: General → Task Configuration
Product: Release Engineering → Firefox Build System
QA Contact: catlee
Summary: Infra issue - C compiler cannot create executables → Include toolchain dependency hashes in downstream toolchain hashes.
Version: --- → unspecified
Severity: blocker → normal
Where does one look to fix the dependency information calculation?
taskcluster/taskgraph/transforms/job/toolchain.py

It's going to be tricky to handle...
gps, do you know who could possible handle this, or can you think of better ways to try and address this?  The long and short of the problem is:

1. We upgraded the version of GCC we used to build clang in bug 1451104.
2. We used the newly-built clang to build various incarnations of cctools.
3. The newly-built cctools apparently has the same hash (?) as the cctools built with the clang built with the older GCC.
4. Build jobs can then get a newer cctools but older versions of other packages (?).
5. The newer and older cctools, however, require different versions of libstdc++.
6. The newer cctools then blows up when used with older tools that don't install the required libstdc++.

I think we could *possibly* work around this by statically linking cctools with libstdc++ (and possibly other things?), but that feels like kind of a hack, and one that'd we have to keep repeating.  (Though we do build clang itself with statically linking libstdc++...)
Flags: needinfo?(gps)
There are several kinds that cache tasks based on the inputs that go into the task. Historically,
these inputs included the name of upstream tasks. This change these tasks to include the digest
of the upstream tasks.

This also bumps the version of the docker and toolchain as every digest is changed for them.
`mach artifact toolchain` gets task definitions from taskgraph, to get the
index path to find the artifacts at. Now that these index paths depend on the
digests of fetch tasks, those kinds need to be loaded as well. This adds a
supported API to get task definitions for a given kind, which loads all the
kind dependencies.
Keywords: leave-open
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/8953bab8246c
[taskgraph] Fix some typos in docker image transform; r=dustin
https://hg.mozilla.org/integration/autoland/rev/aa145dd45f48
[taskgraph] Don't calculate docker context hashes if asked for a fast taskgraph; r=dustin
Duplicate of this bug: 1394658
Attachment #9025752 - Attachment is obsolete: true
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/ac2e2c7315db
[taskgraph] Consistently include the cache digests of parent tasks in downstream cached tasks; r=dustin
Keywords: leave-open
https://hg.mozilla.org/mozilla-central/rev/ac2e2c7315db
Status: NEW → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 65
Assignee: nobody → mozilla
Target Milestone: Firefox 65 → mozilla65
Flags: needinfo?(gps)
You need to log in before you can comment on or make changes to this bug.