Closed Bug 1410148 Opened 2 years ago Closed 2 years ago

Intermittent subprocess.CalledProcessError: Command '[u'/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil', '--arch=x86_64', '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL']' returned non-zero exit status -11

Categories

(Firefox Build System :: General, defect, blocker)

defect
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aryx, Assigned: glandium)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell fixed:other])

Attachments

(4 files)

This seems to be a pretty frequent (~20% of runs) failure which started a few hours ago for OS X Cross Compiled opt.

Ted, can you take a look at the issue, please?

First failure on integration branches: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=8e0c08885611569371c51a2e6ad128fce3879b32&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=138201168&repo=autoland

[task 2017-10-19T14:29:02.423Z] 14:29:02     INFO -  gmake[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library'
[task 2017-10-19T14:29:02.424Z] 14:29:02     INFO -  /builds/worker/workspace/build/src/obj-firefox/_virtualenv/bin/python -m mozbuild.action.dumpsymbols /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL_syms.track
[task 2017-10-19T14:29:02.424Z] 14:29:02     INFO -  Starting Mac pre-processing on file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2017-10-19T14:29:02.424Z] 14:29:02     INFO -  Running Mac pre-processing on file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2017-10-19T14:29:02.425Z] 14:29:02     INFO -  /builds/worker/workspace/build/src/clang/bin/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2017-10-19T14:29:02.425Z] 14:29:02     INFO -  while processing /builds/worker/workspace/build/src/obj-firefox/toolkit/library/x86_64-apple-darwin/release/libgkrust.a(gkrust-dfe50c863337d8a0.0.o):
[task 2017-10-19T14:29:02.425Z] 14:29:02     INFO -  warning: could not find referenced DIE
^^ tons of these

[task 2017-10-19T14:29:03.343Z] 14:29:03     INFO -  while processing /builds/worker/workspace/build/src/obj-firefox/toolkit/library/x86_64-apple-darwin/release/libgkrust.a(gkrust-dfe50c863337d8a0.0.o):
[task 2017-10-19T14:29:03.343Z] 14:29:03     INFO -  warning: could not find referenced DIE
[task 2017-10-19T14:29:03.343Z] 14:29:03     INFO -  #0 0x00000000007a8715 llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil+0x7a8715)
[task 2017-10-19T14:29:03.343Z] 14:29:03     INFO -  #1 0x00000000007a8d66 SignalHandler(int) (/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil+0x7a8d66)
[task 2017-10-19T14:29:03.343Z] 14:29:03     INFO -  #2 0x00007fdd53a9e7e0 __restore_rt (/lib64/libpthread.so.0+0xf7e0)
[task 2017-10-19T14:29:03.344Z] 14:29:03     INFO -  #3 0x00000000006d68a7 llvm::DWARFDebugInfoEntryMinimal::getAttributeValue(llvm::DWARFUnit const*, unsigned short, llvm::DWARFFormValue&) const (/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil+0x6d68a7)
[task 2017-10-19T14:29:03.344Z] 14:29:03     INFO -  #4 0x00000000006d635e llvm::DWARFDebugInfoEntryMinimal::getName(llvm::DWARFUnit const*, llvm::DINameKind) const (/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil+0x6d635e)
^^ tons of these

[task 2017-10-19T14:29:03.396Z] 14:29:03     INFO -  Stack dump:
[task 2017-10-19T14:29:03.396Z] 14:29:03     INFO -  0.	Program arguments: /builds/worker/workspace/build/src/clang/bin/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2017-10-19T14:29:03.397Z] 14:29:03     INFO -  Error running dsymutil: Command '[u'/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil', '--arch=x86_64', '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL']' returned non-zero exit status -11
[task 2017-10-19T14:29:03.397Z] 14:29:03     INFO -  Traceback (most recent call last):
[task 2017-10-19T14:29:03.397Z] 14:29:03     INFO -    File "/builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py", line 888, in <module>
[task 2017-10-19T14:29:03.397Z] 14:29:03     INFO -      main()
[task 2017-10-19T14:29:03.397Z] 14:29:03     INFO -    File "/builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py", line 884, in main
[task 2017-10-19T14:29:03.397Z] 14:29:03     INFO -      dumper.Process(args[2])
[task 2017-10-19T14:29:03.398Z] 14:29:03     INFO -    File "/builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py", line 501, in Process
[task 2017-10-19T14:29:03.398Z] 14:29:03     INFO -      self.ProcessFile(file_to_process)
[task 2017-10-19T14:29:03.398Z] 14:29:03     INFO -    File "/builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py", line 758, in ProcessFile
[task 2017-10-19T14:29:03.398Z] 14:29:03     INFO -      dsymbundle = self.GenerateDSYM(file)
[task 2017-10-19T14:29:03.398Z] 14:29:03     INFO -    File "/builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py", line 792, in GenerateDSYM
[task 2017-10-19T14:29:03.399Z] 14:29:03     INFO -      subprocess.check_call(cmd, stdout=open(os.devnull, 'w'))
[task 2017-10-19T14:29:03.399Z] 14:29:03     INFO -    File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
[task 2017-10-19T14:29:03.399Z] 14:29:03     INFO -      raise CalledProcessError(retcode, cmd)
[task 2017-10-19T14:29:03.399Z] 14:29:03     INFO -  subprocess.CalledProcessError: Command '[u'/builds/worker/workspace/build/src/clang/bin/llvm-dsymutil', '--arch=x86_64', '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL']' returned non-zero exit status -11
[task 2017-10-19T14:29:03.400Z] 14:29:03     INFO -  Running: /builds/worker/workspace/build/src/obj-firefox/_virtualenv/bin/python /builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py -c -a x86_64 --vcs-info --install-manifest=/builds/worker/workspace/build/src/obj-firefox/_build_manifests/install/dist_include,/builds/worker/workspace/build/src/obj-firefox/dist/include -s /builds/worker/workspace/build/src /builds/worker/workspace/build/src/obj-firefox/dist/host/bin/dump_syms /builds/worker/workspace/build/src/obj-firefox/dist/crashreporter-symbols /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
Flags: needinfo?(ted)
This sounds like a reoccurrence of bug 1397382, which we thought we fixed in bug 1381043. The changeset linked in comment 0 is a servo merge, so it's entirely possible that some new Rust code has caused another kind of failure.

We can try backing out https://hg.mozilla.org/mozilla-central/rev/95555b11aaaf, which should wallpaper over this again until we get a new fix.
Flags: needinfo?(ted)
Note it's a different stack trace than bug 1397382.
Yeah, I should have clarified, it's probably a different dsymutil crash that's just also caused by Rust debug info.
broken rust debug info.
Attachment #8928232 - Flags: review?(core-build-config-reviews)
Comment on attachment 8928232 [details]
bug 1410148 - try to work around llvm-dsymutil crashes again by lowering rust debuginfo level on cross-mac builds.

https://reviewboard.mozilla.org/r/199454/#review204560

lgtm.

::: build/moz.configure/toolchain.configure:1366
(Diff revision 1)
>      if opt_level == '0' and not debug_rust:
>          debug_assertions = False
>  
>      if debug_symbols:
> +        if host.kernel == 'Linux' and target.kernel == 'Darwin':
> +            # hack to work around dsymutil failing on cross-OSX builds (bug 1410148)

nit: generally we prefer full sentences.
Attachment #8928232 - Flags: review+
Attachment #8928232 - Flags: review?(core-build-config-reviews)
Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e963baf002da
try to work around llvm-dsymutil crashes again by lowering rust debuginfo level on cross-mac builds. r=nalexander
I'm going to tag this leave-open, even if this patch makes it go away we should figure out why llvm-dsymutil is crashing (again). I suspect we'd have to replicate glandium's work in bug 1381043. Given how many times we've seen this, maybe we should write down some steps for getting a smaller reproducible testcase out of a try build?
Keywords: leave-open
This started again yesterday and is permafailing for the nightly (3 runs failed, furthermore 3 normal builds on that push and 2 dark matter detector ones so far).

I am sorry to bother you on the first day of PTO. Please take a look at this.
Flags: needinfo?(ted)
Severity: major → blocker
This is permafailing on central (not on autoland or inbound), so no OS X nightlies.
I can't push to Try right now, but this patch should unblock us.
Attachment #8938762 - Flags: review?(aryx.bugmail)
Pushed by archaeopteryx@coole-files.de:
https://hg.mozilla.org/mozilla-central/rev/1a76c0899bc6
Attempt to fix OS X cross-compile builds on central by changing rust debug log level to 0. r=Aryx a=bustage-fix-attempt
This fixed the OS X build bustage on central.
Comment on attachment 8938991 [details]
Bug 1410148 - Backport llvm r289565 to clang 3.9.

https://reviewboard.mozilla.org/r/209446/#review215198

This is mostly a rubber stamp, since I'm not intimately familiar with Clang's internals.
Attachment #8938991 - Flags: review+
Attachment #8938991 - Flags: review?(core-build-config-reviews)
Attachment #8938992 - Flags: review?(core-build-config-reviews)
Comment on attachment 8938992 [details]
Bug 1410148 - Fully re-enable debug info on mac builds.

https://reviewboard.mozilla.org/r/209448/#review215200
Attachment #8938992 - Flags: review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/593515920c7e
Backport llvm r289565 to clang 3.9. r=gps
https://hg.mozilla.org/integration/autoland/rev/f77f58a060ad
Fully re-enable debug info on mac builds. r=gps
Flags: needinfo?(ted)
Flags: needinfo?(gps)
Attachment #8938991 - Flags: review?(core-build-config-reviews)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/8696ea000827
Backport llvm r289565 to clang 3.9. r=gps
https://hg.mozilla.org/integration/autoland/rev/5fa379ef34e9
Fully re-enable debug info on mac builds. r=gps
:glandium Looks like this brought a small improvement:

== Change summary for alert #11055 (as of Fri, 29 Dec 2017 21:24:07 GMT) ==

Improvements:

  5%  Strings PerfIsUTF8Example3 osx-10-10 opt      8,336.42 -> 7,914.67

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=11055
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #33)
> :glandium Looks like this brought a small improvement:
> 
> == Change summary for alert #11055 (as of Fri, 29 Dec 2017 21:24:07 GMT) ==
> 
> Improvements:
> 
>   5%  Strings PerfIsUTF8Example3 osx-10-10 opt      8,336.42 -> 7,914.67
> 
> For up to date results, see:
> https://treeherder.mozilla.org/perf.html#/alerts?id=11055

O_o sounds like a misattribution.
Assignee: nobody → mh+mozilla
Status: NEW → RESOLVED
Closed: 2 years ago
Keywords: leave-open
Resolution: --- → FIXED
Whiteboard: [stockwell fixed:other]
glandium: thanks again for dealing with another one of these difficult-to-handle llvm-dsymutil crashes. I filed bug 1429056 to try to figure out a better plan for dealing with this sort of situation in the future.
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.