1377561 - LSAN error found in packager when building with CLang 4.0.1 on TC

Status: NEW

(Firefox Build System :: General, defect)

Description

[Raymond Forbes[:rforbes]]

We are in the process of moving our asan and fuzzing taskcluster builds to clang 4.0. I am doing this by replacing the 3.9 entry in the tooltool manifest to this:

  {
    "version": "clang 4.0.1 gecko build for linux",
    "size": 215309284,
    "visibility": "public",
    "digest": "8f6d386ca1d4606526dd24f366b1dbc1914c6c6d7f54c69c2a2ca0e7cfabe641c1168952d606295feffa9f38ad687084de5efb1e80be3ed2f431ac91de80039b",
    "algorithm": "sha512",
    "filename": "clang.tar.xz",
    "unpack": true
  },


When I push to try I get the following error.

[task 2017-06-30T18:49:31.171441Z] 18:49:31     INFO -  ==24208==LeakSanitizer has encountered a fatal error.
[task 2017-06-30T18:49:31.171641Z] 18:49:31     INFO -  ==24208==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
[task 2017-06-30T18:49:31.171830Z] 18:49:31     INFO -  ==24208==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
[task 2017-06-30T18:49:31.172057Z] 18:49:31     INFO -  Traceback (most recent call last):
[task 2017-06-30T18:49:31.172278Z] 18:49:31     INFO -    File "/home/worker/workspace/build/src/toolkit/mozapps/installer/packager.py", line 328, in <module>
[task 2017-06-30T18:49:31.172458Z] 18:49:31     INFO -      main()
[task 2017-06-30T18:49:31.172626Z] 18:49:31     INFO -    File "/home/worker/workspace/build/src/toolkit/mozapps/installer/packager.py", line 322, in main
[task 2017-06-30T18:49:31.172753Z] 18:49:31     INFO -      copier.copy(args.destination)
[task 2017-06-30T18:49:31.172921Z] 18:49:31     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozpack/copier.py", line 399, in copy
[task 2017-06-30T18:49:31.173059Z] 18:49:31     INFO -      copy_results.append((destfile, f.copy(destfile, skip_if_older)))
[task 2017-06-30T18:49:31.173215Z] 18:49:31     INFO -    File "/home/worker/workspace/build/src/toolkit/mozapps/installer/packager.py", line 126, in copy
[task 2017-06-30T18:49:31.173345Z] 18:49:31     INFO -      errors.fatal('Error while signing %s' % self.path)
[task 2017-06-30T18:49:31.173494Z] 18:49:31     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozpack/errors.py", line 103, in fatal
[task 2017-06-30T18:49:31.173627Z] 18:49:31     INFO -      self._handle(self.FATAL, msg)
[task 2017-06-30T18:49:31.173780Z] 18:49:31     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozpack/errors.py", line 98, in _handle
[task 2017-06-30T18:49:31.173912Z] 18:49:31     INFO -      raise ErrorMessage(msg)
[task 2017-06-30T18:49:31.174062Z] 18:49:31     INFO -  mozpack.errors.ErrorMessage: Error: Error while signing ../../dist/firefox/libsoftokn3.so
[task 2017-06-30T18:49:31.174205Z] 18:49:31     INFO -  /home/worker/workspace/build/src/toolkit/mozapps/installer/packager.mk:39: recipe for target 'stage-package' failed

We are not able to reproduce this locally.

Comment 1

[Raymond Forbes[:rforbes]]

I did try disabling LSAN and I was able to complete the build. One thing to consider as worst case scenario is to disable LSAN during just the packaging phase.

Comment 2

[Christian Holler (:decoder)]

Attachment: Local Failure Log

I was able to reproduce this locally, but only using the clang package mentioned in comment 0. With either clang-4.0 or clang-5.0 from the Ubuntu LLVM repositories it did not reproduce for me. The error happens in the packaging step during a call to shlibsign, the log is attached. It is interesting that there is no actual LSan trace or anything, just a fatal error and the process aborting.