Closed Bug 1545095 Opened 5 years ago Closed 5 years ago

Intermittent mozpack.errors.ErrorMessage: Error: Error while signing ../../dist/firefox/libsoftokn3.so

Categories

(Firefox Build System :: Task Configuration, defect)

Product:
Firefox Build System
Component:
Task Configuration
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox70 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla70
Tracking Flags:
Tracking Status
firefox70 --- fixed

People

(Reporter: NarcisB, Assigned: wcosta)

Details

Attachments

(1 file)

Bug 1545095: Disable LeakSanitizer runtime in asan builds r=glandium
47 bytes, text/x-phabricator-request
Details | Review

The issue was caused by https://github.com/taskcluster/docker-worker/releases/tag/v201904171218

An example of push from autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&selectedJob=240943218&revision=aa8babb8b1ee73c3713c7a8b22fe9315cb302cb6

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=240934515&repo=autoland&lineNumber=36974

Log snippet:
[task 2019-04-17T12:32:29.155Z] 12:32:29 INFO - package> ==31912==LeakSanitizer has encountered a fatal error.
[task 2019-04-17T12:32:29.155Z] 12:32:29 INFO - package> ==31912==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
[task 2019-04-17T12:32:29.155Z] 12:32:29 INFO - package> ==31912==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
[task 2019-04-17T12:32:29.155Z] 12:32:29 ERROR - package> Traceback (most recent call last):
[task 2019-04-17T12:32:29.155Z] 12:32:29 INFO - package> File "/builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.py", line 347, in <module>
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> main()
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> File "/builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.py", line 341, in main
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> copier.copy(args.destination)
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/copier.py", line 431, in copy
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> copy_results.append((destfile, f.copy(destfile, skip_if_older)))
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> File "/builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.py", line 126, in copy
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> errors.fatal('Error while signing %s' % self.path)
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/errors.py", line 103, in fatal
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> self._handle(self.FATAL, msg)
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/errors.py", line 98, in _handle
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> raise ErrorMessage(msg)
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> mozpack.errors.ErrorMessage: Error: Error while signing ../../dist/firefox/libsoftokn3.so
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> /builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.mk:23: recipe for target 'stage-package' failed
[task 2019-04-17T12:32:29.156Z] 12:32:29 ERROR - package> make[5]: *** [stage-package] Error 1
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> make[5]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/browser/installer'
[task 2019-04-17T12:32:29.156Z] 12:32:29 INFO - package> /builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.mk:97: recipe for target 'make-package' failed
[task 2019-04-17T12:32:29.156Z] 12:32:29 ERROR - package> make[4]: *** [make-package] Error 2
[task 2019-04-17T12:32:29.157Z] 12:32:29 INFO - package> /builds/worker/workspace/build/src/config/rules.mk:400: recipe for target 'default' failed
[task 2019-04-17T12:32:29.157Z] 12:32:29 ERROR - package> make[3]: *** [default] Error 2
[task 2019-04-17T12:32:29.157Z] 12:32:29 INFO - package> /builds/worker/workspace/build/src/browser/build.mk:6: recipe for target 'package' failed
[task 2019-04-17T12:32:29.157Z] 12:32:29 ERROR - package> make[2]: *** [package] Error 2
[task 2019-04-17T12:32:29.157Z] 12:32:29 INFO - /builds/worker/workspace/build/src/build/moz-automation.mk:84: recipe for target 'automation/package' failed
[task 2019-04-17T12:32:29.157Z] 12:32:29 ERROR - make[1]: *** [automation/package] Error 2
[task 2019-04-17T12:32:29.157Z] 12:32:29 INFO - make[1]: *** Waiting for unfinished jobs....

According to [1], LeakSanitizer must run in a privileged container. Indeed, running it in a privileged task works [2]. It seems that we have been lucky so far, but with the docker upgrade, the bug emerged.

[1] https://github.com/google/sanitizers/issues/764
[2] https://tools.taskcluster.net/groups/KVtUXTGRQTKG8VegpaXhJw/tasks/dEp33P_kS4mzWS0L7LaL9w/details

Hi Wander, can you find a more appropriate bug component for this? Thanks

Flags: needinfo?(wcosta)
Component: General → Task Configuration
Flags: needinfo?(wcosta)

These builds are currently running on level 3 workers, which we can't enable privileged mode on. Does running make check under asan add value or could we disable asan in the build task? Another option would be to move make check to a separate task that could be run on a tester instead.

(In reply to Tom Prince [:tomprince] from comment #4)

These builds are currently running on level 3 workers, which we can't enable
privileged mode on. Does running make check under asan add value or could
we disable asan in the build task? Another option would be to move make check to a separate task that could be run on a tester instead.

tester worker-types cannot run privileged tasks either (at least not with current configuration). We may create a special builder worker-type for this. But it iiuc, it cannot run try jobs.

Flags: needinfo?(mozilla)

I wonder if allow-ptrace is enough to enable these jobs?

Flags: needinfo?(mozilla)

(In reply to Tom Prince [:tomprince] from comment #6)

I wonder if allow-ptrace is enough to enable these jobs?

I tested with allowPtrace feature, but it didn't work. Since privileged containers are a safe route to security vulnerabilities, my propose is to run this task in generic-worker. What do you think, Tom?

Flags: needinfo?(mozilla)

:chamanchester Is there some way to make shlibsign (and I guess anything else that is part of the build) not run under ASAN?

Flags: needinfo?(cmanchester)

(In reply to Tom Prince [:tomprince] from comment #8)

:chamanchester Is there some way to make shlibsign (and I guess anything else that is part of the build) not run under ASAN?

I expect there would be... decoder do you know how to achieve this?

Flags: needinfo?(cmanchester) → needinfo?(choller)
Flags: needinfo?(mozilla)

(In reply to Chris Manchester (Offline until July 8th) from comment #9)

(In reply to Tom Prince [:tomprince] from comment #8)

:chamanchester Is there some way to make shlibsign (and I guess anything else that is part of the build) not run under ASAN?

I expect there would be... decoder do you know how to achieve this?

Host tools like shlibsign used to be not built with ASan because it was not enabled for the host compiler. I assume this got changed at some point (maybe due to Rust).

Anyways, I think it might be easier to look into the error here rather than disabling ASan. I don't know why LSan would complain this way (especially not intermittently), but we could disable that part by passing ASAN_OPTIONS="detect_leaks=0" in the environment in ToolLauncher to avoid this happening.

Flags: needinfo?(choller)

In newer docker versions the LeakSanitizer will fail unless it runs in a
privileged container. During the gecko build process in asan builds, the
support tools built as part of the build process will also run with
LeakSanitizer by default, but there is no need for such.

We then disable runtime leak sanitizer in asan builds to avoid erros in
builds inside a docker container.

Assignee: nobody → wcosta
Status: NEW → ASSIGNED
Pushed by wcosta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/471550ac81dc
Disable LeakSanitizer runtime in asan builds r=glandium

https://hg.mozilla.org/mozilla-central/rev/471550ac81dc

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox70: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: