Rust panic stacks are broken

RESOLVED FIXED in Firefox 65

Status

()

defect
RESOLVED FIXED
7 months ago
3 months ago

People

(Reporter: bholley, Assigned: glandium)

Tracking

(Blocks 1 bug)

unspecified
mozilla65
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox65 fixed)

Details

Attachments

(9 attachments)

(Reporter)

Description

7 months ago
Posted file example stack
I'm not sure exactly how new this is, but I noticed it recently.

Currently when I get a panic in Rust code, the stack is truncated as in the attachment. This means I need to fire up a debugger to see the context of the problem.
(Reporter)

Comment 1

7 months ago
Posted file my .mozconfig
(Reporter)

Updated

7 months ago
Flags: needinfo?(mwoerister)

Comment 2

7 months ago
I haven't seen this before but I don't use macos much. I've opened an issue in the Rust bug tracker:
https://github.com/rust-lang/rust/issues/54839
Flags: needinfo?(mwoerister)
QA Contact: nfroyd
QA Contact: nfroyd
(Reporter)

Comment 3

7 months ago
Looks like other people can reproduce this as well. Michael, are you the right person to continue investigation here? If not, please flag Anthony to find an owner.
Flags: needinfo?(mwoerister)
(Reporter)

Updated

7 months ago
Blocks: rust-great

Comment 4

6 months ago
Hey, sorry for the late response. I was on PTO.

I don't have any real experience with backtrace generation and Alex' comment on GH seems to suggest that this is something specific to Firefox (or how FF uses Rust), so I might not be the best person to investigate. I'll ask Anthony to re-assign.
Flags: needinfo?(mwoerister) → needinfo?(ajones)

Comment 5

6 months ago
I talked to Anthony and he said that Mike Hommey might be a good fit for doing an initial investigation. Mike, would you mind taking a look at this?
Flags: needinfo?(ajones) → needinfo?(mh+mozilla)
(Reporter)

Comment 6

6 months ago
I'm seeing stacktrace brokenness beyond mac. See the busted mochitest-gpu jobs at [1].

On windows, I see:

22:17:05     INFO - GECKO(6548) | stack backtrace:
22:17:05     INFO - GECKO(6548) |    0:     0x7fff32a14bc2 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    1:     0x7fff3317662e - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    2:     0x7fff32a2bc98 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    3:     0x7fff32a2b951 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    4:     0x7fff32a2c621 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    5:     0x7fff33213a97 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    6:     0x7fff3328d183 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    7:     0x7fff3327b1a3 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    8:     0x7fff3327773c - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |    9:     0x7fff32a92a52 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |   10:     0x7fff33214538 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |   11:     0x7fff3317580b - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |   12:     0x7fff331ef36e - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |   13:     0x7fff32a2b131 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |   14:     0x7fff32ab3888 - XRE_GetBootstrap
22:17:05     INFO - GECKO(6548) |   15:     0x7fff63462773 - BaseThreadInitThunk
22:17:05     INFO - GECKO(6548) |   16:     0x7fff4bfc3e9a - DllBlocklist_Initialize
22:17:05     INFO - GECKO(6548) |   17:     0x7fff64120d50 - RtlUserThreadStart


On linux, I see:

[task 2018-10-26T21:30:31.627Z] 21:30:31     INFO - GECKO(2256) |    0:     0x7f753b34b2be - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h4e8f20df13e312d1
[task 2018-10-26T21:30:31.629Z] 21:30:31     INFO - GECKO(2256) |    1:     0x7f753b32b286 - std::sys_common::backtrace::print::ha948e198817dc736
[task 2018-10-26T21:30:31.630Z] 21:30:31     INFO - GECKO(2256) |    2:     0x7f753b32599d - std::panicking::default_hook::{{closure}}::hbce70b8b8e9560c0
[task 2018-10-26T21:30:31.632Z] 21:30:31     INFO - GECKO(2256) |    3:     0x7f753b325710 - std::panicking::default_hook::h445b4bc93ff2c15f
[task 2018-10-26T21:30:31.635Z] 21:30:31     INFO - GECKO(2256) |    4:     0x7f753b3260a3 - std::panicking::rust_panic_with_hook::h542dacac3f3a20d8
[task 2018-10-26T21:30:31.637Z] 21:30:31     INFO - GECKO(2256) |    5:     0x7f753acd7e66 - <unknown>
[task 2018-10-26T21:30:31.639Z] 21:30:31     INFO - GECKO(2256) | Redirecting call to abort() to mozalloc_abort


Which happens to match what I see locally.


[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0081382b64ef4535bc46054743df72cb19c4286c
(Reporter)

Comment 7

6 months ago
Updating the title, since the bustage here doesn't seem mac-specific.
Summary: Rust panic stacks on mac are truncated → Rust panic stacks are broken
(Assignee)

Comment 8

6 months ago
Those automation logs at least show that those crash reports *do* get proper stack traces via minidump stack walking, which is the most important thing on automation. As a matter of fact, I don't understand why we'd want rust to print stack traces on its own. That is inconsistent with C++, and the unwinding code in rust, as this bug shows, may not work appropriately.

If we *do* want those printed out (and I don't think we should ; or maybe only for debug builds), why should we not print stacks for C++ too? At which point it would mean using our own stack walker for those, which would likely work better for rust stack traces too.
Flags: needinfo?(mh+mozilla)
(Reporter)

Comment 9

6 months ago
I don't particularly care which code does the stack printing, but I do think it's important (and urgent) to have the stacks printed, which they presently are not in local builds.
(Assignee)

Comment 10

6 months ago
But why rust needs to be special here? Why aren't "normal" crashes enough? And why do all builds need this (isn't it actually an anti-feature on the builds we ship?) Without knowing what the goal is, I can't tell what, specifically, needs to be fixed.
(Assignee)

Comment 11

6 months ago
FWIW, you might be shooting yourself in the foot with this part of your mozconfig:
> export LDFLAGS=-Wl,-no_compact_unwind
(Reporter)

Comment 14

6 months ago
(In reply to Mike Hommey [:glandium] from comment #11)
> FWIW, you might be shooting yourself in the foot with this part of your
> mozconfig:
> > export LDFLAGS=-Wl,-no_compact_unwind

Thanks, I've gone ahead and removed that. Even so, I still don't get useful backtraces for Rust like I do for C++. See the comparison between the attachment in comment 12 and comment 13. They add fatal crashes in adjacent code on the Renderer thread (in the C++ that creates the WR instance, and in the subsequent Rust code that initializes it). For C++, I get a useful stack. For Rust, I do not. This was not always the case.
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 15

6 months ago
So... bug 1005730 added stacks for C++ in debug and asan builds. We should just use the same stack walker. (Although there would seem to be a legitimate problem with rust's unwinder itself)
Assignee: nobody → mh+mozilla
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 16

6 months ago
Status: after some fighting, I got rust to use the same crash/stack walking code as C++... and interestingly, MozStackWalk fails to go past gkrust_shared::install_rust_panic_hook::{{closure}}...
(Assignee)

Comment 17

6 months ago
I have yet to figure out exactly what was wrong but what I did confirm is that there is no problem with the current code with rust 1.30, while there is with 1.29. So... upgrade your rust. There are some nice things to win from switching the rust panic handler to using MozStackWalk, and I'll file one or several separate bugs with what I came up with.
(Assignee)

Comment 18

6 months ago
So, there appears to be multiple different problems on different platforms.

- On Linux and Mac, we're just hitting similar problems as bug 1453220, where this old comment I made on the corresponding rust issue still applied on 1.29: https://github.com/rust-lang/rust/issues/49867#issuecomment-380344472. Somehow, though, 1.30 fixed things, although it doesn't seem requires_uwtables was changed... Alex, do you know what's up? Experimentally, it seems to be tied to the level of debuginfo (as in, it seems that it's the use of -C debuginfo=2 that unlocked eh_frame with 1.30, but didn't in earlier versions).

- Interestingly, enabling LTO on < 1.30 _does_ emit the eh_frame data, so this wasn't actually a problem on opt builds on automation. And I got proper stack traces from debug builds with LTO enabled.

- On Windows, the rust unwinder just doesn't work properly. In fact, my first attempt with the C++ stack walking code mentioned in comment 16 *did* fix the stack traces on Windows.

- Things have been in this broken state forever, and I'm not sure why we've tolerated it. I think we need some kind of test that we can properly walk stacks through C++ and rust.

Anyways, as mentioned in comment 17, the good news is that the upgrade to 1.30 that landed two days ago fixed things on Linux and Mac.
Flags: needinfo?(acrichton)
We also have bug 1493630 which seems to show similar issues in some of our crash reports.
I think we had an LLVM update between 1.29 and 1.30, but beyond that I'm not sure what would have changed here. Perhaps a fixed llvm bug though?
Flags: needinfo?(acrichton)
(Reporter)

Comment 21

6 months ago
Confirmed that this works with rust 1.30. Thanks!
(Assignee)

Comment 22

6 months ago
One sad fact is that MozStackWalk apparently doesn't walk stacks properly on the rust side... which can actually be a problem for other things, so that will need to be fixed.
(Assignee)

Comment 23

6 months ago
Let me retract comment 22. Here is the actual status for rust panic backtraces vs. MozStackWalk:

- On linux:
  - On automation, rust prints proper back traces, but leaves C++ symbols mangled.
  - On automation, MozStackWalk prints traces with frames like ??? [libxul.so +0xoffset] which we postprocess to a proper symbol (with fix_linux_stack.py).
  - On local builds, things are unchanged for both.
- On Mac:
  - On automation, rust prints back traces that, in many cases, aren't symbolicated properly ; both on rust and C++ code.
  - On automation, MozStackWalk prints similar back traces that aren't symbolicated properly, but we postprocess them with fix_macosx_stack, which gives the right symbols.
  - On local builds, both produce symbolicated traces, even without postprocessing with fix_stack_using_bpsyms for MozStackWalk.
- On Windows:
  - On automation, rust prints back traces that aren't symbolicated properly ; both on rust and C++ code.
  - On automation, MozStackWalk prints the same, badly symbolicated back traces, but we postprocess them with fix_stack_using_bpsyms, which gives the right symbols.
  - On local builds, both produce symbolicated traces, even without postprocessing with fix_stack_using_bpsyms for MozStackWalk.

In all cases, MozStackWalk doesn't demangle /some/ rust symbols (those that have a mangling that has an extra layer of mangling on top of the itanium C++ ABI mangling (AIUI)).

Overall, I still think going with MozStackWalk everywhere is the right thing to do.
(Assignee)

Comment 24

5 months ago
Ideally, we'd want the function to stay in Assertions.cpp, but that's
only part of MFBT proper, and that doesn't have access to WalkTheStack
like MOZ_CRASH has from being in Assertion.h, when included from Gecko
code. Moving WalkTheStack to mozglue, putting it close together with
MozStackWalk would be prefered, but that causes problems linking MFBT
tests (which don't have access to mozglue), and other things.

Overall, this was too deep a rabbit hole, and moving MOZ_CrashOOL to
Assertions.h is much simpler. Since it's essentially the same as
MOZ_CRASH, except it allows non-literal strings, we can make it inlined,
and leave it to the compiler to drop the filename argument when it's not
used.
(Assignee)

Comment 25

5 months ago
Because we don't really need all this repetition.

Depends on D11718
(Assignee)

Comment 26

5 months ago
The current rust panic hook keeps a string for the crash reporter, and
goes on calling the default rust panic hook, which prints out a crash
stack...  when RUST_BOOTSTRAP is set *and* when that works. Notably, on
both mac and Windows, it only really works for local builds, but fails
for debug builds from automation, although on automation itself, we also
do stackwalk from crash minidumps, which alleviates the problem.
Artifact debug builds are affected, though.

More importantly, C++ calls to e.g. MOZ_CRASH have a similar but
different behavior, in that they dump a stack trace on debug builds, by
default (with exceptions, see below for one). The format of those stack
traces is understood by the various fix*stack*py scripts under
tools/rb/, that are used by the various test harnesses both on
automation and locally.

Additionally, the current rust panic hook, as it calls the default rust
panic hook, ends up calling abort() on non-Windows platforms, which ends
up being verbosely redirected to mozalloc_abort per
https://dxr.mozilla.org/mozilla-central/rev/237e4c0633fda8e227b2ab3ab57e417c980a2811/memory/mozalloc/mozalloc_abort.cpp#79
which then calls MOZ_CRASH. Theoretically, /that/ would also print a
stack trace, but doesn't because currently the stack trace printing code
lives in libxul, and MOZ_CRASH only calls it when compiled from
libxul-code, which mozalloc_abort is not part of.

With this change, we make the rust panic handler call back into
MOZ_CRASH directly. This has multiple advantages:
- This is more consistent cross-platforms (Windows is not special
anymore).
- This is more consistent between C++ and rust (stack traces all look
the same, and can all be post-processed by fix*stack*py if need be)
- This is more consistent in behavior, where debug builds will show
those stack traces without caring about environment variables.
- It demangles C++ symbols in rust-initiated stack traces (for some
reason that didn't happen with the rust panic handler)

A few downsides:
- the loss of demangling for some rust symbols.
- the loss of addresses in the stacks, although they're not entirely
useful
- extra empty lines.

The first should be fixable later one. The latter two are arguably
something that should be consistent across C++ and rust, and should be
changed if necessary, independently of this patch.

Depends on D11719
(Assignee)

Comment 27

5 months ago
Previously, our panic hook was only really useful when the crash
reporter is used, because all it did apart from calling rust's default
panic handler was to keep a pointer to the panic message for the crash
reporter.

Now that it just redirects to the Gecko crash code, it doesn't need to
be tied to the crash reporter. In fact, to ensure it's consistently used
in all cases, we ought to install it early on. Use a static initializer
for that.

Depends on D11720

Comment 28

5 months ago
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/bfb4ee856c71
Move MOZ_CrashOOL to Assertions.h. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/d0b6c1fc149d
Use MOZ_CrashOOL in MOZ_CrashPrintf. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/164a5a49fd25
Change the rust panic hook to delegate to Gecko's crash code. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/cfeee3d5ed6a
Install the rust panic hook early. r=froydnj
Backed out 4 changesets (bug 1496503) for xpcshell failures at toolkit/crashreporter/test/unit/test_crash_rust_panic.js

Backout: https://hg.mozilla.org/integration/autoland/rev/d8a262837cd3956ec6a6aa3082d0ff0e7352c79c

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=211607883&revision=cfeee3d5ed6a215072a17a037c7a707bd03b909e

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=211607883&repo=autoland&lineNumber=7646

22:19:27     INFO -  TEST-START | toolkit/crashreporter/test/unit/test_crash_rust_panic.js
22:19:27  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/crashreporter/test/unit/test_crash_rust_panic.js | xpcshell return code: 0
22:19:27     INFO -  TEST-INFO took 544ms
22:19:27     INFO -  >>>>>>>
22:19:27     INFO -  PID 6363 | [6363, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2749
22:19:27     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
22:19:27     INFO -  PID 6363 | [6364, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2749
22:19:27     INFO -  PID 6363 | Hit MOZ_CRASH(OH NO) at toolkit/library/rust/shared/lib.rs:155
22:20:43     INFO -  PID 6363 | #01: MOZ_CrashOOL(char const*, int, char const*) [mfbt/Assertions.h:306]
22:20:43     INFO -  PID 6363 | #02: __rust_alloc_zeroed
22:20:43     INFO -  PID 6363 | #03: __rust_alloc_zeroed
22:20:43     INFO -  PID 6363 | #04: std::panicking::rust_panic_with_hook [/rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libcore/sync/atomic.rs:2007]
22:20:43     INFO -  PID 6363 | #05: std::panicking::continue_panic_fmt [/rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libstd/panicking.rs:0]
22:20:43     INFO -  PID 6363 | #06: _$LT$std..ffi..c_str..CStr$u20$as$u20$core..cmp..PartialOrd$GT$::partial_cmp::hc2054f4cb5d78ab0
22:20:43     INFO -  PID 6363 | #07: __rust_alloc_zeroed
22:20:43     INFO -  PID 6363 | #08: nsDebugImpl::RustPanic(char const*) [xpcom/base/nsDebugImpl.cpp:157]
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 30

5 months ago
O_o the test passes on Linux but fails on Mac/Windows.
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 31

5 months ago
And now that I know why it fails on Mac/Windows, I don't know why it doesn't on Linux...

Comment 32

5 months ago
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/43e44f8439ec
Move MOZ_CrashOOL to Assertions.h. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/963d8ac1cfee
Use MOZ_CrashOOL in MOZ_CrashPrintf. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/a0f255b660ce
Change the rust panic hook to delegate to Gecko's crash code. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/033a89b3e00d
Install the rust panic hook early. r=froydnj
Backed out 4 changesets (bug 1496503) for Valgrind bustage. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=211697304&repo=autoland&lineNumber=45911

INFO -  14:59.37 ==37079== Warning: set address range perms: large range [0xc9efae74000, 0xc9f3ae74000) (noaccess)
[task 2018-11-14T15:46:05.732Z] 15:46:05     INFO -  15:03.88 --37079-- Archiving syms at 0x1b986180-0x1b9f7daa in /builds/worker/workspace/build/src/obj-firefox/security/nss/lib/freebl/freebl_freeblpriv3/libfreeblpriv3.so due to munmap()
[task 2018-11-14T15:46:05.732Z] 15:46:05     INFO -  15:03.88 --37079-- Scanning and archiving ExeContexts ...
[task 2018-11-14T15:46:06.232Z] 15:46:06     INFO -  15:04.38 --37079-- Scanned 2,986,880 ExeContexts, archived 3,245 ExeContexts
[task 2018-11-14T15:46:06.248Z] 15:46:06     INFO -  15:04.40 --37079-- Archiving syms at 0x1b945820-0x1b97002c in /builds/worker/workspace/build/src/obj-firefox/security/nss/lib/softoken/softoken_softokn3/libsoftokn3.so due to munmap()
[task 2018-11-14T15:46:06.248Z] 15:46:06     INFO -  15:04.40 --37079-- Scanning and archiving ExeContexts ...
[task 2018-11-14T15:46:06.748Z] 15:46:06     INFO -  15:04.90 --37079-- Scanned 2,986,920 ExeContexts, archived 6,998 ExeContexts
[task 2018-11-14T15:46:06.950Z] 15:46:06     INFO -  15:05.10 --37079-- Archiving syms at 0x1e6014d0-0x1e602168 in /usr/lib/x86_64-linux-gnu/gconv/MACINTOSH.so due to munmap()
[task 2018-11-14T15:46:06.950Z] 15:46:06     INFO -  15:05.10 --37079-- Scanning and archiving ExeContexts ...
[task 2018-11-14T15:46:07.438Z] 15:46:07     INFO -  15:05.59 --37079-- Scanned 2,989,333 ExeContexts, archived 26 ExeContexts
[task 2018-11-14T15:46:07.607Z] 15:46:07     INFO -  15:05.76 --37079-- Archiving syms at 0x16b14180-0x16b1b148 in /lib/x86_64-linux-gnu/libnss_files-2.13.so due to munmap()
[task 2018-11-14T15:46:07.607Z] 15:46:07     INFO -  15:05.76 --37079-- Scanning and archiving ExeContexts ...
[task 2018-11-14T15:46:08.112Z] 15:46:08     INFO -  15:06.27 --37079-- Scanned 2,989,434 ExeContexts, archived 120 ExeContexts
[task 2018-11-14T15:46:08.127Z] 15:46:08     INFO -  15:06.28 --37079-- Archiving syms at 0x1ac81000-0x1ac840b8 in /lib/x86_64-linux-gnu/libnss_dns-2.13.so due to munmap()
[task 2018-11-14T15:46:08.128Z] 15:46:08     INFO -  15:06.28 --37079-- Scanning and archiving ExeContexts ...
[task 2018-11-14T15:46:08.632Z] 15:46:08     INFO -  15:06.79 --37079-- Scanned 2,989,445 ExeContexts, archived 134 ExeContexts
[task 2018-11-14T15:46:08.648Z] 15:46:08     INFO -  15:06.80 ==37079==
[task 2018-11-14T15:46:08.648Z] 15:46:08     INFO -  15:06.80 ==37079== HEAP SUMMARY:
[task 2018-11-14T15:46:08.648Z] 15:46:08     INFO -  15:06.80 ==37079==     in use at exit: 1,879,698 bytes in 14,724 blocks
[task 2018-11-14T15:46:08.648Z] 15:46:08     INFO -  15:06.80 ==37079==   total heap usage: 2,233,547 allocs, 2,218,823 frees, 1,495,940,639 bytes allocated
[task 2018-11-14T15:46:08.649Z] 15:46:08     INFO -  15:06.80 ==37079==
[task 2018-11-14T15:46:08.652Z] 15:46:08     INFO -  15:06.81 ==37079== Searching for pointers to 14,008 not-freed blocks
[task 2018-11-14T15:46:08.693Z] 15:46:08     INFO -  15:06.85 ==37079== Checked 13,757,280 bytes
[task 2018-11-14T15:46:08.693Z] 15:46:08     INFO -  15:06.85 ==37079==
[task 2018-11-14T15:46:09.556Z] 15:46:09     INFO -  15:07.71 TEST-UNEXPECTED-FAIL | valgrind-test | 4,159 bytes in 1 blocks are definitely lost at malloc / FcDirScanConfig / ??? / FcDirCacheScan
[task 2018-11-14T15:46:09.556Z] 15:46:09     INFO -  15:07.71 ==37079== 4,159 bytes in 1 blocks are definitely lost in loss record 8,381 of 8,434
[task 2018-11-14T15:46:09.556Z] 15:46:09     INFO -  15:07.71 ==37079==    at 0x4C2B240: malloc+112 (vg_replace_malloc.c:298)
[task 2018-11-14T15:46:09.557Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x908C047: FcDirScanConfig (fcdir.c:158)
[task 2018-11-14T15:46:09.557Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x908C047: ??? (fcdir.c:133)
[task 2018-11-14T15:46:09.557Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x908C30F: FcDirCacheScan+127 (fcdir.c:259)
[task 2018-11-14T15:46:09.557Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x908876A: FcConfigAddDirList+74 (fccfg.c:355)
[task 2018-11-14T15:46:09.557Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x9088837: FcConfigAppFontAddDir+87 (fccfg.c:1975)
[task 2018-11-14T15:46:09.558Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD332D4: gfxFcPlatformFontList::ActivateBundledFonts()+52 (gfxFcPlatformFontList.cpp:2564)
[task 2018-11-14T15:46:09.558Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD32EFB: gfxFcPlatformFontList::InitFontListForPlatform()+27 (gfxFcPlatformFontList.cpp:1634)
[task 2018-11-14T15:46:09.558Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD9A719: gfxPlatformFontList::InitFontList()+457 (gfxPlatformFontList.cpp:363)
[task 2018-11-14T15:46:09.558Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD415F2: gfxPlatformGtk::CreatePlatformFontList()+34 (gfxPlatformGtk.cpp:260)
[task 2018-11-14T15:46:09.558Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD3ABFF: Init (gfxPlatformFontList.h:112)
[task 2018-11-14T15:46:09.559Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD3ABFF: gfxPlatform::Init()+2927 (gfxPlatform.cpp:914)
[task 2018-11-14T15:46:09.559Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xFD3A05F: gfxPlatform::GetPlatform()+31 (gfxPlatform.cpp:519)
[task 2018-11-14T15:46:09.559Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x112821D4: mozilla::widget::GfxInfoBase::GetContentBackend(nsTSubstring<char16_t>&)+20 (GfxInfoBase.cpp:1523)
[task 2018-11-14T15:46:09.559Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xF176121: ??? (xptcinvoke_asm_x86_64_unix.S:106)
[task 2018-11-14T15:46:09.559Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xF8CF52F: Invoke (XPCWrappedNative.cpp:1723)
[task 2018-11-14T15:46:09.560Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xF8CF52F: Call (XPCWrappedNative.cpp:1268)
[task 2018-11-14T15:46:09.560Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xF8CF52F: XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode)+3567 (XPCWrappedNative.cpp:1232)
[task 2018-11-14T15:46:09.560Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xF8D0ABF: GetAttribute (xpcprivate.h:1571)
[task 2018-11-14T15:46:09.560Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0xF8D0ABF: XPC_WN_GetterSetter(JSContext*, unsigned int, JS::Value*)+463 (XPCWrappedNativeJSOps.cpp:1060)
[task 2018-11-14T15:46:09.560Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C859FB: CallJSNative (Interpreter.cpp:468)
[task 2018-11-14T15:46:09.561Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C859FB: js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct)+891 (Interpreter.cpp:560)
[task 2018-11-14T15:46:09.561Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C86A43: InternalCall (Interpreter.cpp:614)
[task 2018-11-14T15:46:09.561Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C86A43: Call (Interpreter.cpp:633)
[task 2018-11-14T15:46:09.561Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C86A43: js::CallGetter(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::MutableHandle<JS::Value>)+211 (Interpreter.cpp:760)
[task 2018-11-14T15:46:09.561Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: CallGetter (NativeObject.cpp:2304)
[task 2018-11-14T15:46:09.562Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: GetExistingProperty<js::CanGC> (NativeObject.cpp:2359)
[task 2018-11-14T15:46:09.562Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: NativeGetPropertyInline<js::CanGC> (NativeObject.cpp:2615)
[task 2018-11-14T15:46:09.563Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: js::NativeGetProperty(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::Value>, JS::Handle<jsid>, JS::MutableHandle<JS::Value>)+2729 (NativeObject.cpp:2652)
[task 2018-11-14T15:46:09.563Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7E0FA: GetProperty (ObjectOperations-inl.h:122)
[task 2018-11-14T15:46:09.564Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7E0FA: GetObjectElementOperation (Interpreter-inl.h:558)
[task 2018-11-14T15:46:09.564Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7E0FA: GetElementOperation (Interpreter-inl.h:675)
[task 2018-11-14T15:46:09.564Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7E0FA: Interpret(JSContext*, js::RunState&)+34650 (Interpreter.cpp:3285)
[task 2018-11-14T15:46:09.564Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7575C: js::RunScript(JSContext*, js::RunState&)+300 (Interpreter.cpp:447)
[task 2018-11-14T15:46:09.564Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C85D48: js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct)+1736 (Interpreter.cpp:587)
[task 2018-11-14T15:46:09.565Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C86A43: InternalCall (Interpreter.cpp:614)
[task 2018-11-14T15:46:09.565Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C86A43: Call (Interpreter.cpp:633)
[task 2018-11-14T15:46:09.565Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C86A43: js::CallGetter(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::MutableHandle<JS::Value>)+211 (Interpreter.cpp:760)
[task 2018-11-14T15:46:09.565Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: CallGetter (NativeObject.cpp:2304)
[task 2018-11-14T15:46:09.565Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: GetExistingProperty<js::CanGC> (NativeObject.cpp:2359)
[task 2018-11-14T15:46:09.566Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: NativeGetPropertyInline<js::CanGC> (NativeObject.cpp:2615)
[task 2018-11-14T15:46:09.566Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x126867F9: js::NativeGetProperty(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::Value>, JS::Handle<jsid>, JS::MutableHandle<JS::Value>)+2729 (NativeObject.cpp:2652)
[task 2018-11-14T15:46:09.566Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C8906E: GetProperty (ObjectOperations-inl.h:122)
[task 2018-11-14T15:46:09.566Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C8906E: GetProperty (ObjectOperations-inl.h:130)
[task 2018-11-14T15:46:09.567Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C8906E: js::GetProperty(JSContext*, JS::Handle<JS::Value>, JS::Handle<js::PropertyName*>, JS::MutableHandle<JS::Value>)+430 (Interpreter.cpp:4916)
[task 2018-11-14T15:46:09.567Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7CA7A: GetPropertyOperation (Interpreter.cpp:223)
[task 2018-11-14T15:46:09.567Z] 15:46:09     INFO -  15:07.71 ==37079==    by 0x12C7CA7A: Interpret(JSContext*, js::RunState&)+28890 (Interpreter.cpp:3158)
[task 2018-11-14T15:46:09.567Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x12C7575C: js::RunScript(JSContext*, js::RunState&)+300 (Interpreter.cpp:447)
[task 2018-11-14T15:46:09.568Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x12C85D48: js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct)+1736 (Interpreter.cpp:587)
[task 2018-11-14T15:46:09.568Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x12C86228: js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>)+40 (Interpreter.cpp:633)
[task 2018-11-14T15:46:09.568Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x128284F5: JS_CallFunctionValue(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>)+1269 (jsapi.cpp:2911)
[task 2018-11-14T15:46:09.568Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0xF8C8CCD: nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, nsXPTMethodInfo const*, nsXPTCMiniVariant*)+3549 (XPCWrappedJSClass.cpp:1174)
[task 2018-11-14T15:46:09.568Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0xF176E4F: PrepareAndDispatch+511 (xptcstubs_x86_64_linux.cpp:127)
[task 2018-11-14T15:46:09.569Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0xF1762AE: SharedStub+90 (in /builds/worker/workspace/build/src/obj-firefox/toolkit/library/libxul.so)
[task 2018-11-14T15:46:09.569Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0xF14D65A: NS_CreateServicesFromCategory(char const*, nsISupports*, char const*, char16_t const*)+586 (nsCategoryManager.cpp:777)
[task 2018-11-14T15:46:09.569Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x1245A9FE: nsXREDirProvider::DoStartup()+366 (nsXREDirProvider.cpp:1103)
[task 2018-11-14T15:46:09.569Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x12450FCD: XREMain::XRE_mainRun()+2237 (nsAppRunner.cpp:4620)
[task 2018-11-14T15:46:09.570Z] 15:46:09     INFO -  15:07.72 ==37079==    by 0x12451C9F: XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&)+1151 (nsAppRunner.cpp:4935)
[task 2018-11-14T15:46:09.570Z] 15:46:09     INFO -  15:07.72 ==37079==
[task 2018-11-14T15:46:09.570Z] 15:46:09     INFO -  15:07.72 {
[task 2018-11-14T15:46:09.570Z] 15:46:09     INFO -  15:07.72    <insert_a_suppression_name_here>
[task 2018-11-14T15:46:09.571Z] 15:46:09     INFO -  15:07.72    Memcheck:Leak
[task 2018-11-14T15:46:09.571Z] 15:46:09     INFO -  15:07.72    match-leak-kinds: definite
[task 2018-11-14T15:46:09.571Z] 15:46:09     INFO -  15:07.72    fun:malloc
[task 2018-11-14T15:46:09.571Z] 15:46:09     INFO -  15:07.72    fun:FcDirScanConfig
[task 2018-11-14T15:46:09.572Z] 15:46:09     INFO -  15:07.72    obj:/usr/lib/x86_64-linux-gnu/libfontconfig.so.1.5.0
[task 2018-11-14T15:46:09.572Z] 15:46:09     INFO -  15:07.72    fun:FcDirCacheScan
[task 2018-11-14T15:46:09.572Z] 15:46:09     INFO -  15:07.72    fun:FcConfigAddDirList
[task 2018-11-14T15:46:09.572Z] 15:46:09     INFO -  15:07.72    fun:FcConfigAppFontAddDir
[task 2018-11-14T15:46:09.573Z] 15:46:09     INFO -  15:07.72    fun:_ZN21gfxFcPlatformFontList20ActivateBundledFontsEv
[task 2018-11-14T15:46:09.573Z] 15:46:09     INFO -  15:07.72    fun:_ZN21gfxFcPlatformFontList23InitFontListForPlatformEv
[task 2018-11-14T15:46:09.573Z] 15:46:09     INFO -  15:07.72    fun:_ZN19gfxPlatformFontList12InitFontListEv
[task 2018-11-14T15:46:09.574Z] 15:46:09     INFO -  15:07.72    fun:_ZN14gfxPlatformGtk22CreatePlatformFontListEv
[task 2018-11-14T15:46:09.574Z] 15:46:09     INFO -  15:07.72    fun:Init
[task 2018-11-14T15:46:09.574Z] 15:46:09     INFO -  15:07.72    fun:_ZN11gfxPlatform4InitEv
[task 2018-11-14T15:46:09.574Z] 15:46:09     INFO -  15:07.72    fun:_ZN11gfxPlatform11GetPlatformEv
[task 2018-11-14T15:46:09.575Z] 15:46:09     INFO -  15:07.72    fun:_ZN7mozilla6widget11GfxInfoBase17GetContentBackendER12nsTSubstringIDsE
[task 2018-11-14T15:46:09.575Z] 15:46:09     INFO -  15:07.72    obj:/builds/worker/workspace/build/src/obj-firefox/toolkit/library/libxul.so
[task 2018-11-14T15:46:09.575Z] 15:46:09     INFO -  15:07.72    fun:Invoke
[task 2018-11-14T15:46:09.575Z] 15:46:09     INFO -  15:07.72    fun:Call
[task 2018-11-14T15:46:09.576Z] 15:46:09     INFO -  15:07.72    fun:_ZN16XPCWrappedNative10CallMethodER14XPCCallContextNS_8CallModeE
[task 2018-11-14T15:46:09.576Z] 15:46:09     INFO -  15:07.72    fun:GetAttribute
[task 2018-11-14T15:46:09.576Z] 15:46:09     INFO -  15:07.72    fun:_Z19XPC_WN_GetterSetterP9JSContextjPN2JS5ValueE
[task 2018-11-14T15:46:09.576Z] 15:46:09     INFO -  15:07.72    fun:CallJSNative
[task 2018-11-14T15:46:09.577Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js23InternalCallOrConstructEP9JSContextRKN2JS8CallArgsENS_14MaybeConstructE
[task 2018-11-14T15:46:09.577Z] 15:46:09     INFO -  15:07.72    fun:InternalCall
[task 2018-11-14T15:46:09.577Z] 15:46:09     INFO -  15:07.72    fun:Call
[task 2018-11-14T15:46:09.577Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js10CallGetterEP9JSContextN2JS6HandleINS2_5ValueEEES5_NS2_13MutableHandleIS4_EE
[task 2018-11-14T15:46:09.578Z] 15:46:09     INFO -  15:07.72    fun:CallGetter
[task 2018-11-14T15:46:09.578Z] 15:46:09     INFO -  15:07.72    fun:GetExistingProperty<js::CanGC>
[task 2018-11-14T15:46:09.578Z] 15:46:09     INFO -  15:07.72    fun:NativeGetPropertyInline<js::CanGC>
[task 2018-11-14T15:46:09.578Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js17NativeGetPropertyEP9JSContextN2JS6HandleIPNS_12NativeObjectEEENS3_INS2_5ValueEEENS3_I4jsidEENS2_13MutableHandleIS7_EE
[task 2018-11-14T15:46:09.579Z] 15:46:09     INFO -  15:07.72    fun:GetProperty
[task 2018-11-14T15:46:09.579Z] 15:46:09     INFO -  15:07.72    fun:GetObjectElementOperation
[task 2018-11-14T15:46:09.579Z] 15:46:09     INFO -  15:07.72    fun:GetElementOperation
[task 2018-11-14T15:46:09.579Z] 15:46:09     INFO -  15:07.72    fun:_ZL9InterpretP9JSContextRN2js8RunStateE
[task 2018-11-14T15:46:09.580Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js9RunScriptEP9JSContextRNS_8RunStateE
[task 2018-11-14T15:46:09.580Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js23InternalCallOrConstructEP9JSContextRKN2JS8CallArgsENS_14MaybeConstructE
[task 2018-11-14T15:46:09.580Z] 15:46:09     INFO -  15:07.72    fun:InternalCall
[task 2018-11-14T15:46:09.580Z] 15:46:09     INFO -  15:07.72    fun:Call
[task 2018-11-14T15:46:09.581Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js10CallGetterEP9JSContextN2JS6HandleINS2_5ValueEEES5_NS2_13MutableHandleIS4_EE
[task 2018-11-14T15:46:09.581Z] 15:46:09     INFO -  15:07.72    fun:CallGetter
[task 2018-11-14T15:46:09.581Z] 15:46:09     INFO -  15:07.72    fun:GetExistingProperty<js::CanGC>
[task 2018-11-14T15:46:09.581Z] 15:46:09     INFO -  15:07.72    fun:NativeGetPropertyInline<js::CanGC>
[task 2018-11-14T15:46:09.582Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js17NativeGetPropertyEP9JSContextN2JS6HandleIPNS_12NativeObjectEEENS3_INS2_5ValueEEENS3_I4jsidEENS2_13MutableHandleIS7_EE
[task 2018-11-14T15:46:09.582Z] 15:46:09     INFO -  15:07.72    fun:GetProperty
[task 2018-11-14T15:46:09.582Z] 15:46:09     INFO -  15:07.72    fun:GetProperty
[task 2018-11-14T15:46:09.582Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js11GetPropertyEP9JSContextN2JS6HandleINS2_5ValueEEENS3_IPNS_12PropertyNameEEENS2_13MutableHandleIS4_EE
[task 2018-11-14T15:46:09.583Z] 15:46:09     INFO -  15:07.72    fun:GetPropertyOperation
[task 2018-11-14T15:46:09.583Z] 15:46:09     INFO -  15:07.72    fun:_ZL9InterpretP9JSContextRN2js8RunStateE
[task 2018-11-14T15:46:09.583Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js9RunScriptEP9JSContextRNS_8RunStateE
[task 2018-11-14T15:46:09.583Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js23InternalCallOrConstructEP9JSContextRKN2JS8CallArgsENS_14MaybeConstructE
[task 2018-11-14T15:46:09.584Z] 15:46:09     INFO -  15:07.72    fun:_ZN2js4CallEP9JSContextN2JS6HandleINS2_5ValueEEES5_RKNS_13AnyInvokeArgsENS2_13MutableHandleIS4_EE
[task 2018-11-14T15:46:09.584Z] 15:46:09     INFO -  15:07.72    fun:_Z20JS_CallFunctionValueP9JSContextN2JS6HandleIP8JSObjectEENS2_INS1_5ValueEEERKNS1_16HandleValueArrayENS1_13MutableHandleIS6_EE
[task 2018-11-14T15:46:09.584Z] 15:46:09     INFO -  15:07.72    fun:_ZN19nsXPCWrappedJSClass10CallMethodEP14nsXPCWrappedJStPK15nsXPTMethodInfoP17nsXPTCMiniVariant
[task 2018-11-14T15:46:09.584Z] 15:46:09     INFO -  15:07.72    fun:PrepareAndDispatch
[task 2018-11-14T15:46:09.585Z] 15:46:09     INFO -  15:07.72    fun:SharedStub
[task 2018-11-14T15:46:09.585Z] 15:46:09     INFO -  15:07.72    fun:_Z29NS_CreateServicesFromCategoryPKcP11nsISupportsS0_PKDs
[task 2018-11-14T15:46:09.585Z] 15:46:09     INFO -  15:07.72    fun:_ZN16nsXREDirProvider9DoStartupEv
[task 2018-11-14T15:46:09.585Z] 15:46:09     INFO -  15:07.72    fun:_ZN7XREMain11XRE_mainRunEv
[task 2018-11-14T15:46:09.585Z] 15:46:09     INFO -  15:07.72    fun:_ZN7XREMain8XRE_mainEiPPcRKN7mozilla15BootstrapConfigE
[task 2018-11-14T15:46:09.586Z] 15:46:09     INFO -  15:07.72 }
[task 2018-11-14T15:46:09.586Z] 15:46:09     INFO -  15:07.72 ==37079== LEAK SUMMARY:
[task 2018-11-14T15:46:09.586Z] 15:46:09     INFO -  15:07.72 ==37079==    definitely lost: 4,159 bytes in 1 blocks
[task 2018-11-14T15:46:09.586Z] 15:46:09     INFO -  15:07.72 ==37079==    indirectly lost: 11,392 bytes in 354 blocks
[task 2018-11-14T15:46:09.587Z] 15:46:09     INFO -  15:07.72 ==37079==      possibly lost: 24,962 bytes in 342 blocks
[task 2018-11-14T15:46:09.587Z] 15:46:09     INFO -  15:07.72 ==37079==    still reachable: 1,702,634 bytes in 12,701 blocks
[task 2018-11-14T15:46:09.587Z] 15:46:09     INFO -  15:07.72 ==37079==                       of which reachable via heuristic:
[task 2018-11-14T15:46:09.587Z] 15:46:09     INFO -  15:07.72 ==37079==                         length64           : 7,392 bytes in 111 blocks
[task 2018-11-14T15:46:09.587Z] 15:46:09     INFO -  15:07.72 ==37079==                         newarray           : 2,128 bytes in 53 blocks
[task 2018-11-14T15:46:09.588Z] 15:46:09     INFO -  15:07.72 ==37079==         suppressed: 31,583 bytes in 610 blocks
[task 2018-11-14T15:46:09.588Z] 15:46:09     INFO -  15:07.72 ==37079== Reachable blocks (those to which a pointer was found) are not shown.
[task 2018-11-14T15:46:09.588Z] 15:46:09     INFO -  15:07.72 ==37079== To see them, rerun with: --leak-check=full --show-leak-kinds=all
[task 2018-11-14T15:46:09.588Z] 15:46:09     INFO -  15:07.72 ==37079==
[task 2018-11-14T15:46:09.588Z] 15:46:09     INFO -  15:07.73 ==37079== ERROR SUMMARY: 334 errors from 334 contexts (suppressed: 89 from 63)
[task 2018-11-14T15:46:09.588Z] 15:46:09     INFO -  15:07.73 --37079--
[task 2018-11-14T15:46:09.589Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      2 Bug 794372 /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:121 suppressed: 12,224 bytes in 382 blocks
[task 2018-11-14T15:46:09.589Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      1 Bug 794374 /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:130 suppressed: 4,192 bytes in 131 blocks
[task 2018-11-14T15:46:09.589Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      6 Bug 1187649 /builds/worker/workspace/build/src/build/valgrind/x86_64-pc-linux-gnu.sup:25 suppressed: 15,456 bytes in 6 blocks
[task 2018-11-14T15:46:09.589Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:     32 Bug 794369 /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:103 suppressed: 8,752 bytes in 43 blocks
[task 2018-11-14T15:46:09.589Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      1 Bug 1187649 /builds/worker/workspace/build/src/build/valgrind/x86_64-pc-linux-gnu.sup:33 suppressed: 288 bytes in 1 blocks
[task 2018-11-14T15:46:09.589Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      2 See bug 793535 /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:74 suppressed: 224 bytes in 7 blocks
[task 2018-11-14T15:46:09.590Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:     19 _g_io_module_get_default /builds/worker/workspace/build/src/build/valgrind/x86_64-pc-linux-gnu.sup:186 suppressed: 1,244 bytes in 19 blocks
[task 2018-11-14T15:46:09.590Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      6 Bug 794370 /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:112 suppressed: 576 bytes in 12 blocks
[task 2018-11-14T15:46:09.590Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      1 PR_SetEnv requires its argument to be leaked, but does not appear on stacks. (See bug 793549.) /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:29 suppressed: 156 bytes in 1 blocks
[task 2018-11-14T15:46:09.590Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      8 PR_SetEnv requires its argument to be leaked, but does not appear on stacks. (See bug 793534.) /builds/worker/workspace/build/src/build/valgrind/cross-architecture.sup:15 suppressed: 145 bytes in 8 blocks
[task 2018-11-14T15:46:09.590Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      2 Bug 1496486: style::properties::longhands::font_language_override::cascade_property /builds/worker/workspace/build/src/build/valgrind/x86_64-pc-linux-gnu.sup:538
[task 2018-11-14T15:46:09.591Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      5 X on SUSE11 writev uninit padding /usr/lib/valgrind/default.supp:397
[task 2018-11-14T15:46:09.591Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      1 Bug 1418083 Servo::TraverseSubtree, January 2018 /builds/worker/workspace/build/src/build/valgrind/x86_64-pc-linux-gnu.sup:300
[task 2018-11-14T15:46:09.591Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:      1 Bug 1248365: libfontconfig-1 /builds/worker/workspace/build/src/build/valgrind/x86_64-pc-linux-gnu.sup:827
[task 2018-11-14T15:46:09.591Z] 15:46:09     INFO -  15:07.73 --37079-- used_suppression:     27 dl-hack3-cond-1 /usr/lib/valgrind/default.supp:1248
[task 2018-11-14T15:46:09.591Z] 15:46:09     INFO -  15:07.73 ==37079==
[task 2018-11-14T15:46:09.591Z] 15:46:09     INFO -  15:07.73 ==37079== ERROR SUMMARY: 334 errors from 334 contexts (suppressed: 89 from 63)
[task 2018-11-14T15:46:10.037Z] 15:46:10    ERROR - Return code: 1
[task 2018-11-14T15:46:10.037Z] 15:46:10  WARNING - setting return code to 2
[task 2018-11-14T15:46:10.038Z] 15:46:10    FATAL - 'mach valgrind-test' did not run successfully. Please check log for errors.
[task 2018-11-14T15:46:10.038Z] 15:46:10    FATAL - Running post_fatal callback...
[task 2018-11-14T15:46:10.038Z] 15:46:10    FATAL - Exiting -1
[task 2018-11-14T15:46:10.038Z] 15:46:10     INFO - [mozharness: 2018-11-14 15:46:10.038462Z] Finished valgrind-test step (failed)
[task 2018-11-14T15:46:10.038Z] 15:46:10     INFO - Running post-run listener: _shutdown_sccache
[task 2018-11-14T15:46:10.038Z] 15:46:10     INFO - Running command: ['/builds/worker/workspace/build/src/sccache2/sccache', '--stop-server'] in /builds/worker/workspace/build/src
[task 2018-11-14T15:46:10.039Z] 15:46:10     INFO - Copy/paste: /builds/worker/workspace/build/src/sccache2/sccache --stop-server
[task 2018-11-14T15:46:10.043Z] 15:46:10     INFO -  Stopping sccache server...
[task 2018-11-14T15:46:10.043Z] 15:46:10     INFO -  Compile requests               4163
[task 2018-11-14T15:46:10.043Z] 15:46:10     INFO -  Compile requests executed      3879
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Cache hits                     3783
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Cache misses                     96
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Cache timeouts                    0
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Cache read errors                 0
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Forced recaches                   0
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Cache write errors                0
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Compilation failures              0
[task 2018-11-14T15:46:10.044Z] 15:46:10     INFO -  Cache errors                      0
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Non-cacheable compilations        0
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Non-cacheable calls             100
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Non-compilation calls           184
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Unsupported compiler calls        0
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Average cache write           0.137 s
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Average cache read miss      22.837 s
[task 2018-11-14T15:46:10.045Z] 15:46:10     INFO -  Average cache read hit        0.100 s
[task 2018-11-14T15:46:10.046Z] 15:46:10     INFO -  Cache location             S3, bucket: Bucket(name=taskcluster-level-3-sccache-us-east-1, base_url=http://taskcluster-level-3-sccache-us-east-1.s3.amazonaws.com/)
[task 2018-11-14T15:46:10.046Z] 15:46:10     INFO - Return code: 0
[task 2018-11-14T15:46:10.046Z] 15:46:10     INFO - Running post-run listener: _summarize
[task 2018-11-14T15:46:10.046Z] 15:46:10    ERROR - # TBPL FAILURE #
[task 2018-11-14T15:46:10.046Z] 15:46:10     INFO - [mozharness: 2018-11-14 15:46:10.046516Z] FxDesktopBuild summary:
[task 2018-11-14T15:46:10.046Z] 15:46:10    ERROR - # TBPL FAILURE #

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=033a89b3e00d8210043a4696ab7ad66c2bd88fe3

Backout:
https://hg.mozilla.org/integration/autoland/rev/38516fcc409fdc8f82ae10fa96fe474e7b4ac251
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 34

5 months ago
There is *no* reason for the changes from this bug to trigger that "error", which, in itself is interesting:
- it looks very much like existing suppressions, except for one frame that valgrind didn't symbolicate.
- it's intermittent, which in itself makes no sense. This leak *always* happens. The code path that leads to it should always be the same, and the symbolication should also always be the same.

I'll note that the changeset right before the backout rebuilt the valgrind docker image, which may have affected how things go. Let's reland, and see how it goes. If the leak happens again, then I'll add a _third_ suppression variant for this leak.
Flags: needinfo?(mh+mozilla)

Comment 35

5 months ago
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/c61a46d741f9
Move MOZ_CrashOOL to Assertions.h. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/2c9ec3b001ff
Use MOZ_CrashOOL in MOZ_CrashPrintf. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/35f916cb6452
Change the rust panic hook to delegate to Gecko's crash code. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/fca9442066f5
Install the rust panic hook early. r=froydnj
Depends on: 1507049
You need to log in before you can comment on or make changes to this bug.