1311173 - MOZ_ASSERT not always fully resolving symbols in crash dump

Status: NEW

(Testing :: web-platform-tests, defect)

Description

[Jean-Yves Avenard [:jya]]

This can be seen here:
https://treeherder.mozilla.org/logviewer.html#?job_id=37849497&repo=mozilla-inbound

While I lodge it under testing on recommendations on #developers, I've seen the same on a local central build running on my Linux box.

When an MOZ_ASSERT was hit, all symbols showed their location as lixbul.so

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This was on a revision from after bug 1238043 landed :(

Comment 2

[Andrew McCreight [:mccr8]]

It is symbolicating something, but not everything. Near the assertion, there is:

Assertion failure: aStart <= aEnd, at /home/worker/workspace/build/src/obj-firefox/dist/include/Intervals.h:56
#01: mozilla::TrackBuffersManager::Buffered() const (/home/worker/workspace/build/application/firefox/libxul.so)

...where the libxul.so makes it look like some kind of symbolication script processed it, but it just ate the detailed info.

Then down below in the larger stack that has all the register information there is:
1  libxul.so!mozilla::TrackBuffersManager::Buffered [TrackBuffersManager.cpp:c7aeff40316c : 333 + 0x32]

Comment 3

[James Graham [:jgraham]]

One day I will understand how that stuff is intended to work. In the meantime I need ted to teach me. Again.

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Andrew McCreight [:mccr8] from comment #2)
> It is symbolicating something, but not everything. Near the assertion, there
> is:
> 
> Assertion failure: aStart <= aEnd, at
> /home/worker/workspace/build/src/obj-firefox/dist/include/Intervals.h:56
> #01: mozilla::TrackBuffersManager::Buffered() const
> (/home/worker/workspace/build/application/firefox/libxul.so)
> 
> ...where the libxul.so makes it look like some kind of symbolication script
> processed it, but it just ate the detailed info.

So, I noticed in the middle of the assertion stack:
 [task 2016-10-18T16:37:25.511129Z] 16:37:25 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/NjtJJm9oR4CxU6pxB4Gq7g/artifacts/public/build/target.crashreporter-symbols.zip 

...which is weird because earlier in the build log there is:
[task 2016-10-18T16:24:29.723539Z] 16:24:29     INFO - Downloading and extracting to /home/worker/workspace/build/symbols these dirs * from https://queue.taskcluster.net/v1/task/NjtJJm9oR4CxU6pxB4Gq7g/artifacts/public/build/target.crashreporter-symbols.zip

But yeah, the harness is still being invoked with the URL to the symbols.zip:
[task 2016-10-18T16:25:26.019141Z] 16:25:26     INFO - Running command: ['/home/worker/workspace/build/venv/bin/python', '-u', '/home/worker/workspace/build/tests/web-platform/runtests.py', '--log-raw=-', '--log-raw=/home/worker/workspace/build/blobber_upload_dir/wpt_raw.log', '--log-errorsummary=/home/worker/workspace/build/blobber_upload_dir/wpt_errorsummary.log', '--binary=/home/worker/workspace/build/application/firefox/firefox', '--symbols-path=https://queue.taskcluster.net/v1/task/NjtJJm9oR4CxU6pxB4Gq7g/artifacts/public/build/target.crashreporter-symbols.zip', '--stackwalk-binary=/usr/local/bin/linux64-minidump_stackwalk', '--stackfix-dir=/home/worker/workspace/build/tests/bin', '--test-type=testharness', '--disable-e10s', '--total-chunks=12', '--this-chunk=11', '--prefs-root=/home/worker/workspace/build/tests/web-platform/prefs', '--processes=1', '--config=/home/worker/workspace/build/tests/web-platform/wptrunner.ini', '--ca-cert-path=/home/worker/workspace/build/tests/web-platform/certs/cacert.pem', '--host-key-path=/home/worker/workspace/build/tests/web-platform/certs/web-platform.test.key', '--host-cert-path=/home/worker/workspace/build/tests/web-platform/certs/web-platform.test.pem', '--certutil-binary=/home/worker/workspace/build/tests/bin/certutil'] in /home/worker/workspace/build

That's long, but the '--symbols-path=https://queue.taskcluster.net/v1/task/NjtJJm9oR4CxU6pxB4Gq7g/artifacts/public/build/target.crashreporter-symbols.zip' is the notable bit.

So presumably the WPT harness is passing a URL as the `symbols_path` into the stack fixer script, which isn't doing anything useful. (mozcrash can handle a URL to a symbols zip, but the stack fixer script needs a directory of unzipped symbols). I don't know where this gets handled for other harnesses, I assume somewhere in mozharness. For debug tests we always download the symbols.zip before running the harness (for this exact reason) and then pass the symbol directory in as --symbols-path.


> Then down below in the larger stack that has all the register information
> there is:

This is the output of minidump_stackwalk, FYI.