1673103 - Mangled output from dump_syms

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect, P3)

Description

[Steve Fink [:sfink] [:s:]]

See bug 1639153 comment 114. I ran dump_syms libmozjs-83a1.so >stdout.txt 2>stderr.txt, and stdout.txt contained mostly valid output but some corruption that caused decoding to choke. (It was trying to decode it as ASCII, which I still don't understand, but decoding it as UTF-8 would produce a different error. It's invalid gunk.)

You can reproduce with https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=HJ0Rcb51RvWd0bGZdHCICw.0&revision=8d2bbdde2e6c2639a8861ec6364da3c0a55306b0 by creating an interactive task. After the job has run, do:

cd workspace/sm-package/mozjs-83.0a1.0/obj-spider/dist
LD_LIBRARY_PATH=$HOME/fetches/gcc/lib64 ./host/bin/dump_syms bin/libmozjs-83a1.so | less

If you scroll down for a while (or search for "1765"), you should see a line like:

FILE 1765 /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/fmt/k^B@^A^E^N^C^U<90>^B^G

That line should say

FILE 1765 /rustc/4fb7144ed159f94491249e86d5bbd033b5d60550/src/libcore/fmt/float.rs

based on

$HOME/fetches/rustc/lib/rustlib/src/rust/src/libcore/fmt/float.rs

Comment 1

[Lars T Hansen [:lth]]

Gabriele, is there a way we can up the priority on this bug? It's blocking work that we've been trying to land for a long time and this looks like the last stumbling block.

Comment 2

[Gabriele Svelto [:gsvelto]]

I'll see if I can work on it this week since I was already touching dump_syms for other stuff. I've also CC'd Calixte in the hope he has some spare cycles before I do :-)

Comment 3

[Steve Fink [:sfink] [:s:]]

I'm not that familiar with all the moving parts here, but my personal opinion is that the blocked work is significant enough that we might want to temporarily demote SM(pkg) to tier 3 (or hack the output of dump_syms or something) in order to land.

Comment 4

[Calixte Denizet (:calixte)]

:sfink, could you attach libmozjs-83a1.so in the bug ?

Comment 5

[Steve Fink [:sfink] [:s:]]

(In reply to Calixte Denizet (:calixte) from comment #4)

:sfink, could you attach libmozjs-83a1.so in the bug ?

If you can tell me how to exfiltrate a 100MB file from a worker. I used to have a way, but that was when I had an external server to tunnel SSH connections through. (It's 440MB uncompressed, and xz gets it down to 100MB).

Hm... I guess I could hack the job to copy it into the uploads directory even if it fails. I can try that tomorrow.

Comment 6

[Gabriele Svelto [:gsvelto]]

(In reply to Steve Fink [:sfink] [:s:] from comment #5)

Hm... I guess I could hack the job to copy it into the uploads directory even if it fails. I can try that tomorrow.

+1

This should be an easy fix if we have access to the file.

Comment 7

[Steve Fink [:sfink] [:s:]]

Ok, that worked. You can get the file at https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AaKDx8_3RJmab-3NyKSElg/runs/0/artifacts/public/build/libmozjs-83a1.so (and it did error in the same way).

Comment 8

[Calixte Denizet (:calixte)]

It's very weird... I can't reproduce the bug on my machine so I guess the string for this filename is correct in debug info.
So the only thing I can imagine here is that there is something buggy with fs::canonicalize:
https://github.com/mozilla/dump_syms/blob/1e64e56720131bcf0ab01f201ed89a337c73069d/src/linux/source.rs#L70
Anyway I'll replace invalid chars with some ? or whatever to at least avoid invalid strings in sym files.

Comment 9

[Calixte Denizet (:calixte)]

So it isn't an issue with fs::canonicalize since we get a string from the path (new_path is None since we don't use self.mapping for the moment):
https://github.com/mozilla/dump_syms/blob/1e64e56720131bcf0ab01f201ed89a337c73069d/src/linux/source.rs#L106
and this string is put in array and it's dumped at some point:
https://github.com/mozilla/dump_syms/blob/1e64e56720131bcf0ab01f201ed89a337c73069d/src/linux/elf.rs#L72

So normally we should have a valid utf-8 string when it's dumped...
I've absolutely no idea on how we can have an invalid string in the output.

@sfink, could you try ./host/bin/dump_syms bin/libmozjs-83a1.so -o /tmp/mozjs.sym && less /tmp/mozjs.sym and see if you've the same issue ?

Comment 10

[Steve Fink [:sfink] [:s:]]

Ok. I'll have to wait for a new interactive job to finish building the library, but then I think my tab will hold it alive for some time. I think you can probably join me assuming the auth flow lets you, at https://firefox-ci-tc.services.mozilla.com/shell/?runId=0&socketUrl=wss%3A%2F%2Felvnh6qaaaaxk3rnjijtgpbll65d3vdh4a4qt5qyzxc6ghw5.taskcluster-worker.net%3A33406%2FSrlWRmr6TYyyDzLFy22dMA%2Fshell.sock&taskId=BVkR96DZR827Sze3gx_OLg&v=1

Comment 11

[Calixte Denizet (:calixte)]

Ok I think I understand the problem (thx for sharing this shell).
You aren't using the "new" dump_syms but the old buggy one from breakpad.
Maybe you need to add something like:
https://searchfox.org/mozilla-central/source/browser/config/mozconfigs/linux64/tsan#10
and be sure you fetch dump_syms:
https://searchfox.org/mozilla-central/source/taskcluster/ci/toolchain/dump-syms.yml#18

Comment 12

[Steve Fink [:sfink] [:s:]]

Ah! Thank you, that makes a lot more sense. Thank you for checking that out.

I have a try push running with it now.

Comment 13

[Steve Fink [:sfink] [:s:]]

Argh. Needs a bit more work -- my patch fixed SM(pkg) but broke the other SM(...) jobs. So the real problem is fixed, at least.

Comment 14

[Steve Fink [:sfink] [:s:]]

Attachment: Bug 1673103 - SM(pkg): Upload artifacts even if job fails

Comment 15

[Steve Fink [:sfink] [:s:]]

Attachment: Bug 1673103 - Switch SM jobs to use newer dump_syms

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/12355fe29698
SM(pkg): Upload artifacts even if job fails r=calixte
https://hg.mozilla.org/integration/autoland/rev/def6e060e7dd
Switch SM jobs to use newer dump_syms r=calixte

Comment 17

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/12355fe29698
https://hg.mozilla.org/mozilla-central/rev/def6e060e7dd