Closed Bug 1253070 Opened 4 years ago Closed 3 years ago

Flip MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS option on by default

Categories

(Toolkit :: Crash Reporting, defect)

All
Windows
defect
Not set

Tracking

()

RESOLVED FIXED
mozilla52
Tracking Status
firefox47 --- affected
firefox52 --- fixed

People

(Reporter: abbGZcvu_bugzilla.mozilla.org, Assigned: blassey)

References

Details

Attachments

(1 file, 1 obsolete file)

When downloadBuild.py downloads symbols, it produces the normal directory structures (\symbols\binary_name.pdb\[hash]\) but each folder contains a "binary_name.sym" file, rather than a "binary_name.pdb" file. I'm not sure what the former are, but I would expect the latter. So does WinDbg/cdb; I am getting no symbol information from these ".sym" files.
(In reply to SkyLined from comment #0)
> When downloadBuild.py downloads symbols

What you mean is probably the following file:

https://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1456930532/firefox-47.0a1.en-US.win32.crashreporter-symbols.zip

In this zip file, I see the firefox.pdb folder, and firefox.sym is located within the subfolders of this folder, which is probably what Skylined is referring to here.

:glandium/Ted, do you have an idea what might be going on? (or feel free to pass on the baton to someone else more appropriate)
Flags: needinfo?(ted)
Flags: needinfo?(mh+mozilla)
Keywords: crash, testcase
Version: 47 Branch → Trunk
Component: Untriaged → Breakpad Integration
Product: Core → Toolkit
Guessing this is related to Breakpad.
This is expected behavior. We don't upload the native debug symbols alongside the build, because we don't use them in automation and don't want to waste the time uploading/downloading them there. We upload native debug symbols from nightly/release builds to the symbol server. (I don't know what downloadBuild.py is, FWIW.)
Flags: needinfo?(ted)
downloadBuild.py is merely a file in the fuzzing harness to download specific Firefox/js builds (as well as other files, including the above-mentioned firefox-47.0a1.en-US.win32.crashreporter-symbols.zip):

https://github.com/MozillaSecurity/funfuzz/blob/29139149d01ac93d040d08d8bc6b11885ff0497a/util/downloadBuild.py

downloadBuild.py isn't the issue here, though.

Wrt. *.crashreporter-symbols.zip, will it make sense to start uploading native debug aurora/nightly/treeherder per-push symbols so fuzzers can use the builds for testing and get symbols?

Is it too much overhead (space requirements) to ask? We could restrict to store only the past month's  symbols, for example.
Flags: needinfo?(ted)
I think space requirements are probably manageable nowadays since all the builds are stored in S3. I still don't want to put the symbols into the .symbols.zip package, since as I stated in comment 3 we have to download that package in automation for debug builds and getting stacks out of crashes, and the native symbols add a lot of space (and thus overhead in downloading the file). What we could do is flip the MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS option on by default, which uploads a separate crashreporter-symbols-full.zip alongside the existing symbols.zip:
https://dxr.mozilla.org/mozilla-central/search?tree=mozilla-central&q=MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS&redirect=true

Your script could then fetch that file which would give you the debug symbols you need.

(As an aside, you might want to consider switching to or relying on mozdownload, since it solves the same problem but is well-maintained: https://github.com/mozilla/mozdownload )
Flags: needinfo?(ted)
mozdownload in turn requires pip for installing, which seems to turn this into a chain of dependent tools and scripts I would need to run in sequence. This is much more fragile than a single, simple, stand-alone script I can run to download binary + symbols.

If a separate file with full symbols could be made available in the same folder, I can easily update the code for downloadBuild.py to fetch that. This would solve my problem.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> What we could do is flip the
> MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS option on by default, which uploads a
> separate crashreporter-symbols-full.zip alongside the existing symbols.zip:
> https://dxr.mozilla.org/mozilla-central/search?tree=mozilla-
> central&q=MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS&redirect=true
> 
> Your script could then fetch that file which would give you the debug
> symbols you need.

Updating summary since this is what we need.
Summary: Symbols only contain ".sym" files and no ".pdb" files. → Flip MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS option on by default
Flags: needinfo?(mh+mozilla)
Can I assume flipping this option is not a very complex process and I can expect builds with debug symbols soon? I'd like to update the build I fuzz frequently to avoid reporting issues that have already been addressed. I'd also like to start reporting the issues I found so far. Both are not practically possible without symbols, as I cannot get a proper stack to automatically detect duplicates or manually analyze the crashes.
Is there anything I can do to help expedite this?
(In reply to SkyLined from comment #9)
> Is there anything I can do to help expedite this?

You could write and test the patch! It's probably 5 lines or so. Removing the ifdef here:
https://dxr.mozilla.org/mozilla-central/rev/3e04659fdf6aef792f7cf9840189c6c38d08d1e8/toolkit/mozapps/installer/upload-files.mk#491

and just moving that `$(call QUOTED_WILDCARD ...` line up into the UPLOAD_FILES list above, then removing the other reference to this variable for completeness:
https://dxr.mozilla.org/mozilla-central/rev/3e04659fdf6aef792f7cf9840189c6c38d08d1e8/old-configure.in#7515
Flags: needinfo?(berendjanwever)
Sorry, been too busy maintaining my own code to work on yours... will look into this next week.
Flags: needinfo?(berendjanwever)
Attached patch upload_full_symbols.patch (obsolete) — Splinter Review
Assignee: nobody → blassey.bugs
Attachment #8802299 - Flags: review?(ted)
forgot to qref
Attachment #8802299 - Attachment is obsolete: true
Attachment #8802299 - Flags: review?(ted)
Attachment #8802524 - Flags: review?(ted)
Comment on attachment 8802524 [details] [diff] [review]
upload_full_symbols.patch

Review of attachment 8802524 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!
Attachment #8802524 - Flags: review?(ted) → review+
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5d3e8e87351c
Flip MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS option on by default. r=ted
Keywords: checkin-needed
Pushed by philringnalda@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e60f475551d
followup, make it actually, you know, build
https://hg.mozilla.org/mozilla-central/rev/5d3e8e87351c
https://hg.mozilla.org/mozilla-central/rev/3e60f475551d
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
You need to log in before you can comment on or make changes to this bug.