1440611 - some linux crashes missing symbolication

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Julien Cristau [:jcristau]]

There's an unusual number of crashes on yesterday's nightly with libxul.so@... signatures:

https://crash-stats.mozilla.com/search/?release_channel=nightly&build_id=20180222101929&cpu_arch=amd64&signature=~libxul.so%40&product=Firefox&version=60.0a1&platform=Linux&date=%3E%3D2018-02-16T12%3A40%3A00.000Z&date=%3C2018-02-23T12%3A40%3A17.000Z&_sort=-date&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform

The symbols are available on the symbol server, and calixte tells me he's able to symbolicate these addresses:

11:36 < calixte> fyi: addr2line -f -C -e ./libxul.so.dbg 0x203fc00
11:36 < calixte> mozilla::FrameLayerBuilder::GetOldLayerForFrame(nsIFrame*, unsigned int, mozilla::DisplayItemData*)

Comment 1

[Calixte Denizet (:calixte)]

:ted, do you have any idea here ? Could it be related to bug 1440282 ?

Comment 2

[Marcia Knous [:marcia]]

Hello Will - Is this something you can help with? Also can we get someone to be triage owner of this component? Thanks.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Is this an ongoing issue that spans multiple builds or is it isolated to the build in the url in the description?

Is there a better url that shows the crashes having problems? That url only shows 23 which doesn't seem unusual to me given we get tens of thousands of crashes a day.

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Calixte Denizet (:calixte) from comment #1)
> :ted, do you have any idea here ? Could it be related to bug 1440282 ?

I looked into this briefly and we do seem to have valid symbols for these crashes but the .sym files are missing functions for these addresses. I didn't find time to look into it more deeply. It is entirely possible that it's related to that issue, or potentially the same issue.

Comment 5

[Tyson Smith [:tsmith]]

This is preventing us from properly bucketing crashes found by our fuzzers.

Comment 6

[Kim Moir [:kmoir] ET]

:ajones Also, this bug needs an owner

Comment 7

[Gabriele Svelto [:gsvelto]]

I haven't looked at the crashes but if comment 4 is true and only some functions are missing then this might be an instance of bug 1440282. In my first quick test roughly ~8% of the functions are lost when dumping symbols w/o DW_AT_ranges.

Comment 8

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Gabriele - is this still an issue?

Comment 9

[Gabriele Svelto [:gsvelto]]

There are still some unsymbolicated crashes on Nightly:

https://crash-stats.mozilla.com/search/?release_channel=nightly&cpu_arch=amd64&signature=~libxul.so%40&product=Firefox&platform=Linux&date=%3E%3D2019-01-14T09%3A13%3A30.000Z&date=%3C2019-01-21T09%3A13%3A30.000Z&_sort=-date&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

I fixed one more bug related to this (bug 1510574) so this might be a separate issue, I'll try to investigate, leaving the NI? for now.

Comment 10

[Gabriele Svelto [:gsvelto]]

This might be bug 1510698.

Comment 11

[Gabriele Svelto [:gsvelto]]

Unfortunately there's still crash reports with unsymbolicated stacks for recent builds, so they must be unrelated to the previous fixes we landed. I'll have to look at them individually.

Comment 12

[Jason Kratzer [:jkratzer]]

We're no longer seeing this issue during fuzzing.

Comment 13

[Gabriele Svelto [:gsvelto]]

(In reply to Jason Kratzer [:jkratzer] from comment #12)

We're no longer seeing this issue during fuzzing.

Good, this part of the crash reporting machinery has been extensively overhauled and supports things we only dreamed we would support when this bug was filed so chances are it was fixed along the way.