711973 - [skiplist] Add '(libxul\.so|xul\.dll|XUL)@0x.*' to the prefix list

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Scoobidiver (away)]

Please add:
'xul.dll@0x.*'
'libxul.so0x.*'
to the skiplist as irrelevantSignatureRegEx.

Comment 1

[Robert Kaiser]

Just as with bug 711954, I don't think the irrelevant list is the right thing to do here.

Comment 2

[Scoobidiver (away)]

(In reply to Robert Kaiser (:[e-mail]) from comment #1)
> Just as with bug 711954, I don't think the irrelevant list is the right
> thing to do here.
Same answer as in bug 711954: the crash address move between each build, making almost impossible to track it across all Firefox versions or even builds.
For instance, if libxul.so is not added to the skiplist, bug 711852 and bug 711853 will become unmaintainable as 5 different crash addresses should be added for each build.
See also:
https://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&range_value=4&range_unit=weeks&query_search=signature&query_type=contains&query=xul.dll%400x&reason=&build_id=&process_type=any&hang_type=any&do_query=1
https://crash-stats.mozilla.com/query/query?product=Fennec&version=ALL%3AALL&range_value=4&range_unit=weeks&query_search=signature&query_type=contains&query=libxul.so%400x&reason=&build_id=&process_type=any&hang_type=any&do_query=1

Comment 3

[Robert Kaiser]

And same reply as there: Hiding crashes places in our own code is not a solution for that. Finding out why we don't get symbols for those places is what we should try to solve.

Comment 4

[Scoobidiver (away)]

(In reply to Robert Kaiser (:[e-mail]) from bug 711954 comment #5)
> A bug should only ever be filed after searching if there is a similar one in
> existence - no matter if it's a crash or other bug.
So, each day, Naoki will add crash signatures to bug 691281, bug 711706, bug 711852 and bug 711853 and will remove the same amount as the number of signatures are limited in order to track them in Socorro.

Comment 5

[Robert Kaiser]

(In reply to Scoobidiver from comment #4)
> So, each day, Naoki will add crash signatures to bug 691281, bug 711706, bug
> 711852 and bug 711853 and will remove the same amount as the number of
> signatures are limited in order to track them in Socorro.

If his prerogative to do that. But the solution is not to hide the location of crashes happening in our own code, we need to deal with the cause of not seeing symbols there.

Comment 6

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Perhaps adding to the prefix skiplist 'libxul\.so\s*@\s*0x.*' (reg ex taken from bug 683385 ) so that the engineers have more meaning behind this crash signature would be a solution?

Purpose is to give more meaning to the crash signature seen here: 
https://crash-stats.mozilla.com/report/index/2671fadc-bb78-4319-88ba-0a5a42111226

Not sure about the xul.dll that looks more like a windows crash signature.  Which crash were you looking at scoobi?

Comment 7

[Robert Kaiser]

(In reply to Naoki Hirata :nhirata from comment #6)
> Perhaps adding to the prefix skiplist 'libxul\.so\s*@\s*0x.*' (reg ex taken
> from bug 683385 ) so that the engineers have more meaning behind this crash
> signature would be a solution?

Prefix would be acceptable, I think, but in the end, any crashes like that sound a lot like we are missing symbols there.

> Not sure about the xul.dll that looks more like a windows crash signature. 

xul.dll is the same as libxul.so, just on a different system - if we add prefixing, we should IMHO do it for both and also XUL which is the Mac variant.

Comment 8

[Robert Kaiser]

Ted, any idea what's actually up in those cases and why we don't get the addresses resolved properly? What should be our course of action here?

Comment 9

[Robert Kaiser]

Laura, in the CrashKill meeting we found we want the step from comment #6 be done ASAP as FennecAndroid code is going to beta soon and we need this for diagnosing FennecAndroid crashes. How soon can this be in production?

Comment 10

[(not currently active) Ted Mielczarek]

Kairo: I haven't had time to look into this yet, sorry.

Comment 11

[Robert Kaiser]

(In reply to Ted Mielczarek [:ted, :luser] from comment #10)
> Kairo: I haven't had time to look into this yet, sorry.

Ted, I filed bug 716869 on this on the Breakpad side, it seems rather urgent. We'll leave the bug here for a short-term alleviation of the problem via the prefix skiplist, the other one is for going to the core of the problem, and from what I've seen it seems to be a regression in 12 trunk.

Comment 12

[[github robot]]

Commits pushed to https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/1746167f33d2110e1bcf8f56be43ba8808192f1b
bug 711973 - add libxul to skiplist

https://github.com/mozilla/socorro/commit/a430bd8ab1254fa29e18f02f6428da9be454cb16
Merge pull request #272 from rhelmer/bug711973-skiplist-changes

bug 711973 - add libxul to skiplist

Comment 13

[[github robot]]

Commits pushed to https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/dde3a1b4809b30145dbc268134416f0909186137
bug 711973 - add libxul to skiplist

https://github.com/mozilla/socorro/commit/41929f9384b9f348f7f35af1e6ac2fc6e1dde43d
Merge pull request #273 from rhelmer/stage

bug 711973 - add libxul to skiplist

Comment 14

[Robert Helmer [:rhelmer]]

I've slipped this into the 2.4 release, currently scheduled for Friday: https://wiki.mozilla.org/Socorro:Releases

If this needs to go out sooner than that please let me know ASAP.

Comment 15

[Scoobidiver (away)]

(In reply to [github robot] from comment #13)
> https://github.com/mozilla/socorro/commit/
> 41929f9384b9f348f7f35af1e6ac2fc6e1dde43d
> Merge pull request #273 from rhelmer/stage
It seems that comment 0 has been taken into account instead of comment 6, and xul.dll and XUL has also been added to the skiplist.
In addition, it's not in irrelevantSignatureRegEx as requested, but in prefixSignatureRegEx.

Comment 16

[Robert Kaiser]

(In reply to Scoobidiver from comment #15)
> and xul.dll and XUL has also been added to the skiplist.
> In addition, it's not in irrelevantSignatureRegEx as requested, but in
> prefixSignatureRegEx.

This is as designed and the right way to go here. The Socorro team has done the right things there. As of comment #6, we asked them to use the prefix skiplist as we can't just ignore all frames of libxul because it may be valid frames that need to be looked at, esp. in the cases where symbols are missing. Also, because similar things have been seen with Mac and Windows builds, we asked them to add the regexes for all three systems for libxul, so it's libxul.so, xul.dll and XUL. That's all correct and as it should be.

Comment 17

[Scoobidiver (away)]

Comment 6 and comment 9 asked things to be done only for libxul.so like in bug 683385 that was about irrelevantSignatureRegEx. So there's a misunderstanding.

Comment 18

[Robert Kaiser]

Scoobidiver, the misunderstanding is between you and us then. Adding the Windows and Mac variants was discussed on IRC.

Comment 19

[Scoobidiver (away)]

Per comment 18, I renamed the bug summary and added again the dependent desktop bugs.