555599 - [skiplist] Add nsTArray_base* socorro skiplists

Status: RESOLVED DUPLICATE of bug 630230

(Socorro :: Infra, task)

Description

[Wayne Mery (:wsmwk)]

Several of our topcrashes of the past year have been aggregated under the signature
  memmove | nsTArray_base::ShiftData

for example bug 460036, bug 524064, bug 524266, bug 524673, bug 534858

and we have more crashes to kill
- in v3.0.3 like bp-3988fdb2-1806-4128-9f3c-85b212100327 which I believe is an unresolved topcrash
- on trunk like bp-885365f2-f13d-4d97-b994-f5a2c2100326
so it would be a great help for future crashes to be more unique (I wish we wold have asked for this sooner). 

Please arrange for these crashes to be attributed to the caller of nsTArray_base::ShiftData.

Comment 1

[Wayne Mery (:wsmwk)]

[I forgot to commit this yesterday]

TB devs, we have some crash variations of nsTArray, some listed below and some marked dependent on this bug. 

Should we make the skip to be anything that starts with nsTArray* ?

Also, (see bug 530684 comment 4, ...) do we want a compound signature, so signatures become
   memmove | nsTArray_base::ShiftData | blah.  
Or ignore top frames, which may simplify our crash bugs where in some cases Mac crashes sigs get the same sig as other OSes, as in 
   blah


bp-3988fdb2-1806-4128-9f3c-85b212100327 v3.0.3 (OOM?)

0	mozcrt19.dll	memmove	 MEMCPY.ASM:188
1	xpcom_core.dll	nsTArray_base::ShiftData	objdir-tb/mozilla/xpcom/build/nsTArray.cpp:173
2	thunderbird.exe	nsTArray<nsListenerStruct>::RemoveElementsAt	objdir-tb/mozilla/dist/include/xpcom/nsTArray.h:664
3	thunderbird.exe	nsAutoTObserverArray<nsListenerStruct,2>::Clear	objdir-tb/mozilla/dist/include/xpcom/nsTObserverArray.h:242
4	thunderbird.exe	nsEventListenerManager::Disconnect	content/events/src/nsEventListenerManager.cpp:1230 


bp-885365f2-f13d-4d97-b994-f5a2c2100326
0	mozcrt19.dll	memmove	 MEMCPY.ASM:188
1	xpcom_core.dll	nsTArray_base::ShiftData	objdir-tb/mozilla/xpcom/build/nsTArray.cpp:173
2	thunderbird.exe	nsCSSStyleSheetInner::RemoveSheet	layout/style/nsCSSStyleSheet.cpp:909
3	thunderbird.exe	nsCSSStyleSheet::~nsCSSStyleSheet	layout/style/nsCSSStyleSheet.cpp:1050
4	thunderbird.exe	nsCSSStyleSheet::`vector deleting destructor'	
5	thunderbird.exe	nsCSSStyleSheet::Release	layout/style/nsCSSStyleSheet.cpp:1074

Comment 2

[Laura Thomson :laura]

Please clarify priority for this bug.

Comment 3

[Wayne Mery (:wsmwk)]

Because this will help us address a top 5 crasher for us, we'd like this as soon as possible. On par with what is done for crashkill for firefox

We'd like this to be as nsTArray_base* and compound signature, so nsTArray_base appears in the signature - unless someone else says otherwise (which so far has been no one)

Comment 4

[Wayne Mery (:wsmwk)]

help.

I thought (according to the website) that skiplist requests were handled in a weekly review process. But this request seems to have gotten lost. Please implement

Comment 5

[Austin King [:ozten]]

(In reply to comment #4)
This bug isn't in the search linked to from:
https://wiki.mozilla.org/Breakpad/SkipList
It is missing:
* whiteboard CrashKill

Is this a CrashKill priority?

Comment 6

[Wayne Mery (:wsmwk)]

see comment 5. I have no idea how Thunderbird fits into crashkill definitions and procedures. This bug addresses topcrash signatures for Thunderbird, so if [crashkill] is what is needed to get this resolved ...

Comment 7

[Wayne Mery (:wsmwk)]

(committing this info so I don't lose it)

ref bug 520678 comment 0 and bug 520678 comment 23.  Jonas' bug 520678 does prefixSignatureRegEx skip for nsTArray_base::ShiftData.  However, it does not seem to be sufficient for full differentiation of stacks. See the bug list of bp-39738046-be6d-4f98-b763-4e25f2100807 [1].  (waiting on response in bug 520678) 

Further, highly speculative as yet, perhaps we need relax the skiplist from nsTArray_base::ShiftData to be nsTArray_base* or nsTArray* ...
 nsTArray_base* Firefox - http://bit.ly/9MWlDC
 nsTArray_base* Thunderbird - http://bit.ly/b5ac7a
 nsTArray* Thunderbird - http://bit.ly/bkvF5U
 nsTArray* Firefox - http://bit.ly/cVxLnw

Note: I can't see the bugs at https://wiki.mozilla.org/Breakpad/SkipList#Changelog ... so I have no idea what skiplists have actually been implemented in the past year.  The only example I can see is bug 514505 comment 12.


[1] bug list of bp-39738046-be6d-4f98-b763-4e25f2100807

OPEN
Bug 489577 NEW freezes while loading video then crashes while exiting (Move Networks player?) [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)]
Bug 555618 NEW crash [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)] - [@ nsTArray<nsListenerStruct>::RemoveElementsAt] - [@ nsAutoTObserverArray<nsListenerStruct,2>::Clear]
Bug 555619 NEW crash [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)] - [@ nsCSSStyleSheetInner::RemoveSheet]
Bug 568883 NEW crash [@memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)] - [@ nsMsgDBView::RemoveRows]

DUPLICATE
Bug 540416 RESOLVED crash [@ nsImapFlagAndUidState::ExpungeByIndex(unsigned int)] and [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)] | nsImapFlagAndUidState::ExpungeByIndex

INVALID
Bug 519771 RESOLVED Crash [@memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int) ]

FIXED
Bug 507114 RESOLVED [Win] Topcrasher for Firefox 3.5.1 [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)][@ nsObserverList::FillObserverArray]
Bug 524673 VERIFIED crash [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)]
Bug 524064 RESOLVED crash [@memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)] - [@ nsMsgDBView::RemoveRows]
Bug 534858 RESOLVED Crash on repeated collapse/expand of threads with subthreads killed by filter [@ memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int)] - nsMsgDBView::RemoveRows


ref: bug 535548 comment 7 about compound signatures
ref: http://code.google.com/p/socorro/wiki/SignatureGeneration
ref: https://wiki.mozilla.org/Breakpad/SkipList

Comment 8

[Laura Thomson :laura]

Checking in.

Current config in prod includes:
prefixSignatureRegEx = '|'.join([
...
  'nsTArray_base<.*',
  'nsTArray<.*',
...])

Is this now giving you the stacks you want?

Comment 9

[Laura Thomson :laura]

Ping?

Comment 10

[Wayne Mery (:wsmwk)]

sorry for not replying earlier. I don't know to what extent this has lessened, if at all, but this still occurs [1]. Some examples...

memmove | nsTArray_base<nsTArrayDefaultAllocator>::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int) | nsHttpConnectionMgr::PruneDeadConnectionsCB(nsHashKey*, void*, void*) 

mozalloc_abort(char const* const) | mozalloc_handle_oom() | moz_xmalloc | nsTArray_base<nsTArrayInfallibleAllocator>::EnsureCapacity(unsigned int, unsigned int) | nsTArray<nsRefPtr<mozilla::layers::ColorLayer>, nsTArrayDefaultAllocator>::AppendElements

memmove | nsTArray_base<nsTArrayInfallibleAllocator>::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int) | nsImapFlagAndUidState::ExpungeByIndex(unsigned int) 


[1] https://crash-stats.mozilla.com/query/query?product=Thunderbird&version=ALL%3AALL&range_value=2&range_unit=weeks&date=09%2F22%2F2011+11%3A17%3A18&query_search=signature&query_type=contains&query=nsTArray_base&reason=&build_id=&process_type=any&hang_type=any&do_query=1

Comment 11

[Laura Thomson :laura]

Ok, the behavior you describe is as intended - these functions are in the prefix list and the signature reflects that by prepending the function name and moving on to the next one - look at the end of the signatures you pasted.

There is one other way we can do this, which is to put those functions on the irrelevant list, meaning they won't be shown in the signature at all.   Is that what you would prefer?

It's worth reading up on how skiplists work to understand fully what I mean:
https://wiki.mozilla.org/Breakpad/Skip_List_How_To

Comment 12

[Laura Thomson :laura]

Waiting on a response, detargeting

Comment 13

[Robert Kaiser]

1) Wayne, is something still needed to be done here?
2) If so, what exactly should be added to which skiplist (prefix or ignore list)?
3) Once you know that, can you clear up with string devs if that sounds good to be done?

Comment 14

[Robert Kaiser]

Oops, the comment was on the wrong bug, commented it on the correct one, sorry.

Still, we already skiplisted a number of nsTArray stuff in bug 630230, and this looks to be a dupe of that nowadays.