Closed Bug 627727 Opened 9 years ago Closed 8 years ago

growing spike in crashes [@ mozalloc_abort(char const* const) ][@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ] (was [@ mozalloc_handle_oom() ][@ mozalloc_handle_oom ]) just after transition to 4.0b10pre

Categories

(Core :: General, defect, critical)

x86
All
defect
Not set
critical

Tracking

()

RESOLVED INVALID
Tracking Status
blocking2.0 --- -

People

(Reporter: chofmann, Unassigned)

References

Details

(Keywords: crash)

Crash Data

Attachments

(1 file, 1 obsolete file)

from Jan 1-12 we were pretty consistently running at 0-10 crashes per day on mozalloc_handle_oom() crashes in latest mozilla-central builds.  From Jan 13-18 we bumped to 15-17 crashes per day with one day at 26 crashes.  The last two days we have seen 24 and 27 crashes.

The signature has moved up near the top ten in recent days.

Need to keep an eye out on this, and we might need skip listing or to try and find some change that might have put increased memory preasure on the overall program to get a better diagnosis of what might be behind the increase.

There is a wide variety of stacks behind the signature:  here is a sample breakdown.


....Signature number: 10-mozalloc_handle_oom
______ distribution of 20 different stacks, looking at top 10 frames
      3  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
0|3|xul.dll|nsTArray<mozilla::FrameLayerBuilder::ClippedDisplayItem,nsTArrayDefaultAllocator>::AppendElements<mozilla::FrameLayerBuilder::ClippedDisplayItem>(mozilla::FrameLayerBuilder::ClippedDisplayItem const *,unsigned int)
0|4|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|5|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|6|xul.dll|mozilla::FrameLayerBuilder::BuildContainerLayerFor(nsDisplayListBuilder *,mozilla::layers::LayerManager *,nsIFrame *,nsDisplayItem *,nsDisplayList const &)
0|7|xul.dll|nsDisplayOwnLayer::BuildLayer(nsDisplayListBuilder *,mozilla::layers::LayerManager *)
0|8|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|9|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)

      2  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsCSSRuleProcessor::RefreshRuleCascade(nsPresContext *)
0|3|xul.dll|nsCSSRuleProcessor::RulesMatching(AnonBoxRuleProcessorData *)
0|4|xul.dll|EnumRulesMatching<AnonBoxRuleProcessorData>
0|5|xul.dll|nsStyleSet::FileRules(int (*)(nsIStyleRuleProcessor *,void *),void *,nsIContent *,nsRuleWalker *)
0|6|xul.dll|nsStyleSet::ResolveAnonymousBoxStyle(nsIAtom *,nsStyleContext *)
0|7|xul.dll|nsCSSFrameConstructor::ConstructRootFrame(nsIFrame * *)
0|8|xul.dll|PresShell::InitialReflow(int,int)
0|9|xul.dll|nsContentSink::StartLayout(int)

      1  stacks like
7|0|mozalloc.dll|mozalloc_abort(char const * const)
7|1|mozalloc.dll|mozalloc_handle_oom()
7|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
7|3|xul.dll|nsHttpHeaderArray::SetHeader(nsHttpAtom,nsACString_internal const &,int)
7|4|xul.dll|nsHttpHeaderArray::ParseHeaderLine(char const *,nsHttpAtom *,char * *)
7|5|xul.dll|nsHttpResponseHead::ParseHeaderLine(char const *)

      1  stacks like
3|0|mozalloc.dll|mozalloc_abort(char const * const)
3|1|mozalloc.dll|mozalloc_handle_oom()
3|2|xul.dll|GCGraphBuilder::NoteScriptChild(unsigned int,void *)
3|3|xul.dll|JSContextParticipant::Traverse(void *,nsCycleCollectionTraversalCallback &)
3|4|xul.dll|nsCycleCollector::MarkRoots(GCGraphBuilder &)
3|5|xul.dll|nsCycleCollector::BeginCollection(int,nsICycleCollectorListener *)
3|6|xul.dll|nsCycleCollectorRunner::Run()
3|7|xul.dll|nsThread::ProcessNextEvent(int,int *)
3|8|xul.dll|
3|9|xul.dll|nsThreadStartupEvent::`scalar deleting destructor'(unsigned int)

      1  stacks like
16|0|mozalloc.dll|mozalloc_abort(char const * const)
16|1|mozalloc.dll|mozalloc_handle_oom()
16|2|xul.dll|nsHtml5TreeBuilder::flushCharacters()
16|3|xul.dll|nsHtml5TreeBuilder::comment(unsigned short *,int,int)
16|4|xul.dll|nsHtml5Tokenizer::stateLoop(int,unsigned short,int,unsigned short *,int,int,int)
16|5|xul.dll|nsHtml5Tokenizer::tokenizeBuffer(nsHtml5UTF16Buffer *)
16|6|xul.dll|nsHtml5StreamParser::ParseAvailableData()
16|7|xul.dll|nsHtml5StreamParser::DoDataAvailable(unsigned char *,unsigned int)
16|8|xul.dll|nsHtml5DataAvailable::Run()
16|9|xul.dll|nsThread::ProcessNextEvent(int,int *)

      1  stacks like
13|0|mozalloc.dll|mozalloc_abort(char const * const)
13|1|mozalloc.dll|mozalloc_handle_oom()
13|2|xul.dll|nsHtml5Tokenizer::stateLoop(int,unsigned short,int,unsigned short *,int,int,int)
13|3|xul.dll|nsHtml5Tokenizer::tokenizeBuffer(nsHtml5UTF16Buffer *)
13|4|xul.dll|nsHtml5StreamParser::ParseAvailableData()
13|5|xul.dll|nsHtml5StreamParser::DoDataAvailable(unsigned char *,unsigned int)
13|6|xul.dll|nsHtml5DataAvailable::Run()
13|7|xul.dll|nsThread::ProcessNextEvent(int,int *)
13|8|xul.dll|nsThreadStartupEvent::`scalar deleting destructor'(unsigned int)
13|9|xul.dll|nsThread::ThreadFunc(void *)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|xptiInterfaceInfoManager::ReadXPTFileFromInputStream(nsIInputStream *)
0|3|xul.dll|nsComponentManagerImpl::ManifestXPT(nsComponentManagerImpl::ManifestProcessingContext &,int,char * const *)
0|4|xul.dll|ParseManifestCommon
0|5|xul.dll|ParseManifest(NSLocationType,nsIZipReader *,char const *,char *,bool)
0|6|xul.dll|nsComponentManagerImpl::RegisterJarManifest(nsIZipReader *,char const *,bool)
0|7|xul.dll|nsComponentManagerImpl::ManifestManifest(nsComponentManagerImpl::ManifestProcessingContext &,int,char * const *)
0|8|xul.dll|ParseManifestCommon
0|9|xul.dll|ParseManifest(NSLocationType,nsIZipReader *,char const *,char *,bool)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|XPC_WN_GetterSetter(JSContext *,unsigned int,unsigned __int64 *)
0|3|mozjs.dll|js::Invoke(JSContext *,js::CallArgs const &,unsigned int)
0|4|mozjs.dll|js::ExternalInvoke(JSContext *,js::Value const &,js::Value const &,unsigned int,js::Value *,js::Value *)
0|5|mozjs.dll|js::ExternalGetOrSet(JSContext *,JSObject *,int,js::Value const &,JSAccessMode,unsigned int,js::Value *,js::Value *)
0|6|mozjs.dll|js::Shape::get(JSContext *,JSObject *,JSObject *,JSObject *,js::Value *)
0|7|mozjs.dll|js_GetPropertyHelper(JSContext *,JSObject *,int,unsigned int,js::Value *)
0|8|mozjs.dll|js::Interpret(JSContext *,JSStackFrame *,unsigned int,JSInterpMode)
0|9|mozjs.dll|js::RunScript(JSContext *,JSScript *,JSStackFrame *)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
0|3|xul.dll|nsTArray<nsHtml5TreeOperation,nsTArrayDefaultAllocator>::MoveElementsFrom<nsHtml5TreeOperation,nsTArrayDefaultAllocator>(nsTArray<nsHtml5TreeOperation,nsTArrayDefaultAllocator> &)
0|4|xul.dll|nsHtml5TreeOpStage::MoveOpsAndSpeculativeLoadsTo(nsTArray<nsHtml5TreeOperation,nsTArrayDefaultAllocator> &,nsTArray<nsHtml5SpeculativeLoad,nsTArrayDefaultAllocator> &)
0|5|xul.dll|nsHtml5ExecutorReflusher::Run()
0|6|xul.dll|mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *)
0|7|xul.dll|MessageLoop::RunHandler()
0|8|xul.dll|MessageLoop::Run()
0|9|xul.dll|nsBaseAppShell::Run()

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
0|3|xul.dll|nsTArray<mozilla::FrameLayerBuilder::DisplayItemData,nsTArrayDefaultAllocator>::AppendElements<mozilla::FrameLayerBuilder::DisplayItemData>(mozilla::FrameLayerBuilder::DisplayItemData const *,unsigned int)
0|4|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|5|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|6|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|7|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|8|xul.dll|mozilla::FrameLayerBuilder::BuildContainerLayerFor(nsDisplayListBuilder *,mozilla::layers::LayerManager *,nsIFrame *,nsDisplayItem *,nsDisplayList const &)
0|9|xul.dll|nsDisplayOwnLayer::BuildLayer(nsDisplayListBuilder *,mozilla::layers::LayerManager *)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
0|3|xul.dll|nsTArray<mozilla::FrameLayerBuilder::ClippedDisplayItem,nsTArrayDefaultAllocator>::AppendElements<mozilla::FrameLayerBuilder::ClippedDisplayItem>(mozilla::FrameLayerBuilder::ClippedDisplayItem const *,unsigned int)
0|4|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|5|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|6|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
0|7|xul.dll|mozilla::FrameLayerBuilder::BuildContainerLayerFor(nsDisplayListBuilder *,mozilla::layers::LayerManager *,nsIFrame *,nsDisplayItem *,nsDisplayList const &)
0|8|xul.dll|nsDisplayList::PaintForFrame(nsDisplayListBuilder *,nsIRenderingContext *,nsIFrame *,unsigned int)
0|9|xul.dll|nsLayoutUtils::PaintFrame(nsIRenderingContext *,nsIFrame *,nsRegion const &,unsigned int,unsigned int)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
0|3|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const &,nsDisplayItem::HitTestState *,nsTArray<nsIFrame *,nsTArrayDefaultAllocator> *)
0|4||
0|5|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const &,nsDisplayItem::HitTestState *,nsTArray<nsIFrame *,nsTArrayDefaultAllocator> *)
0|6|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const &,nsDisplayItem::HitTestState *,nsTArray<nsIFrame *,nsTArrayDefaultAllocator> *)
0|7|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const &,nsDisplayItem::HitTestState *,nsTArray<nsIFrame *,nsTArrayDefaultAllocator> *)
0|8|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const &,nsDisplayItem::HitTestState *,nsTArray<nsIFrame *,nsTArrayDefaultAllocator> *)
0|9|xul.dll|nsImageBoxFrame::GetPrefSize(nsBoxLayoutState &)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsTArray_base<nsTArrayDefaultAllocator>::EnsureCapacity(unsigned int,unsigned int)
0|3|xul.dll|mozilla::imagelib::RasterImage::SetSourceSizeHint(unsigned int)
0|4|xul.dll|imgRequest::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|5|xul.dll|nsCOMPtr<nsICancelable>::operator=(nsICancelable *)
0|6|xul.dll|nsStreamListenerTee::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|7|xul.dll|nsHttpChannel::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|8|xul.dll|nsInputStreamPump::OnStateTransfer()

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|nsStyledElement::ParseStyleAttribute(nsAString_internal const &,nsAttrValue &,int)
0|3|xul.dll|nsGenericHTMLElement::ParseAttribute(int,nsIAtom *,nsAString_internal const &,nsAttrValue &)
0|4|xul.dll|nsGenericElement::SetAttr(int,nsIAtom *,nsIAtom *,nsAString_internal const &,int)
0|5|xul.dll|nsGenericHTMLElement::SetAttr(int,nsIAtom *,nsIAtom *,nsAString_internal const &,int)
0|6|xul.dll|nsHtml5TreeOperation::Perform(nsHtml5TreeOpExecutor *,nsIContent * *)
0|7|xul.dll|nsHtml5TreeOpExecutor::RunFlushLoop()
0|8|xul.dll|nsHtml5ExecutorReflusher::Run()
0|9|xul.dll|nsThread::ProcessNextEvent(int,int *)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|mozilla::imagelib::RasterImage::InitDecoder(bool)
0|3|xul.dll|mozilla::imagelib::RasterImage::Init(imgIDecoderObserver *,char const *,char const *,unsigned int)
0|4|xul.dll|imgRequest::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|5|xul.dll|ProxyListener::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|6|xul.dll|nsStreamListenerTee::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|7|xul.dll|nsHttpChannel::OnDataAvailable(nsIRequest *,nsISupports *,nsIInputStream *,unsigned int,unsigned int)
0|8|xul.dll|nsInputStreamPump::OnStateTransfer()

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|gfxSkipChars::TakeFrom(gfxSkipCharsBuilder *)
0|3|xul.dll|BuildTextRunsScanner::BuildTextRunForFrames(void *)
0|4|xul.dll|BuildTextRunsScanner::FlushFrames(int,int)
0|5|xul.dll|BuildTextRunsScanner::ScanFrame(nsIFrame *)
0|6|xul.dll|BuildTextRunsScanner::ScanFrame(nsIFrame *)
0|7|xul.dll|BuildTextRuns
0|8|xul.dll|nsTextFrame::EnsureTextRun(gfxContext *,nsIFrame *,nsLineList_iterator const *,unsigned int *)
0|9|xul.dll|nsTextFrame::ReflowText(nsLineLayout &,int,nsIRenderingContext *,int,nsHTMLReflowMetrics &,unsigned int &)

      1  stacks like
0|0|mozalloc.dll|mozalloc_abort(char const * const)
0|1|mozalloc.dll|mozalloc_handle_oom()
0|2|xul.dll|
0|3|xul.dll|
0|4|xul.dll|
0|5|mozjs.dll|FinalizeArenaList<JSObject_Slots2>
0|6|mozjs.dll|MarkAndSweep
0|7|mozjs.dll|GCUntilDone
0|8|mozjs.dll|JS_GC
0|9|xul.dll|
Keywords: crash
OS: Mac OS X → Windows XP
For the last week, here is a table that summarizes crash-stats:
            Top crasher           % of all crashes        % of crashes/ADU      
4.0b10pre       #10                    1.21%                   0.43%
4.0b9           #23                    0.5%                    0.08%
4.0b8           #36                    0.36                    0.03%

There are 5 times more crashes/ADU in 4.0b10pre than in 4.0b9.

More reports at:
http://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=mozalloc_handle_oom%28%29

Chris,
Can you give us a breakdown by the frame #4 component (Layout, Javascript, ...) in 4.0b8, b9, b10pre?
For instance, in 4.0b8: 10% in Layout, 15% in Javascript, ...
We don't really have good tools to search for patterns and modules lower in the stack, and I don't know if the sample size is big enough to be very accurate, but I looked at some data that I did have available it looks like there is quite a bit of shifting between the releases if these samples are reflective of the larger body of reports.

b8 was a lot of plugins, layout, cookies, gfx

b9 was FrameLayerBuilder, _cairo_d2d_fill, Image

b10 is FrameLayerBuilder, nsHtml5, nsHttpHeaderArray

here is what's in the 4th frame of stacks from a 20 report sample from 4.0b8

count  frame info

  10 0|3|xul.dll|nsPluginHost::ReadPluginInfo()
   2 0|3|xul.dll|HashMgr::load_config(char const *,char const *)
   1 0|3|xul.dll|nsXULElement::Create(nsXULPrototypeElement *,nsIDocument *,int,mozilla::dom::Element * *)
   1 0|3|xul.dll|nsIFrame::BuildDisplayListForChild(nsDisplayListBuilder *,nsIFrame *,nsRect const &,nsDisplayListSet const &,unsigned int)
   1 0|3|xul.dll|nsCookieService::GetCookieFromRow<nsCOMPtr<mozIStorageRow> >(nsCOMPtr<mozIStorageRow> &)
   1 0|3|xul.dll|nsCSSRuleProcessor::RulesMatching(AnonBoxRuleProcessorData *)
   1 0|3|xul.dll|nsBlockFrame::BuildDisplayList(nsDisplayListBuilder *,nsRect const &,nsDisplayListSet const &)
   1 0|3|xul.dll|mozilla::`anonymous namespace'::ContainerState::CreateOrRecycleThebesLayer(nsIFrame *)
   1 0|3|xul.dll|gfxFontGroup::MakeSpaceTextRun(gfxTextRunFactory::Parameters const *,unsigned int)
   1 0|3|xul.dll|

here is what's in the 4th frame of stacks from a 20 report sample from 4.0b9 

count  frame info

   6 0|3|xul.dll|mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &,mozilla::FrameLayerBuilder::Clip const &)
   3 0|3|xul.dll|nsTArray<unsigned long,nsTArrayDefaultAllocator>::AppendElements(unsigned int)
   3 0|3|xul.dll|
   2 0|3|xul.dll|_cairo_d2d_fill
   1 0|3|xul.dll|nsTArray<nsCounterNode *,nsTArrayDefaultAllocator>::AppendElements<nsCounterNode *>(nsCounterNode * const *,unsigned int)
   1 0|3|xul.dll|nsImageLoader::Load(imgIRequest *)
   1 0|3|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const &,nsDisplayItem::HitTestState *,nsTArray<nsIFrame *,nsTArrayDefaultAllocator> *)
   1 0|3|xul.dll|mozilla::FramePropertyTable::Set(nsIFrame *,mozilla::FramePropertyDescriptor const *,void *)
   1 0|3|xul.dll|imgFrame::Draw(gfxContext *,gfxPattern::GraphicsFilter,gfxMatrix const &,gfxRect const &,nsIntMargin const &,nsIntRect const &)
   1 0|3|xul.dll|TextRunWordCache::MakeTextRun(unsigned char const *,unsigned int,gfxFontGroup *,gfxTextRunFactory::Parameters const *,unsigned int)

and here is what's in a 20 report sample for 4.0b10 pre

count  frame info

   4 0|3|xul.dll|nsTArray<mozilla::FrameLayerBuilder::ClippedDisplayItem,nsTArrayDefaultAllocator>::AppendElements<mozilla::FrameLayerBuilder::ClippedDisplayItem>(mozilla::FrameLayerBuilder::ClippedDisplayItem
   2 0|3|xul.dll|nsCSSRuleProcessor::RulesMatching(AnonBoxRuleProcessorData *)
   1 7|3|xul.dll|nsHttpHeaderArray::SetHeader(nsHttpAtom,nsACString_internal const
   1 3|3|xul.dll|JSContextParticipant::Traverse(void
   1 16|3|xul.dll|nsHtml5TreeBuilder::comment(unsigned short *,int,int)
   1 13|3|xul.dll|nsHtml5Tokenizer::tokenizeBuffer(nsHtml5UTF16Buffer *)
   1 0|3|xul.dll|nsTArray<nsHtml5TreeOperation,nsTArrayDefaultAllocator>::MoveElementsFrom<nsHtml5TreeOperation,nsTArrayDefaultAllocator>(nsTArray<nsHtml5TreeOperation,nsTArrayDefaultAllocator>
   1 0|3|xul.dll|nsTArray<mozilla::FrameLayerBuilder::DisplayItemData,nsTArrayDefaultAllocator>::AppendElements<mozilla::FrameLayerBuilder::DisplayItemData>(mozilla::FrameLayerBuilder::DisplayItemData
   1 0|3|xul.dll|nsGenericHTMLElement::ParseAttribute(int,nsIAtom
   1 0|3|xul.dll|nsDisplayList::HitTest(nsDisplayListBuilder *,nsRect const
   1 0|3|xul.dll|nsComponentManagerImpl::ManifestXPT(nsComponentManagerImpl::ManifestProcessingContext
   1 0|3|xul.dll|mozilla::imagelib::RasterImage::SetSourceSizeHint(unsigned int)
   1 0|3|xul.dll|mozilla::imagelib::RasterImage::Init(imgIDecoderObserver *,char
   1 0|3|xul.dll|BuildTextRunsScanner::BuildTextRunForFrames(void *)
   1 0|3|xul.dll|
   1 0|3|mozjs.dll|js::Invoke(JSContext *,js::CallArgs const &,unsigned int)
I analyzed manually the latest 50 crashes in 4.0b10pre. Here is the breakdown by component:
Layout 32%  JavaScript 28%  XPCOM 12%  Gfx 8%  Parser 6%  Plugins 4%  Modules 4%  Netwerk 2%  XSLT 2%  XML 2%

So it is either a Layout or a JavaScript regression.
The regression range for the spike is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=c0e05d518f57&tochange=54184cfa6f0e
Possible layout culprits: bug 617860
Severity: normal → critical
blocking2.0: --- → ?
Component: General → Layout
QA Contact: general → layout
Version: unspecified → Trunk
Because of bug 627802 in Socorro, ADUs in 4.0b10pre were wrong, so comment 1 was wrong. For the last week, here is a table that summarizes crash-stats:
            Top crasher           % of all crashes        % of crashes/ADU      
4.0b10pre       #12                    1.13%                    0.13%
4.0b9           #23                    0.52%                    0.06%
4.0b8           #23                    0.47%                    0.04%
There are 2 times more crashes/ADU in 4.0b10pre than in 4.0b9.
As the Layout and JavaScript crashes count for half of crashes, the spike is probably due to a Layout issue that has an impact on JavaScript.
This appears to be actual OOM conditions which are now being caught by infallible TArray. We need to group these very precisely by the differening frame (frame 4), and get bugs filed on the particular top issues. In general, these are intentional aborts which protect us from memory corruption issue later on, but if we have variable-size allocations controlled by content, we should make them fallible and null-check if possible.
We should get a Socorro bug on file to update the SkipList with frames 1,2 and 3, then:
https://wiki.mozilla.org/Breakpad/SkipList
http://code.google.com/p/socorro/wiki/SignatureGeneration
Component: Layout → General
QA Contact: layout → general
Benjamin: During crashkill the team needs clarification on which skiplist you want to use - skip frames 1-3?

(In reply to comment #5)
> This appears to be actual OOM conditions which are now being caught by
> infallible TArray. We need to group these very precisely by the differening
> frame (frame 4), and get bugs filed on the particular top issues. In general,
> these are intentional aborts which protect us from memory corruption issue
> later on, but if we have variable-size allocations controlled by content, we
> should make them fallible and null-check if possible.
I think we should add

mozalloc_handle_oom
nsTArray_base<.*
nsTArray<.*

to prefixSignatureRegEx. We've done similar with all things nsCOMPtr related which I think has worked well.

bsmedberg, if this sounds good, I'll file a bug.
Also, for what it's worth, nsTArray is not on the stack more often than it is. This could be either because we really aren't using nsTArray, or because the nsTArray frames are getting inlined away. Though it sounds somewhat strange that the memory-allocating nsTArray functions would get inlined.
It is #7 top crasher in 4.0b10 and it still represents 0.14% of crashes/ADU over the last 3 days (see 4.0b10pre stats in comment 4).

> bsmedberg, if this sounds good, I'll file a bug.
It becomes urgent to file this bug.
OS: Windows XP → All
Summary: growing spike in crashes [@ mozalloc_handle_oom() ] on mozilla-central just after transition to 4.0b10pre → growing spike in crashes [@ mozalloc_handle_oom() ][@ mozalloc_handle_oom ] on mozilla-central just after transition to 4.0b10pre
sicking can you file?  I think bsmedberg is still on maternity leave, and we probably need to make faster progress on this top crash.
I grabbed a new, and larger, sample with 100 reports from jan 28.

top section of the report contains a summary of whats in the 4th frame, and the lower section of the report show a longer summary of all the stacks.

some of the top problems appear to be in

mozilla::`anonymous namespace'::ContainerState::ProcessDisplayItems(nsDisplayList const &amp;,mozilla::FrameLayerBuilder::Clip const &amp;)

html 5

      1 9|3|xul.dll|nsHtml5TreeBuilder::flushCharacters()
      1 9|3|xul.dll|nsHtml5StreamParser::DoDataAvailable(unsigned char *,unsigned int)
      1 18|3|xul.dll|nsHtml5TreeBuilder::flushCharacters()
      1 17|3|xul.dll|nsHtml5TreeBuilder::flushCharacters()
      1 14|3|xul.dll|nsHtml5StreamParser::DoDataAvailable(unsigned char *,unsigned int)
      1 13|3|xul.dll|nsHtml5TreeBuilder::createElement(int,nsIAtom *,nsHtml5HtmlAttributes *)
      1 11|3|xul.dll|nsHtml5TreeBuilder::endTag(nsHtml5ElementName *)
      1 10|3|xul.dll|nsHtml5Tokenizer::tokenizeBuffer(nsHtml5UTF16Buffer *)
      1 0|3|xul.dll|nsHtml5TreeOperation::Perform(nsHtml5TreeOpExecutor *,nsIContent * *)
      1 0|3|xul.dll|nsHtml5Tokenizer::tokenizeBuffer(nsHtml5UTF16Buffer *)
      1 0|3|xul.dll|nsTArray&lt;nsHtml5TreeOperation,nsTArrayDefaultAllocator&gt;::MoveElementsFrom&lt;nsHtml5TreeOperation,nsTArrayDefaultAllocator&gt;(nsTArray&lt;nsHtml5TreeOperation,nsTArrayDefaultAllocator&gt; &amp;)

js

      1 4|3|xul.dll|JSContextParticipant::Traverse(void *,nsCycleCollectionTraversalCallback &amp;)
      1 3|3|xul.dll|JSContextParticipant::Traverse(void *,nsCycleCollectionTraversalCallback &amp;)
      1 3|3|mozjs.dll|js::gc::MarkObject


css

     1 0|3|xul.dll|nsCSSRuleProcessor::RulesMatching(AnonBoxRuleProcessorData *)
      1 0|3|xul.dll|nsCSSRuleProcessor::RefreshRuleCascade(nsPresContext *)
      1 0|3|xul.dll|nsCSSFrameConstructor::ProcessChildren(nsFrameConstructorState &amp;,nsIContent *,nsStyleContext *,nsIFrame *,int,nsFrameItems &amp;,int,PendingBinding *)
      1 0|3|xul.dll|`anonymous namespace'::CSSParserImpl::ParseSelectorList(nsCSSSelectorList * &amp;,unsigned short)
      1 0|3|xul.dll|`anonymous namespace'::CSSParserImpl::ParseSelectorGroup(nsCSSSelectorList * &amp;)
      1 0|3|xul.dll|mozilla::css::Loader::ParseSheet(nsIUnicharInputStream *,mozilla::css::SheetLoadData *,int &amp;)

and more details are in the attachment
Attachment #508118 - Attachment is obsolete: true
None of these look serious enough to block.
blocking2.0: ? → -
Depends on: 630230
Depends on: 631607
Summary: growing spike in crashes [@ mozalloc_handle_oom() ][@ mozalloc_handle_oom ] on mozilla-central just after transition to 4.0b10pre → growing spike in crashes [@ mozalloc_abort(char const* const) ][@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ] (was [@ mozalloc_handle_oom() ][@ mozalloc_handle_oom ]) just after transition to 4.0b10pre
I filed bug 633119 to differentiate each kind of crashes.
Depends on: 626768
The spike appears to be related to the new drawing code, which got backed out.

2011020900 	92 - 100.000%
2011020800 	1 - 100.000%
2011020700 	3 - 100.000%
2011020600 	2 - 100.000%
2011020500 	3 - 100.000% 
2011020400 	1 - 100.000%
Depends on: 633445
Depends on: 633473
Keywords: topcrash
Depends on: 633903
Crash Signature: [@ mozalloc_abort(char const* const) ] [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ] [@ mozalloc_handle_oom() ] [@ mozalloc_handle_oom ]
This was opened due to a spike that appeared in 4.0 that seems to be gone. We also added some of these to the skip list so new bugs will/are be filed with different signatures further down the stack. I am going to resolve this invalid for now because the spike is gone and the bug isn't really actionable now. If people have issues, please comment and reopen. I don't think it's a top crash anymore.
Status: NEW → RESOLVED
Crash Signature: [@ mozalloc_abort(char const* const) ] [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ] [@ mozalloc_handle_oom() ] [@ mozalloc_handle_oom ] → [@ mozalloc_abort(char const* const) ] [@ mozalloc_abort(char const* const) | mozalloc_handle_oom() ] [@ mozalloc_handle_oom() ] [@ mozalloc_handle_oom ]
Closed: 8 years ago
Keywords: topcrash
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.