117153 - build crashes on startup with PAC enabled

Status: RESOLVED FIXED

(Core :: XPCOM, defect, P1)

Description

[Christopher Blizzard (:blizzard)]

This affects 0.9.7 builds as well as builds from the tip at least as of Dec 27,
2001.

If I have my pac file enabled, I will crash often (half the time) on startup. 
Here's a stack trace of both the UI thread (thread 1) and thread 6 which is
accessing the PAC file.

(gdb) thread 1
[Switching to thread 1 (Thread 1024 (LWP 26972))]#0  0x405eeb85 in __sigsuspend
    (set=0xbfffe250) at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
45
../sysdeps/unix/sysv/linux/sigsuspend.c: No such file or directory.
	in ../sysdeps/unix/sysv/linux/sigsuspend.c
Current language:  auto; currently c
(gdb) where
#0  0x405eeb85 in __sigsuspend (set=0xbfffe250)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:45
#1  0x4030c1c9 in __pthread_wait_for_restart_signal (self=0x40314f40)
    at pthread.c:969
#2  0x4030df09 in __pthread_alt_lock (lock=0x406f4fa0, self=0x0)
    at restart.h:34
#3  0x4030ad16 in __pthread_mutex_lock (mutex=0x406f4f90) at mutex.c:120
#4  0x40640fe8 in __libc_free (mem=0x8188c28) at malloc.c:3152
#5  0x401ad95e in PL_DHashFreeTable (table=0x402a8ca0, ptr=0x8188c28)
    at pldhash.c:69
#6  0x401ae30b in ChangeTable (table=0x402a8ca0, deltaLog2=1) at pldhash.c:468
#7  0x401ae411 in PL_DHashTableOperate (table=0x402a8ca0, key=0xbfffe524, 
    op=PL_DHASH_ADD) at pldhash.c:513
#8  0x401afd34 in GetAtomHashEntry (aString=@0xbfffe510) at nsAtomTable.cpp:276
#9  0x401afd73 in NS_NewAtom (aString=@0xbfffe510) at nsAtomTable.cpp:284
#10 0x4167f033 in nsCSSSelector::SetTag (this=0xbfffe650, aTag=@0xbfffe510)
    at nsCSSStyleRule.cpp:470
#11 0x41671eac in CSSParserImpl::ParseTypeOrUniversalSelector (
    this=0x426b7638, aDataMask=@0xbfffe5f0, aSelector=@0xbfffe650, 
    aParsingStatus=@0xbfffe5ec, aErrorCode=@0xbfffe740, aIsNegated=0)
    at nsCSSParser.cpp:1882
#12 0x416730b7 in CSSParserImpl::ParseSelector (this=0x426b7638, 
    aErrorCode=@0xbfffe740, aSelector=@0xbfffe650) at nsCSSParser.cpp:2347
#13 0x4167148b in CSSParserImpl::ParseSelectorGroup (this=0x426b7638, 
    aErrorCode=@0xbfffe740, aList=@0xbfffe6c0) at nsCSSParser.cpp:1598
#14 0x41670fd4 in CSSParserImpl::ParseSelectorList (this=0x426b7638, 
    aErrorCode=@0xbfffe740, aListHead=@0xbfffe700) at nsCSSParser.cpp:1498
#15 0x41670dd9 in CSSParserImpl::ParseRuleSet (this=0x426b7638, 
    aErrorCode=@0xbfffe740, 
    aAppendFunc=0x4166df5c <AppendRuleToSheet(nsICSSRule *, void *)>, 
    aData=0x426b7638) at nsCSSParser.cpp:1438
#16 0x4166eb90 in CSSParserImpl::Parse (this=0x426b7638, aInput=0x426b79a8, 
    aInputURL=0x426c1818, aResult=@0xbfffe8a4) at nsCSSParser.cpp:614
#17 0x4166acd5 in CSSLoaderImpl::ParseSheet (this=0x426b69e8, aIn=0x426b79a8, 
    aLoadData=0x426b7920, aCompleted=@0xbfffe8a8, aSheet=@0xbfffe8a4)
    at nsCSSLoader.cpp:943
#18 0x4166bce6 in CSSLoaderImpl::LoadSheet (this=0x426b69e8, aKey=@0xbfffe940, 
    aData=0x426b7920) at nsCSSLoader.cpp:1272
#19 0x4166cda7 in CSSLoaderImpl::LoadChildSheet (this=0x426b69e8, 
    aParentSheet=0x426b6cf0, aURL=0x426c1818, aMedia=@0xbfffea20, 
    aDefaultNameSpaceID=-1, aIndex=2, aParentRule=0x426b78ac)
    at nsCSSLoader.cpp:1573
#20 0x41670104 in CSSParserImpl::ProcessImport (this=0x426b6e08, 
    aErrorCode=@0xbfffebc0, aURLSpec=@0xbfffeac0, aMedia=@0xbfffea20, 
    aAppendFunc=0x4166df5c <AppendRuleToSheet(nsICSSRule *, void *)>, 
    aData=0x426b6e08) at nsCSSParser.cpp:1163
#21 0x4166fe88 in CSSParserImpl::ParseImportRule (this=0x426b6e08, 
    aErrorCode=@0xbfffebc0, 
    aAppendFunc=0x4166df5c <AppendRuleToSheet(nsICSSRule *, void *)>, 
    aData=0x426b6e08) at nsCSSParser.cpp:1122
#22 0x4166f8e2 in CSSParserImpl::ParseAtRule (this=0x426b6e08, 
    aErrorCode=@0xbfffebc0, 
    aAppendFunc=0x4166df5c <AppendRuleToSheet(nsICSSRule *, void *)>, 
    aData=0x426b6e08) at nsCSSParser.cpp:981
#23 0x4166eb66 in CSSParserImpl::Parse (this=0x426b6e08, aInput=0x426b72a8, 
    aInputURL=0x426b6900, aResult=@0x419730ac) at nsCSSParser.cpp:610
#24 0x4166acd5 in CSSLoaderImpl::ParseSheet (this=0x426b69e8, aIn=0x426b72a8, 
    aLoadData=0x426b6c10, aCompleted=@0xbfffede8, aSheet=@0x419730ac)
    at nsCSSLoader.cpp:943
#25 0x4166cfa2 in CSSLoaderImpl::LoadAgentSheet (this=0x426b69e8, 
    aURL=0x426b6900, aSheet=@0x419730ac, aCompleted=@0xbfffede8, aObserver=0x0)
    at nsCSSLoader.cpp:1612
#26 0x415a3534 in nsContentDLF::CreateInstance (this=0x426b68e8, 
    aCommand=0x412efe7a "view", aChannel=0x8198140, aLoadGroup=0x8160c00, 
    aContentType=0xbffff194 "text/html", aContainer=0x8160674, aExtraInfo=0x0, 
    aDocListener=0xbffff240, aDocViewer=0xbffff050) at nsContentDLF.cpp:187
#27 0x412bae4f in nsDocShell::NewContentViewerObj (this=0x8160650, 
    aContentType=0xbffff194 "text/html", request=0x8198140, 
    aLoadGroup=0x8160c00, aContentHandler=0xbffff240, aViewer=0xbffff050)
    at nsDocShell.cpp:3642
#28 0x412ba7ae in nsDocShell::CreateContentViewer (this=0x8160650, 
    aContentType=0xbffff194 "text/html", request=0x8198140, 
    aContentHandler=0xbffff240) at nsDocShell.cpp:3533
#29 0x412cedda in nsDSURIContentListener::DoContent (this=0x8160898, 
    aContentType=0xbffff194 "text/html", aIsContentPreferred=0, 
    request=0x8198140, aContentHandler=0xbffff240, aAbortProcess=0xbffff1e0)
    at nsDSURIContentListener.cpp:107
#30 0x411ed389 in nsDocumentOpenInfo::DispatchContent (this=0x8198030, 
    request=0x8198140, aCtxt=0x0) at nsURILoader.cpp:355
#31 0x411ecbb1 in nsDocumentOpenInfo::OnStartRequest (this=0x8198030, 
    request=0x8198140, aCtxt=0x0) at nsURILoader.cpp:226
#32 0x40b55764 in nsStreamIOChannel::OnStartRequest (this=0x8198140, 
    request=0x81982f4, context=0x0) at nsInputStreamChannel.cpp:470
#33 0x40be2734 in nsOnStartRequestEvent::HandleEvent (this=0x8193bf8)
    at nsRequestObserverProxy.cpp:161
#34 0x40b601c8 in nsARequestObserverEvent::HandlePLEvent (plev=0x8193bf8)
    at nsRequestObserverProxy.cpp:115
#35 0x4020d570 in PL_HandleEvent (self=0x8193bf8) at plevent.c:590
#36 0x4020dce1 in PL_ProcessEventsBeforeID (aSelf=0x80d1f20, aID=153)
    at plevent.c:1256
#37 0x4097f1b5 in processQueue (aElement=0x80d1f20, aData=0x99)
    at nsAppShell.cpp:464
#38 0x401ca06c in nsVoidArray::EnumerateForwards (this=0x80a8770, 
    aFunc=0x4097f188 <processQueue(void *, void *)>, aData=0x99)
    at nsVoidArray.cpp:660
#39 0x4097f1f8 in nsAppShell::ProcessBeforeID (aID=153) at nsAppShell.cpp:472
#40 0x40987fd2 in handle_gdk_event (event=0x41e57828, data=0x0)
    at nsGtkEventHandler.cpp:908
#41 0x40461d7f in gdk_event_dispatch () from /usr/lib/libgdk-1.2.so.0
#42 0x40495773 in g_main_dispatch () from /usr/lib/libglib-1.2.so.0
#43 0x40495d39 in g_main_iterate () from /usr/lib/libglib-1.2.so.0
#44 0x40495eec in g_main_run () from /usr/lib/libglib-1.2.so.0
#45 0x403b0333 in gtk_main () from /usr/lib/libgtk-1.2.so.0
#46 0x4097ee69 in nsAppShell::Run (this=0x80de088) at nsAppShell.cpp:349
#47 0x4091b6a5 in nsAppShellService::Run (this=0x80dd758)
    at nsAppShellService.cpp:302
#48 0x0805ab75 in main1 (argc=1, argv=0xbffff794, nativeApp=0x0)
    at nsAppRunner.cpp:1264
#49 0x0805b81b in main (argc=1, argv=0xbffff794) at nsAppRunner.cpp:1594
#50 0x405dc627 in __libc_start_main (main=0x805b614 <main>, argc=1, 
    ubp_av=0xbffff794, init=0x8055178 <_init>, fini=0x8064d64 <_fini>, 
    rtld_fini=0x4000dcc4 <_dl_fini>, stack_end=0xbffff78c)
    at ../sysdeps/generic/libc-start.c:129
(gdb) thread 6
[Switching to thread 6 (Thread 4101 (LWP 26978))]#0  0x4067b071 in
__libc_nanosleep () from /lib/i686/libc.so.6
(gdb) where
#0  0x4067b071 in __libc_nanosleep () from /lib/i686/libc.so.6
#1  0x4067aef1 in __sleep (seconds=300)
    at ../sysdeps/unix/sysv/linux/sleep.c:85
#2  0x0805bf84 in ah_crap_handler (signum=11) at nsSigHandlers.cpp:143
#3  0x4030ca85 in pthread_sighandler (signo=11, ctx=
      {gs = 47, __gsh = 0, fs = 0, __fsh = 0, es = 43, __esh = 0, ds = 43, __dsh
= 0, edi = 0, esi = 134872800, ebp = 1105157892, esp = 1105157804, ebx =
1081042612, edx = 0, ecx = 3, eax = 1081035648, trapno = 14, err = 6, eip =
1080296993, cs = 35, __csh = 0, eflags = 66054, esp_at_signal = 1105157804, ss =
43, __ssh = 0, fpstate = 0x41df5c30, oldmask = 2147483648, cr2 = 12}) at
signals.c:97
#4  <signal handler called>
#5  0x40640621 in chunk_alloc (ar_ptr=0x406f4b80, nb=24) at malloc.c:2878
#6  0x40640428 in __libc_malloc (bytes=16) at malloc.c:2811
#7  0x4072300d in __builtin_new (sz=16) from /usr/lib/libstdc++-libc6.2-2.so.3
#8  0x40247ba2 in nsXPIDLCString::PrepareForUseAsOutParam (this=0x41df60bc)
    at nsXPIDLString.cpp:207
#9  0x08062c94 in nsXPIDLCString::getter_Copies_t::operator char ** (
    this=0x41df5fe4) at ../../dist/include/string/nsXPIDLString.h:320
#10 0x40a108f4 in nsPrefBranch::GetComplexValue (this=0x81189c0, 
    aPrefName=0x41214000 "helpers.private_mime_types_file", aType=@0x8097040, 
    _retval=0x41df614c) at nsPrefBranch.cpp:316
#11 0x40a1e74a in nsPrefService::GetComplexValue (this=0x8118990, 
    aPrefName=0x41214000 "helpers.private_mime_types_file", aType=@0x8097040, 
    aValue=0x41df614c) at nsPrefService.h:57
#12 0x40a0e44e in nsPref::CopyUnicharPref (this=0x8118908, 
    pref=0x41214000 "helpers.private_mime_types_file", _retval=0x809fee8)
    at nsPref.cpp:395
#13 0x411fbaa9 in LookUpTypeAndDescription (aFileExtension=@0x41df623c, 
    aMajorType=@0x41df65fc, aMinorType=@0x41df655c, aDescription=@0x41df64bc)
    at ./unix/nsOSHelperAppService.cpp:277
#14 0x411ff4be in nsOSHelperAppService::GetFromExtension (this=0x81439a8, 
    aFileExt=0x8136a18 "pac", _retval=0x41df67bc)
    at ./unix/nsOSHelperAppService.cpp:1161
#15 0x411f8fab in nsExternalHelperAppService::GetTypeFromExtension (
    this=0x81439a8, aFileExt=0x8136a18 "pac", aContentType=0x426bb540)
    at nsExternalHelperAppService.cpp:1454
#16 0x411f96b4 in nsExternalHelperAppService::GetTypeFromFile (this=0x81439a8, 
    aFile=0x41e5d638, aContentType=0x426bb540)
    at nsExternalHelperAppService.cpp:1589
#17 0x40b4e636 in nsFileIO::Open (this=0x426bb5b0, contentType=0x426bb540, 
    contentLength=0x426bb574) at nsFileStreams.cpp:220
#18 0x40b51a3f in nsFileTransport::Process (this=0x426bb520, progressSink=0x0)
    at nsFileTransport.cpp:671
#19 0x40b518b4 in nsFileTransport::Run (this=0x426bb520)
    at nsFileTransport.cpp:638
#20 0x40213ac9 in nsThreadPoolRunnable::Run (this=0x8193800)
    at nsThread.cpp:904
#21 0x40211475 in nsThread::Main (arg=0x8193818) at nsThread.cpp:120
#22 0x402e5352 in _pt_root (arg=0x8193898) at ptthread.c:214
#23 0x40309c6f in pthread_start_thread (arg=0x41df6be0) at manager.c:284

Comment 1

[Alec Flett]

ugh, this might be (or probably is) related to the bug to move nsAutoConfig out
of prefs...:(

Comment 2

[Christopher Blizzard (:blizzard)]

Has anyone else been able to reproduce this?  This is a 0.9.8 bug and we're not
shipping with it this time.

Comment 3

[Brian Nesse (gone)]

If this shows up in the 0.9.7 builds then it is not related to the autoconfig 
landing (bug 89137)... that landed post 0.9.7.

Chris, are you certain this is PAC specific? The stack in your thead 1 trace 
looks similar to a crash on launch that I'm currently seeing on today's OS9 
commercial build. I have no pac file.

Comment 4

[Christopher Blizzard (:blizzard)]

It's triggered by reading proxy information from a PAC file.  I don't know if
it's caused by the PAC code directly but if I turn it off then I don't get
crashes on a regular basis.

Comment 5

[basic]

tingley: could the async reading of the PAC file in the nsPAC.js component be
causing this?

Comment 6

[Chase Tingley]

If it is, it's happening in a very roundabout way.

blizzard, I assume thread 6 is the one that's crashing?  the thread's just
executing generic file channel code, and it's not immediately clear why it's
dying.  I have some memory that your machine is multi-processor?

fwiw, I've never seen this crash.

thread 6 appears to be doing a helper app lookup on ".pac"; cc'ing bz, who (I
think) knows something about helper apps.

Comment 7

[Christopher Blizzard (:blizzard)]

I have an SMP machine, yeah.

Comment 8

[Boris Zbarsky [:bzbarsky]]

Could this code be running before the pref service is properly initialized or 
something like that?

Comment 9

[Christopher Blizzard (:blizzard)]

I've got this whittled down to a single checkin.  It's dbaron's:

http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=&file=mozilla%2F&filetype=match&who=&whotype=match&sortby=Change+Size&hours=2&date=explicit&mindate=11%2F27%2F2001+21%3A21%3A00&maxdate=11%2F27%2F2001+21%3A23%3A00&cvsroot=%2Fcvsroot

If I back out my tree to right before that checkin I was able to start the
browser 60 times without a single crash.  With that code in my tree I crash
often on startup.

Comment 10

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

What happens if you run *with* the changes in that patch that are in
mozilla/string and mozilla/js/src/xpconnect/src/, but without all the other
changes in that patch?

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(That is, if you run having backed out the changes outside of string and
xpconnect, as if I checked in the string/xpconnect stuff first and the other
stuff later.)

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

And could you send me either a URL or a sample PAC file so I can test?

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

attachment 34902 [details] from bug 79057 should serve as an example PAC file...

Comment 14

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: debug log for 2 crashes I caught in the debugger

I caught 2 crashes in the debugger.  One other was roughly the same.  One other
was much earlier, and in the PromiseFlatString called from NS_NewAtom.

There are a few interesting(?) similarities between these 2 problems.  (I was
actually looking in the debugger at the refcount assertion, which was the first
sign of anything wrong, although in fact it would probably lead to a leak
rather than a crash.)  In both cases, it was in the destructor of an
nsXPIDLCString that had been assigned into once using getter_Copies and read
once using implicit conversion to |const char*|.  In both cases, mDataStart was
valid, mStorageLength was valid (0, correct before RecalculateBoundaries) but
mDataEnd and mFlags were corrupted (the former to garbage, the latter to 0,
forcing the refcount to 0).

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

The other crashes I've seen were much weirder, though, and show signs of memory
corruption, I think.  I don't think the known threadsafety problems are at fault
here, though.

Comment 16

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'm less sure it's not the threadsafety problem -- deleting the shared empty
buffer handle could easily lead to corruption of whatever is allocated at it's
place, or, more likely, things pointed to by that memory.  Of course, after
about 15 tries I've been unable to get this in the debugger again (partly
because gdb is useless after a crash, so I have to get an assertion).

Comment 17

[Christopher Blizzard (:blizzard)]

I backed out all of the non-string changes in my build and I'm still seeing the
crash.  So it looks like the string changes are the culprit.  I'm just double
checking again...

Comment 18

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I just (after figuring out that I've been using a gdb I compiled a year or so
ago that happened to be better at the time) got a look at the empty buffers
(since I added global variables that point to them as well, for debugging), and
found corruption:

(gdb) p *nsXPIDLCString_emptyBuffer
$2 = {<nsBufferHandle<char>> = {
    mDataStart = 0x3f3f3f <Address 0x3f3f3f out of bounds>, 
    mDataEnd = 0x80b1d10 ""}, mFlags = 33554433, mStorageLength = 1}
(gdb) p *nsXPIDLString_emptyBuffer
$3 = {<nsBufferHandle<short unsigned int>> = {mDataStart = 0x81f32a8, 
    mDataEnd = 0x81f32a8}, mFlags = 33554433, mStorageLength = 1}

That mFlags is 0x02000001, but the mDataStart of the first one is corrupted --
perhaps it's hit a refcount of 0 and has now bounced back up to 1.

Comment 19

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Attachment: proposed patch

I was able to start, with PAC enabled, 10 times in a row using this patch.  I
might need to make some similar changes to the xpconnect string class -- I'll
need to check.

Comment 20

[jag (Peter Annema)]

Comment on attachment 64951 [details] [diff] [review]
proposed patch

sr=jag

Comment 21

[Christopher Blizzard (:blizzard)]

I was able to start up mozilla 40 times with no crashes after applying this patch.

SHIP IT.

Comment 22

[Alec Flett]

Comment on attachment 64951 [details] [diff] [review]
proposed patch

r=alecf
wow, I had no idea there was this much plumbing behind that

Comment 23

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Fix checked in 2002-01-15 19:08 PST.

Comment 24

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

*** Bug 116311 has been marked as a duplicate of this bug. ***