106009 - PAC instantiation hangs Regxpcom Solaris nightly build packaging process

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Nathan Bidwell]

As I write this, the latest Solaris nightly build on ftp.mozilla.org is from
10/15/2001.  That was a week ago.  (Not to mention that that build has a rather
broken mail client for me...)  Is this intentional?

Comment 1

[Jim Crumley]

Confirming and changing product to mozilla.org.

CCing [e-mail] in hopes that more info might be out there.

Related to bug 105981 or 105988?

Comment 2

[hacker formerly known as [e-mail]]

Nope, not related to those bugs.  There hasn't been a sol26 nightly build log
since Oct 15, which is weird.   It's still in the crontabs on granite & aesir is
up. Running the nightly script by hand to see what it turns up.

Comment 3

[Jon Granrose]

the 8am builds fail because IC is messing with the network and cvs hangs.  the
8pm builds fail because the cvs process from 8am is still running.  linux
finishes because it starts at 4am before they start breaking things.

Comment 4

[Jim Crumley]

Well, I'm posting this with Solaris build 2001102222, so a nightly build was
made last night. 

Comment 5

[hacker formerly known as [e-mail]]

Marking fixed (for lack of IC_screwed_us resolution).

Comment 6

[Nathan Bidwell]

And now there's an even newer nightly build up, so it seems everything is
working correctly.  Thank you.  Now back to my regularly scheduled testing.

BTW, what/who is IC?

Comment 7

[Jim Crumley]

Reopening becaause Solaris nighlies aren't showing up again.  The last available
build seems to be 2001110210

Comment 8

[hacker formerly known as [e-mail]]

John, aesir needs to be resurrected after the network massacre so that solaris
nightlies can live again.

Comment 9

[Peter O'Shea]

In addition to Solaris builds, source code isn't getting put onto the ftp server.

Comment 10

[Jon Granrose]

source balls are on branch, sol26 builds are on aesir, both down right now. 
antitux is supposed to get all our unix systems up by COB Friday so source
tarballs and sol26 builds should start showing up Saturday morning at the latest.

Comment 11

[Jim Crumley]

Closing since Solaris nightlies and source are both back on ftp.  Thanks!

Comment 12

[Jim Crumley]

Re-opening because the Solaris build on ftp is 2001122110.

Happy holidays!

Comment 13

[Jon Granrose]

someone broke the solaris build in nsPluginModule.cpp a while back.  I don't
know why the tinderboxen are green... could be someone hacked configure, it's an
official-only problem, or a parallel build problem, or ???

Comment 14

[Jon Granrose]

btw - you shouldn't keep reopening the same bug.  this is a completely different
problem from the original reported bug so it makes things confusing.  In the
future, should the Solaris builds fail to appear on ftp.mozilla.org, it should
be a new bug since it may or may not be the same problem.

Comment 15

[hacker formerly known as [e-mail]]

Tinderboxes are green because they use a) a more recent version of gcc
(speedracer's 2.95.3 vs aesir's 2.7.2.1) or b) Forte (nebiros).  Since we've
dropped support for gcc 2.7.2.x, we should upgrade the compiler on aesir.

Comment 16

[Jim Dunn]

John what is the plugin problem on solaris?
There is currently an open bug that is tracking
issues on a variety of platforms
http://bugzilla.mozilla.org/show_bug.cgi?id=106806

if this is also a solaris problem, then this bug
should be updated and we should probably also make
the platform to ALL unix, since linux suffers from
this as well (the .so.1 issue)

Comment 17

[hacker formerly known as [e-mail]]

*** Bug 117712 has been marked as a duplicate of this bug. ***

Comment 18

[Jon Granrose]

reassigning to asasaki since antitux is busy with other work right now.

Aki - Can you upgrade gcc on aesir to 2.95.3?  You'll need to coordinate with
lpham to make sure the build automation is looking in the right place to find
the new compiler once it's in place.  Thanks.

Comment 19

[Jon Granrose]

actually reassigning this time...

Comment 20

[Aki Sasaki]

Installed in /opt/gcc-2.95.3 which is softlinked to /opt/gcc (so you shouldn't
have to change anything in the env).  Old /opt/gcc moved to /opt/gcc-2.95.2
which can be removed once everything is working.

There are a lot of old cltbld processes on aesir... should I kill those?

Comment 21

[lpham]

Yes, please kill them.  Look like they are the old builds.  Thanks.  Loan

Comment 22

[GavinS]

There is now a nightly build for Jan 7 - so thanks!

But, the tar file is only 3.5MB :(

I've raised bug#118701 for this.

Comment 23

[Aki Sasaki]

Hm, looks like it's *building* fine (able to run the package from aesir), but
regxpcom is hanging, and it doesn't get past the packaging phase unless I kill
that...  Want to rerun it but I may hit the 8pm build...?  Meanwhile, there's a
new build up...

Comment 24

[Aki Sasaki]

Again this is the case... build finishes, regxpcom hangs indefinitely after the
"registering smime account manager extension" line.  Once I kill regxpcom, the
rest of the packaging and the push to the ftp site happens.

Anyone have any idea why regxpcom might be hanging on aesir?

Comment 25

[Jon Granrose]

IIRC regxpcom is hanging in smime, right?  Paste the regxpcom output into the
bug, then check lxr or ask on #mozilla for who's been doing smime and regxpcom
work and cc them here to get their input.  This will probably need to be
reassigned to an engineer to fix one or the other.

Comment 26

[hacker formerly known as [e-mail]]

truss output showed that regxpcom was hanging in poll().  It appeared to occur
after all of the components have been registered and the components.reg file had
already been created.

Comment 27

[Aki Sasaki]

 27077  ./run-mozilla.sh ./regxpcom
 27078  *** Registering -venkman handler.
 27079  *** Registering -chat handler.
 27080  *** Registering x-application-irc handler.
 27081  *** Registering irc protocol handler.
 27082  *** Registering smime account manager extension.
 27083  Terminated

Comment 28

[Aki Sasaki]

dougt, kaie -- what are your thoughts on regxpcom hanging after smime
registration? TIA.

Comment 29

[Doug Turner (:dougt)]

could you attach some stacktraces of the threads (probably just one thread)
involved?

Comment 30

[Aki Sasaki]

Attachment: output of `rm component.reg; truss ./regxpcom 2>&1 | tee > regxpcom.log`

output from truss, sleeping in poll().

Comment 31

[Kai Engert [:KaiE:]]

I think that output in comment 27 does not give a hint that it could have
something to do with smime. From what I have seen on Unix optimized builds,
that's always the exact output of the first run of a new build.

Comment 32

[Aki Sasaki]

Probably true, but it doesn't get to the "terminated" bit until I kill the
regxpcom process, which could be many hours after the smime line appears in the
log...

Comment 33

[Kai Engert [:KaiE:]]

Aki, to find out whether it is indeed a problem with the smime extension, or
something else, could you please do the following?

As a first test, when the build has finished, just remove
  mailnews/extensions/smime/build/libmsgsmime.so

Another test, you could add another debugging output line to that module.
In
  mailnews/extensions/smime/src/smime-service.js
locate
  SMIMEModule.registerSelf =
  function (compMgr, fileSpec, location, type)
  {
    dump("*** Registering smime account manager extension.\n");
    ....
  }

And add some more dump output lines. For example, replace that complete function
with:

SMIMEModule.registerSelf =
function (compMgr, fileSpec, location, type)
{
  dump("*** Registering smime account manager extension.\n");
  compMgr =
compMgr.QueryInterface(Components.interfaces.nsIComponentManagerObsolete);
  dump("*** smime 2.\n");
  compMgr.registerComponentWithType(SMIME_EXTENSION_SERVICE_CID,
                                      "SMIME Account Manager Extension Service",
                                      SMIME_EXTENSION_SERVICE_CONTRACTID, fileSpec,
                                      location, true, true, type);
  dump("*** smime 3.\n");
  catman =
Components.classes["@mozilla.org/categorymanager;1"].getService(nsICategoryManager);
  dump("*** smime 4.\n");
  catman.addCategoryEntry("mailnews-accountmanager-extensions",
                            "smime account manager extension",
                            SMIME_EXTENSION_SERVICE_CONTRACTID, true, true);
  dump("*** smime 5.\n");
}

If you build again and start, if we see the line with "*** smime 5", I think it
can't be the smime extension.

Comment 34

[Aki Sasaki]

Ok, done.  Looks like it's regxpcom =)
dougt -- do you want ownership of this bug?  Or do you have recommendations as
to who would be best able to fix it?
thanks.

Comment 35

[Aki Sasaki]

reassigning.

Comment 36

[Doug Turner (:dougt)]

does someone with a sun build want to look at this.  pavlov, do you have a uild
that I could peek at?

Comment 37

[Doug Turner (:dougt)]

over to pavlov.  

Comment 38

[Jim Crumley]

Since Solaris builds are back and working, the summary should involve the
problem that is left.

Comment 39

[Dan Price]

There have been no new Solaris nightlies for 5 days now....
so I think they aren't working again (or whatever workaround was
in place has stopped working).

Comment 40

[Nathan Bidwell]

I'm assuming that this is still the bug holding up nightly builds.  If so, could
someone please massage things by hand for a build newer than 1-15-2002?  Thank you.

Comment 41

[timeless]

*** Bug 122813 has been marked as a duplicate of this bug. ***

Comment 42

[Dave Booth]

My work firewall stops me pulling from CVS and I dont have the time to pull
regular tarballs so building it myself is out.
I'm sure I'm not alone in being dependent on nightlies for my mozilla testing so
I'm guessing theres probably quite a few solaris issues going unnoticed until
the nightlies come back simply by virtue of there being fewer users out there
running the latest codebase.
The longer this goes on the bigger a deal it gets. Any feedback at all on
progress to a fix would be welcome.

Comment 43

[Christiaan Fluit]

I just found out that there is a new nightly build available in
http://ftp.mozilla.org/pub/mozilla/nightly/latest/, with build date 2002013122.

At first it complained about not being able to find run-mozilla.sh, so I just
copied the file from the previous nightly build (2002011510) and now it runs
beautifully :)

Comment 44

[GavinS]

The missing run-mozilla.sh is bug#122942, now fixed.

Thanks to whoever did the manual update of the Solaris nightly (or fixed the
problem) - now I can play with the new Page Info stuff :)

Comment 45

[hacker formerly known as [e-mail]]

Before anyone gets too happy, it was the missing run-mozilla.sh that caused the
nightly to get delivered.  Since run-mozilla.sh didn't exist, regxpcom could not
run and then hang. 

Comment 46

[Peter O'Shea]

With all due respect, why is this bug being "futured"?  Do the drivers not care
about testing on Solaris?  Yes, it's a minority platform, but so are SGI IRIX
and HPUX, both of which have up-to-date nightlies.  

Comment 47

[Jim Crumley]

Even if the root problem with Regxpcom isn't worth the effort to fix at this
point, it seems like it should be possible to make a work around that would
allow nightlies to be distibuted. It seems like an ugly hack to the build
process like automatically killing the Regxpcom process (or removing
run-mozilla.sh which allowed 2002013122 to be built) would work.

Comment 48

[Jim Dunn]

Or just turn off --enable-crypto on the solaris nightly's to see if 
that fixes it... 

--disable-crypto turns off MOZ_PSM which turns off BUILD_SMIME
in mozilla/mailnews/extensions/Makefile.in

All you have to do is set BUILD_PSM="FALSE" in
ns/build/unix/verification/seamonkey-build
in the sol26 stanza... line:

if this fixes your problem, then someone needs to debug
--enable-crypto and smime in the sol26 stanza.  
But in the meantime there will be nightly builds.

Comment 49

[Kai Engert [:KaiE:]]

> Or just turn off --enable-crypto on the solaris nightly's to see if 
> that fixes it... 


I thought Aki confirmed in his comment 34 that it is NOT the crypto component?

Comment 50

[Jim Dunn]

I am not saying it is a crypto issue...
I am saying if we want to get rid of smime from 
the build... the quickest and easiest way to do 
that is to turn off PSM in the nightlies...

Since no PSM, no smime... and then regxpcom
CAN'T have an issue trying to load it.

btw that was line 869 for doing so.

Then whoever is the champion of sol26 should
figure out what is going on... just like I
do when the darn hpux nightlies mess up.

Comment 51

[Dave Booth]

Whatever the issue is, whether its crypto or something else, this still should
NOT be futured. Sure, it doesnt have the visibility of the wintel or linux
platforms but one of the main drivers for adopting a solution like Mozilla is
that it is truly cross-platform. Inhibiting testing on a major unix platform
does not bode well for this continuing. I wonder how many solaris bugs will go
unreported between now and the 0.9.9 release if the nightlies remain hosed? Do
we really want to suddenly see them all show up at that point rather than have
them reported and fixed along the way from nightly trunk builds?

Comment 52

[Jim Dunn]

I am not saying FUTURED... I am trying to get you
a nightly build.  Granted I am suggesting turning
off crypto to get you that (so you can test everything
else).  If I am reading this correctly you guys
(who care about solaris) haven't had a nightly
build in like forever.  

I will shut up, I don't care...
I don't care if solaris nightly builds work or not.
I don't care if crytpo is on or not.
I was just trying to suggest a way to get 
nightlies going again and to narrow down the
problem and not leave it to AKI who hasn't touch
a solaris build in "like forever".

un-ccing myself, do whatever you want.

Comment 53

[piers]

Don't know if this is relevent, but nebiros SunOS/sparc 5.7 Clobber seems to have been orange for ages. Is this the same problem? 

Also, i386 Solaris 2.6 nightlies seem to be being built fine, it's just the sparc ones (if that helps at all).

Comment 54

[Jon Granrose]

Pav - if there is too much on your plate right now to work on this bug, is there
anyone else that could take a stab at it in the meantime?

Comment 55

[Jim Dunn]

We are going to try my suggestion for turning off BUILD_PSM
in the sol26 builds.  We are only going to do this for
this weekend only.  Hopefully we will get nightlies (remember
they won't have PSM or smime) and then on Mon we will turn
it back on.  This will help us narrow down the issue

Comment 56

[hacker formerly known as [e-mail]]

Umm, I think you're barking up the wrong tree with the PSM issue (but feel free
to prove me wrong).  regxpcom is hanging at the end of its run.  components.reg
has already been written out correctly.  Removing smime from the components dir
does not fix the hanging problem (tested manually on the day I ran truss). 

Does anyone know what is being poll'ed?

Comment 57

[Stuart Parmenter]

er, i'm sorry.  i'm not sure why this bug is assigned to me.
-> cls

Comment 58

[hacker formerly known as [e-mail]]

So, this is weird.  Regxpcom works fine in a standalone xpcom build on sheep. 
If I build all of Mozilla (except crypto), I see the hang but according to the
truss log, it's not hanging in poll any longer.  It appeared to be hanging while
processing some of the uconv libs.

I used the following build options:
--enable-extensions=default,irc
--without-system-nspr
--without-system-zlib
--without-system-jpeg
--without-system-png
--without-system-mng
--disable-debug
--enable-optimize
--disable-tests

Comment 59

[hacker formerly known as [e-mail]]

With a debug build, I'm seeing the same hang when building all of Mozilla minus
PSM.  The trace shows that the poll() is coming from necko.

(gdb) bt 
#0  0xfee9990c in _poll () from /usr/lib/libc.so.1
#1  0xfef1b22c in poll () from /usr/lib/libthread.so.1
#2  0xff08847c in PR_Poll (pds=0xcd138, npds=1, timeout=3500000)
    at ../../../../../mozilla/nsprpub/pr/src/pthreads/ptio.c:3963
#3  0xfdabd924 in nsSocketTransportService::Run (this=0xad068)
    at ../../../../mozilla/netwerk/base/src/nsSocketTransportService.cpp:469
#4  0xff224ae8 in nsThread::Main (arg=0x81500)
    at ../../../mozilla/xpcom/threads/nsThread.cpp:120
#5  0xff08a600 in _pt_root (arg=0xa6308)
    at ../../../../../mozilla/nsprpub/pr/src/pthreads/ptthread.c:214
(gdb) 

Stepping thru gdb shows that the AutoRegister() call returned without any errors
(ret = 0).  The hang occurs during XPCOM shutdown.   Or more specifically, after
stepping thru NS_XPCOMShutdown, it's hanging in nsTimerImpl::Shutdown().  It
appears to spawn 2 more LWP threads when this occurs.  One of those threads is
the one shown in the stacktrace above.  The extra threads are spawned when
mThread->Join() is called from TimerThread::Shutdown() .

Pavlov, back to you.

Comment 60

[Igor Nekrestyanov]

Could someone clarify which version of Solaris has the problem?

I am able to reproduce regxpcom hang on one of Solaris 8 boxes
(and after rempval of components.reg problem can be repeated)
but it is not reproducible on several others Solaris 7/8/9 boxes 
(note: i am testing *same* build shared over NFS)

This leads me to idea that problem may be solved by instation of appropriate
solaris patches. Did anyone try to investigate this?

I am not sure which patches are necessary to fix the problem 
but i i would recommend to try patch 106541 for solaris 7
 
http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=fpatches%2F106541&zone_32=libc.so.1

(it contains fix for bug 
4207080 hang in poll, application does not get notified of data on stream head)

For solaris 8 patch 108991 may be usefull.
It also has libc.so fixes and it is one of installed patches on 
system that does not have problem and too old version of this patch is installed 
on system that has the problem.

 

Comment 61

[Jon Granrose]

I wouldn't be surprised if our build system was in need of some patching.

Aki - can you check the patch status on the system, and make sure it's up the
latest and greatest patch cluster, as well as the patches mentioned above?

cls - do you know of any reason we shouldn't upgrade, or any particular patches
we should avoid?

Comment 62

[hacker formerly known as [e-mail]]

I've heard that the very latest patch cluster from Sun introduces some
instabilities.  Pavlov and/or Roland would know specifically which one.

Comment 63

[Brendan Eich [:brendan]]

comment 59 makes no sense to me -- why would Join spawn threads?  cc'ing wtc.

/be

Comment 64

[Roland Mainz]

No clue what's going on.
Both Solaris 2.7+latest patches and Solaris 2.8+latest patches (except Xsun
patch 108652-47, we are still using rev -46) are working here...

Comment 65

[Aki Sasaki]

this is solaris 2.6.  downloading the latest recommended patches... which are
from 2/5/02.  i've had decent luck with the recommended patch bundles, so i'll
install these and just keep an eye out for news on any bad patches.

Comment 66

[Wan-Teh Chang]

Brendan asked:
> comment 59 makes no sense to me -- why would Join spawn threads?
> cc'ing wtc

I have no idea either.  Sorry.

Comment 67

[Aki Sasaki]

patch cluster installed, aesir rebooted.  we'll see if that fixes the 8pm build.

Comment 68

[Aki Sasaki]

regxpcom is still hanging and the recommended patch cluster had a libc fix,
can't find anything else in the 2.6 patchreport about it.

Comment 69

[Dan Price]

16 nights and new nightly build... could someone at least manually push one?

Comment 70

[Dan Price]

Grr.  I meant: 16 nights and *no* new nightly build.  Too many chocolate cookies
for me today.

Comment 71

[Roland Mainz]

Can't we setup another machine for creating Solaris 2.7 or 2.8 nighly tarballs
build with Sun Workshop ?

Comment 72

[Roland Mainz]

Comment on attachment 64581 [details]
output of `rm component.reg; truss ./regxpcom 2>&1 | tee > regxpcom.log`

Can someone provide a log from a hang woth `rm component.reg; truss -u ::
./regxpcom 2>&1 | tee > regxpcom.log`, please ?

Comment 73

[Aki Sasaki]

killed regxpcom, should be another package available on the site.
there is no -u option available in our version of truss... did you want the same
output, but more recent, or different output?
also, I believe I accidentally added the ">" to the tee comment, which shouldn't
be there... ignore it.

Comment 74

[Jim Crumley]

Thanks for the new nightly build.  Any chance of putting "kill regxpcom" in a
cron job?

Comment 75

[Jon Granrose]

no.  this bug has to be fixed.

Comment 76

[Igor Nekrestyanov]

roland: even on non-hanging node truss -u results in 500M+ log file.
         on node with problems it never stop to grow.
         
Back to the idea about solaris patches - I tried one more solaris 2.8 system 
that also did not have the regxpcom hang. However, it does have 
strict subset of patches installed on node with problems :(
Therefore either problem is introduced in one of additional patches
or it is somethere else in the environment.

Comment 77

[piers]

There's a new Solaris build:

2002-02-25-21-trunk/mozilla-sparc-sun-solaris2.6.tar.gz

but it doesn't start because of bug 127817. If 127817 is related to security code
as suggested in 127817 comment #3, does it mean that it's security stuff that's 
stopping Solaris builds normally?

Comment 78

[Dave Booth]

May be a red herring, but take a look at the gzipped truss output file I
attached to bug 129567 - Is this related? If it looks similar, then maybe we can
compare patch revs or something... see if we can find a patch that if applied
causes the problem and can be backed out to make it go away?

Comment 79

[Jim Crumley]

Any possibility of manually getting another nightly Solaris build uploaded (or
at least deleting the current broken one)?  The most recent Solaris nightly is
still the build from 20020225, which is broken due to bug 129749.  We've had a
couple of duplicates of that long since fixed bug because that build is the only
Solaris nightly available.

Comment 80

[Tobias Oetiker]

Attachment: install script we use for building 0.9.9 on solaris 8 box with which produces a hanging regxpcom

we build with gcc (2.95.3).

Comment 81

[Tobias Oetiker]

Attachment: truss of a hanging regxpcom run created with truss -f ./regxpcom the kill at the end is because I ressed control c

I have comperssed the file because it is reather large

Comment 82

[Aki Sasaki]

not sure if my attempt earlier in the week to get a new build up worked well or
not (killed all the regxpcom procs and i think the various build procs
interfered with each other), but there's one from today.

Comment 83

[Henrik Johansson]

We are building RPM:s on FreeBSD 4.3, RedHat 7.1 & 7.2 and Solaris 2.6, 7 & 8
and I have seen the problem with the hanging regxpcom many times on Solaris. 
I have been starting regxpcom thru truss and strace and got huge logfiles, so if 
someone are interested let me know. 
Workaround:
Our SPEC-file (RPM) places this script in the '.../mozilla/dist/bin'-directory. 
I tested it today, when building 0.9.9, regxpcom ran from 7 to 42s on FreeBSD, 
RedHat and Solaris 2.6, and ended normally. 
On Solaris 7 & 8 it was killed after the timeout and ran about 3s the second
time. 
#!/app/cueshell/bin/cueshell  # This is a bash-alias
dist_bin=`dirname $0`
MOZILLA_FIVE_HOME=$dist_bin
LD_LIBRARY_PATH=$dist_bin:$LD_LIBRARY_PATH
export MOZILLA_FIVE_HOME LD_LIBRARY_PATH
case `uname -s` in 
  SunOS)
    echo "`date`: Starting regxpcom"
    ( $dist_bin/regxpcom; echo "`date`: regxpcom done.") &
    waiting=0
    while [ $waiting -lt 1800 ]; do
      if ps -p $! >/dev/null ; then 
        waiting=`expr $waiting + 30`
        sleep 30
        echo "`date`: Waited $waiting seconds for regxpcom"
      else
        echo "`date`: Waiting done."
        waiting=1800
      fi
    done
    if ps -p $! >/dev/null ; then 
      echo "`date`: Kills regxpcom "
      /usr/sbin/fuser -k $dist_bin/regxpcom
      echo "`date`: Restarting regxpcom"
       $dist_bin/regxpcom; echo "`date`: regxpcom done." 
    fi 
    ;;
  *)
    echo "`date`: Starting regxpcom"
     $dist_bin/regxpcom; echo "`date`: regxpcom done." 
    ;;
esac
$dist_bin/regchrome 
touch $dist_bin/chrome/user-skins.rdf $dist_bin/chrome/user-locales.rdf

Comment 84

[Kenneth Herron]

I've been troubleshooting this using the 20020315xx nightly, and I think I have
some useful information.

First of all, if components/nsProxyAutoConfig.js is removed from an installed
copy of mozilla, then regxpcom will run to completion and exit as it should.
However, regxpcom isn't hanging while registering this component; it's hanging
in the call to NS_ShutdownXPCOM() just before regxpcom exits.

It appears that nsProxoyAutoConfig.js causes an nsDNSService thread to be
created, which in turn creates a TimerThread. Later at shutdown time, xpcom
tries to kill the timer thread, but it isn't dying.

I'm going to attach a copy of the /usr/proc/bin/pstack output for a well-hung
regxpcom instance. You'll note the following:

1) thread #1 is performing NS_ShutdownXPCOM() and is waiting for a _thrp_join()
call to complete. This is actually a pthread_join() call in the source. I think
the '6' in the _thrp_join() argument list means thread #6.

2) lwp #1/thread #6 is within a TimerThread::Run call, blissfully waiting for a
call to pthread_cond_wait() to complete.

3) thread #5 is within a nsDNSService::Run call. According to truss, thread 5
was spawned by thread 4, which is inside an nsSocketTransportService::Run call.

I have trusses from running regxpcom with and without the proxy autoconfig
component present.  When it's not present, regxpcom never gets beyond four
threads; #5 and #6 are never created. The trusses are quite large so I won't
attach them.

Comment 85

[Kenneth Herron]

Attachment: Call stack for each thread within a hung regxpcom process.

Comment 86

[timeless]

my bet is that the problem is:
235                  var PacMan = new nsProxyAutoConfig() ;

Comment 87

[Mike Shaver (:shaver emeritus)]

I doubt it, since that will just call this nothing function:
55   function nsProxyAutoConfig() {};

Is it possible that some other component is causing network activity, which is
in turn causing the proxyautoconfig stuff to get kicked off?

If that's the case, we probably need regxpcom to do more mozilla-like things in
its shutdown process.  Cc:ing Jud, because embedders on Solaris might well run
into this problem as well, if they don't do the shutdown perfectly.

Comment 88

[Darin Fisher]

FWIW: PAC download is triggered whenever the PAC preference is modified.  see
nsProtocolProxyService::PrefsChanged.

Comment 89

[benc]

(wonders if the bug on nsDNSshutdown leaking, which he can't find, is related)

Comment 90

[Doug Turner (:dougt)]

regxpcom and InitXPCOM does not create any event queue for the main thread.  /me
wonders if the timer or DNS threads require one present?


attaching hack to test this theory.

Comment 91

[Kenneth Herron]

A detailed truss suggests that there may be a race condition within TimerThread
(xpcom/threads/TimerThread.cpp). TimerThread::Shutdown() is running before
TimerThread::Run(). This is breaking the method that Shutdown() uses to tell
Run() to exit.

Shutdown() checks a condition variable and a flag:

    // notify the cond var so that Run() can return
    if (mCondVar && mWaiting)
      PR_NotifyCondVar(mCondVar);

but Run() hasn't been called yet so the test fails. Shutdown() falls through and
eventually calles nsThread::Join() to harvest the Run() thread.

Some time later, the Run() thread starts executing, and eventually goes to sleep
on PR_WaitCondVar(mCondVar). Deadlock.

I'm attaching a truss clip which illustrates the problem; the truss includes
calls to libxpcom and libnspr4. I apologize for not including more data; these
truss runs take a long time to complete and produce huge amounts of output. The
one I'm excerpting is 41MB, for example.

Comment 92

[Kenneth Herron]

Attachment: Truss excerpt from a hung regxpcom process.

Comment 93

[Doug Turner (:dougt)]

Attachment: hack to test requirement of event queue

If this works, we should fix the problem much cleaner by having InitXPCOM
startup the event queue directory and Shutdown clean it up.  See 135531.

Comment 94

[Kenneth Herron]

Attachment: Proposed TimerThread.cpp, TimerThread.h patch

I tried adding "sleep(1)" to TimerThread::Shutdown() just before the
timerthread lock is acquired. This has the desired effect; the Shutdown()
thread gives up its timeslice, giving the OS time to schedule the Run() thread.
By the time Shutdown() wakes up, the Run() thread is in the state that
Shutdown() expects. But of course this is just a hack, not a proper solution.

TimerThread uses a flag "mProcessing" to indicate whether TimerThread::Run()
should keep going or not, but the logic isn't quite right. The flag is
initialized false.  Run() sets it true on entry, then keeps looping until it
sees the flag become false. Shutdown() sets the flag back to false when it
wants Run() to return. But if Shutdown() runs before Run(), then Run() can't
tell that Shutdown() has already been called and already written to the flag.

The attached patch replaces the mProcessing flag with an mShutdown flag. This
flag is initialized to false. It's set to true in Shutdown(). Run() never
writes to this flag, but it keeps looping as long as the flag is false.

With either the added sleep() call or the mShutdown patch, regxpcom no longer
hangs shutting down xpcom.  Instead, the last few lines that it prints are as
follows:

*** Registering irc protocol handler.
nNCL: registering deferred (0)
nNCL: registering deferred (0)
Getting service on shutdown. Denied.
  ContractID: @mozilla.org/js/xpc/ContextStack;1
	 IID: {a1339ae0-05c1-11d4-8f92-0010a4e73d9a}
###!!! ASSERTION: Component Manager being held past XPCOM shutdown.: 'cnt ==
0', file nsXPComInit.cpp, line 582
###!!! Break: at file nsXPComInit.cpp, line 582

As far as I can tell, this is an unrelated problem. It may be bug 135330
rearing its head; the source distribution I'm using is from 4/3/2002.

Comment 95

[timeless]

the getting @ shutdown is bug 134728

in general, shutdown problems have *many* bugs, although searching for bugs 
filed by me is a good start.

Comment 96

[Gagan]

->Doug

Comment 97

[Doug Turner (:dougt)]

Comment on attachment 78085 [details] [diff] [review]
Proposed TimerThread.cpp, TimerThread.h patch

r=dougt.  Thanks for fixing this.

Comment 98

[Doug Turner (:dougt)]

brendan, can you super review?  You blame to alot of this code.

Comment 99

[Frank Belew]

I've applied this patch to my mozilla 0.9.9 tree, and regxpcom no longer hangs
on package creation.

Comment 100

[hacker formerly known as [e-mail]]

Was it hanging without this patch?  We've had solaris nightlies for the past few
days (since the 12th apparently). So either the problem resolved itself or
someone added a workaround to the build automation, which I don't see.

Comment 101

[Frank Belew]

Yes, it would consistently hang on building 0.9.8 and 0.9.9 without this patch
on solaris 7

Comment 102

[Brendan Eich [:brendan]]

Comment on attachment 78085 [details] [diff] [review]
Proposed TimerThread.cpp, TimerThread.h patch

sr=[e-mail]

dougt: I took cvsblame in making fixes to pavlov's busted threading code, but I
won't take all blame here.  I do feel pretty foolish for taking this stuff so
close to 1.0 (0.9.8, IIRC -- at least I made pav wait till then, instead of
checking in on the last day of 0.9.7 as he wanted to).

/be

Comment 103

[Doug Turner (:dougt)]

Checked into the trunk:

Checking in TimerThread.cpp;
/cvsroot/mozilla/xpcom/threads/TimerThread.cpp,v  <--  TimerThread.cpp
new revision: 1.12; previous revision: 1.11
done
Checking in TimerThread.h;
/cvsroot/mozilla/xpcom/threads/TimerThread.h,v  <--  TimerThread.h
new revision: 1.4; previous revision: 1.3
done

Comment 104

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 78085 [details] [diff] [review]
Proposed TimerThread.cpp, TimerThread.h patch

a=[e-mail] for branch checkin

Comment 105

[Doug Turner (:dougt)]

Checked into branch.

Checking in TimerThread.cpp;
/cvsroot/mozilla/xpcom/threads/TimerThread.cpp,v  <--  TimerThread.cpp
new revision: 1.6.4.4; previous revision: 1.6.4.3
done
Checking in TimerThread.h;
/cvsroot/mozilla/xpcom/threads/TimerThread.h,v  <--  TimerThread.h
new revision: 1.3.4.2; previous revision: 1.3.4.1
done

Kenneth, thank you for the patch.

Comment 106

[Asa Dotzler [:asa]]

adding fixed1.0.0 keyword (branch resolution). This bug has comments saying it
was fixed on the 1.0 branch and a bonsai checkin comment that agrees. To verify
the bug has been fixed on the 1.0 branch please replace the fixed1.0.0 keyword
with verified1.0.0.

Comment 107

[benc]

updating component and qa...
From reading carefully, it seems like this goes to XPCOM Regsitry. Also, the
summary seems out of date, is PAC really the root cause of this?