625991 - Crash when running Mozmill builds in offline mode [@ nsServerSocket::nsServerSocket]

Status: VERIFIED FIXED

(Core :: Networking, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b10pre) Gecko/20110114 Firefox/4.0b10pre ID:20110114030359

With bug 546640 fixed we should be able to run Mozmill tests even when Firefox is in offline mode. On my box Firefox gets started but crashes right after a couple of seconds before any test gets run.

Steps:
1. Remove your network cable
2. Start Mozmill: mozmill -b /Applications/Minefield.app -t mozmill-tests/firefox
3. Wait some seconds

Crash info: bp-53116cab-09bd-4c86-aa4b-124d72110114

Stack:
0 	XUL 	nsServerSocket::nsServerSocket 	netwerk/base/src/nsServerSocket.cpp:84
1 	XUL 	nsServerSocketConstructor 	netwerk/build/nsNetModule.cpp:100
2 	XUL 	nsComponentManagerImpl::CreateInstance 	xpcom/components/nsComponentManager.cpp:1228
3 	XUL 	nsJSCID::CreateInstance 	js/src/xpconnect/src/xpcjsid.cpp:774
4 	XUL 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:208
5 	XUL 	XPCWrappedNative::CallMethod 	js/src/xpconnect/src/xpcwrappednative.cpp:3066
6 	XUL 	XPC_WN_CallMethod 	js/src/xpconnect/src/xpcwrappednativejsops.cpp:1593
7 	XUL 	js::Invoke 	js/src/jscntxtinlines.h:692
8 	XUL 	js::ExternalInvoke 	js/src/jsinterp.cpp:858
9 	XUL 	JS_CallFunctionName 	js/src/jsinterp.h:961

Could be a problem for offline mode or simply on our side. Needs more investigation first before I will move it to network.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

CLI output of Mozmill:

jsbridge: Exception: socket failed to bind, kill all firefox processes
Traceback (most recent call last):
  File "/Volumes/data/testing/envs/mozmill-2/bin/mozmill", line 8, in <module>
    load_entry_point('mozmill==2.0a', 'console_scripts', 'mozmill')()
  File "/Volumes/data/build/tools/mozmill-2.0/mozmill/mozmill/__init__.py", line 637, in cli
    CLI(args).run()
  File "/Volumes/data/build/tools/mozmill-2.0/mozmill/mozmill/__init__.py", line 609, in run
    self.run_tests(MozMill, normal_tests, runner, results)
  File "/Volumes/data/build/tools/mozmill-2.0/mozmill/mozmill/__init__.py", line 579, in run_tests
    mozmill.run(tests)
  File "/Volumes/data/build/tools/mozmill-2.0/mozmill/mozmill/__init__.py", line 267, in run
    self.report_disconnect()               
  File "/Volumes/data/build/tools/mozmill-2.0/mozmill/mozmill/__init__.py", line 293, in report_disconnect
    test = self.current_test
AttributeError: 'MozMill' object has no attribute 'current_test'

Comment 2

[Heather Arthur [:harth]]

This must be coming from httpd.js, it's the only place on master that still uses nsIServerSocket.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Definitely a regression in Firefox between b7 and b8. Regression range for now is between 101115 and 101203. I will check it in detail tomorrow.

Comment 4

[Josh Matthews [:jdm]]

We're crashing addrefing a null gSocketTransportService.  This is a regression from bug 614286, which causes nsSocketTransportService::Init to return NS_ERROR_OFFLINE in this situation.  This has the effect of failing the do_GetService call, and the nsSocketTransportService object is released and destroyed, nulling out gSocketTransportService.  Ehsan, thoughts on how to avoid this problem?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's not a regression from bug 614286, because it started to fail between 10111703 and 10111806.

PASS: http://hg.mozilla.org/mozilla-central/rev/ad227939db82
FAIL: http://hg.mozilla.org/mozilla-central/rev/789f0f85f75a

Checkins: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=ad227939db82&tochange=789f0f85f75a

There are too many check-ins and a tracemonkey merge. I could hg bisect if necessary. Asking for blocking because it's 100% reproducible.

Comment 6

[(no longer active)]

FWIW,

Comment 7

[(no longer active)]

meant to say: FWIW, jdm's analysis in comment 4 is completely correct.

The obvious fix here would be to null check gSocketTransportService, however that would just break other things because we should be able to create server sockets on 127.0.0.1 even if we're offline...

Boris, what do you think is the best way to address this problem?

Comment 8

[Boris Zbarsky [:bzbarsky]]

So how did this (server sockets on 127.0.0.1) use to work before?  In offline more we used to shut down the socket transport service, no?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #8)
> So how did this (server sockets on 127.0.0.1) use to work before?  In offline
> more we used to shut down the socket transport service, no?

Correct, no connection even not to localhost are possible. That's why we had to go through js-ctypes and NSPR to create a connection with our command line Mozmill process.

Comment 10

[Boris Zbarsky [:bzbarsky]]

OK, so then the "obvious fix" would just restore the status quo.  Sounds good enough enough for 2.0 to me.....  I'd rather not be doing more complicated surgery on this code at this point.

Comment 11

[(no longer active)]

Attachment: Patch (v1)

Henrik, can you please test to make sure that this patch actually fixes the crash?  I haven't tested this except to verify that it compiles.  :-)

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ehsan, can you please create a tryserver build? Thanks.

Comment 13

[(no longer active)]

(In reply to comment #12)
> Ehsan, can you please create a tryserver build? Thanks.

Already done: <http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[e-mail]-613e8e63e2ea/>

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I tried that build with our latest Mozmill 2 version and right now we are throwing hundreds of assertions of this kind:

###!!! ASSERTION: no socket transport service: 'sts', file /builds/slave/try-osx-dbg/build/netwerk/base/src/nsServerSocket.cpp, line 84
WARNING: unable to create server socket: file /builds/slave/try-osx-dbg/build/netwerk/base/src/nsServerSocket.cpp, line 307

Not sure if it is wanted or not.

Comment 16

[(no longer active)]

It should probably block since it's a topcrasher and a regression.

Comment 17

[Sheila Mooney]

Johnny, Damon, sounds like this should block, can one of you guys triage it?

Comment 18

[(no longer active)]

(In reply to comment #14)
> I tried that build with our latest Mozmill 2 version and right now we are
> throwing hundreds of assertions of this kind:
> 
> ###!!! ASSERTION: no socket transport service: 'sts', file
> /builds/slave/try-osx-dbg/build/netwerk/base/src/nsServerSocket.cpp, line 84
> WARNING: unable to create server socket: file
> /builds/slave/try-osx-dbg/build/netwerk/base/src/nsServerSocket.cpp, line 307
> 
> Not sure if it is wanted or not.

Could you please attach a stack of the assertion?  Or better yet, teach me how to fish: how to run the mozmill suite locally in a way which triggers this bug?

Also, am I correct in assuming that the patch that I attached causes Firefox not to crash during th mozmill tests?

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Steps to run with Mozmill 2.0:
* Prepare your local environment with Python installed (i.e. on Windows with MozillaBuild)
* Clone https://github.com/mozautomation/mozmill.git
* Execute the setup_development.py script
* Go offline
* Start Mozmill with "mozmill -b %path_to_app%"

The last step should be enough without any test and should also crash Minefield some seconds after start.

The tryserver build is not available anymore, so I can't currently test it on my own.

Comment 20

[(no longer active)]

(In reply to comment #19)
> * Start Mozmill with "mozmill -b %path_to_app%"

And where is the mozmill executable?  :-)

> The tryserver build is not available anymore, so I can't currently test it on
> my own.

It's OK, I'll test it myself.

Comment 21

[(no longer active)]

(This build has the patch applied, FWIW: <http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/[e-mail]-29b683880abc/>)

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #20)
> (In reply to comment #19)
> > * Start Mozmill with "mozmill -b %path_to_app%"
> 
> And where is the mozmill executable?  :-)

When you have ran the setup_development.py script the binary is in your system path. Should be under /usr/local/bin.

Comment 23

[(no longer active)]

(In reply to comment #22)
> (In reply to comment #20)
> > (In reply to comment #19)
> > > * Start Mozmill with "mozmill -b %path_to_app%"
> > 
> > And where is the mozmill executable?  :-)
> 
> When you have ran the setup_development.py script the binary is in your system
> path. Should be under /usr/local/bin.

It's not.  Here's the output of setup_development.py for me:

ehsanakhgari:~/moz/mozmill [10:34:00]$ sudo ./setup_development.py 
running develop
install_dir /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
running egg_info
writing mozprocess.egg-info/PKG-INFO
writing top-level names to mozprocess.egg-info/top_level.txt
writing dependency_links to mozprocess.egg-info/dependency_links.txt
writing entry points to mozprocess.egg-info/entry_points.txt
reading manifest file 'mozprocess.egg-info/SOURCES.txt'
writing manifest file 'mozprocess.egg-info/SOURCES.txt'
running build_ext
Creating /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mozprocess.egg-link (link to .)
mozprocess 0.1a is already the active version in easy-install.pth

Installed /Users/ehsanakhgari/moz/mozmill/mozprocess
Processing dependencies for mozprocess==0.1a
Finished processing dependencies for mozprocess==0.1a
running develop
install_dir /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
running egg_info
writing mozprofile.egg-info/PKG-INFO
writing top-level names to mozprofile.egg-info/top_level.txt
writing dependency_links to mozprofile.egg-info/dependency_links.txt
writing entry points to mozprofile.egg-info/entry_points.txt
reading manifest file 'mozprofile.egg-info/SOURCES.txt'
writing manifest file 'mozprofile.egg-info/SOURCES.txt'
running build_ext
Creating /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mozprofile.egg-link (link to .)
mozprofile 0.1a is already the active version in easy-install.pth
Installing addon_id script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Installed /Users/ehsanakhgari/moz/mozmill/mozprofile
Processing dependencies for mozprofile==0.1a
Finished processing dependencies for mozprofile==0.1a
running develop
install_dir /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
running egg_info
writing requirements to mozrunner.egg-info/requires.txt
writing mozrunner.egg-info/PKG-INFO
writing top-level names to mozrunner.egg-info/top_level.txt
writing dependency_links to mozrunner.egg-info/dependency_links.txt
writing entry points to mozrunner.egg-info/entry_points.txt
reading manifest file 'mozrunner.egg-info/SOURCES.txt'
writing manifest file 'mozrunner.egg-info/SOURCES.txt'
running build_ext
Creating /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mozrunner.egg-link (link to .)
mozrunner 3.0a is already the active version in easy-install.pth
Installing mozrunner script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Installed /Users/ehsanakhgari/moz/mozmill/mozrunner
Processing dependencies for mozrunner==3.0a
Searching for mozprofile==0.1a
Best match: mozprofile 0.1a
mozprofile 0.1a is already the active version in easy-install.pth
Installing addon_id script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /Users/ehsanakhgari/moz/mozmill/mozprofile
Searching for mozprocess==0.1a
Best match: mozprocess 0.1a
mozprocess 0.1a is already the active version in easy-install.pth

Using /Users/ehsanakhgari/moz/mozmill/mozprocess
Finished processing dependencies for mozrunner==3.0a
running develop
install_dir /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
running egg_info
writing requirements to jsbridge.egg-info/requires.txt
writing jsbridge.egg-info/PKG-INFO
writing top-level names to jsbridge.egg-info/top_level.txt
writing dependency_links to jsbridge.egg-info/dependency_links.txt
writing entry points to jsbridge.egg-info/entry_points.txt
reading manifest file 'jsbridge.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'jsbridge/xpi'
writing manifest file 'jsbridge.egg-info/SOURCES.txt'
running build_ext
Creating /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/jsbridge.egg-link (link to .)
jsbridge 3.0a is already the active version in easy-install.pth
Installing jsbridge script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Installed /Users/ehsanakhgari/moz/mozmill/jsbridge
Processing dependencies for jsbridge==3.0a
Searching for mozrunner==3.0a
Best match: mozrunner 3.0a
mozrunner 3.0a is already the active version in easy-install.pth
Installing mozrunner script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /Users/ehsanakhgari/moz/mozmill/mozrunner
Searching for mozprofile==0.1a
Best match: mozprofile 0.1a
mozprofile 0.1a is already the active version in easy-install.pth
Installing addon_id script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /Users/ehsanakhgari/moz/mozmill/mozprofile
Searching for mozprocess==0.1a
Best match: mozprocess 0.1a
mozprocess 0.1a is already the active version in easy-install.pth

Using /Users/ehsanakhgari/moz/mozmill/mozprocess
Finished processing dependencies for jsbridge==3.0a
running develop
install_dir /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
running egg_info
writing requirements to mozmill.egg-info/requires.txt
writing mozmill.egg-info/PKG-INFO
writing top-level names to mozmill.egg-info/top_level.txt
writing dependency_links to mozmill.egg-info/dependency_links.txt
writing entry points to mozmill.egg-info/entry_points.txt
reading manifest file 'mozmill.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'mozmill.egg-info/SOURCES.txt'
running build_ext
Creating /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mozmill.egg-link (link to .)
mozmill 2.0a is already the active version in easy-install.pth
Installing mozmill script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Installed /Users/ehsanakhgari/moz/mozmill/mozmill
Processing dependencies for mozmill==2.0a
Searching for ManifestDestiny==0.1.5
Best match: ManifestDestiny 0.1.5
Processing ManifestDestiny-0.1.5-py2.6.egg
ManifestDestiny 0.1.5 is already the active version in easy-install.pth
Installing update-manifest script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin
Installing copy-manifest script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin
Installing query-manifest script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin
Installing create-manifest script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ManifestDestiny-0.1.5-py2.6.egg
Searching for mozrunner==3.0a
Best match: mozrunner 3.0a
mozrunner 3.0a is already the active version in easy-install.pth
Installing mozrunner script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /Users/ehsanakhgari/moz/mozmill/mozrunner
Searching for jsbridge==3.0a
Best match: jsbridge 3.0a
jsbridge 3.0a is already the active version in easy-install.pth
Installing jsbridge script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /Users/ehsanakhgari/moz/mozmill/jsbridge
Searching for mozprofile==0.1a
Best match: mozprofile 0.1a
mozprofile 0.1a is already the active version in easy-install.pth
Installing addon_id script to /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Using /Users/ehsanakhgari/moz/mozmill/mozprofile
Searching for mozprocess==0.1a
Best match: mozprocess 0.1a
mozprocess 0.1a is already the active version in easy-install.pth

Using /Users/ehsanakhgari/moz/mozmill/mozprocess
Finished processing dependencies for mozmill==2.0a


I'm on a Mac, if that makes any difference.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #23)
> /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin

Hm. Everything gets installed to this folder. Do you have a separate Python installation running on this box? This bin folder should then normally be in the system path.

Comment 25

[(no longer active)]

(In reply to comment #24)
> (In reply to comment #23)
> > /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin
> 
> Hm. Everything gets installed to this folder. Do you have a separate Python
> installation running on this box? This bin folder should then normally be in
> the system path.

Yeah, I do have the mozmill executable under there.  Don't know why that directory is not on my $PATH, but it doesn't matter, I'll just invoke it directly.  I'll work on this tomorrow.

Comment 26

[(no longer active)]

(In reply to comment #19)
> * Start Mozmill with "mozmill -b %path_to_app%"

Here's what I get:

ehsanakhgari:~/moz/mozilla-central [03:17:21]$ /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/mozmill -b ./obj-ff-dbg/dist/MinefieldDebug.app/Contents/MacOS/firefox-bin 
Usage: mozmill [options]

mozmill: error: No tests found. Please specify tests with -t or -m

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Follow comment 19 or for ease of use simply clone the following repository:
http://hg.mozilla.org/qa/mozmill-tests/

Then follow my steps in comment 0.

Comment 28

[(no longer active)]

Bug 627332 and bug 620472 have wallpaper'ed the assertion here, but with them backed out locally, I see that the assertion is simply coming from nsServerSocket's ctor.  The assertion there is stupid, and should be removed!

Comment 29

[(no longer active)]

Attachment: Patch (v2)

Comment 30

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 508048 [details] [diff] [review]
Patch (v2)

r=me

Comment 31

[(no longer active)]

http://hg.mozilla.org/mozilla-central/rev/3dd2962dade5

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Verified fixed with Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b13pre) Gecko/20110310 Firefox/4.0b13pre ID:20110310030427