590121 - Null-pointer deref leading to crash [@ nsTArray_base] when serializing nsHttpResponseHead

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Name=Fennec
Version=2.0a1pre
BuildID=20100824024116
SourceRepository=http://hg.mozilla.org/mobile-browser
SourceStamp=f662427a3796
Copyright=Copyright (c) 2008 Mozilla.org
ID={a23983c0-fd0e-11dc-95ff-0800200c9a66}

[Gecko]
MinVersion=1.9.2b5pre
MaxVersion=2.0b5pre


I downloaded the latest build of gtk maemo5 mozilla-central fennec and put it on my n900.  I do this normally every week and the last 4 weeks I have had no trouble.

Today I tried to run mochitests:
python runtests.py --appname=../fennec/fennec --certificate-path=../certs --utility-path=../bin

When I do this, the browser starts up and immediately crashes.  Vivien has experienced this on a desktop build as well.

If I turn off IPC (browser.tabs.remote=false), I don't see the crash, but I don't see any page load, just a blank browser (no about: page, 1 tab).

In trying to narrow this down, I do this (browser.tabs.remote=true):
 - launch fennec with mochitest profile 'fennec -no-rmeote -profile /tmp/blah', fennec loads as expected
 - launch fennec with mochitest profile + url 'fennec -no-remote -profile /tmp/blah http://www.google.com', loads and immediately crashes
 - launch fennec 'fennec', fennec launches and runs as expected
 - launch fennec with url 'fennec http://www.google.com', fennec launches and runs as expected

So it appears that the -profile or -no-remote flags are causing a crash.  Doing a mock run of mochitests a nd CTRL+C will leave the profile intact so it can be examined.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

in looking more into this (on a desktop build), I see the crash whenever I
enter a URL into the urlbar and I have this in my preferences:
user_pref("network.proxy.type", 2);
user_pref("network.proxy.autoconfig_url", "data:text/plain,function
FindProxyForURL(url, host){return 'DIRECT';}");


NOTE: the autoconfig_url is very simplified in this example compared to what we
test with, but it should help simplify the problem.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

lastly, adding a browser.tabs.remote=false in the user.js section of the profile allows me to enter a url http://127.0.0.1:8888/tests/ and run mochitests manually.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

hmm, it seems that just the browser-tabs.remote=false on desktop with no other changes allows me to run tests successfully.

Seems like it is narrowed down to remote=true and network.proxy.* setup.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this is really a problem, what is the ETA to fix this?

Comment 5

[Mark Finkle (:mfinkle) (use needinfo?)]

CC'ing some network guys to see if any of them can help with e10s and network.proxy.* causing problems.

Maybe there are already bugs filed on it.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

any chance we can get a fix for this?  This morning I pulled the latest code for m-c/m-b and this is still a problem.

Comment 7

[Benjamin Smedberg]

What's the crash stack? Can you pull it off the device?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

this fails on linux desktop as well.  I am not sure where all the crash information is located, I get a exit with code -11 on my linux desktop when running this.  

The reproduction steps are dead simple.  If there is something I should add to my mozconfig to help out, let me know and I can do that.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

ok, I did a debug build (--enable-debug in my mozconfig) and ran this inside of gdb.  I found that inside of gdb this doesn't crash the same way as it does outside of the debugger.

what doesn't work:

Here is the cli I run:
./fennec -no-remote -profile /tmp/tmpfA59bG
 - type in http://www.google.com upon load
 - browser crashes

Now in gdb, I do this:
gdb --args ./fennec -no-remote -profile /tmp/tmpfA59bG
 - type in http://www.google.com upon load
 - no crash!


Now if I do this:
gdb --args ./fennec -no-remote -profile /tmp/tmpfA59bG

I see this:
(gdb) bt
#0  0x011fce0b in void mozilla::net::PHttpChannelParent::Write<nsHttpResponseHead>(nsHttpResponseHead const&, IPC::Message*) () from /home/joel/mozilla/mobilebase-i686-pc-linux-gnu/dist/bin/libxul.so
#1  0x011fc761 in mozilla::net::PHttpChannelParent::SendRedirect1Begin (this=0xb20204c0, newChannel=0xb20a07c0, newUri=..., redirectFlags=@0xbfffeccc, responseHead=...) at PHttpChannelParent.cpp:229
#2  0x0048cc2b in mozilla::net::HttpChannelParentListener::AsyncOnChannelRedirect (this=0xb2032100, oldChannel=0xb7d95630, newChannel=0xb7d957d0, redirectFlags=4, callback=0x4)
    at /home/joel/mozilla/mobilesrc/netwerk/protocol/http/HttpChannelParentListener.cpp:227
#3  0x003cfdc8 in nsAsyncRedirectVerifyHelper::DelegateOnChannelRedirect (this=0xb2032fd0, sink=0xb2032104, oldChannel=0xb7d95630, newChannel=0xb7d957d0, flags=4)
    at /home/joel/mozilla/mobilesrc/netwerk/base/src/nsAsyncRedirectVerifyHelper.cpp:180
#4  0x003d01a1 in nsAsyncRedirectVerifyHelper::Run (this=0xb2032fd0) at /home/joel/mozilla/mobilesrc/netwerk/base/src/nsAsyncRedirectVerifyHelper.cpp:274
#5  0x012c27bf in nsThread::ProcessNextEvent (this=0xb7dd4dd0, mayWait=0, result=0xbfffed9c) at /home/joel/mozilla/mobilesrc/xpcom/threads/nsThread.cpp:547
#6  0x01273c70 in NS_ProcessNextEvent_P (thread=0xb22f1220, mayWait=0) at nsThreadUtils.cpp:250
#7  0x0118269b in mozilla::ipc::MessagePump::Run (this=0xb7dd9a30, aDelegate=0xb7d237c0) at /home/joel/mozilla/mobilesrc/ipc/glue/MessagePump.cpp:110
#8  0x01302da5 in MessageLoop::RunInternal (this=0xb7d237c0) at /home/joel/mozilla/mobilesrc/ipc/chromium/src/base/message_loop.cc:219
#9  0x01302e54 in MessageLoop::RunHandler (this=0xb22f1220) at /home/joel/mozilla/mobilesrc/ipc/chromium/src/base/message_loop.cc:202
#10 MessageLoop::Run (this=0xb22f1220) at /home/joel/mozilla/mobilesrc/ipc/chromium/src/base/message_loop.cc:176
#11 0x01086f4a in nsBaseAppShell::Run (this=0xb715dfb0) at /home/joel/mozilla/mobilesrc/widget/src/xpwidgets/nsBaseAppShell.cpp:175
#12 0x00ebf9e4 in nsAppStartup::Run (this=0xb53cd460) at /home/joel/mozilla/mobilesrc/toolkit/components/startup/src/nsAppStartup.cpp:191
#13 0x003bde1a in XRE_main (argc=5, argv=0xbffff414, aAppData=0xb7d10380) at /home/joel/mozilla/mobilesrc/toolkit/xre/nsAppRunner.cpp:3662
#14 0x0804988b in main (argc=5, argv=0xbffff414) at /home/joel/mozilla/mobilesrc/mobile/app/nsBrowserApp.cpp:155
(gdb) 


Hope this helps!!

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

err, the crashing gdb command is: 'gdb --args ./fennec -no-remote -profile /tmp/tmpfA59bG http://www.google.com'

Comment 11

[Jason Duell]

cc-ing bz/biesi, who are hopefuly more familiar with the proxy code.

Comment 12

[Boris Zbarsky [:bzbarsky]]

So at the crash point.. why did we crash?  Are we dereferencing a bogus or null pointer?  Something else?

Jason, your hope may be .. in vain.  :(

Comment 13

[Brian Crowder]

Yeah, we're dereferencing a null pointer:

#0  0xb5b9db40 in nsTArray_base::Length (this=0x0) at ../../../../dist/include/nsTArray.h:66
#1  0xb706a356 in IPC::ParamTraits<nsTArray<nsHttpHeaderArray::nsEntry> >::Write (aMsg=0xae159b20, aParam=...)
    at ../../dist/include/IPC/IPCMessageUtils.h:256
#2  0xb7068cb8 in WriteParam<nsTArray<nsHttpHeaderArray::nsEntry> > (m=0xae159b20, p=...)
    at /home/crowder/mobilla/ipc/chromium/src/chrome/common/ipc_message_utils.h:124
#3  0xb706912f in IPC::ParamTraits<nsHttpHeaderArray>::Write (aMsg=0xae159b20, aParam=...) at ../../dist/include/mozilla/net/PHttpChannelParams.h:149
#4  0xb7068cd2 in WriteParam<nsHttpHeaderArray> (m=0xae159b20, p=...) at /home/crowder/mobilla/ipc/chromium/src/chrome/common/ipc_message_utils.h:124
#5  0xb7069149 in IPC::ParamTraits<nsHttpResponseHead>::Write (aMsg=0xae159b20, aParam=...)
    at ../../dist/include/mozilla/net/PHttpChannelParams.h:168
#6  0xb7068d62 in WriteParam<nsHttpResponseHead> (m=0xae159b20, p=...)
    at /home/crowder/mobilla/ipc/chromium/src/chrome/common/ipc_message_utils.h:124
#7  0xb706a154 in mozilla::net::PHttpChannelParent::Write<nsHttpResponseHead> (this=0xae3f0580, __v=..., __msg=0xae159b20)
    at ../../ipc/ipdl/_ipdlheaders/mozilla/net/PHttpChannelParent.h:242
#8  0xb706691e in mozilla::net::PHttpChannelParent::SendRedirect1Begin (this=0xae3f0580, newChannel=0xae157d00, newUri=..., 
    redirectFlags=@0xbfffeb0c, responseHead=...) at PHttpChannelParent.cpp:229
#9  0xb5d07dbf in mozilla::net::HttpChannelParentListener::AsyncOnChannelRedirect (this=0xae1039a0, oldChannel=0xb44a3490, newChannel=0xb44a3630, 
    redirectFlags=4, callback=0xae159af4) at /home/crowder/mobilla/netwerk/protocol/http/HttpChannelParentListener.cpp:227
#10 0xb5beb5e4 in nsAsyncRedirectVerifyHelper::DelegateOnChannelRedirect (this=0xae159af0, sink=0xae1039a4, oldChannel=0xb44a3490, 
    newChannel=0xb44a3630, flags=4) at /home/crowder/mobilla/netwerk/base/src/nsAsyncRedirectVerifyHelper.cpp:180
#11 0xb5bebb04 in nsAsyncRedirectVerifyHelper::Run (this=0xae159af0) at /home/crowder/mobilla/netwerk/base/src/nsAsyncRedirectVerifyHelper.cpp:274
#12 0xb717e44e in nsThread::ProcessNextEvent (this=0xb44e0d30, mayWait=0, result=0xbfffec3c) at /home/crowder/mobilla/xpcom/threads/nsThread.cpp:547
#13 0xb710b7dd in NS_ProcessNextEvent_P (thread=0xb44e0d30, mayWait=0) at nsThreadUtils.cpp:250
#14 0xb6f9ed00 in mozilla::ipc::MessagePump::Run (this=0xb44e3af0, aDelegate=0xb44238a0) at /home/crowder/mobilla/ipc/glue/MessagePump.cpp:110
#15 0xb71e2ad7 in MessageLoop::RunInternal (this=0xb44238a0) at /home/crowder/mobilla/ipc/chromium/src/base/message_loop.cc:219
#16 0xb71e2a57 in MessageLoop::RunHandler (this=0xb44238a0) at /home/crowder/mobilla/ipc/chromium/src/base/message_loop.cc:202
#17 0xb71e29fb in MessageLoop::Run (this=0xb44238a0) at /home/crowder/mobilla/ipc/chromium/src/base/message_loop.cc:176
#18 0xb6e42236 in nsBaseAppShell::Run (this=0xb31cb3d0) at /home/crowder/mobilla/widget/src/xpwidgets/nsBaseAppShell.cpp:175
#19 0xb6bb4839 in nsAppStartup::Run (this=0xb1299640) at /home/crowder/mobilla/toolkit/components/startup/src/nsAppStartup.cpp:191
#20 0xb5bb4d34 in XRE_main (argc=5, argv=0xbffff3d4, aAppData=0xb4410380) at /home/crowder/mobilla/toolkit/xre/nsAppRunner.cpp:3662
#21 0x08049927 in main (argc=5, argv=0xbffff3d4) at /home/crowder/mobilla/mobile/app/nsBrowserApp.cpp:155
(gdb) f 7
#7  0xb706a154 in mozilla::net::PHttpChannelParent::Write<nsHttpResponseHead> (this=0xae3f0580, __v=..., __msg=0xae159b20)
    at ../../ipc/ipdl/_ipdlheaders/mozilla/net/PHttpChannelParent.h:242
242	        IPC::WriteParam(__msg, __v);
(gdb) f 8
#8  0xb706691e in mozilla::net::PHttpChannelParent::SendRedirect1Begin (this=0xae3f0580, newChannel=0xae157d00, newUri=..., 
    redirectFlags=@0xbfffeb0c, responseHead=...) at PHttpChannelParent.cpp:229
229	    Write(responseHead, __msg);
(gdb) f 9
#9  0xb5d07dbf in mozilla::net::HttpChannelParentListener::AsyncOnChannelRedirect (this=0xae1039a0, oldChannel=0xb44a3490, newChannel=0xb44a3630, 
    redirectFlags=4, callback=0xae159af4) at /home/crowder/mobilla/netwerk/protocol/http/HttpChannelParentListener.cpp:227
227	                                     *oldHttpChannel->GetResponseHead());
(gdb) p oldHttpChannel.mResponseHead
$15 = {mRawPtr = 0x0}

Comment 14

[Boris Zbarsky [:bzbarsky]]

Ok, so the thing is... we handle proxies, especially proxy autodetect and PAC as internal-type redirects.  The redirects happen without us having made any request anywhere, so there is no response head on the channel being redirected from.

Comment 16

[Jason Duell]

Attachment: Fixes null responseHead in redirect code

Just does the same logic (create temp respHead object) for redirects as we do in regular OnStart:

   http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/HttpChannelParent.cpp#309

I.e. pass empty temp object if respHead is null

Comment 17

[[e-mail]]

Comment on attachment 473310 [details] [diff] [review]
Fixes null responseHead in redirect code

r=dwitte

Comment 18

[Jason Duell]

http://hg.mozilla.org/mozilla-central/rev/a84393d459c8