46989 - http cookies don't work if components.reg not present (linux only)

Status: VERIFIED FIXED

(Core :: Networking: Cookies, defect, P3)

Description

[Matthew Cline]

(NOTE: This bug is a partial reincarnation of bug 45468)

On Linux, if Mozilla is started up with no component.reg file present,
then the cookie service doesn't get started until after the first HTTP
request; if you start up Mozilla on a homepage which uses cookies
(like Slashdot), this can be a problem.

The file component.reg is normally missing after Mozilla is installed
fresh on a system.

As an example, here are the HTTP request headers as sent from Mozilla
when it was started up with no component.reg file:

=============================
Slashdot (homepage, first page loaded):
GET / HTTP/1.1
Host: www.slashdot.org
User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.14 i686; en-US; m18) Gecko/20000528
Accept: */*
Accept-Language: en
Accept-Encoding: gzip,deflate,compress,identity
Keep-Alive: 300
Connection: keep-alive

Bugzilla (viewed after slashdot):
GET / HTTP/1.1
Cookie: Bugzilla_login=[e-mail]; Bugzilla_logincookie=64197;
LASTORDER=bugs.bug_id; COLUMNLIST=opendate severity platform status resolution
summary; SPLITHEADER=0; PLATFORM=Browser; VERSION-mozilla.org=other;
VERSION-Browser=other; VERSION-MailNews=other; VERSION-Webtools=other
Host: bugzilla.mozilla.org
User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.14 i686; en-US; m18) Gecko/20000528
Accept: */*
Accept-Language: en
Accept-Encoding: gzip,deflate,compress,identity
Keep-Alive: 300
Connection: keep-alive

Slashdot (homepage, viewed after Bugzilla):
GET / HTTP/1.1
Cookie: user=[deleted; I'm not telling anyone my Slashdot cookie :-) ]
Host: www.slashdot.org
User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.14 i686; en-US; m18) Gecko/20000528
Accept: */*
Accept-Language: en
Accept-Encoding: gzip,deflate,compress,identity
Keep-Alive: 300
Connection: keep-alive

==============================

The same behavior happened if I made Bugzilla my homepage: Mozilla
didn't send any cookie until I visited some other page, then went back
to Bugzilla.

Comment 1

[Stephen P. Morse]

Does this problem occur every time the browser starts up or is it only on the 
very first execution after a new install?

Comment 2

[Matthew Cline]

Only happens on the first execution after a new install.

Comment 3

[neeti]

*** Bug 46031 has been marked as a duplicate of this bug. ***

Comment 4

[neeti]

*** Bug 46512 has been marked as a duplicate of this bug. ***

Comment 5

[Stephen P. Morse]

At first I thought that this bug was unimportant since it only happens once in 
the lifetime of the browser.  But now that it has two dups, it gains in 
importance.  Switching from M19 to M18 and putting on beta3 list.

Comment 6

[Suresh Duddi (gone)]

Wonder why this happens. Theoritically, if component.reg isn't there, autoreg 
starts to register all dlls under component. Then the first htpp request starts 
up cookie. The bugs says it doesn't and the second one starts cookies.

We got to see what the exact failure is. We should be able to look at the 
xpcom-log to see what is happening as in if cookie service creation is forgotten 
in that first path or creation fails.

To get xpcom log:
% setenv NSPR_LOG_MODULES nsComponentManager:5
% setenv NSPR_LOG_FILE xpcom.log
% rm component.reg
% ./mozilla
<quit>

Attach file xpcom.log to the bug.

Comment 7

[Stephen P. Morse]

Tried twice to attach the xpcom.log file to this report as dp requested but kept 
getting an error message.  Perhaps there's a limit to how big a file you can 
attach (it's 1.5 meg).  So I sent it to dp as a private e-mail instead.

Comment 8

[Stephen P. Morse]

I just ran the following test.  The problem does not seem to be occuring.

1. Instrumented code so that I know what is happening when
2. Cleared my cache subdirectory
3. Removed my components.reg file
4. Put a dummy cookie for mozilla.org in my cookies.txt file (mozilla.org is my 
startup page by default)

Below is the output I obtained.  Note that the cookie is being sent on the first 
GET.

^^^^^ in nsCookieHTTPNotify::RegisterProc
^^^^^ in nsCookieHTTPNotify::nsCookieHTTPNotify
^^^^^ entering nsCookieHTTPNotify::Init
^^^^^ leaving nsCookieHTTPNotify::Init
^^^^^ in nsHTTPRequest::SetHeader, setting: <if-modified-since> <(null)>
^^^^^ entering nsCookieHTTPNotify::ModifyRequest
^^^^^ entering nsCookieHTTPNotify::SetupCookieService
^^^^^ in nsCookieService::nsCookieService
^^^^^ entering nsCookieService::Init
^^^^^ leaving nsCookieService::Init
^^^^^ leaving nsCookieHTTPNotify::SetupCookieService
^^^^^ in nsHTTPRequest::SetHeader, setting: <Cookie> 
<SID=1402a8c0%24398ed063%240166>
^^^^^ leaving nsCookieHTTPNotify::ModifyRequest
^^^^^ entering nsHTTPPipelinedRequest::WriteRequest
^^^^^ in nsHTTPRequest::SetHeader, setting: <host> <www.mozilla.org>
^^^^^ in nsHTTPRequest::SetHeader, setting: <user-agent> <Mozilla/5.0 (Windows; 
U; WinNT4.0; en-US; m18) Gecko/20000806>
^^^^^ in nsHTTPRequest::SetHeader, setting: <accept> <*/*>
^^^^^ in nsHTTPRequest::SetHeader, setting: <accept-language> <en>
^^^^^ in nsHTTPRequest::SetHeader, setting: <accept-encoding> 
<gzip,deflate,compress,identity>
^^^^^ in nsHTTPRequest::SetHeader, setting: <keep-alive> <300>
^^^^^ in nsHTTPRequest::SetHeader, setting: <connection> <keep-alive>

^^^^^sending the following headers to server
GET / HTTP/1.1
Cookie: SID=1402a8c0%24398ed063%240166
Host: www.mozilla.org
User-Agent: Mozilla/5.0 (Windows; U; WinNT4.0; en-US; m18) Gecko/20000806
Accept: */*
Accept-Language: en
Accept-Encoding: gzip,deflate,compress,identity
Keep-Alive: 300
Connection: keep-alive
^^^^^end of headers sent to server

^^^^^ leaving nsHTTPPipelinedRequest::WriteRequest
^^^^^ entering nsHTTPPipelinedRequest::WriteRequest
^^^^^ exiting from nsHTTPPipelinedRequest::WriteRequest because nothing more to 
write
^^^^^ in nsHTTPRequest::SetHeader, setting: <referer> <http://www.mozilla.org/>
^^^^^ in nsHTTPRequest::SetHeader, setting: <if-modified-since> <(null)>
^^^^^ entering nsCookieHTTPNotify::ModifyRequest
^^^^^ entering nsCookieHTTPNotify::SetupCookieService
^^^^^ leaving nsCookieHTTPNotify::SetupCookieService
^^^^^ in nsHTTPRequest::SetHeader, setting: <Cookie> 
<SID=1402a8c0%24398ed063%240166>
^^^^^ leaving nsCookieHTTPNotify::ModifyRequest
^^^^^ entering nsHTTPPipelinedRequest::WriteRequest
^^^^^ in nsHTTPRequest::SetHeader, setting: <host> <www.mozilla.org>
^^^^^ in nsHTTPRequest::SetHeader, setting: <user-agent> <Mozilla/5.0 (Windows; 
U; WinNT4.0; en-US; m18) Gecko/20000806>
^^^^^ in nsHTTPRequest::SetHeader, setting: <accept> <*/*>
^^^^^ in nsHTTPRequest::SetHeader, setting: <accept-language> <en>
^^^^^ in nsHTTPRequest::SetHeader, setting: <accept-encoding> 
<gzip,deflate,compress,identity>
^^^^^ in nsHTTPRequest::SetHeader, setting: <keep-alive> <300>
^^^^^ in nsHTTPRequest::SetHeader, setting: <connection> <keep-alive>

^^^^^writing the following headers to server
GET /images/mozilla-banner.gif HTTP/1.1
Referer: http://www.mozilla.org/
Cookie: SID=1402a8c0%24398ed063%240166
Host: www.mozilla.org
User-Agent: Mozilla/5.0 (Windows; U; WinNT4.0; en-US; m18) Gecko/20000806
Accept: */*
Accept-Language: en
Accept-Encoding: gzip,deflate,compress,identity
Keep-Alive: 300
Connection: keep-alive
^^^^^end of headers written to server

^^^^^ leaving nsHTTPPipelinedRequest::WriteRequest
^^^^^ entering nsHTTPPipelinedRequest::WriteRequest
^^^^^ exiting from nsHTTPPipelinedRequest::WriteRequest because nothing more to 
write

Comment 9

[Matthew Cline]

I've tested this with build 2000080808 on Linux and the problem has gone away.
Marking FIXED.

Comment 10

[Stephen P. Morse]

Thanks for retesting this and closing it.  I just realized that this was 
reported as linux-only (I didn't notice that before).  The tests that I ran 
above were done on a win32 box so that explains why I couldn't see the problem.  
And now you saved me the trouble of repeating that intrumentation on a linux 
box.

Comment 11

[Stephen P. Morse]

Reopening this bug because it looks like it might not have vanished after all.  
There is currently a bug on bugscape (1915) which smells a lot like this one.  
The bug was opened on bugscape because at first it was thought to occur in 
commercial tree only.  But that was later disproved.

There are a log of confusing comments in that bug report, so rather than dump 
all those comments here I am only including the coherent ones ;-)


***************************

Leaf opened bug report on 8-8 with the following description:

my.yahoo.com prompts for a login/password, after i enter this correctly, i get
shunted to an error page.

mail.com does the same thing, and bugscape/bugzilla forgets that i logged in as
well, prompting login after each transaction (the password manager is very
useful in this instance!). But i have no workaround for mail.com and 
my.yahoo.com.

I see this in the commercial build 2000080808 on linux, but not in mozilla 
builds.

********** Additional Comments From Stephen P. Morse 2000-08-09 16:25 

I finished building a commercial linux build and I tried it.  The first time I 
logged into my.yahoo.com, I indeed got to the error page as described in this 
report.  But since then I have been unable to reproduce it.  I even tried using 
a new profile and still it worked correctly (except for the layout bug that I 
mentioned above).

I'm wondering if this failure occurs only on the very first execution of the 
browser after an intall or a build.  Can anyone confirm or deny this assumption?

*********** Additional Comments From Tom Everingham 2000-08-09 16:37 

yep, that's what is happening.  Once you exit and re-start it works fine.

********* Additional Comments From Stephen P. Morse 2000-08-09 16:43

More observations.  I rebuilt and tried again.  Again the failure.  This time I 
kept returning to the my.yahoo.com page and trying to log in and each time I got 
the failure.  Finally I exited the browser.  Then reentered.  No more problems.

So the failure appears to be occuring on the first browser session after a 
rebuild (or re-install).  Again, can someone comment if that is what they are 
observing as well.

******* Additional Comments From Stephen P. Morse 2000-08-09 16:47 

Aha, now that the conditions are better understood (first browser session only), 
I am able to reproduce this problem on a mozilla build as well.  So this is not 
a commercial-only problem!

************ Additional Comments From Tom Everingham 2000-08-09 17:09 

rechecked the pr2 release and it still passes.  This is only showing up in the 
m18 builds.

******************** Additional Comments From leaf 2000-08-09 17:16

Very interesting. I see the workaround working now (i could have sworn i
restarted once or twice, but obviously i was mistaken). But i still don't have
the problem with new installs of mozilla (maybe my existing profile is affecting
the behaviour, though).

************ Additional Comments From Tom Everingham 2000-08-09 17:34 

I'm also failing in this simple test case that sets a cookie via minimal http 
commands  http://bubblegum:4321/01cookies.html   (on first session that is).  
Cookie doesn't appear to be set. 

********** Additional Comments From Stephen P. Morse 2000-08-09 17:53

And the problem is definitely not happening on win32.  Rebuilt win32 and still 
no problem.  So it is linux only, both commericial and mozilla.



*********** Additional Comments From Stephen P. Morse 2000-08-10 11:34

And some more information.  It is indeed a cookie problem (so many times cookies 
get blamed unfairly).  The cookies are collected but are not being sent out on 
that first run.

And still more information.  You do not need to rebuild or reinstall to see the 
problem.  Simply delete your component.reg file.  This is beginning to smell 
like bugzilla bug 46989.

Comment 12

[Stephen P. Morse]

Correction:  above I said that cookies [for my.yahoo.com] are getting collected 
but are not being sent out on that first run.  They are not even getting 
collected.  The http response contains a set-cookie header which the browser 
sees, but the cookie does not get set (as demonstrated by using the cookie 
viewer).

However if I visit home.netscape.com, it's cookies are getting set.  Then if I 
return to my.yahoo.com, again it's cookies are not getting set.  Still 
investigating.  I'm suspecting that this has something to do with cookies set 
via javascript versus cookies set in the http response header.

Comment 13

[Stephen P. Morse]

Here's where we are failing in the linux build.  In the Init() method of 
nsCookieHTTPNotify.cpp there is the following code:

    rv = pNetModuleMgr->RegisterModule
      (NS_NETWORK_MODULE_MANAGER_HTTP_REQUEST_PROGID,(nsIHTTPNotify *)this);
    if (NS_FAILED(rv)) return rv;

    rv = pNetModuleMgr->RegisterModule
      (NS_NETWORK_MODULE_MANAGER_HTTP_RESPONSE_PROGID, (nsIHTTPNotify *)this);
    return rv;

We are failing on the first register-module call.  So we are not installing the 
cookie observers for either http request or http response.  That explains all 
the symptoms and it's now clear why javascript cookies would get set but not 
http cookies.

So now to figure out why the failure when attempting to register the observers.

Comment 14

[Stephen P. Morse]

And working backwards, the failure is in nsNetModuleManager::RegisterModule is 
at the following line of code:

    nsNetModRegEntry *newEntry = new nsNetModRegEntry(aTopic, aNotify, &rv);
    if (!newEntry) return NS_ERROR_OUT_OF_MEMORY;

    if (NS_FAILED(rv)) {  //  <----------- rv is bad here
        delete newEntry;
        return rv;
    }

dp, if you get any brainstorms as I'm swimming through this, please chime in.

Comment 15

[Stephen P. Morse]

And in the constructor for nsNetModuleRegEntry we are getting back a bad value 
on the following call:

    *result = eventQService->GetThreadEventQueue
        (NS_CURRENT_THREAD, getter_AddRefs(mEventQ)); 

Comment 16

[Stephen P. Morse]

And in GetThreadEventQueue is the following code (slightly abreviated):

  nsCOMPtr<nsIEventQueue> queue =
    getter_AddRefs((nsIEventQueue*)mEventQTable.Get(&key));
  if (queue) {
  } else {
    rv = NS_ERROR_FAILURE;
  }

and indeed queue is coming up as null.

Comment 17

[Suresh Duddi (gone)]

So could it be possible that the event queue is empty because we are in the
wrong thread.

Gagan, what do you think. The call thread starts from nsHTTPProtocolHandler ->
nsHTTPCookieNotify. I belive the protocol handler can be on any thread right!

This looks like either nsHTTPCookieNotify should find the right thread or necko
problem.

Comment 18

[Stephen P. Morse]

Modifying summary from:

   Cookie service created *after* 1st HTTP request when
   component.reg not present (Linux only)

to the more accurate:

   http cookies don't work if components.reg not present (linux only)

Comment 19

[Stephen P. Morse]

What dp said made a lot of sense.  From the output written to the console, I see 
that we install the cookie observers immediately after the autoregistration 
finishes.  So if the autoregistration does a thread switch and forgets to switch 
back, this could cause us to be on the wrong thread when the cookie observers 
are installed.  And it would also explain why everything works fine when 
component.reg exists (because we never need to execute autoregistration.

Comment 20

[Judson Valeski]

what build are you using. Last week I fixed the "cookies are loaded until 
*after* first http request" problem (see 
http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&fi
le=nsHTTPHandler.cpp&root=/cvsroot&subdir=mozilla/netwerk/protocol/http/src&comm
and=DIFF_FRAMESET&rev1=1.123&rev2=1.124 ).

Comment 21

[Judson Valeski]

GetThreadEvnetQueue() fails when a thread event q hasn't been created for the 
given thread. everything (but socket IO) is happening in one thread so I dn't 
suspect a switching problem.

Here's where we create the thread event q for apprunner 
http://lxr.mozilla.org/seamonkey/source/xpfe/appshell/src/nsAppShellService.cpp#
141 , are we not hitting that or is it failing for some reason. we need to be 
hitting that *before* any http stuff or cookie stuff happens.

Comment 22

[Stephen P. Morse]

I'm using a tree that I pulled yesterday.

Comment 23

[Stephen P. Morse]

Jud, you got it.  The call to nsAppShellService::Initialize (which is the code 
you cited) is occurring *after* the registering of the http request and response 
observers for the cookie module.  So what do we do about it?

The sequence of events is:

   - autregistration
   - registering of cookie-module observers
   - chrome registration 
   - creation of UI thread

Comment 24

[Stephen P. Morse]

And the sequence of events when component.reg already exists is as follows:

   - autoeregistration (which doesn't register anything)
   - creation of UI thread
   - registration of cookie-module observers
   (don't see anything in console output for chrome registration)

So why is the sequence of events different in the two cases?

Comment 25

[Stephen P. Morse]

And on win32 (which does not exhibit the problem) the sequence of events is:

   - autregistration
   - creation of UI thread
   - registering of cookie-module observers
   (no mention of chrome registration) 

Unlike linux, we get this same sequence of events whether or not the 
component.reg file is present.   Hmmm!

Comment 26

[Judson Valeski]

I'm going to defer to dougt on the sequence difference.

Comment 27

[Stephen P. Morse]

And yet another clue:

In all cases, the creation of the UI thread is called indirectly from main1 (in 
nsApprunner.cpp) at the line that reads:

   rv = appShell->Initialize( cmdLineArgs, nativeApp );

In all the good cases (win32 with and without component.reg, linux with 
component.reg) the registration of the cookie observers is called indirectly 
from the following line that is a few lines beyond the above line:

   rv = profileMgr->StartupWithArgs(cmdLineArgs);

So the creation of the UI thread occurs first and everybody is happy.  In the 
bad case (linux without component.reg), the registration of the cookie observers 
is occurring much sooner.  I have not yet figured out from where.  Stay tuned.

Comment 28

[Stephen P. Morse]

In the bad case the registring of the cookie observers is occuring indirectly 
from the following line in main1 which is several line before the lines cited in 
the previous comment:

   NS_SetupRegistry_1( needAutoreg );

So now I need to figure out why this line triggers the observer registration in 
linux but not in win32.

Comment 29

[Doug Turner (:dougt)]

Does anyone have a problem with simply constructing the eventQ inside of main1():

around here:

http://lxr.mozilla.org/seamonkey/source/xpfe/bootstrap/nsAppRunner.cpp#757


In our embedding examples, creation of the UI eventQ comes much sooner in the 
startup process. 

Comment 30

[Stephen P. Morse]

Thanks Doug, that solved the problem.  Attaching diffs for your code review.

Comment 31

[Stephen P. Morse]

Attachment: diff for nsAppShellService.cpp

Comment 32

[Stephen P. Morse]

Attachment: diff for nsAppRunner.cpp

Comment 33

[Doug Turner (:dougt)]

Can we remove the kEventQueueServiceCID from nsAppShellService.cpp?

otherwise r=dougt.  

Comment 34

[Judson Valeski]

yea. please remove extraneous cid stuff.

Comment 35

[Stephen P. Morse]

No, it's used elsewhere in the file.

Fix checked in.

Comment 36

[Tom Everingham]

verified:
Linux 2000081320

Comment 37

[Matthias Versen [:Matti]]

mid-air collision ? / bugzilla cleanup
Reopening (current State: verified and no resolution)

Comment 38

[Matthias Versen [:Matti]]

fixed

Comment 39

[Matthias Versen [:Matti]]

and fixed