156435 - view-source: a request is sent but no cookies

Status: RESOLVED DUPLICATE of bug 787304

(Core :: Networking: Cookies, defect, P3)

Description

[Georg Maaß]

There is a general problem with view-source and sessions. For the future there
should be an option to generate the view-source alternatively from

a) DOM or 
b) from cache or 
c) from a resent request, where the resent request must be exactly as the
original request including protocoll and all HTTP headers.

At the moment there is only the variant creating the view-source from sending a
new request, but this new request sent to the server is not exactly equal to
ther original request. I.e. the cookies (containing the session id) are missing.

The first stage of solving that problems should be to resent the request exactly
as the original request including all HTTP headers. In future releases also the
other two variants may be provided, but first the problem of resending the
request without the original HTTP headers should be fixed.

The problem occures on Mozilla 1.0

Comment 1

[Boris Zbarsky [:bzbarsky]]

> a) DOM or 

This exists in current nightlies (view selection source)

> b) from cache or 

This is what 1.0 does (unless the document is not in the cache)

> c) from a resent request,

This should only happen if the document is not in the cache (eg. the cache is
off).  The request does not include cookies that went with the original request
only in the case when the cookies got deleted after loading the original
document.....

Is there an actual page that demonstrates this problem?

Comment 2

[Matthias Versen [:Matti]]

-> ViewSource

Comment 3

[Georg Maaß]

> > b) from cache or 
>
> This is what 1.0 does (unless the document is not in the cache)

If this is, what it does, then the cache mechanism of Mozilla seams to be quite
different to that of NN4. With NN4 I never detected a repetition of the request,
when I tried to look into the source when disc cache and memory cache are both
turned off.

What I need is a way to see, which code _was_ sent by the server but without
using that code again for new request. In my Web Development environment caching
is allways turned off to allways get the response from the server and never from
cache. But when I want to inspect the response, I do not like the browser to
make a new request; instead of sending a new request to the server, getting a
new response, it should display the code of the last response.

For Web Development a cache mode "cache for source view only" would be nice.
This could advice mozilla to take stuff from cache only if this stuff is
requested for source view or similiar activities.

> The request does not include cookies that went with the original request
> only in the case when the cookies got deleted after loading the original
> document.....

Session-IDs handles by cookies will be not transmited, if the cookies are not
resent too. This means that the request made by the source-view is not the same
as the original request. But it looks like a not logged in user. As result of
this, the server may answer with a completely different response.

This should be fixed.

And the "cache for source-view only" mode should be added, because a request may
result in server side modifications, which might cause a different result, each
time the same request is sent. (I.e. counters, etc.)

In my actual project there is a CMS. There the user may delete stuff on the
server or rearrange stuff. Resending the same request may find the server in a
completly different situation i.e. objects deleted by the first request are no
longer available, when a repetion of the request again tries to delete them.
This problem can be fixed only by a "cache for source-view only" mode, which
alowws me to cache the responses only for usage by the source-view.

Comment 4

[Boris Zbarsky [:bzbarsky]]

> then the cache mechanism of Mozilla seams to be quite different to that of NN4

Yes.  It is quite different.

> to allways get the response from the server and never from cachea
[sic]

You sound like you want the "Compare to cache every time" option under cache
prefs...  Or file a separate bug on the cache component to support this extra cache.

> when a repetion of the request again tries to delete them

Such requests would be POST requests, and we ask whether the user wants to
resend the POST data on such requests....

Back to the real problem here, the cookie issue.  I have not been able to
reproduce this in my testing.  Is there a publicly available testcase, or even
one you could point me to via a private mail, that would allow me to reproduce
the problem?

Comment 5

[Georg Maaß]

> > when a repetion of the request again tries to delete them
>
> Such requests would be POST requests, and we ask whether the user wants to
> resend the POST data on such requests....

No, such request do not need to be POST request. On POST requests looking into
the source is imposible. It asks whether to repost the data, but then it
displays the content instead of the source in the "source window". For this
other problem there is an other bug pending. I don't know the number now.

The problem I speak about also occures on GET requests. And it does not target
only the Cookie headers but obiously all HTTP headers, because on my private
pages the server switches the language of the content depending on the HTTP
header infos about the favourite language. If that info disapears it switches to
English as default language. This happens, when I toggle displaying images.
Mozilla then does rerequest the page but without the HTTP headers. As a result
of this, the server switches from German language to English.

So I think we need more advanced control on what to cache and what for to use
cached stuff. To prevent such problem on Development platforms, where caching is
normaly switched off.

Comment 6

[Boris Zbarsky [:bzbarsky]]

> For this other problem there is an other bug pending. I don't know the number
> now.

It's not pending, it's fixed.  Just not on the 1.0.x branch.

The headers sent with a re-request for view-source should be completely
identical to the headers on the original request.  If they are not, that is a
bug.  Not a cache bug, a networking bug.  View-source and session history use
the same exact mechanism, so they should suffer the same problems in this case.  

Again, could you please set up a publicly available testcase that demonstrates
this problem (since you have a setup that reproduces it already)?  That would
greatly expedite diagnosing and fixing the problem.

Comment 7

[Georg Maaß]

I'll try to extract one from the not yet public project onto a public part,
where you can test your self. This may take a few days, until I go back to the
project.

Comment 8

[Georg Maaß]

For the moment I can not present a sample page for this problem because for
unknow reasons the problem does not occure any longer after I temporyily
switched ON caching.

Now I have to fight with the opposite problem. In spite of being disk cache und
memory cache are both set to zero and cache setting is set to "every time I view
the page" only the markup is requested from the server where images and
stylesheets are taken from anywhere else.

That is very curious. The origin problem disappeared when I set the disk Cache
and memory cache to 1 MB and the cache setting to "when the page is out of
date". But setting this again to: 0 MB disk cache and 0 MB memory cache and
"every time I view the page" does not reproduce the original behaviour but still
keeps stylesheets and images to be cached.

Might there be something destroyed in my browser profile? If so, how can I
repair this to get back a well defined and well know behaviour?

Comment 9

[Georg Maaß]

Configure your Mozilla this way in the advanced preferences dialog:


Memory Cache: [0] kB
Disk Cache:   [0] kB
Compare the page in cache to then page in network
[x] every time i vew the page


Now a sample URL:

http://gruenderportal.itanix.vnett.de/login.asp?login=login

username / login: bugzilla
password: bugzilla

now hit the status button in the bottom navigation of the page or just type
http://gruenderportal.itanix.vnett.de/status.asp into the location bar.

Now look into the source of the page and compare it to the page. Sometimes this
source is the code generated for the page, if the user is not logged in (or
cookies are off) sometimes it is the code for the logged in page content as
visible in the window. This is very confusing to me.

Comment 10

[Georg Maaß]

There might be a relation to the proxy settings. The problem occures with
SOCKETs v. 4. I dit not verify ist with SOCKETs v. 5. But with SOCKETs v. 5 I
can't reach some domains through our proxy firewall, so I switched to SOCKETs v. 4.

May be that this is a necessary detail to know about the problem

Comment 11

[Boris Zbarsky [:bzbarsky]]

I've been unable to reproduce this problem, so this _is_ very likely a proxy
settings issue...

Comment 12

[benc]

neeti: who should own this?

Comment 13

[Georg Maaß]

Hm,

if it is a proxy problem how can we detect whether it is a bug in communication
of Mozilla with the proxy or a bug in the proxy it self?

Comment 14

[Boris Zbarsky [:bzbarsky]]

set NSPR_LOG_MODULES=nsHttp:5
set NSPR_LOG_FILE=c:\httplog.txt

Start up Mozilla.  Load the problem page.  View source.  Quit mozilla.  Do
minimal amounts of http requests while you do all this (eg, turn off images
beforehand).

Then attach the log file to this bug.  That will have a log of what _Mozilla_ is
sending and receiving.  If it's sending the cookies, then it's a proxy bug. 
Otherwise, it's more likely to be a Mozilla bug.

Comment 15

[Georg Maaß]

> set NSPR_LOG_MODULES=nsHttp:5
> set NSPR_LOG_FILE=c:\httplog.txt

where do I have to do this?

I assume that this should be done in user.js but where does this file reside? I
don't find it on my Win 2000 machine.

If I must create that file, where must I create it to give Mozilla a chance to
find it?

Comment 16

[Boris Zbarsky [:bzbarsky]]

You do that in a command prompt.  Then you start mozilla from that command prompt.

Comment 17

[benc]

The easiest way is to use another browser...

Otherwise, you can do incoming & outgoing packet traces (much more difficult and
time consuming).

Comment 18

[Georg Maaß]

Attachment: most important parts of the http log file of Mozilla 1.0 2002053012

As you may see the rerequest sent to generate the source view does not contain
the Cookie header. As a result of this, the response data for this rerequest is
quite different to the response data of the original request. Without the
session cookie the server tells me to login where it sends the real content, if
a valid session cookie is present.

Comment 19

[Georg Maaß]

Attachment: zipped complete http log file

This is the complete http log file zipped with winzip.

Comment 20

[benc]

-> viewsource
Networking people can help analyze the problem, but the code fix is likely to be
in the View-soure code.

Comment 21

[Boris Zbarsky [:bzbarsky]]

There is no view-source code involved in this stuff. This is either a networking
bug, a session history bug, or a cookie bug.   Most likely, this is running into
our "do not send cookies to non-HTTP protocols" stuff or something like that. 
Reassigning accordingly.

Comment 22

[Jeremiah Blatz]

I have experienced this bug in version 1.4a. If you clear your cache, than view source of, say, an 
ASP page that uses session cookies, you get a new session. Mozilla is pretty good about htis, but 
Phoenix never wants to use the cache, so it tickles this bug all the time.

Comment 23

[Boris Zbarsky [:bzbarsky]]

The Phoenix problem is just a (known) bug in the Phoenix view source impl.

Comment 24

[Steven Garrity [:sgarrity]]

This is made irrelevant by bug 153884

Comment 25

[Boris Zbarsky [:bzbarsky]]

Steven, bug 153884 is Camino-only.  This bug is reported on Windows.  Camino
does not run on Windows.

Comment 26

[kaeng]

Funny, that this two year old bug is still unconfirmed.

I'm experiencing the same with Firefox 1.0.4 under Linux. I'm using the plugin
"Live HTTP headers" to track the request. When I want to view the source of a
page, it is reloaded (which itself seems quite unnecessary). But in this new
request no cookies are sent.

Comment 27

[Peter 'vt100' Schwindt]

confirming to (maybe) get things going

OS -> All, according to comment 26

Comment 28

[Darin Fisher]

-> default owner

Comment 29

[Jonathan Addleman]

Hmm.. funny that a 5+ year old bug is still listed as NEW. I'm still seeing this in firefox 3.0b5 as well.

A site I develop for uses session cookies for authentication. On any page, if I do a view source, I just see the source for the "please log in" page. It's rather annoying! I can select some/all of the text and view source for the selection, but then it's slightly modified from what the server actually send - <br /> turns into <br> for example.

Comment 30

[Mikel Ward]

This is happening for me on a Confluence Wiki page where I have to log in.

Both File->Save As... and View->Page Source... are showing me the log in page, while the displayed page is the actual document content.

Can't easily tell what headers it's sending as it's over HTTPS.  (Will try creating a tunnel and using tcpdump when I have time.)

Comment 31

[[e-mail]]

-> me to look into sometime in the (likely distant) future, but don't let that stop anyone from picking it up...

Comment 32

[Daniel Veditz [:dveditz]]

This does NOT happen to me, even on https sites; the cookies get sent for view-source.

Does it happen on _every_ site for people who have the problem, or just some? Is there maybe some redirection that's making us think it's a 3rd-party cookie, and you have 3rd-party cookies blocked? Does it happen to you here on bugzilla? if you view-source this page do you find "Log out <yourname>" links, or "Log in" links when you view-source a page where you're logged in (such as this bug)?

If it's every site for you I'm at a loss. If it's only some sites can you give us some URLs of sites where we'll be able to get a login account (e.g. no banks or subscription sites, please). The wiki mentioned in comment 30 sounds promising if it's public.

Comment 33

[Georg Maaß]

With Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1.16) Gecko/20080703 SeaMonkey/1.1.11 it works on this page, where I get the Log out link

Comment 34

[Georg Maaß]

Does it realy repost the form here, when I type Command-U to view the source? I don't believe, because I in case of reposting the form I expect a mid air collision error message, but what I see here is the source I've got, when pressing the Commit button below. So this bugzilla page is probably not a valid test case for the problem.

Comment 35

[Georg Maaß]

In the response from bugzilla I got following HTTP headers:

HTTP/1.1 200 OK
Fri, 19 Sep 2008 08:59:34 GMT
Apache/2.2.3 (Red Hat)
mrapp51
timeout=300, max=1000
Keep-Alive
chunked
text/html; charset=UTF-8

There is nothing telling the browser not to cache the response, so in this case the source-view is probably done from the cache and no REPOST is done. So bugzilla is not a valid test case for this problem.

Comment 36

[Daniel Veditz [:dveditz]]

The source view is not done from the cache for bugzilla. I can see the request being made and cookies are sent. Looking for the log-out/log-in text was just handy for people to look for who don't have debugging tools. There's no POST here (unless you've just made a comment); for my tests I used GET on the bug itself.

Either way Bugzilla is not a valid testcase because you do not experience the problem here, answering my question from comment 32: it's only some sites. So we're not going to get much further until we find a site that affects both you and us.

Comment 37

[kaeng]

To present an example I just tried one of the pages (https and frames) that was the reason for my comment three years ago, because I know it hadn't changed since then. (Un)fortunately with Firefox 3.0.1 I can view the source, so I am no longer able to confirm this bug. Looking at comment #30 though it seems there still are some configurations where it doesn't work.

Comment 38

[Marc Buchmann]

Confirmed on 3.0.1 - Seems to occur when a long-idle window containing a session cookie is subsequently view-sourced, and the page's contents have been kicked out of the cache. FF re-hits the server for a fresh copy of the page contents, and the page's cookies are not sent with that request.

In most sites using cookie-based login, this will generally cause a redirect to another page ("your session timed out", login page, etc...) and that new page is what's view-sourced.

Interesting that FF can still somehow display the original page, with all contents, and with full functionality, yet not be able to display its source without a round trip to the server.

Comment 39

[Nathan Bryant]

I experienced this issue several years ago on Firefox version 1 or 2. (Can't remember.) At the time, I was working at a different company. We developed an in-house web application. I think some of the other developers decided to add Cache-control: no-store or Cache-control: no-cache (can't remember which) headers to our application. This may have had an impact on the issue.

Comment 40

[Georg Maaß]

The prefered source for the view-source should be the memory cache or disk cache because make a new request even with all HTTP headers from the original request results in a new situation on a webserver running an application with transactions. The simplest case of an applcation with transactions based on request is a request counter like known from the hit counters.

So if view-source causes a new request, thus causes the request counter to increment an to response with a new increased value, which is not the original value the user likes to see, when using view-source.

Comment 41

[Matthew Eernisse]

If Firefox is going to pull the page down again from the server (which seems to me not what any right-thinking person would want, given that the implication of "view source" is "view the source that has resulted in this stuff in the browser window I'm looking at"), I would at least expect Firefox to make precisely the same request as the original.

I'm sitting here right now looking at a case where "viewing source" is not sending the same session cookie that was in the original request, so what I get back is manifestly not the same source as what's loaded in the browser window.

I'll try to futz around with cache settings to ameliorate this, but the default behavior seems pretty broken for the implied meaning of "view source."

Comment 42

[Seungbeom Kim]

This bug makes the view source feature useless for websites that use cookies for authorization. Is anything being done to fix it?

Comment 44

[Daniel Veditz [:dveditz]]

(In reply to comment #42)
> This bug makes the view source feature useless for websites that use cookies
> for authorization. Is anything being done to fix it?

We weren't able to reproduce it when filed (see Boris's comments) and I can't reproduce it now. On a basic web page (this bug, in fact) when I view source it a) loads from cache if it can, b) resends the request with cookies if it's not in the cache.

There's a different problem on dynamic "Web 2.0" pages where the original source differs from the "generated" document. Both are useful in different sitations. view-source gets the original server content, for the generated page select all, right-click, and select "View Selection Source". Which choice should be the default "view source" is argued in other bugs and out of place here, but both are available.

_this_ bug is about cookie sending related to View Source as currently implemented. Boris and I (Mozilla Developers) are seeing cookies. George and Matthew aren't (and maybe Marc in comment 38, but it's equally possible in that case that the session timed out on the server side). In order to "fix" the problem we're going to have to see it: either a site where we can reproduce the problem, or a low-level http/cookie log as described by Boris earlier.

Comment 45

[skid.matt+fb]

Attachment: httplog.txt and bug156435.php that demonstrate the issue.

These are the results of running FF3.6.13 with the following env variables:
  NSPR_LOG_MODULES=nsHttp:5
  NSPR_LOG_FILE=~/httplog.txt

The steps taken were:
1.  Opened Firefox, loading the included 'http://www.example.com/bug156435.php' (which was set as the home page)
2. Reloaded the page.
3. Chose 'Page Source' from the 'View Menu'.
4. Reloaded the page again.
5. Quit.

The result was as previously described -- the "count" cookie was not sent when viewing the source.

Comment 47

[[e-mail]]

Reassigning to nobody. If anyone wants to work on this, feel free!

Comment 49

[Donald Sarratt]

Attachment: A short PHP script for reproducing the error, along with sample HTTP headers

(In reply to Daniel Veditz [:dveditz] from comment #44)
> (In reply to comment #42)
> > This bug makes the view source feature useless for websites that use cookies
> > for authorization. Is anything being done to fix it?
> 
> We weren't able to reproduce it when filed (see Boris's comments) and I
> can't reproduce it now. On a basic web page (this bug, in fact) when I view
> source it a) loads from cache if it can, b) resends the request with cookies
> if it's not in the cache.

> _this_ bug is about cookie sending related to View Source as currently
> implemented. Boris and I (Mozilla Developers) are seeing cookies. George and
> Matthew aren't (and maybe Marc in comment 38, but it's equally possible in
> that case that the session timed out on the server side). In order to "fix"
> the problem we're going to have to see it: either a site where we can
> reproduce the problem, or a low-level http/cookie log as described by Boris
> earlier.

https://bitbucket.org/ is currently displaying this problem, as do many django sites.

Setting a "Vary: Cookie" header will trigger this behaviour. I tested this by writing a tiny PHP script and serving it through nginx via PHP-FPM. The test script and HTTP header exchanges are attached. With the first exchange "Vary: Cookie" was not set, and a subsequent View Page Source sent the cookie as it should. After reloading the page with "Vary: Cookie" set, a subsequent View Page Source didn't send the cookie. Hopefully this gets us a reproducible test so someone can get around to fixing this. :)

Comment 50

[Daniel Roesler]

I'm not sure how to mark something as duplicate, but Bug #307089 is a duplicate of this bug. Am I not allowed to mark duplicates?

Also, this also now affects the Firefox Developer Tools Debugger.

When you load the dev tools debugger, the page is re-requested without the current cookies. If you are logged into a site via cookie-based sessions (django commonly uses these), the debugger shows the source code to the login page (since the website thinks you're logged out).

Reproduction steps:
1) Login to https://bitbucket.org/ (or another django-based site)
2) Navigate to your profile page.
3) Open the developer tools (right click on the page and select "Inspect Element").
4) Click the "Debugger" tab in the dev tools.
5) The source code will show the source code of the login page, not the current page.

Comment 51

[Chris Adams]

(In reply to Daniel Roesler from comment #50)
> 1) Login to https://bitbucket.org/ (or another django-based site)
> 2) Navigate to your profile page.
> 3) Open the developer tools (right click on the page and select "Inspect
> Element").
> 4) Click the "Debugger" tab in the dev tools.
> 5) The source code will show the source code of the login page, not the
> current page.

This does not reproduce in FF31 with the default settings. Does your about:config show any custom settings for browser.cache?

Comment 52

[Daniel Roesler]

(In reply to Chris Adams from comment #51)
> This does not reproduce in FF31 with the default settings. Does your
> about:config show any custom settings for browser.cache?

I've tested again, and narrowed the reproduction steps down to third-party cookies.

Reproduction steps:
1) Preferences > Privacy > Use Custom Settings > Third Party Cookies: Never
2) Login to https://bitbucket.org/
3) Navigate to your settings page (e.g. https://bitbucket.org/account/user/<username>/)
4) Open the developer tools (right click on the page and select "Inspect Element").
5) Click the "Debugger" tab in the dev tools.
6) Select the "accounts/user/<username>" source.
7) The source code will show the source code of the login page, not the current page.

Can you please confirm? Even though the session cookie is not third party, the debugger seems to think it is.

Comment 53

[Donald Sarratt]

(In reply to Daniel Roesler from comment #52)
> (In reply to Chris Adams from comment #51)
> > This does not reproduce in FF31 with the default settings. Does your
> > about:config show any custom settings for browser.cache?
> 
> I've tested again, and narrowed the reproduction steps down to third-party
> cookies.
> 
> Reproduction steps:
> 1) Preferences > Privacy > Use Custom Settings > Third Party Cookies: Never
> 2) Login to https://bitbucket.org/
> 3) Navigate to your settings page (e.g.
> https://bitbucket.org/account/user/<username>/)
> 4) Open the developer tools (right click on the page and select "Inspect
> Element").
> 5) Click the "Debugger" tab in the dev tools.
> 6) Select the "accounts/user/<username>" source.
> 7) The source code will show the source code of the login page, not the
> current page.
> 
> Can you please confirm? Even though the session cookie is not third party,
> the debugger seems to think it is.

I confirm this replicates the issue in Firefox 42.0. Interestingly, it does not occur if Third Party Cookies are set to 'From Visited'. The behaviour appears to switch as soon as the cookie setting is changed - you don't have to restart Firefox or even refresh the page.

Comment 54

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258