436442 - RSS pages have broken referer behavior

Status: RESOLVED INACTIVE

(Firefox Graveyard :: RSS Discovery and Preview, defect)

Description

[Sean Middleditch]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9b5) Gecko/2008043010 Fedora/3.0-0.60.beta5.fc9 Firefox/3.0b5
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9b5) Gecko/2008043010 Fedora/3.0-0.60.beta5.fc9 Firefox/3.0b5

For any RSS page, Firefox 3.0b5 will first load the page, and then immediately load it again.  THis is not visible to the user.  However, on the second page load, the Referer header is not set.

I caught this while trying to add hotlink protection to a playlist file (implemented using Media RSS).  When loading the player in IE, Opera, or Safari, I see a single request for the page with the referer set properly and everything works.  With Firefox, I see two page requests, and the second one has no referer.  When the second page request forwards to an error page due to the hotlink protection, the end user ends up seeing that instead of the original request which served the RSS just fine.

If I change the output to in any way not look like RSS (changing the Content-Type header has no effect - Firefox seems to be looking at the actual content) this behavior goes away.  Something as simple as changing the root element of the returned XML document from <rss> to <rst> will result in Firefox not re-loading the page (with or without the referer).

Reproducible: Always

Steps to Reproduce:
1. open an RSS file in firefox
2. look at the access logs and see the two requests, one without a referer
Actual Results:  
page loads twice, the second time without a referer

Expected Results:  
the page should only load once (and with the referer)

Fedora 9 build of Firefox 3 b5, default theme.  The Web Developer extension and Flash Block extensions (latest versions of both) are installed.

Comment 1

[Sylvain Pasche]

I've created a page with links to RSS files (http://developer.mozilla.org/webwatch/?feed=rss or http://rss.cnn.com/rss/cnn_topstories.rss), but I don't see these files being requested more than once when I click on a RSS link and the correct Referer header is sent (I'm using 3.0 rc2).

Do you know of a public page where you can reproduce this? It may also be useful to test in a profile with no extensions.

Comment 2

[Sean Middleditch]

I tried RC1 with a pure profile.  Same result.  Here is a test page:

http://middleditch.us/sean/bug/firefox-rss-referer/test.php

When you click the link, the following page is RSS that should list the referer in the item.  In firefox it just says "Referer: " but in IE, Opera, or Safari it will properly say "Referer: http://middleditch.us/sean/bug/firefox-rss-referer/test.php"

I don't seem to be getting the page relaod behavior on this test, but I am getting the missing referer.  The page reload was confirmed on my (unfortunately non-public) dev project site because if I added debugging statements, the output would work and I'd see the referer in both the debug output and the XML (the XML would be invalid because of the debugging text dumped out before the XML).  I then looked at the logs and saw that every page load created two entries, one with a referer and then one without.

Here are my Apache logs from a test with the above URL.  I clicked the link on the test landing page once and once only.

68.32.92.23 - - [30/May/2008:23:35:45 -0400] "GET /sean/bug/firefox-rss-referer/test.php HTTP/1.1" 200 108 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9b5) Gecko/2008043010 Fedora/3.0-0.60.beta5.fc9 Firefox/3.0b5"
68.32.92.23 - - [30/May/2008:23:35:47 -0400] "GET /sean/bug/firefox-rss-referer/test2.php HTTP/1.1" 200 384 "http://middleditch.us/sean/bug/firefox-rss-referer/test.php" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9b5) Gecko/2008043010 Fedora/3.0-0.60.beta5.fc9 Firefox/3.0b5"
68.32.92.23 - - [30/May/2008:23:35:47 -0400] "GET /sean/bug/firefox-rss-referer/test2.php HTTP/1.1" 200 325 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9b5) Gecko/2008043010 Fedora/3.0-0.60.beta5.fc9 Firefox/3.0b5"

Note the two entries for test2.php, and how only the first one has a referer.

Comment 3

[Sylvain Pasche]

Here's what happen: the channel with the rss file is first opened when you click on the link and then opened again in FeedConverter.js, handleResult().

When using a php script there is no expiry or validation headers, so another http request hits the wire, this time without the Referer header. One thing you can do to avoid this is to use a header that enables the browser to use the cache and not perform this second http request. In your PHP script, you can add:

header("Cache-Control: max-age=300");

So that Firefox will use the rss file in the cache for 5 minutes instead of requesting it two times.

Maybe what could be done here is to use a LOAD_FROM_CACHE flag on the new channel so that content is forced to be taken from cache.

Comment 4

[Sylvain Pasche]

Attachment: use LOAD_FROM_CACHE on the chromeChannel to avoid two requests

Here's the idea. I don't get the two requests with this.

Comment 5

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.