160903 - Referrer sometimes omitted

Status: RESOLVED WORKSFORME

(Core :: Networking: HTTP, defect)

Description

[a_geek]

Hello,

I've got a small CGI that calculates the location of an image based on,
amongst other things, the referrer passed to it. This CGI program
sometimes receives the referrer, and sometimes not (saying "undefined"
for the variables where I dissect the referer).

The relevant code in the CGI looks like this (CGI.pm at 2.76):

$q = new CGI;
$ref = $q->referer();
$ref =~ /v=([^;]+);/;
$vh = $1;

(for what the relevant part of it is, in this case extracting the
parameter 'v' from the referer). This code sometimes (not always)
barfs on "$vh = $1;", but not when I use Netscape 4.77 to view the
same script. The page to load contains several images produced
in this manner, all via the same CGI, and some load while others
don't. Some of those that don't load do it because of this problem.

I have HTTP1.1 keepalive on and pipelining off (Linux,
build id 2002060721).

Comment 1

[James Ross]

Could the reported please give the/a URL that this occurs on, so the bug can be
confirmed or otherwise.

Comment 2

[Boris Zbarsky [:bzbarsky]]

are any of the urls involved https?

Comment 3

[a_geek]

Hi,

yes, all URLs are HTTPS. I don't serve mixed content since that doesn't
work - some browsers decide to display only the secure content, so if
I want the page to be complete, it must be served secure only.

Comment 4

[Boris Zbarsky [:bzbarsky]]

OK.  Then the next question is, do all the URLs have the same hostname and port?

Comment 5

[a_geek]

hi,

wrt. comment #1, I can't give URLs in a reasonable way since the whole
thing is also password protected and not for general access (as you have
already guessed). I probably can, however, post the software i use to
serve the images. This will take some time to roll for publication, though.

I'll also retry with HTTP/1.0 (over SSL, of course) and see if the problem
persists, as well as trying other browsers.

Comment 6

[a_geek]

To comment #4: yes, these are all to the same host and port.

It's a simple page like

...
<img src="img/pic1.png">
<img src="img/pic2.png">
<img src="img/pic3.png">
...

and they all get served through the same CGI acting upon the referer and
the user (the one logged in).

The images are stored on disk outside the web server's tree, and their
individual paths are calculated from the data collected above. Then the
CGI just opens the files, sends a HTTP header containing, amongst others,
the MIME type, and just shoves the image out to the browser.

Comment 7

[James Ross]

Ok, I don't think the image serving software is the problem here, but we do need
a base case that it happens on to check out what's going on.

Do we have any idea if it only happens over https or does it also happen with http?

Comment 8

[a_geek]

To comment #7:

I *guess* that it only happens over HTTPS. I developed the software
under HTTP (all on my local Linux worstation) and then pushed it to
the OpenBSD (3.1-stable) production system when I found no errors.
There I re-tested with HTTPS. There the problem appears, and only
there, not on my local machine.

Since there are several differences between these systems, I'm still
not quite sure if it's a browser or server bug, but will try to reproduce
locally with HTTPS, too.

The list of differences is as follows:

Development system: Debian unstable, Apache 1.3.26 with DSOs, CGI.pm
at 2.81

Production system: OpenBSD 3.1-stable, Apache 1.3.24 patched, all static,
CGI.pm at 2.76.

(patched = OpenBSD's patches for various security problems applied, see
http://www.openbsd.org/errata.html)

Comment 9

[a_geek]

Hello,

I was able to check that the error does not occur whith IE6 against the
production system. All images come through as they should, using HTTP/1.1.
Seeing this work on other browsers I suspect that the error really is on
Mozilla's end.

My suspicion is that his IE setup is pretty much as it comes out of the
box, but what that means apart from having HTTP/1.1 on, I don't know.

Comment 10

[Darin Fisher]

without having read this bug fully... perhaps bug 141641 is related?

Comment 11

[James Ross]

Having read through the comments to bug #141641, I would say this is not the
same bug, at least as described, but may be related. Bug #141641 seems to be
about sending the referrer to other domains when using https. This bug however
(as I understand it) is about the referrer when using the same protocol (https),
server and port.

Please could the reporter confirm that the issue here is the referrer not being
sent at all, and not just the referrer being incomplete.

Comment 12

[a_geek]

Hello,

to comment #11:

I've just confirmed that the referer sometimes isn't sent at all. Well, at
the same time I found out that IE 5 has the same problem :((( SO while
Mozilla may be doing something wrong, I'll have to rewrite the logic of
this app either way since I need to _reliably_ pass the required info to
my program.

Comment 13

[a_geek]

Attachment: environment when the request fails (sanitized)

to comments #11, #12

Comment 14

[James Ross]

If it's also happening in IE, I'm tempted to say it's something at the server
end, or possibly a proxy/cache between you and the server that's doing something
wrong.

As for the enviroment, there seems to be a 404 on the image on the page, which
then redirects to your image script - could this be related to the referrer
missing? Could you not put the URL to the image script in the IMG tag itself,
with a parameter to indicate which image is needed? (something like <img
src="img/index.cgi?pic=img/pic1.png">?)

Comment 15

[a_geek]

Hi,

wrt comment #14: I can't do it exactly this way without great effort,
but will try something similar.

Please note that so far the record is like this:

NS4, IE6: works ok
Moz1, IE5: doesn't work ok

SSL only, HTTP w/o SSL works fine with Mozilla, too.

(Is it ok to assume that IE5 has no bugs?)

Comment 16

[benc]

I'm working on setting up some more through Refer: testing, but I haven't gotten
to https: cases yet.

Can you give us an update w/ a new mozilla version and a current problem
description?

Comment 17

[Alexander Opitz]

Reporter can you reproduce this bug with a newer build (1.4 final)?
If not, then please close this bug as worksforme. Thanks.

Comment 18

[Andrew Hagen]

No response in over a month. WFM.