509303 - buglist.cgi redirects to proxied port number

Status: RESOLVED FIXED

(Bugzilla :: Query/Bug List, defect)

Description

[Tamer Embaby]

User-Agent:       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; FDM; eMusic DLM/4; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)
Build Identifier: 3.4.1

I use the following arch.:

Frontend: Apache 2.2 - reverse proxy (fe.company.com - default port 80)
Backend: Apache 2.0 - bugzilla installation (be.company.com - port 7200)
Feature affected: Advanced search
OS: Solaris 10

I used this with 3.2.X and was working correctly when search for bugs using the "advanced search" feature, as follows:

Hit "Search" from main Bugzilla bar, choose "Advacned search", directly click "Search" beside "Summary" combo, bust lsit returns correctly, here is URLs used:

To access the advanced search page:
http://fe.company.com/bugz/query.cgi
Here what I get when I click search:
http://fe.company.com/bugz/buglist.cgi?query_format=advanced;bug_status=NEW...

Now I upgraded to 3.4.1 and since then the above scenario doesn't work anymore, since Bugzilla redirect to frontend hostname and backend port, here is URLs used:

To access the advanced search page:
http://fe.company.com/bugz/query.cgi
And here what I get when I click search:
http://fe.company.com:7200/bugz/buglist.cgi?query_format=advanced;bug_status=NEW...

The second URL is incorrect so the advanced search functionality doesn't work for my Bugzilla installation now.

Probably this might be related to one of the cgi->redirect() in buglist.cgi after the $cgi->clean_search_url()?

Thanks in advance.

Reproducible: Always

Steps to Reproduce:
0. Have frontend reverse porxy Apache, backend Bugzilla Apache, and access bugzilla using frontend URL
1. Hit "Search" from main Bugzilla bar
2. Choose "Advacned search"
3. Click "Search"
Actual Results:  
Malformed URL redirect with mixed:
- Frontend hostname
- Backend port number

Expected Results:  
Correct URL redirect with:
- Frontend hostname
- Frontend port number

Comment 1

[Max Kanat-Alexander]

What is the "urlbase" setting on your Bugzilla?

Comment 2

[Tamer Embaby]

urlbase: http://fe.company.com/bugz/

Comment 3

[Frédéric Buclin]

(In reply to comment #0)
> Probably this might be related to one of the cgi->redirect() in buglist.cgi
> after the $cgi->clean_search_url()?

Yes, I think so: print $cgi->redirect(-url => $cgi->self_url()); No idea how self_url() looks like.

Comment 4

[Max Kanat-Alexander]

Okay, yeah, this is a problem with the configuration of your reverse proxy, I think.

Comment 5

[Tamer Embaby]

OK, any idea how to work around this now (except for changing the backend to port 80 as this is not possible)?  Should this be fixed in Bugzilla::CGI?

This was introduced with 3.4.1 (probabely 3.4). I believe it shouldn't do this behavior (returning only the port of the backend while leaving hostname of the frontend - this way even mod_proxy_html would not work).

Comment 6

[Max Kanat-Alexander]

(In reply to comment #5)
> This was introduced with 3.4.1 (probabely 3.4). I believe it shouldn't do this
> behavior (returning only the port of the backend while leaving hostname of the
> frontend - this way even mod_proxy_html would not work).

  What was introduced is that Bugzilla 3.4 is now doing redirects in certain cases. (This is how the Search URL Shortening works.) By the HTTP specification, redirects require the full URL, which CGI.pm determines using the environment variables on your system. It is only using the data that Apache is providing it, as far as I am aware.

Comment 7

[Tamer Embaby]

(In reply to comment #6)
>   What was introduced is that Bugzilla 3.4 is now doing redirects in certain
> cases. (This is how the Search URL Shortening works.) By the HTTP
> specification, redirects require the full URL, which CGI.pm determines using
> the environment variables on your system. It is only using the data that Apache
> is providing it, as far as I am aware.

Yeah, but I guess it's not doing it the right way, because it's getting me only "the port" of the backend while leaving the hostname of the frontend intact.  It should use the "urlbase" value in this case and append the request path to it instead, and not the HTTP_PORT CGI variable for example.

So for example instead of using:
$cgi->redirect(-url => $cgi->self_url()); 

Use:
$cgi->redirect(-url => "$urlbase" . "SEARCH_URL_ETC"); 

It's important to have a non-port-80 Bugzilla's "advanced search" feature working when it's behind reverse proxy.

Comment 8

[Max Kanat-Alexander]

We actually want to use correct_urlbase() as little as possible--I'm already trying to eliminate it from forms in the UI and so forth. I'm pretty sure that proxies can be configured to set this all correctly for you so that Bugzilla does not have to be modified.

Comment 9

[Tamer Embaby]

OK, I have cheap workaround for the issue by adding extra PoxyPassReverse:

[1] ProxyPassReverse        /bugz           http://fe.company.com:7200/bugz

The orignal reverse proxy rules were:
[2] ProxyPass               /bugz           http://be.company.com:7200/bugz
[3] ProxyPassReverse        /bugz           http://be.company.com:7200/bugz

I still believe Bugzila is not behaving correctly since:
If it will return full URL to comply with the HTTP specs, it should return the HTTP header:
Location: http://be.company.com:7200/bugz

And in this case the FE proxy would perform correct reverse proxying using [2] and [3] proxy directives.

BUT the issue is, Bugzilla is returning the HTTP header:
Location: http://fe.company.com:7200/bugz

Which is not correct.  I believe the CGI.pm should not know anything about "fe.company.com" except for the "urlbase", and if it will use the "urlbase" to constuct the "Location:" header it should use it without appending "7200" of the BE portal.

Comment 10

[Daniel Piddock]

I have also experienced this bug. The problem lies in CGI.pm's url(), called by self_url(). It messes up port numbers when generating the full URL:
> $url = "$protocol://";
> my $vh = http('x_forwarded_host') || http('host') || '';
> $vh =~ s/\:\d+$//; # some clients add the port number (incorrectly). Get rid of it.
> if ($vh) {
>    $url .= $vh;
> } else {
>    $url .= server_name();
> }
> my $port = $self->server_port;
> $url .= ":" . $port
>   unless (lc($protocol) eq 'http' && $port == 80)
>         || (lc($protocol) eq 'https' && $port == 443);

Both Apache and lighttpd set the port in HTTP_X_FORWARDED_HOST for a good reason, which is then stripped by CGI.pm before being replaced by the local port number. This creates the original poster's problem. 

It's a bug, but not exactly bugzilla's fault. I wish CGI.pm didn't mess about munging the proxy data in with local url as it all breaks horribly, especially if bugzilla lives at a different path on the backend. Proxying creates a huge mess when you move to absolute urls based on ENV variables.

Fix it? Admins using reverse proxies add in more ProxyPassReverse (per comment 9) or bugzilla goes back to generating redirect urls (or anything absolute) from the urlbase param.

Comment 12

[Frédéric Buclin]

Max, is it severe enough to block 3.4.3 (assuming it's a Bugzilla bug)?

Comment 13

[Max Kanat-Alexander]

No, not quite severe enough to be a blocker--ProxyPass situations aren't extremely common.

Comment 14

[Oded Arbel]

I have the same problem with Bugzilla 3.4.2 (after upgrading from 3.2.3), though in my case the reverse proxy is rewriting the URL path:

Frontend URL: http://tracker.company.com/
Backend URL: http://dmzserver/bugzilla

The reason is that I use the same apache installation to host a few other web applications that are externally accessible through different virtual hosts (to make it easier on users). When buglist.cgi is POSTed to from the advanced search dialog it redirects to the full path of the backend: http://tracker.company.com/bugzilla/buglist.cgi

I think the solution is to mangle the URL returned from cgi->self_url by replacing everything up to "buglist.cgi" with $urlbase.

My current workaround was to change the advanced search's form from POST to GET. I'm not sure what adverse effects this will have, but I would really love to see this solved for 3.4.3.

P.S. If bugzilla developers are aware of this issue, why is this ticket still marked UNCONFIRMED?

Comment 15

[Oded Arbel]

Sorry for the bug spam, I just had to comment on this:

(In reply to comment #13)
> No, not quite severe enough to be a blocker--ProxyPass situations aren't
> extremely common.

I believe that reverse proxy situations are extremely common - in virtually /all/ companies I worked in/with/around, the Bugzilla installation is on a private network and exposed to interested parties through a reverse proxy. Sometimes the mapping does not modify ports or paths and will work with the 3.4 change, but often it does not and will break when they upgrade to 3.4.

Comment 16

[Max Kanat-Alexander]

(In reply to comment #14)
> P.S. If bugzilla developers are aware of this issue, why is this ticket still
> marked UNCONFIRMED?

  Because I think it might be a bug in CGI.pm, not in Bugzilla, but neither I nor any of the other Bugzilla developers have investigated it enough to actually report it to the CGI.pm developers. (If it was a bug, and we did report it, I'm sure they'd fix it--they're very good about that.)

Comment 17

[Oded Arbel]

Well, it is a bug somewhere (either in Bugzilla or CGI) as the behavior is broken under certain valid conditions (regardless of rarity for a second here).

My take on this is that Bugzilla ships their own CGI (as Bugzilla::CGI) so regardless if it is a bug also in the upstream CGI, it is a bug in code shipped by Bugzilla under the Bugzilla package - hence this is a Bugzilla bug.

One way to fix it might be to get it fixed upstream in CGI and then restream the new release to Bugzilla::CGI. But from looking at the CGI code, I think it works correctly in the scope of the CGI package. The fix I believe should be either on buglist.cgi after calling $cgi->self_url, or in Bugzilla's private copy of CGI.

Comment 18

[Max Kanat-Alexander]

(In reply to comment #17)
> My take on this is that Bugzilla ships their own CGI (as Bugzilla::CGI)

  No, we only override certain CGI.pm functions. You'll notice that redirect() is not one of them.

> One way to fix it might be to get it fixed upstream in CGI and then restream
> the new release to Bugzilla::CGI. But from looking at the CGI code, I think it
> works correctly in the scope of the CGI package. The fix I believe should be
> either on buglist.cgi after calling $cgi->self_url, or in Bugzilla's private
> copy of CGI.

  It might be possible to fix it in Bugzilla::CGI.

  If anybody can trace down the root cause of the problem and provide a solution, feel free to post a patch here (for Bugzilla) or to the CPAN RT (for CGI.pm).

Comment 19

[Oded Arbel]

Attachment: My fix for the problem

My previous workaround didn't work for me because Mylyn apparently uses POST for its queries, so I did this instead - just replace everything before buglist.cgi with the urlbase

Comment 20

[Oded Arbel]

(In reply to comment #18)
> (In reply to comment #17)
> > My take on this is that Bugzilla ships their own CGI (as Bugzilla::CGI)
> 
>   No, we only override certain CGI.pm functions. You'll notice that redirect()
> is not one of them.

Yes, sorry - my bad. I actually read it correctly but then forgot about it :-(

Comment 22

[Max Kanat-Alexander]

I've filed the following bug against CGI.pm:

  https://rt.cpan.org/Ticket/Display.html?id=51562

Comment 23

[Max Kanat-Alexander]

A fix for this was just checked in to the CGI.pm git repository that contains the latest bug-fixes, and I'd expect the next-released version of CGI.pm will have this fixed in it.

Comment 24

[Max Kanat-Alexander]

This is fixed in CGI.pm 3.49, so anybody who upgrades to that version will have this bug fixed for them.

Comment 25

[Max Kanat-Alexander]

Attachment: v1

Okay, this updates the requirement to CGI 3.49 to fix this problem. Probably doesn't hurt to require this version in general anyway, since there have been so many CGI.pm bug-fixes lately.

Comment 26

[Max Kanat-Alexander]

  I think we only need to update the requirement on HEAD. For the branches, we can just tell people to update CGI.pm if they hit this problem.

Comment 27

[Frédéric Buclin]

Comment on attachment 453504 [details] [diff] [review]
v1

r=LpSolit for head only.

Comment 28

[Max Kanat-Alexander]

Thanks for the review, LpSolit! :-)

Committing to: bzr+ssh://bzr.mozilla.org/bugzilla/trunk/
modified Bugzilla/Install/Requirements.pm
Committed revision 7238.

Comment 31

[John Burgess]

I'm afraid I'm experiencing this problem with CGI.pm version 3.49.
Server is windows 2008 Build 6002 (SP1).  Bugzilla 3.6.2 with perl 5.10.1 and Apache 2.2.14, all installed using the new windows installer package, after which I imported my old (bugzilla 3.2.3) mysql and ran checksetup.pl again.