761046 - Several security-related headers intermittently repeated on mod_perl bugzilla installations

Status: RESOLVED FIXED

(Bugzilla :: Bugzilla-General, defect)

Description

[Reed Loden [:reed]]

I noticed this when trying to figure out why the PoC in bug 761043 works according to the author...

$ curl -I https://bugzilla.mozilla.org/buglist.cgi
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pp-app-bugs01
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
Date: Mon, 04 Jun 2012 02:10:43 GMT
Refresh: 10; URL=query.cgi
Transfer-Encoding: chunked
Connection: Keep-Alive
X-frame-options: SAMEORIGIN, SAMEORIGIN

I cannot reproduce this upstream, so seems bmo-specific. Zeus doing something to the headers to double them for Strict-Transport-Security and X-Frame-Options? If so, that's really bad, as it seems the browsers are throwing out the invalid headers, allowing things that shouldn't be allowed.

Filing as critical, as there's an active PoC that seems to be abusing this issue in order to work.

Comment 1

[Brandon Burton [:solarce]]

Confirmed this appears to be being added by Zeus, will see why


Server directly:

bburton@andesite ~$ curl -k -v -H "Host: bugzilla.mozilla.org" http://pp-app-bugs01.phx.mozilla.com:443/ -o /dev/null     ‹1.9.2-p290›
* About to connect() to pp-app-bugs01.phx.mozilla.com port 443 (#0)
*   Trying 10.8.82.11...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to pp-app-bugs01.phx.mozilla.com (10.8.82.11) port 443 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Accept: */*
> Host: bugzilla.mozilla.org
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Jun 2012 02:35:11 GMT
< Server: Apache
< Strict-transport-security: max-age=2629744; includeSubDomains
< X-frame-options: SAMEORIGIN
< X-Backend-Server: pp-app-bugs01
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
< 
{ [data not shown]
100 19374    0 19374    0     0   115k      0 --:--:-- --:--:-- --:--:--  136k* Connection #0 to host pp-app-bugs01.phx.mozilla.com left intact

* Closing connection #0

Through Zeus

bburton@andesite ~$ curl -v https://bugzilla.mozilla.org/buglist.cgi -o /dev/null                                         ‹1.9.2-p290›
* About to connect() to bugzilla.mozilla.org port 443 (#0)
*   Trying 63.245.217.60...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to bugzilla.mozilla.org (63.245.217.60) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using RC4-SHA
* Server certificate:
* 	 subject: serialNumber=jOqjuRFD1V/oclcVMdxx19LpRe5oLYFY; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; OU=IT; CN=bugzilla.mozilla.org
* 	 start date: 2011-11-23 02:31:32 GMT
* 	 expire date: 2013-11-24 21:05:00 GMT
* 	 subjectAltName: bugzilla.mozilla.org matched
* 	 issuer: C=US; O=GeoTrust, Inc.; CN=GeoTrust SSL CA
* 	 SSL certificate verify ok.
> GET /buglist.cgi HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: bugzilla.mozilla.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Apache
< X-Backend-Server: pp-app-bugs01
< Vary: Accept-Encoding
< Content-Type: text/html; charset=UTF-8
< Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
< Date: Mon, 04 Jun 2012 02:35:31 GMT
< Refresh: 10; URL=query.cgi
< Transfer-Encoding: chunked
< X-frame-options: SAMEORIGIN, SAMEORIGIN
< 
{ [data not shown]
100 15885    0 15885    0     0  55760      0 --:--:-- --:--:-- --:--:-- 60862* Connection #0 to host bugzilla.mozilla.org left intact

* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]

Comment 2

[Brandon Burton [:solarce]]

Nothing immediately obvious in https://pp-zlb08.phx.mozilla.net:9090/apps/zxtm/?name=bugzilla-ssl&section=Virtual%20Servers%3AEdit or the Rules being applied to this virtual server, but a couple do append headers via Traffic Script, so maybe a bug in Zeus? 

Add Jake, Corey, and Jeremy in hopes one of them has some quick insight. If not, I'll open a case with Zeus in the morning

Comment 3

[Brandon Burton [:solarce]]

:reed , can you add me to 761043? If it's a serious exploit, is it something that looks like we can block by IP or header with a Zeus TS rule for now?

Comment 4

[Reed Loden [:reed]]

(In reply to Brandon Burton [:solarce] from comment #3)
> :reed , can you add me to 761043? If it's a serious exploit, is it something
> that looks like we can block by IP or header with a Zeus TS rule for now?

No, not really... The PoC is just abusing X-Frame-Options not working properly, allowing data from bug searches loaded in an <iframe> to possibly be leaked to an outside party.

Does it have anything to do with the redirect that buglist.cgi performs? Is Zeus handling redirects weirdly somehow?

Comment 5

[Brandon Burton [:solarce]]

Zeus isn't doing any redirects in its rules, it just does some cookie manipulation and the other rules for this virtual server are protection related

However, I briefly disabled both the rules applied and the security class (which is only configured to block certain IPs) and neither changed the header behavior, so it's not clear to me what's going on here

Comment 6

[Reed Loden [:reed]]

Just to confirm the origin servers aren't doing something dumb, can you do this:

curl -k -v -H "Host: bugzilla.mozilla.org" http://pp-app-bugs01.phx.mozilla.com:443/buglist.cgi -o /dev/null

Comment 7

[Brandon Burton [:solarce]]

Yes, I did that in c1, going through Zeus is causing those headers to get appended

Comment 8

[Reed Loden [:reed]]

(In reply to Brandon Burton [:solarce] from comment #7)
> Yes, I did that in c1, going through Zeus is causing those headers to get
> appended

No, you didn't include /buglist.cgi in the #c1 test. Can you test with /buglist.cgi?

Comment 9

[Reed Loden [:reed]]

(In reply to Reed Loden [:reed] from comment #8)
> (In reply to Brandon Burton [:solarce] from comment #7)
> > Yes, I did that in c1, going through Zeus is causing those headers to get
> > appended
> 
> No, you didn't include /buglist.cgi in the #c1 test. Can you test with
> /buglist.cgi?

or, I should say, from what #c1 says, it doesn't look like you included /buglist.cgi in the command you ran. Just wanted to confirm that.

Comment 10

[Brandon Burton [:solarce]]

Oh, I see, so yes, it does appear to be a /buglist.cgi thing

here is /buglist.cgi 

bburton@andesite ~$ curl -k -v -H "Host: bugzilla.mozilla.org" http://pp-app-bugs01.phx.mozilla.com:443/buglist.cgi -o /dev/null
* About to connect() to pp-app-bugs01.phx.mozilla.com port 443 (#0)
*   Trying 10.8.82.11...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to pp-app-bugs01.phx.mozilla.com (10.8.82.11) port 443 (#0)
> GET /buglist.cgi HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Accept: */*
> Host: bugzilla.mozilla.org
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Jun 2012 03:10:34 GMT
< Server: Apache
< Refresh: 10; URL=query.cgi
< Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
< X-frame-options: SAMEORIGIN, SAMEORIGIN
< X-Backend-Server: pp-app-bugs01
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
< 
{ [data not shown]
100 15885    0 15885    0     0  65474      0 --:--:-- --:--:-- --:--:-- 79425* Connection #0 to host pp-app-bugs01.phx.mozilla.com left intact

* Closing connection #0

And also note no issue when just requests / to bmo through Zeus

bburton@andesite ~$ curl -v https://bugzilla.mozilla.org/ -o /dev/null                                               6 ↵  ‹1.9.2-p290›
* About to connect() to bugzilla.mozilla.org port 443 (#0)
*   Trying 63.245.217.60...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to bugzilla.mozilla.org (63.245.217.60) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using RC4-SHA
* Server certificate:
* 	 subject: serialNumber=jOqjuRFD1V/oclcVMdxx19LpRe5oLYFY; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; OU=IT; CN=bugzilla.mozilla.org
* 	 start date: 2011-11-23 02:31:32 GMT
* 	 expire date: 2013-11-24 21:05:00 GMT
* 	 subjectAltName: bugzilla.mozilla.org matched
* 	 issuer: C=US; O=GeoTrust, Inc.; CN=GeoTrust SSL CA
* 	 SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: bugzilla.mozilla.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Apache
< X-Backend-Server: pp-app-bugs02
< Vary: Accept-Encoding
< Content-Type: text/html; charset=UTF-8
< Strict-transport-security: max-age=2629744; includeSubDomains
< Date: Mon, 04 Jun 2012 03:10:54 GMT
< Transfer-Encoding: chunked
< X-frame-options: SAMEORIGIN
< 
{ [data not shown]
100 19374    0 19374    0     0  28789      0 --:--:-- --:--:-- --:--:-- 70708* Connection #0 to host bugzilla.mozilla.org left intact

* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]

Also fine on a random bug

bburton@andesite ~$ curl -v https://bugzilla.mozilla.org/show_bug.cgi\?id\=760659 -o /dev/null                            ‹1.9.2-p290›
* About to connect() to bugzilla.mozilla.org port 443 (#0)
*   Trying 63.245.217.60...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to bugzilla.mozilla.org (63.245.217.60) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using RC4-SHA
* Server certificate:
* 	 subject: serialNumber=jOqjuRFD1V/oclcVMdxx19LpRe5oLYFY; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; OU=IT; CN=bugzilla.mozilla.org
* 	 start date: 2011-11-23 02:31:32 GMT
* 	 expire date: 2013-11-24 21:05:00 GMT
* 	 subjectAltName: bugzilla.mozilla.org matched
* 	 issuer: C=US; O=GeoTrust, Inc.; CN=GeoTrust SSL CA
* 	 SSL certificate verify ok.
> GET /show_bug.cgi?id=760659 HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: bugzilla.mozilla.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: Apache
< X-Backend-Server: pp-app-bugs03
< Vary: Accept-Encoding
< Content-Type: text/html; charset=UTF-8
< Strict-transport-security: max-age=2629744; includeSubDomains
< Date: Mon, 04 Jun 2012 03:12:19 GMT
< Transfer-Encoding: chunked
< X-frame-options: SAMEORIGIN
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{ [data not shown]
100 43269    0 43269    0     0  47317      0 --:--:-- --:--:-- --:--:-- 48671* Connection #0 to host bugzilla.mozilla.org left intact

* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]

So it does appear to be a /buglist.cgi issue

Comment 11

[Brandon Burton [:solarce]]

For thoroughness, 02 and 03 are also returning the same

bburton@andesite ~$ curl -k -v -H "Host: bugzilla.mozilla.org" http://pp-app-bugs02.phx.mozilla.com:443/buglist.cgi -o /dev/null
* About to connect() to pp-app-bugs02.phx.mozilla.com port 443 (#0)
*   Trying 10.8.82.12...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to pp-app-bugs02.phx.mozilla.com (10.8.82.12) port 443 (#0)
> GET /buglist.cgi HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Accept: */*
> Host: bugzilla.mozilla.org
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Jun 2012 03:14:55 GMT
< Server: Apache
< Refresh: 10; URL=query.cgi
< Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
< X-frame-options: SAMEORIGIN, SAMEORIGIN
< X-Backend-Server: pp-app-bugs02
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
< 
{ [data not shown]
100 15885    0 15885    0     0  57649      0 --:--:-- --:--:-- --:--:-- 66187* Connection #0 to host pp-app-bugs02.phx.mozilla.com left intact

* Closing connection #0
bburton@andesite ~$ curl -k -v -H "Host: bugzilla.mozilla.org" http://pp-app-bugs03.phx.mozilla.com:443/buglist.cgi -o /dev/null
* About to connect() to pp-app-bugs03.phx.mozilla.com port 443 (#0)
*   Trying 10.8.82.13...   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0connected
* Connected to pp-app-bugs03.phx.mozilla.com (10.8.82.13) port 443 (#0)
> GET /buglist.cgi HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Accept: */*
> Host: bugzilla.mozilla.org
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Jun 2012 03:15:02 GMT
< Server: Apache
< Refresh: 10; URL=query.cgi
< Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
< X-frame-options: SAMEORIGIN, SAMEORIGIN
< X-Backend-Server: pp-app-bugs03
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
< 
{ [data not shown]
100 15885    0 15885    0     0  60506      0 --:--:-- --:--:-- --:--:-- 70600* Connection #0 to host pp-app-bugs03.phx.mozilla.com left intact

* Closing connection #0

Comment 12

[Reed Loden [:reed]]

I cannot reproduce this upstream on landfill.bugzilla.org, but I can on bugzilla-stage-tip.mozilla.org. Can you find out what version of the Perl CGI module is being used (might also be a copy in the lib/ folder under the bmo directory)? I'll dig into bmo-specific modifications to see if any of them would cause this, but I'm unaware of any changes of that nature. From the data I have, seems like a server-side library issue.

Comment 13

[Reed Loden [:reed]]

From IRC:
[20:19:24] <solarce> reed: does $CGI::VERSION='3.52'; sound like what you're looking for?
[20:19:38] <reed> solarce: yeah, can you also give me the output of `rpm -q perl` as well?
[20:19:42] <solarce> reed: i am not very wise in the ways of perl
[20:19:53] <solarce> reed: perl-5.8.8-32.el5_7.6
[20:20:09] <reed> did you get that version from the lib/ directory or from perl itself?
[20:20:10] <solarce> these appear to be rhel 5.7 boxes
[20:20:19] <solarce> reed: the CGI is from lib/

landfill has $CGI::VERSION='3.51'; (from /var/www/html/bugzilla-4.0-branch/lib/CGI.pm), yet this shows fine:

$ curl -I https://landfill.bugzilla.org/bugzilla-4.0-branch/buglist.cgi
HTTP/1.1 200 OK
Date: Mon, 04 Jun 2012 03:26:45 GMT
Server: Apache/2.2.3 (CentOS)
Refresh: 10; URL=query.cgi
Strict-transport-security: max-age=604800
X-frame-options: SAMEORIGIN
Connection: close
Content-Type: text/html; charset=UTF-8

3.52 seems to only be doc-related changes over 3.51...

Comment 14

[:glob ✱]

(In reply to Reed Loden [:reed] from comment #12)
> I'll dig into bmo-specific modifications to see if any of
> them would cause this, but I'm unaware of any changes of that nature. From
> the data I have, seems like a server-side library issue.

running the same test against my development bmo environment, which is also running CGI v3.52, results in a single SAMEORIGIN response every time:

mod_cgi:

~ $ wget -S -O /dev/null http://fedora/bmo_4.0/buglist.cgi
--2012-06-04 21:50:22--  http://fedora/bmo_4.0/buglist.cgi
Resolving fedora (fedora)... 10.37.129.3
Connecting to fedora (fedora)|10.37.129.3|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Mon, 04 Jun 2012 13:50:22 GMT
  Server: Apache/2.2.22 (Fedora)
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN
  Connection: close
  Transfer-Encoding: chunked
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]

mod_perl:

~ $ wget -S -O /dev/null http://modperl.fedora/buglist.cgi
--2012-06-04 21:55:48--  http://modperl.fedora/buglist.cgi
Resolving modperl.fedora (modperl.fedora)... 10.37.129.3
Connecting to modperl.fedora (modperl.fedora)|10.37.129.3|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Mon, 04 Jun 2012 13:55:47 GMT
  Server: Apache/2.2.22 (Fedora)
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN
  Connection: close
  Transfer-Encoding: chunked
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]


i don't see the double responses consistently from bugzilla-stage-tip:

~ $ wget -S -O /dev/null http://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:17:18--  http://bugzilla-stage-tip.mozilla.org/buglist.cgi
[snip]
  X-frame-options: SAMEORIGIN

~ $ wget -S -O /dev/null http://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:17:35--  http://bugzilla-stage-tip.mozilla.org/buglist.cgi
[snip]
  X-frame-options: SAMEORIGIN, SAMEORIGIN

~ $ wget -S -O /dev/null http://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:18:53--  http://bugzilla-stage-tip.mozilla.org/buglist.cgi
[snip]
  X-frame-options: SAMEORIGIN

~ $ wget -S -O /dev/null http://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:18:57--  http://bugzilla-stage-tip.mozilla.org/buglist.cgi
[snip]
  X-frame-options: SAMEORIGIN

~ $ wget -S -O /dev/null https://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:19:09--  https://bugzilla-stage-tip.mozilla.org/buglist.cgi
[snip]
  X-frame-options: SAMEORIGIN, SAMEORIGIN

~ $ wget -S -O /dev/null https://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:20:54--  https://bugzilla-stage-tip.mozilla.org/buglist.cgi
  X-frame-options: SAMEORIGIN, SAMEORIGIN

~ $ wget -S -O /dev/null https://bugzilla-stage-tip.mozilla.org/buglist.cgi
--2012-06-04 22:20:56--  https://bugzilla-stage-tip.mozilla.org/buglist.cgi
  X-frame-options: SAMEORIGIN

bugzilla-stage-tip is a single webhead.

Comment 15

[David Lawrence [:dkl]]

Same for me locally:

[dkl@localhost ~]$ wget -S -O /dev/null http://fedora/stable/buglist.cgi
--2012-06-04 10:59:01--  http://fedora/stable/buglist.cgi
Resolving fedora... ::1, 127.0.0.1
Connecting to fedora|::1|:80... failed: Connection refused.
Connecting to fedora|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Mon, 04 Jun 2012 14:59:01 GMT
  Server: Apache/2.2.22 (Fedora)
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN
  Connection: close
  Transfer-Encoding: chunked
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]

[dkl@localhost ~]$ rpm -q perl
perl-5.14.2-211.fc17.x86_64
[dkl@localhost ~]$ rpm -q perl-CGI
perl-CGI-3.52-211.fc17.noarch

Comment 16

[Frédéric Buclin]

Once you call bmo with some data, the problem goes away. The first call without arguments, the 2nd one with:

# curl -Ik https://bugzilla.mozilla.org/buglist.cgi
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pp-app-bugs02
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
Date: Mon, 04 Jun 2012 15:15:42 GMT
Refresh: 10; URL=query.cgi
Transfer-Encoding: chunked
Connection: Keep-Alive
X-frame-options: SAMEORIGIN, SAMEORIGIN


# curl -Ik https://bugzilla.mozilla.org/buglist.cgi?short_desc=longblob
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pp-app-bugs02
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Strict-transport-security: max-age=2629744; includeSubDomains
Date: Mon, 04 Jun 2012 15:15:58 GMT
Transfer-Encoding: chunked
Content-disposition: inline; filename="bugs-2012-06-04.html"
Connection: Keep-Alive
Set-Cookie: LASTORDER=bug_status%2Cpriority%2Cassigned_to%2Cbug_id; path=/; expires=Fri, 01-Jan-2038 00:00:00 GMT
X-frame-options: SAMEORIGIN

Comment 17

[:glob ✱]

i've added this test script to bugzilla-stage-tip:

> #!/usr/bin/perl
> use strict;
> use lib qw(. lib);
> if ($ENV{QUERY_STRING} eq 'bug') {
>     use Bugzilla;
>     use Bugzilla::Error;
>     my $cgi = Bugzilla->cgi;
>     print $cgi->header(-refresh=> '10; URL=query.cgi');
>     ThrowUserError("buglist_parameters_required");
> } elsif ($ENV{QUERY_STRING} eq 'cgi') {
>     use CGI;
>     my $cgi = CGI->new;
>     print $cgi->header(-refresh=> '10; URL=query.cgi');
>     ThrowUserError("buglist_parameters_required");
> } else {
>     print "Content-type: text/plain\n\nfallthrough?\n";
> }

calling it as https://bugzilla-stage-tip/761046_test.cgi?bug results in duplicates, as expected.

however if you call it as https://bugzilla-stage-tip/761046_test.cgi?cgi this results in a single X-frame-options header.  this is interesting because this header isn't being created by the perl code at all.  Bugzilla->cgi adds it, but CGI->new doesn't.

~ $ curl -Ik https://bugzilla-stage-tip/761046_test.cgi?cgi
HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
Content-Type: text/html; charset=UTF-8
Date: Mon, 04 Jun 2012 17:31:24 GMT
Refresh: 10; URL=query.cgi
Transfer-Encoding: chunked
Connection: Keep-Alive
X-frame-options: SAMEORIGIN

Comment 19

[Daniel Veditz [:dveditz]]

Gecko doesn't handle duplicate combined headers, or any extension really.
https://mxr.mozilla.org/mozilla-central/source/docshell/base/nsDSURIContentListener.cpp#290

We need a DOM bug filed on that.

Comment 20

[Brandon Burton [:solarce]]

So this has turned out to be a mod_perl + Bugzilla behavior? And is not something IT/WebOps needs to take action to fix at this time?

Comment 21

[Frédéric Buclin]

(In reply to Brandon Burton [:solarce] from comment #20)
> So this has turned out to be a mod_perl + Bugzilla behavior? And is not
> something IT/WebOps needs to take action to fix at this time?

It's unclear yet what the causes are. This seems related to mod_perl, but we need more data to be sure.

Comment 22

[Reed Loden [:reed]]

Moving this back to Bugzilla for further analysis.

Comment 23

[Frédéric Buclin]

(In reply to Frédéric Buclin from comment #21)
> It's unclear yet what the causes are. This seems related to mod_perl, but we
> need more data to be sure.

I enabled mod_perl in Fedora 17, and the header is fine. I see no duplicated headers. So it's at least not always reproducible, even with mod_perl enabled.

Comment 24

[Gervase Markham [:gerv]]

glob: I don't get the same results as you with b-s-t:

gerv@mink:~$ curl -Ik https://bugzilla-stage-tip.mozilla.org/761046_test.cgi?cgi
HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
Content-Type: text/plain; charset=ISO-8859-1
Date: Thu, 14 Jun 2012 09:57:30 GMT
Refresh: 10; URL=query.cgi
Transfer-Encoding: chunked
Connection: Keep-Alive

I don't get an X-Frame-Options header at all, and the charset is different. The only difference is that I used the full name for the machine. Any idea what's going in? Has it been reconfigured in the last week?

(The answer for ?bug is exactly the same as above.)

Gerv

Comment 25

[Frédéric Buclin]

CC'ing Denis, Eclipse's webmaster, as they are also affected by the same issue. The issue is not bmo specific.

Comment 26

[Denis Roy]

Thanks LpSolit.  We're seeing the same thing as you guys are at Eclipse.org.

- Happens only on mod_perl installs.  Plain CGI sites don't exhibit this
- Happens even if we hit the webheads directly (ie, the load balancer isn't adding this)
- Happens intermittently

wget -O /dev/null -S https://bugs.eclipse.org/bugs/buglist.cgi
Connecting to bugs.eclipse.org|172.30.206.101|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 26 Jun 2012 17:27:15 GMT
  Server: Apache
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN
  X-NodeID: bugs-vm1


wget -O /dev/null -S https://bugs.eclipse.org/bugs/buglist.cgi
Connecting to bugs.eclipse.org|172.30.206.102|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 26 Jun 2012 17:27:17 GMT
  Server: Apache
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN, SAMEORIGIN
  X-NodeID: bugs-vm2

wget -O /dev/null -S https://bugs.eclipse.org/bugs/buglist.cgi
Connecting to bugs.eclipse.org|172.30.206.101|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 26 Jun 2012 17:27:18 GMT
  Server: Apache
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN
  X-NodeID: bugs-vm1

wget -O /dev/null -S https://bugs.eclipse.org/bugs/buglist.cgi
Connecting to bugs.eclipse.org|172.30.206.102|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 26 Jun 2012 17:27:20 GMT
  Server: Apache
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN
  X-NodeID: bugs-vm2

wget -O /dev/null -S https://bugs.eclipse.org/bugs/buglist.cgi
Connecting to bugs.eclipse.org|172.30.206.101|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 26 Jun 2012 17:27:21 GMT
  Server: Apache
  Refresh: 10; URL=query.cgi
  X-frame-options: SAMEORIGIN, SAMEORIGIN
  X-NodeID: bugs-vm1

Comment 27

[Frédéric Buclin]

(In reply to Denis Roy from comment #26)
> - Happens intermittently

This is the weird part. This makes it hard to determine who is the culprit.

Comment 28

[Frédéric Buclin]

Not a blocker as we have no idea where the problem is and can only be reproduced randomly.

Comment 29

[Reed Loden [:reed]]

Now it's even worse with the new security headers... We've got to find a fix for this.

$ curl -I https://bugzilla.mozilla.org/buglist.cgi
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pp-app-bugs03
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
Date: Thu, 13 Sep 2012 15:53:54 GMT
Refresh: 10; URL=query.cgi
X-xss-protection: 1; mode=block, 1; mode=block
Transfer-Encoding: chunked
X-content-type-options: nosniff, nosniff
Connection: Keep-Alive
X-frame-options: SAMEORIGIN, SAMEORIGIN

Comment 30

[:glob ✱]

what's happening is when you call buglist.cgi without a query string, bugzilla calls CGI->header twice:

once in buglist.cgi:

  if (length($buffer) == 0) {
      print $cgi->header(-refresh=> '10; URL=query.cgi');
      ThrowUserError("buglist_parameters_required");
  }

and then again in Bugzilla::Error.pm:

  print Bugzilla->cgi->header();
  $template->process($name, $vars)
    || ThrowTemplateError($template->error());


it appears intermittent because the first time a worker generates this response, it doesn't duplicate the headers.  subsequent requests to that worker always results in the duplicated values.  i don't know why this happens :)

denis: check your main (not virtual host) apache logs -- you'll probably find Apache2::SizeLimit is killing off a lot of your workers, and you may want to raise the limit in modperl.pl (on bmo we have it set to 1.6g).


testing with a query string |curl -I https://bugzilla.mozilla.org/buglist.cgi?id=1| i've never seen any duplicates.  also on my dev environment when i commented out the redirect print statement, the duplicates vanished.

dropping the priority as the scope of this issue is very limited.

i'll work on a patch for this soon.

Comment 31

[Denis Roy]

(In reply to Byron Jones ‹:glob› from comment #30)
> denis: check your main (not virtual host) apache logs -- you'll probably
> find Apache2::SizeLimit is killing off a lot of your workers, and you may
> want to raise the limit in modperl.pl (on bmo we have it set to 1.6g).

I noticed that a while back when we switched to mod_perl and adjusted limit to 300M if memory serves me correctly.  I'm glad it's not just me -- This is OT but perhaps raising the default limit would be a good idea? Otherwise people who switch from cgi to mod_perl only to have each process killed off will fail to see the performance benefit.

Comment 32

[David Lawrence [:dkl]]

(In reply to Byron Jones ‹:glob› from comment #30)
> what's happening is when you call buglist.cgi without a query string,
> bugzilla calls CGI->header twice:
> 
> once in buglist.cgi:
> 
>   if (length($buffer) == 0) {
>       print $cgi->header(-refresh=> '10; URL=query.cgi');
>       ThrowUserError("buglist_parameters_required");
>   }
> 
> and then again in Bugzilla::Error.pm:
> 
>   print Bugzilla->cgi->header();
>   $template->process($name, $vars)
>     || ThrowTemplateError($template->error());
> 

Could we not simply set a request cache value that is set the first time a header is generate and then look for the value for subsequent call to header()?

=== modified file 'Bugzilla/CGI.pm'
--- Bugzilla/CGI.pm	2012-09-09 18:32:05 +0000
+++ Bugzilla/CGI.pm	2012-09-13 18:35:17 +0000
@@ -275,6 +275,8 @@
 sub header {
     my $self = shift;
 
+    return if Bugzilla->request_cache->{header_done};
+
     # If there's only one parameter, then it's a Content-Type.
     if (scalar(@_) == 1) {
         # Since we're adding parameters below, we have to name it.
@@ -314,6 +316,8 @@
     # the MIME type away from the declared Content-Type.
     unshift(@_, '-x_content_type_options' => 'nosniff');
 
+    Bugzilla->request_cache->{header_done} = 1;
+
     return $self->SUPER::header(@_) || "";
 }

One downside is if the one or more subsequent calls to header() have special headers added or other special reasons to call header().

dkl

Comment 33

[:glob ✱]

(In reply to David Lawrence [:dkl] from comment #32)
> Could we not simply set a request cache value that is set the first time a
> header is generate and then look for the value for subsequent call to
> header()?

i think a better way to address this would be to add a 'redirect' option to ThrowUserError, which inserts the redirect into ThrowUserError's cgi->header call.

Comment 34

[:glob ✱]

Attachment: patch v1

this patch adds a CGI_HEADER parameter to Throw* functions which is passed to cgi->header()

as this is the only place in bugzilla where we do "show an error then redirect", a simpler fix would be to drop the redirect (i have no strong preference either way).

Comment 35

[Frédéric Buclin]

It's a pity that $cgi->redirect doesn't take a -refresh argument where you could specify when the redirect should occur.

Comment 36

[Frédéric Buclin]

(In reply to Byron Jones ‹:glob› from comment #34)
> as this is the only place in bugzilla where we do "show an error then
> redirect", a simpler fix would be to drop the redirect (i have no strong
> preference either way).

I agree that there is no special reason to have a redirect here and nowhere else. For consistency and simplicity (especially on branches), I strongly prefer to remove the redirect from buglist.cgi (which exists since Bugzilla 2.17 (July 2002!), see bug 96003).

Comment 37

[:glob ✱]

Attachment: patch v2

this patch removes the redirect, and the duplicate headers.

Comment 38

[Frédéric Buclin]

Comment on attachment 661711 [details] [diff] [review]
patch v2

r=LpSolit, but note that this patch doesn't prevent my PoC from working, due to bug 761667. So this only fixes the duplicated header problem.

Comment 39

[Frédéric Buclin]

I don't see how this could be exploitable as you must call buglist.cgi with no arguments to trigger duplicated headers, see comment 30. So this is not a security issue => 4.2 and newer only (and I don't want to change the behavior of buglist.cgi on older branches). Before making this bug public, attachment 629851 [details] must be marked as private. No comment contains sensitive data, so we are fine here.

Comment 40

[:glob ✱]

Comment on attachment 661711 [details] [diff] [review]
patch v2

>=== modified file 'buglist.cgi'
>--- buglist.cgi	2012-09-01 21:43:00 +0000
>+++ buglist.cgi	2012-09-17 07:49:18 +0000
>@@ -41,7 +41,6 @@
> 
> my $buffer = $cgi->query_string();
> if (length($buffer) == 0) {
>-    print $cgi->header(-refresh=> '10; URL=query.cgi');
>     ThrowUserError("buglist_parameters_required");
> }
> 
>

Comment 41

[:glob ✱]

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified buglist.cgi
Committed revision 8393.

Committing to: bzr+ssh://[e-mail]/bugzilla/4.4/
modified buglist.cgi
Committed revision 8390.

Committing to: bzr+ssh://[e-mail]/bugzilla/4.2/
modified buglist.cgi
Committed revision 8140.