Bugzilla Auth Delegation is not working with SNI-enabled server due to Squid proxy server

RESOLVED WONTFIX

Status

()

bugzilla.mozilla.org
API
RESOLVED WONTFIX
2 years ago
2 years ago

People

(Reporter: kohei, Assigned: dylan)

Tracking

Production

Details

Attachments

(3 attachments, 2 obsolete attachments)

(Reporter)

Description

2 years ago
:bwinton reported that Bugzilla Authentication Delegation didn't work for my BzDeck app and BMO, resulting in the following error:

> Auth delegation received an HTTP response other than 200 OK 
> from auth consumer. Code: 503 

For some reason the BzDeck server log shows the last request from Bugzilla was made on February 15:

> 63.245.214.162 - - [15/Feb/2016:18:41:33 -0500] "POST /integration
> /bugzilla-auth-callback/ HTTP/1.1" 200 5056 "-" "libwww-perl/5.837"

The server has been upgraded to HTTP/2 on Feb 19, but I'm not sure it's related. Can you please check Bugzila's log? Thanks.
(Assignee)

Updated

2 years ago
Assignee: nobody → dylan
(Assignee)

Comment 1

2 years ago
This is a more serious problem. the config has been overwritten with http://squid.proxy.service.consul:3128/,
which I believe to the value used for AWS. 

Is this address supposed to work from inside SCL3?
Flags: needinfo?(klibby)
Summary: Bugzilla Auth Delegation is not working for BzDeck → BMO Proxy config overwritten with AWS value
(Assignee)

Comment 2

2 years ago
Nope, it appears that does exist in the hosts file and a manual curl works. In addition, proxying is used in github auth and that still works.
Flags: needinfo?(klibby)
Summary: BMO Proxy config overwritten with AWS value → Bugzilla Auth Delegation is not working for BzDeck
yeah, we added that in SCL3 to keep the config (more) simple for failover.
(Reporter)

Comment 4

2 years ago
Sorry I'm late here. I have just tested this again. Downgrading to HTTP/1.1 didn't work. The access (and error) log still doesn't have any output when I accept auth delegation on https://bugzilla.mozilla.org/auth.cgi

Another possibility is...

1. the TLS cert? https://www.bzdeck.com/ uses Let's Encrypt since Oct 30, 2015. Does the Bugzilla server have the latest ca-certificates package?

2. auth delegation permission? Does Bugzilla use a whitelist or blacklist of sites (dis)allowing auth delegation?
(In reply to Kohei Yoshino [:kohei] from comment #4)

> 1. the TLS cert? https://www.bzdeck.com/ uses Let's Encrypt since Oct 30,
> 2015. Does the Bugzilla server have the latest ca-certificates package?

yes. it's ca-certificates-2015.2.6-65.0.1.el6_7, which appears to be just a few weeks old (cf https://bugzilla.redhat.com/show_bug.cgi?id=1297893 and https://bugzilla.redhat.com/show_bug.cgi?id=1299863)

And from a curl -v to https://www.bzdeck.com/ from a web head:

* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: CN=bzdeck.com
* 	start date: Jan 18 23:38:00 2016 GMT
* 	expire date: Apr 17 23:38:00 2016 GMT
* 	common name: bzdeck.com
* 	issuer: CN=Let's Encrypt Authority X1,O=Let's Encrypt,C=US
If someone can give me more info I'm happy to go looking through logs, but :dylan may have already done so.
(Assignee)

Comment 7

2 years ago
I can confirm that bzdeck works fine when I run it on my own server.
(Assignee)

Comment 8

2 years ago
Created attachment 8740972 [details]
proxy-test.pl

squid is not able to connect to bzdeck, with the following error:

"This proxy and the remote host failed to negotiate a mutually acceptable security settings for handling your request. It is possible that the remote host does not support secure connections, or the proxy is not satisfied with the host security credentials."

Attached is a script that generates this error. Note the API key is fake.
Flags: needinfo?(klibby)
(Assignee)

Comment 9

2 years ago
Created attachment 8740981 [details]
bzdeck-ciphers.txt

ciphers bzdeck is using (from :ulfr's cipherscan, https://github.com/jvehent/cipherscan)
(Assignee)

Comment 10

2 years ago
Created attachment 8740982 [details]
reviewboard-ciphers.txt

ciphers reviewboard is using (which auth delegation is able to work with)
(Reporter)

Comment 11

2 years ago
Created attachment 8741001 [details]
bzdeck-ciphers-latest.txt

The SSL conf was the server's default :/ Now following https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=apache-2.4.18&openssl=1.0.2g&hsts=yes&profile=modern

Also renewed the cert and rebooted Apache. Can you please test w/ squid again?

https://www.ssllabs.com/ssltest/analyze.html?d=www.bzdeck.com&hideResults=on
Attachment #8740981 - Attachment is obsolete: true
(Reporter)

Comment 12

2 years ago
Nah, I still get a 503 error while accepting auth delegation request :(
(Reporter)

Comment 13

2 years ago
Created attachment 8741015 [details]
bzdeck-ciphers-latest.txt

Changed to the Generator's "intermediate profile" but the error persists.
Attachment #8741001 - Attachment is obsolete: true
(Reporter)

Comment 14

2 years ago
Squid doesn't support SNI??

http://bugs.squid-cache.org/show_bug.cgi?id=4153
http://serverfault.com/questions/555582/multiple-ssl-certificates-with-squid-reverse-proxy
(Reporter)

Comment 15

2 years ago
Can you please consider replacing Squid with an alternative? As IE6 is dead, SNI becomes a practical solution. I can technically create a new virtual server but can't afford to pay another $5/month for now because BzDeck (nor the Firefox Site Compatibility project that shares the same server) is not a business. Thank you!
sorry, that's a considerably larger issues as the entire data center uses it for outbound traffic; the proxy isn't a BMO specific thing. /cc :digi

it may be worth figuring out if we can stand up bzdeck somewhere, but that's a managerial call ($, people).
Flags: needinfo?(klibby)
If your squid server is http://squid.proxy.service.consul:3128/ this suggests it's part of the nubis project, which is outside of my purview. If your proxy endpoint is proxy.dmz.scl3 or proxy.dmz.phx1 I can help work with you to find a solution. the nubis team is responsible for the supporting infrastructure of applications.
Sorry, I was just informed that the proxy endpoint is a red herring. Configs are identical across AWS and SCL3. Replacing squid isn't on our current roadmap, so the only alternative we can offer is for you to bypass squid and access BzDeck directly if our infosec team is ok with that.
(Reporter)

Comment 19

2 years ago
Good news! I have just solved the issue myself. Not a new virtual server. Let's Encrypt allows us to request one single certificate for multiple domains, not only subdomains under the same domain, so I extended an existing cert to include all the domains and subdomains I have:

> ./letsencrypt-auto auth --agree-tos --apache \
> -d britegrid.com -d www.britegrid.com -d analytics.britegrid.com \
> -d fxsitecompat.com -d www.fxsitecompat.com \
> -d bzdeck.com -d www.bzdeck.com 

Then updated the server config, rebooted Apache, and problem solved!

Thank you so much :dylan and :fubar for your investigation!! Not yet FIXED by the Bugzilla side, so closing as WORKSFORME so far.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
Summary: Bugzilla Auth Delegation is not working for BzDeck → Bugzilla Auth Delegation is not working with SNI-enabled server due to Squid proxy server
(Reporter)

Comment 20

2 years ago
I mean WONTFIX.
Resolution: WORKSFORME → WONTFIX
(Reporter)

Updated

2 years ago
See Also: → bug 1264667
You need to log in before you can comment on or make changes to this bug.