If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Upgrade LWP to 6.26 & add dependency for LWP::Protocol::https 6.07

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
General
RESOLVED FIXED
3 months ago
3 months ago

People

(Reporter: dylan, Assigned: dylan)

Tracking

Production

Details

(Assignee)

Description

3 months ago
Currently, bmo is configured to use a 'http://' proxy scheme which means, among other things, we can't speak to anything using SNI. 
I'm told the proper solution is to use the CONNECT method, to which I have updated BMO's code to support.

However currently BMO's web nodes aren't allowed to do this:

cd /data/www/bugzilla.mozilla.org
perl -Ilocal/lib/perl5 -MLWP::UserAgent -E 'my $ua = LWP::UserAgent->new; $ua->proxy("https", "connect://dc-proxy"); my $r = $ua->get("https://github.com/mozilla-bteam/bmo/pull/85.diff"); say $r->content'

500 error while CONNECT thru proxy: 405 Method Not Allowed

Among other things, this breaks auth delegation between sites that use SNI,
and more pressingly breaks pull-requests-as-patch viewing.
I don't think the 405 is being generated by the proxy, instead this error is coming from LWP. I confirmed this by running your one liner under strace and looking for calls to connect() and stat().

It is in fact looking for LWP::Protocol::connect:
> stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)

I found an article on stackoverflow.com[0] which references this exact issue, and someone commenting they wrote a LWP protocol module to address it[1]. The module doesn't appear to be maintained, not having any releases since 2013.

I turned to LWP source and found a PR[2] related to using CONNECT, but I wasn't able to get it to work. LWP still wants to send a GET to the proxies for a https target. I think LWP might only support CONNECT when it's using SSL to connect to the proxy, which we currently don't support. (But we should)

I don't have the appropriate flows from my dev host to test this with a SSL port on squid, or a dev squid host I can install a bunch of dependencies on to continue testing.

I'll ping you on IRC to follow up

[0] https://stackoverflow.com/questions/12116244/https-proxy-and-lwpuseragent
[1] http://search.cpan.org/~benning/LWP-Protocol-connect/
[2] https://github.com/libwww-perl/libwww-perl/compare/master...dod38fr:fix-https-proxy
(Assignee)

Updated

3 months ago
Assignee: infra → nobody
Component: Proxy ACL Request → General
Product: Infrastructure & Operations → bugzilla.mozilla.org
QA Contact: cshields
Summary: BMO could do with CONNECT access to the outside world → Fix or replace LWP::Protocol::connect
Version: unspecified → Production
(Assignee)

Updated

3 months ago
Assignee: nobody → dylan
(Assignee)

Comment 2

3 months ago
Thanks digi, and sorry for the noise: The root cause is that LWP is too old. We're using the system-provided one currently, so we need to add it to our dependencies.
Summary: Fix or replace LWP::Protocol::connect → Upgrade LWP to 6.26 & add dependency for LWP::Protocol::https 6.07
(Assignee)

Comment 3

3 months ago
LWP::UserAgent upgrade:
   https://github.com/mozilla-bteam/carton-bundles/commit/7d03027254b18e07e85140ae51833cdb61abf401
(Assignee)

Comment 4

3 months ago
(In reply to Brian Hourigan [:digi] from comment #1)
> I don't think the 405 is being generated by the proxy, instead this error is
> coming from LWP. I confirmed this by running your one liner under strace and
> looking for calls to connect() and stat().
> 
> It is in fact looking for LWP::Protocol::connect:
> > stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)
I just wanted to mention that this test should be true if run from the bmo root.
(In reply to Dylan Hardison [:dylan] (he/him) from comment #4)
> (In reply to Brian Hourigan [:digi] from comment #1)
> > I don't think the 405 is being generated by the proxy, instead this error is
> > coming from LWP. I confirmed this by running your one liner under strace and
> > looking for calls to connect() and stat().
> > 
> > It is in fact looking for LWP::Protocol::connect:
> > > stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)
> I just wanted to mention that this test should be true if run from the bmo
> root.

I ran that on stage from /data/www/bugzilla-dev.allizom.org
(Assignee)

Comment 6

3 months ago
Aha! and dev hadn't been updated. 

Still, the root problem is LWP itself was too old. :-)
(Assignee)

Comment 7

3 months ago
To git@github.com:mozilla-bteam/bmo.git
   37120e708..8bb543cad  master -> master
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → FIXED
(Assignee)

Updated

3 months ago
See Also: → bug 1374767
You need to log in before you can comment on or make changes to this bug.