Closed Bug 1372653 Opened 3 years ago Closed 3 years ago

Upgrade LWP to 6.26 & add dependency for LWP::Protocol::https 6.07

Categories

(bugzilla.mozilla.org :: General, enhancement)

Production
enhancement
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: dylan, Assigned: dylan)

References

Details

Currently, bmo is configured to use a 'http://' proxy scheme which means, among other things, we can't speak to anything using SNI. 
I'm told the proper solution is to use the CONNECT method, to which I have updated BMO's code to support.

However currently BMO's web nodes aren't allowed to do this:

cd /data/www/bugzilla.mozilla.org
perl -Ilocal/lib/perl5 -MLWP::UserAgent -E 'my $ua = LWP::UserAgent->new; $ua->proxy("https", "connect://dc-proxy"); my $r = $ua->get("https://github.com/mozilla-bteam/bmo/pull/85.diff"); say $r->content'

500 error while CONNECT thru proxy: 405 Method Not Allowed

Among other things, this breaks auth delegation between sites that use SNI,
and more pressingly breaks pull-requests-as-patch viewing.
I don't think the 405 is being generated by the proxy, instead this error is coming from LWP. I confirmed this by running your one liner under strace and looking for calls to connect() and stat().

It is in fact looking for LWP::Protocol::connect:
> stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)

I found an article on stackoverflow.com[0] which references this exact issue, and someone commenting they wrote a LWP protocol module to address it[1]. The module doesn't appear to be maintained, not having any releases since 2013.

I turned to LWP source and found a PR[2] related to using CONNECT, but I wasn't able to get it to work. LWP still wants to send a GET to the proxies for a https target. I think LWP might only support CONNECT when it's using SSL to connect to the proxy, which we currently don't support. (But we should)

I don't have the appropriate flows from my dev host to test this with a SSL port on squid, or a dev squid host I can install a bunch of dependencies on to continue testing.

I'll ping you on IRC to follow up

[0] https://stackoverflow.com/questions/12116244/https-proxy-and-lwpuseragent
[1] http://search.cpan.org/~benning/LWP-Protocol-connect/
[2] https://github.com/libwww-perl/libwww-perl/compare/master...dod38fr:fix-https-proxy
Assignee: infra → nobody
Component: Proxy ACL Request → General
Product: Infrastructure & Operations → bugzilla.mozilla.org
QA Contact: cshields
Summary: BMO could do with CONNECT access to the outside world → Fix or replace LWP::Protocol::connect
Version: unspecified → Production
Assignee: nobody → dylan
Thanks digi, and sorry for the noise: The root cause is that LWP is too old. We're using the system-provided one currently, so we need to add it to our dependencies.
Summary: Fix or replace LWP::Protocol::connect → Upgrade LWP to 6.26 & add dependency for LWP::Protocol::https 6.07
(In reply to Brian Hourigan [:digi] from comment #1)
> I don't think the 405 is being generated by the proxy, instead this error is
> coming from LWP. I confirmed this by running your one liner under strace and
> looking for calls to connect() and stat().
> 
> It is in fact looking for LWP::Protocol::connect:
> > stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)
I just wanted to mention that this test should be true if run from the bmo root.
(In reply to Dylan Hardison [:dylan] (he/him) from comment #4)
> (In reply to Brian Hourigan [:digi] from comment #1)
> > I don't think the 405 is being generated by the proxy, instead this error is
> > coming from LWP. I confirmed this by running your one liner under strace and
> > looking for calls to connect() and stat().
> > 
> > It is in fact looking for LWP::Protocol::connect:
> > > stat("local/lib/perl5/LWP/Protocol/connect.pm", 0x7ffeb7e68fc0) = -1 ENOENT (No such file or directory)
> I just wanted to mention that this test should be true if run from the bmo
> root.

I ran that on stage from /data/www/bugzilla-dev.allizom.org
Aha! and dev hadn't been updated. 

Still, the root problem is LWP itself was too old. :-)
To git@github.com:mozilla-bteam/bmo.git
   37120e708..8bb543cad  master -> master
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
See Also: → 1374767
You need to log in before you can comment on or make changes to this bug.