Closed Bug 1131242 Opened 9 years ago Closed 8 years ago

Reverse proxy requested for Jenkins instance on webqa-ci.mozilla.com

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: davehunt, Assigned: Atoll)

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/546] ) 

It appears that there's an issue with the reverse proxy used to access Web QA's Jenkins instance via the public facing FQDN.

Proxy: https://webqa-ci.mozilla.com/
Destination: http://webqa-ci1.qa.scl3.mozilla.com:8080/

The Jenkins global configuration page shows the message "It appears that your reverse proxy set up is broken." with a link to https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken for more information. This has a couple of suggestions for what might be broken.

The page also suggests running cURL for more information. Here's is the result I get:

$ curl -iL -e https://webqa-ci.mozilla.com/manage https://webqa-ci.mozilla.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test
HTTP/1.1 302 Found
Location: https://webqa-ci.mozilla.com/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https%3A%2F%2Fwebqa-ci.mozilla.com%2Fmanage/
Content-Length: 0
Server: Jetty(winstone-2.8)

HTTP/1.1 404 http://webqa-ci.mozilla.com/manage vs. https://webqa-ci.mozilla.com/manage
Content-Type: text/html;charset=ISO-8859-1
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1628
Server: Jetty(winstone-2.8)

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 404 http://webqa-ci.mozilla.com/manage vs. https://webqa-ci.mozilla.com/manage</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https%3A%2F%2Fwebqa-ci.mozilla.com%2Fmanage/. Reason:
<pre>    http://webqa-ci.mozilla.com/manage vs. https://webqa-ci.mozilla.com/manage</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>                                                
<!-- snip -->                                              
<br/>                                                

</body>
</html>

It appears that there's at least a mismatch between https and http in the above output. There's also a link to https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache for more help when configuring a reverse proxy using Apache.

I should note that we haven't (yet) experienced any issues with access this Jenkins instance other than the configuration warning message, but ideally we would like to resolve this.
Load balancers are managed bu the WebOps team, moving the bug to them.
Assignee: network-operations → server-ops-webops
Component: NetOps → WebOps: Other
QA Contact: jbarnell → nmaul
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/546]
Jake, would you be able to look into this for us?  Thanks!
Flags: needinfo?(nmaul)
Flags: needinfo?(nmaul) → needinfo?(smani)
I'll take a poke tomorrow.
Assignee: server-ops-webops → smani
Flags: needinfo?(smani)
QA Contact: nmaul → smani
This is probably complaining because what we did with the load balancers is only a forward proxy.  =)  I've tried setting up the load balancer to do reverse proxying 


The fastest way to fix this is to set up Apache on webqa-ci1.qa.scl3 to front jenkins, having it handle the reverse proxy, adding in lines like:

  ProxyPass / http://webqa-ci1.qa.scl3.mozilla.com:8080/
  ProxyPassReverse / http://webqa-ci1.qa.scl3.mozilla.com:8080/
  ProxyTimeout 600


I spent some time trying to get this done on the load balancer but I've had a distinct lack of success.
Stephen,

Since this isn't a webops supported setup (does not use our jenkins puppet module) , I'm hesitant to make these changes. 

With everything else on our plate, this isn't a high priority. How can we help you guys help yourselves? Share an apache config? C's point in comment #4 is valid. I'd rather front Jenkins with Apache since that's way more flexible than Jenkins itself. Happy to chat more on IRC, feel free to hit me up. thanks!
Flags: needinfo?(smani)
What constitutes a WebOps supported setup? I'm unfamiliar with the Jenkins puppet module that's mentioned - what difference would using this make? What's involved with having an Apache instance in-front of Jenkins, and is there some documentation available for doing this? I believe the B2G Jenkins instance has an Apache instance proxying traffic from 80 to 8080 but that still have the reverse proxy warning - of course there could be different reason for this.

I tend to agree that this isn't high priority, but I also do not fully understand the implications of the reverse proxy warnings.
Flags: needinfo?(smani)
Updating bug description and adding sec-review?, since we'll need it to proceed with a reverse proxy (most likely) someday.
Flags: needinfo?(smani) → sec-review?
Summary: Reverse proxy for Jenkins instance on webqa-ci.mozilla.com is not set up correctly → Reverse proxy requested for Jenkins instance on webqa-ci.mozilla.com
This isn't a proxy request. This is a load balancer sort-of front-end request. Clearing sec-review and will try to proceed with some Zeus testing, because I think I've done this before.
Assignee: smani → rsoderberg
Flags: sec-review?
Jenkins no longer warns about this. I'm not sure if that means later versions have fixed an issue that caused it to be erroneously reported, or if somehow this has been resolved. Either way, I'm going to resolve this as works for me.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.