554783 - Add a custom header to backend web servers

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Ryan Doherty (:rdoherty)]

Can we get apache to add the hostname of the server to the end of all webpages served?

Something like:

<!-- pm-app-personas01.mozilla.org Wed Mar 24 15:45:34 PDT 2010 -->

at the end of all html files.

This would help debugging a *lot* when one machine is having issues.

Bonus points if we could use that hostname to access the machine directly and bypass zeus/netscaler.

Live example:
http://messenger.yahoo.com/

Comment 1

[Ryan Doherty (:rdoherty)]

This was a request for IT to add this to all our webservers, not just getpersonas.com

Comment 2

[Reed Loden [:reed]]

http://viewvc.svn.mozilla.org/vc/addons/trunk/site/app/views/elements/footer.thtml?view=annotate#l67

<p id="copyright"><?=___('Copyright')?> <span title="<?=php_uname('n')?>">&#169;</span> 2005&#8211;<?=date('Y')?> Mozilla. <?=___('All rights reserved.')?></p>

<?=php_uname('n')?> :)

Comment 3

[Ryan Doherty (:rdoherty)]

We could do it in php, but it would be great if it was on by default and added via apache. That way any text file served would get it.

Comment 4

[Toby Elliott [:telliott]]

This would be very useful, but we'll need to make sure it's easy to turn off in apache so as not to mess with non-html output

Comment 5

[matthew zeier [:mrz]]

ryan - backend app servers are already accessible through the VPN.

Comment 6

[Ryan Doherty (:rdoherty)]

(In reply to comment #5)
> ryan - backend app servers are already accessible through the VPN.

Huh, I tried pm-app-personas01.mozilla.org and got redirected to mozilla.com while on Mozilla-MPT. What am I doing wrong?

Comment 7

[Reed Loden [:reed]]

(In reply to comment #6)
> (In reply to comment #5)
> > ryan - backend app servers are already accessible through the VPN.
> 
> Huh, I tried pm-app-personas01.mozilla.org and got redirected to mozilla.com
> while on Mozilla-MPT. What am I doing wrong?

You need to send "getpersonas.com" as your "Host:" header.

Example: curl -H'Host: getpersonas.com' http://pm-app-personas01.mozilla.org

Comment 8

[Dave Miller [:justdave]]

How about an HTTP header?  The Bugzilla servers add an X-Backend-Server header with the server name in it, with this puppet rule:

        "/etc/httpd/conf.d/hostname.conf":
            require => File[etc-httpd],
            before => Service[httpd],
            notify => Service[httpd],
            content => "Header set X-Backend-Server $hostname\n";

Same rule could be added to the other clusters really easily.

Comment 9

[Wil Clouser [:clouserw]]

(In reply to comment #8)
> How about an HTTP header?  The Bugzilla servers add an X-Backend-Server header
> with the server name in it, with this puppet rule:

We add an X-AMO-ServedBy header for AMO.  A header is a lot more flexible than an html comment and will work for any web request instead of just html.

Comment 10

[Ryan Doherty (:rdoherty)]

The trouble is if you are viewing a broken page and have to refresh after opening up Firebug or some other network analyzer, you will have to refresh again and again until you see the problem. If a broken host happens often it's a real pita (especially if it involves filling out forms).

Ideal workflow:
1) View broken page
2) View source
3) See hostname
4) Copy hostname into browser/bug report, etc

Not-so-ideal workflow
1) See broken page
2) Open Firebug/LiveHTTPHeaders
3) Refresh
4) Refresh or
5) Fill out form again
6) Locate request in list of requests
7) Locate hostname header
8) Copy hostname into browser/bug report, etc

Header is good for non-html pages, but a majority of our servers serve just html, so this would simplify most of our debugging.

Comment 11

[Dave Miller [:justdave]]

Content injection via apache is asking for trouble.  Way too easy to break stuff.  Just my opinion.  Best to do that in the application in each affected app.

Comment 12

[Shyam Mani [:fox2mike]]

(In reply to comment #11)
> Content injection via apache is asking for trouble.  Way too easy to break
> stuff.  Just my opinion.  Best to do that in the application in each affected
> app.

+1, my thoughts exactly.

Comment 13

[Jeff Balogh (:jbalogh)]

(In reply to comment #10)
> The trouble is if you are viewing a broken page and have to refresh after
> opening up Firebug or some other network analyzer, you will have to refresh
> again and again until you see the problem. If a broken host happens often it's
> a real pita (especially if it involves filling out forms).

You could make a jetpack that's always tracking headers.  I think this would be helpful: https://developer.mozilla.org/en/Monitoring_HTTP_activity

Comment 14

[Ryan Doherty (:rdoherty)]

Ok, I can live with headers and a jetpack :)

Comment 15

[matthew zeier [:mrz]]

I also like the jetpack idea.  Assuming that's the direction, closing this one.

Comment 16

[Ryan Doherty (:rdoherty)]

(In reply to comment #15)
> I also like the jetpack idea.  Assuming that's the direction, closing this one.

We'd still need the headers from every webserver for the jetpack to read.

Comment 17

[Shyam Mani [:fox2mike]]

I'll grab this, and aim to get it done sooner than later ;)

Comment 18

[Shyam Mani [:fox2mike]]

Ryan,

Because of all the <3 IT has for Webdev...

trinity:~ shyam$ curl -L  -I -H "Host: www.getpersonas.com" http://pm-app-personas02.mozilla.org/
HTTP/1.1 302 Found
Date: Thu, 08 Apr 2010 10:02:01 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Location: http://www.getpersonas.com/en-US/
X-Backend-Server: pm-app-personas02
Content-Type: text/html; charset=UTF-8

HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pm-app-personas01
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Date: Thu, 08 Apr 2010 09:57:04 GMT
Transfer-Encoding: chunked
X-Powered-By: PHP/5.2.9
Connection: Keep-Alive
X-Cache-Info: cached

:D I'll leave the bug open till I'm sure I've hit all the webservers.

Comment 19

[Shyam Mani [:fox2mike]]

Personas - check
AMO - Check
Static - Check
Generic - Check
Dist - Check

Guess I got them all. Reopen if you see anything amiss, AMO now has both X-AMO-ServedBy and X-Backend-Server

Comment 20

[Wil Clouser [:clouserw]]

preview.amo and next.amo (bug 557899) too, please!

Comment 21

[Ryan Doherty (:rdoherty)]

<3

Comment 22

[Wil Clouser [:clouserw]]

reopening for comment 20 since you asked, but I'm not in a hurry

Comment 23

[Shyam Mani [:fox2mike]]

(In reply to comment #22)
> reopening for comment 20 since you asked, but I'm not in a hurry

Done.

[shyam@boris ~]$ curl -L -I http://preview.addons.mozilla.org
HTTP/1.1 301 Moved Permanently
Content-Type: text/html
Date: Mon, 19 Apr 2010 08:19:47 GMT
Location: https://preview.addons.mozilla.org/
Connection: Keep-Alive
Content-Length: 0

HTTP/1.1 302 Found
Server: Apache
X-Backend-Server: pm-app-amo24
Vary: Accept-Encoding
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
Content-Type: text/html; charset=UTF-8
Date: Mon, 19 Apr 2010 08:19:46 GMT
Location: https://preview.addons.mozilla.org/en-US/
X-AMO-ServedBy: pm-app-amo24.mozilla.org
Pragma: no-cache
Transfer-Encoding: chunked
Via: Moz-Cache-zlb05
Connection: Keep-Alive
X-Powered-By: PHP/5.2.9
X-Cache-Info: not cacheable; response specified "Cache-Control: no-store"

HTTP/1.1 302 Found
Server: Apache
X-Backend-Server: pm-app-amo24
Vary: Accept-Encoding
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private
Content-Type: text/html; charset=UTF-8
Date: Mon, 19 Apr 2010 08:19:46 GMT
Location: https://preview.addons.mozilla.org/en-US/firefox/
X-AMO-ServedBy: pm-app-amo24.mozilla.org
Pragma: no-cache
Transfer-Encoding: chunked
Via: Moz-Cache-zlb05
Connection: Keep-Alive
X-Powered-By: PHP/5.2.9
X-Cache-Info: not cacheable; response specified "Cache-Control: no-store"

HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pm-app-amo24
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Date: Mon, 19 Apr 2010 08:19:46 GMT
X-AMO-ServedBy: pm-app-amo24.mozilla.org
Transfer-Encoding: chunked
Via: Moz-Cache-zlb05
Connection: Keep-Alive
X-Powered-By: PHP/5.2.9
X-Cache-Info: caching

Took a while to figure out why puppet wouldn't run (and pick up this change automatically). Turned out zamboni was filling up /tmp with a ton of pip stuff. I think there's another bug filed to track that.