1149753 - Migrate TBPL's in-repo .htaccess redirects to the 'virtual-redirect' site on generic

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ed Morley [:emorley]]

In deps of bug 1054977, we've migrated away from TBPL, have stopped the data import cron & have set up redirects (using an .htaccess in the root of the src repo) from TBPL to Treeherder/mcMerge. There is also bug 1149564 filed for decommissioning the TBPL database nodes.

This bug is specifically about removing the TBPL backend parts from the generic cluster. (Another bug will remove flows et al.)

In order to do this, we'll need to:
1) Copy the current in-repo .htaccess redirect rules into Zeus (or wherever).
2) Adjust the DNS/Zeus configs for tbpl.mozilla.org and tbpl-dev.allizom.org so that they use the new redirects in Zeus, and not the www directory for TBPL on generic.
3) Check this all works.
4) Delete the src/www directories on generic, as well as puppet entries/nagios checks/chief for TBPL etc

The redirects currently set in .htaccess that we'll wish to keep, are:
https://hg.mozilla.org/webtools/tbpl/file/default/.htaccess#l5

<IfModule mod_rewrite.c>
  # Redirects for TBPL end of life (bug 1054977).
  RewriteEngine On
  RewriteBase /

  # mcMerge is now on its own hosting, so redirect separately from TBPL:
  # eg: https://tbpl.mozilla.org/mcmerge/?cset=2a404169de2d&tree=mozilla-central
  RewriteRule ^mcmerge(/.*)?$ https://mcmerge.paas.allizom.org/ [R=301,L]

  # Redirecting doesn't make sense for calls to TBPL's backend, so let's
  # return 403 Forbidden, so API consumers hit a more obvious failure mode.
  RewriteRule ^php(/.*)?$ - [F,L]

  # TBPL used non-standard names for the trees, so we have to convert to the
  # canonical repo name used by Treeherder/everywhere else. These rules could
  # be simplified using RewriteMap to just convert many of the repo names to
  # lowercase (though there are exceptions) but it can't be used inside an
  # .htaccess, and it's not worth the hassle of having to get WebOps involved.
  # So we just handle the common trees and let the rest hit the catch-all.

  RewriteCond %{QUERY_STRING} ^(.*)tree=Mozilla-Inbound(.*)$
  RewriteRule ^(.*)$ $1?%1repo=mozilla-inbound%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=B2g-Inbound(.*)$
  RewriteRule ^(.*)$ $1?%1repo=b2g-inbound%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Fx-Team(.*)$
  RewriteRule ^(.*)$ $1?%1repo=fx-team%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Try(.*)$
  RewriteRule ^(.*)$ $1?%1repo=try%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Mozilla-Aurora(.*)$
  RewriteRule ^(.*)$ $1?%1repo=mozilla-aurora%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Mozilla-Beta(.*)$
  RewriteRule ^(.*)$ $1?%1repo=mozilla-beta%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Mozilla-Release(.*)$
  RewriteRule ^(.*)$ $1?%1repo=mozilla-release%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Gaia-Try(.*)$
  RewriteRule ^(.*)$ $1?%1repo=gaia-try%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Thunderbird-Trunk(.*)$
  RewriteRule ^(.*)$ $1?%1repo=comm-central%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Thunderbird-Try(.*)$
  RewriteRule ^(.*)$ $1?%1repo=try-comm-central%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Thunderbird-Aurora(.*)$
  RewriteRule ^(.*)$ $1?%1repo=comm-aurora%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Thunderbird-Beta(.*)$
  RewriteRule ^(.*)$ $1?%1repo=comm-beta%2

  RewriteCond %{QUERY_STRING} ^(.*)tree=Jetpack(.*)$
  RewriteRule ^(.*)$ $1?%1repo=addon-sdk%2

  # Next problem: Treeherder's query params are different in virtually all cases.
  # The repo name mappings above have already converted 'tree' to 'repo', but we
  # still have to handle pusher/rev. Any other params will be discarded for
  # simplicity. In addition, if the params are specified in an order different to
  # the one used by the UI, we don't match and so hit the catch-all, but it's not
  # worth the added complexity to handle this, since it should be rare.
  # The [NE] flag is required to prevent Apache escaping the '#' to '%23'.

  # Filtered by pusher:
  # eg: https://tbpl.mozilla.org/?tree=Try&pusher=[e-mail]
  # Any params other than the repo and pusher are discarded.
  RewriteCond %{QUERY_STRING} repo=([^&]+).*&pusher=([^&]+)
  RewriteRule ^(.*)$ https://treeherder.mozilla.org/#/jobs?repo=%1&author=%2 [NE,R=301,L]

  # Viewing a specific revision:
  # eg: https://tbpl.mozilla.org/?tree=Try&rev=cc63db12dafb
  # Any params other than the repo and revision are discarded.
  RewriteCond %{QUERY_STRING} repo=([^&]+).*&rev=([^&]+)
  RewriteRule ^(.*)$ https://treeherder.mozilla.org/#/jobs?repo=%1&revision=%2 [NE,R=301,L]

  # The standard view for a repo:
  # eg: https://tbpl.mozilla.org/?tree=Mozilla-Inbound
  # Any params other than the repo are discarded.
  RewriteCond %{QUERY_STRING} repo=([^&]+)
  RewriteRule ^(.*)$ https://treeherder.mozilla.org/#/jobs?repo=%1 [NE,R=301,L]

  # For everything else just redirect with the entire query string discarded.
  RewriteRule ^(.*)$ https://treeherder.mozilla.org/? [R=301,L]
</IfModule>

(I've removed the cache directory pass-through that's currently present in the .htaccess rules, since that won't be applicable once the src/www directories are deleted on generic).

Comment 1

[:Atoll]

(In reply to Ed Morley (Away 23rd March -> 1st April) [:edmorley] from comment #0)
> In order to do this, we'll need to:
> 1) Copy the current in-repo .htaccess redirect rules into Zeus (or wherever).
> 2) Adjust the DNS/Zeus configs for tbpl.mozilla.org and tbpl-dev.allizom.org
> so that they use the new redirects in Zeus, and not the www directory for
> TBPL on generic.

I'd like to use the 'virtual-redirect' site on the generic cluster. It's purpose-built to host RewriteRules and nothing else, and it seems like (given the large blob of RewriteRule above) it would be an excellent fit here. That also saves us porting these rules to Zeus.

Comment 2

[:Atoll]

To correct one point, "virtual-redirect.mozilla.org" on the static cluster:

files/generic/httpd-static/etc-httpd/conf/domains/virtual-redirect.mozilla.org.conf

Comment 3

[Ed Morley [:emorley]]

(In reply to Richard Soderberg [:atoll] from comment #1)
> I'd like to use the 'virtual-redirect' site on the generic cluster. It's
> purpose-built to host RewriteRules and nothing else, and it seems like
> (given the large blob of RewriteRule above) it would be an excellent fit
> here. That also saves us porting these rules to Zeus.

Sounds perfect - I didn't know that existed :-)

Comment 4

[Ed Morley [:emorley]]

If it's easier, I'm also happy to just remove the tbpl-dev domain/files entirely, and only redirect prod.

Comment 5

[:Atoll]

+1 to dropping tbpl-dev entirely, unless anyone else objects.

Comment 6

[C. Liang [:cyliang]]

Rewrites for tbpl.mozilla.org is now being handled by the static cluster via the virtual-redirect host outlined above.  Please let me / webops know if there are issues.


I had to make minor tweaks to get things working:
  - Drop RewriteBase (only valid in per-directory config files)
  - adding in a preceding "/" for the mcmerge and php redirects


$ curl -I "https://tbpl.mozilla.org/mcmerge/foo"
HTTP/1.1 301 Moved Permanently
Server: Apache
X-Backend-Server: pp-web04
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 01 Apr 2015 18:31:05 GMT
Location: https://mcmerge.paas.allizom.org/
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: cached

$ curl -I "https://tbpl.mozilla.org/php/getRevisionBuilds.php?branch=gaia-try&ignore_deprecation_notice=1&rev=34400fa5f226&_=1427899045396"
HTTP/1.1 403 Forbidden
Server: Apache
X-Backend-Server: pp-web03
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 01 Apr 2015 18:32:33 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: caching

$ curl -I "https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=17a0c2af1f2e"
HTTP/1.1 301 Moved Permanently
Server: Apache
X-Backend-Server: pp-web04
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 01 Apr 2015 18:33:04 GMT
Location: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=17a0c2af1f2e
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: caching

$ curl -I "https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=17a0c2af1f2e&should_not_pass=this"
HTTP/1.1 301 Moved Permanently
Server: Apache
X-Backend-Server: pp-web02
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 01 Apr 2015 18:33:16 GMT
Location: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=17a0c2af1f2e
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: caching

Comment 7

[Ed Morley [:emorley]]

(Sorry for delay in replying, back from PTO then public holidays)
Looks great - thank you for this and the other bugs! :-)

Happy to just drop tbpl-dev - would you like me to file another bug?

Comment 8

[:Atoll]

(In reply to Ed Morley [:edmorley] from comment #7)
> (Sorry for delay in replying, back from PTO then public holidays)
> Looks great - thank you for this and the other bugs! :-)
> 
> Happy to just drop tbpl-dev - would you like me to file another bug?

Yes, if you wouldn't mind - it'll help us track the decom separate from the redirects here.

Comment 9

[:Atoll]

(If when're decommissioning all of tbpl once these redirects are live a few days, we could just merge the -dev and -prod bugs, too.)

Comment 10

[Ed Morley [:emorley]]

(In reply to Richard Soderberg [:atoll] from comment #8)
> Yes, if you wouldn't mind - it'll help us track the decom separate from the
> redirects here.

Filed bug 1152225 :-)

(In reply to Richard Soderberg [:atoll] from comment #9)
> (If when're decommissioning all of tbpl once these redirects are live a few
> days, we could just merge the -dev and -prod bugs, too.)

We're planning on keeping the tbpl-prod redirects around for something more like a few months. (There are a lot of bug comments that still link to TBPL for Try results etc).

Though reading this back, I'm not sure whether you mean:
(a) Removing everything tbpl-prod, including the redirects in this bug
(b) Removing the tbpl-prod files from generic and leaving the redirects.

I'd be fine with doing (b) now; the redirects have had enough testing now. (In which case we can either close the new bug 1152225, or make that about both, and make this bug just about the redirects, and mark it as fixed).

Comment 11

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #10)
> I'd be fine with doing (b) now; the redirects have had enough testing now.

I've made bug 1152225 about both (but obviously not removing the tbpl-prod DNS entries required for the redirects set up here).

As such, I believe this bug is now complete :-)