1098342 - Change api-dev.bugzilla.mozilla.org DNS to point to bmo

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Greg Cox [:gcox]]

Per https://bugzilla.mozilla.org/show_bug.cgi?id=1082028#c32 (the decommissioning of api-dev.bmo) please set up a '301 generator' to redirect these URLs:

https://api-dev.bugzilla.mozilla.org/latest/* -> https://bugzilla.mozilla.org/bzapi/*
https://api-dev.bugzilla.mozilla.org/1.3/* -> https://bugzilla.mozilla.org/bzapi/*
https://api-dev.bugzilla.mozilla.org/1.2/* -> https://bugzilla.mozilla.org/bzapi/*

Comment 1

[Ed Morley [:emorley]]

(Making summary more keyword rich, so this bug is easier to find by those using the old API endpoint, in case there are any side-effects of switching them over)

Comment 2

[Gervase Markham [:gerv]]

fubar: ashish says you know who can deal with this. :-)

He suggests this needs to be migrated to Zeus and have a rewrite rule written there. With a short DNS TTL so we can switch it back if the first attempt causes screaming which is too loud, due to some unnoticed dependency.

Today is the day we told people we were decommissioning this server, so we are cool to try this. If it sticks, we can shut down the api-dev box.

Gerv

Comment 3

[Kendall Libby [:fubar] (he/him)]

(In reply to Gervase Markham [:gerv] from comment #2)
> fubar: ashish says you know who can deal with this. :-)
> 
> He suggests this needs to be migrated to Zeus and have a rewrite rule
> written there. With a short DNS TTL so we can switch it back if the first
> attempt causes screaming which is too loud, due to some unnoticed dependency.

yes, that's about right.

> Today is the day we told people we were decommissioning this server

Uh, that's news to me. I might be able to get to it this afternoon, but that
assumes zero complications with some delicate treeherder infrastructure work
that's happening very shortly.

Comment 4

[Gervase Markham [:gerv]]

That doesn't mean it has to be _done_ today, it just means that it's OK to do it from today :-)

Gerv

Comment 5

[Gervase Markham [:gerv]]

fubar: do you have an ETA for when this will be set up, so we can throw the switch?

Gerv

Comment 6

[Kendall Libby [:fubar] (he/him)]

zeus VIP set up. worked with gerv to test the redirect rules (needed a little tweaking). DNS change has been committed w/ gerv's OK; will take a few minutes to propagate to naemservers.

Comment 7

[Gervase Markham [:gerv]]

As we perhaps suspected, this switch has turned up some issues - with bzexport. See dependencies. We've reverted the switch, and are reopening this bug. Once the dependencies are fixed, we'll have another go and see if it sticks.

Gerv

Comment 8

[Gervase Markham [:gerv]]

For reference, the switch is to change the CNAME of api-dev.bugzilla.mozilla.org between:
api-dev.community.scl3.mozilla.com (BzAPI box)
and
bugzilla-apidev.vips.scl3.mozilla.com (redirect to BMO)

Gerv

Comment 9

[Ed Morley [:emorley]]

tl;dr from bug 1105141 is that using a 301 redirect causes clients to turn POSTs into payload-less GETs, so we need a non-30x-redirect solution.

It would also likely be wise to switch bzexport to the new endpoint prior to this relanding, to confirm whether bug 1033394 comment 7 is still a problem.

Comment 10

[Gervase Markham [:gerv]]

Hmm. fubar: what options do we have? Can Zeus do some transparent proxying/URL rewriting thing?

edmorley: yes, clearly, switching all code over is what we really want to do. This redirect is a stopgap, not a permanent solution.

Gerv

Comment 11

[Ed Morley [:emorley]]

Bug 1105433 unfortunately definitely blocks this from landing, bug 1098291 may also do so, but needs more investigation.

Comment 12

[Gervase Markham [:gerv]]

(In reply to Ed Morley (moved to Treeherder) [:edmorley] from comment #9)
> tl;dr from bug 1105141 is that using a 301 redirect causes clients to turn
> POSTs into payload-less GETs, so we need a non-30x-redirect solution.

From reading Wikipedia:
http://www.wikiwand.com/en/List_of_HTTP_status_codes#/3xx_Redirection
it seems that HTTP status codes 307 and 308 are designed to engineer a redirect without a change of HTTP method. So perhaps we could start using 307 (seems better supported than 308) rather than 301?

Gerv

Comment 13

[Ed Morley [:emorley]]

(In reply to Gervase Markham [:gerv] from comment #12)
> (In reply to Ed Morley (moved to Treeherder) [:edmorley] from comment #9)
> > tl;dr from bug 1105141 is that using a 301 redirect causes clients to turn
> > POSTs into payload-less GETs, so we need a non-30x-redirect solution.
> 
> From reading Wikipedia:
> http://www.wikiwand.com/en/List_of_HTTP_status_codes#/3xx_Redirection
> it seems that HTTP status codes 307 and 308 are designed to engineer a
> redirect without a change of HTTP method. So perhaps we could start using
> 307 (seems better supported than 308) rather than 301?

Yeah in theory that should work, but in practice I don't believe it will, given:
https://hg.python.org/cpython/file/8cd84e62c1fd/Lib/urllib2.py#l579

(We'll just get an HTTPError exception)

Comment 14

[Gervase Markham [:gerv]]

(In reply to Ed Morley (away until 3rd Jan) [:edmorley] from comment #13)
> Yeah in theory that should work, but in practice I don't believe it will,
> given:
> https://hg.python.org/cpython/file/8cd84e62c1fd/Lib/urllib2.py#l579

:-(

So does Zeus or some other middlebox have a transparent redirect function we can employ, then?

Gerv

Comment 15

[Ed Morley [:emorley]]

The suggestion in bug 1105122 comment 19 sounds like the best way forwards - however at least for now, this bug is blocked on bzapi compatibility layer fixes (bug 1105433) and investigation/possible-fixes (bug 1105257, bug 1098291).

Comment 16

[Ed Morley [:emorley]]

Given bug 1105433 comment 9, I think we needn't block on that bug any more.

As such, I think we're ready to proceed again here.

Note: We cannot do a 301 redirect due to bug 1105122 comment 17, so we'll need to do something like:

(In reply to David Lawrence [:dkl] from bug 1105122 comment #19)
> What if we too twiddle DNS to just make all requests to
> api-dev.bugzilla.mozilla.org and the use mod_rewrite in .htaccess to rewrite
> the path?
> 
> Example:
> RewriteRule ^(latest|1\.2|1\.3)/(.*)$ bzapi/$1 [NE]

Comment 17

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley] from comment #16)
> Given bug 1105433 comment 9, I think we needn't block on that bug any more.
> 
> As such, I think we're ready to proceed again here.
> 
> Note: We cannot do a 301 redirect due to bug 1105122 comment 17, so we'll
> need to do something like:
> 
> (In reply to David Lawrence [:dkl] from bug 1105122 comment #19)
> > What if we too twiddle DNS to just make all requests to
> > api-dev.bugzilla.mozilla.org and the use mod_rewrite in .htaccess to rewrite
> > the path?
> > 
> > Example:
> > RewriteRule ^(latest|1\.2|1\.3)/(.*)$ bzapi/$1 [NE]

Filed bug 1126315 to track the rewrite rule separately.

dkl

Comment 18

[David Lawrence [:dkl]]

(In reply to David Lawrence [:dkl] from comment #17) 
> Filed bug 1126315 to track the rewrite rule separately.
> 
> dkl

fubar, bug 1126315 is committed now so when live, we should be able to just do a simple redirect without additional logic.

dkl

Comment 19

[Gervase Markham [:gerv]]

OK.

Comment 20

[Gervase Markham [:gerv]]

So the plan of record now is to change the DNS for api-dev.bugzilla.mozilla.org to point to the servers of bugzilla.mozilla.org, and use .htaccess-based rewriting on BMO to redirect requests to the compat layer. Can we confirm that this is going to work? Does BMO pay attention to the Host: header?

Once we know it's going to work, we can change the DNS (with a short TTL, because reverting it would be the backout plan) any time after the next BMO push day. Looks like the current TTL is 60 seconds.

Gerv

Comment 21

[Gervase Markham [:gerv]]

:fubar: are you able to have another go at this?

Gerv

Comment 22

[Kendall Libby [:fubar] (he/him)]

DNS is updated.

;; ANSWER SECTION:
api-dev.bugzilla.mozilla.org. 13 IN	CNAME	bugzilla-apidev.vips.scl3.mozilla.com.
bugzilla-apidev.vips.scl3.mozilla.com. 13 IN A	63.245.215.123

Comment 23

[:glob ✱]

i suspect this caused bug 1131036; can the redirect please be disabled until that issue is resolved?

Comment 24

[Kendall Libby [:fubar] (he/him)]

(In reply to Byron Jones ‹:glob› from comment #23)
> i suspect this caused bug 1131036; can the redirect please be disabled until
> that issue is resolved?

Reverted in inventory; DNS will update in ~180s.

Comment 25

[:glob ✱]

the fix for that bug has been pushed to prod; can we re-enable the redirect again please?

Comment 26

[Kendall Libby [:fubar] (he/him)]

Updated in inventory; DNS should be live in ~180s.

Comment 27

[Gervase Markham [:gerv]]

This seems to have stuck :-)) Decommissioning the old server is tracked in bug 1082028.

Gerv