Closed Bug 1098342 Opened 8 years ago Closed 8 years ago

Change api-dev.bugzilla.mozilla.org DNS to point to bmo

Categories

(bugzilla.mozilla.org :: Infrastructure, defect)

Production
defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: gcox, Assigned: fubar)

References

Details

(Making summary more keyword rich, so this bug is easier to find by those using the old API endpoint, in case there are any side-effects of switching them over)
Summary: Redirect api-dev requests to bmo → Redirect api-dev.bugzilla.mozilla.org requests to bmo's bzapi compatibility layer
fubar: ashish says you know who can deal with this. :-)

He suggests this needs to be migrated to Zeus and have a rewrite rule written there. With a short DNS TTL so we can switch it back if the first attempt causes screaming which is too loud, due to some unnoticed dependency.

Today is the day we told people we were decommissioning this server, so we are cool to try this. If it sticks, we can shut down the api-dev box.

Gerv
Flags: needinfo?(klibby)
(In reply to Gervase Markham [:gerv] from comment #2)
> fubar: ashish says you know who can deal with this. :-)
> 
> He suggests this needs to be migrated to Zeus and have a rewrite rule
> written there. With a short DNS TTL so we can switch it back if the first
> attempt causes screaming which is too loud, due to some unnoticed dependency.

yes, that's about right.

> Today is the day we told people we were decommissioning this server

Uh, that's news to me. I might be able to get to it this afternoon, but that
assumes zero complications with some delicate treeherder infrastructure work
that's happening very shortly.
Flags: needinfo?(klibby)
That doesn't mean it has to be _done_ today, it just means that it's OK to do it from today :-)

Gerv
fubar: do you have an ETA for when this will be set up, so we can throw the switch?

Gerv
Flags: needinfo?(klibby)
zeus VIP set up. worked with gerv to test the redirect rules (needed a little tweaking). DNS change has been committed w/ gerv's OK; will take a few minutes to propagate to naemservers.
Assignee: nobody → klibby
Flags: needinfo?(klibby)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Blocks: 1105122
As we perhaps suspected, this switch has turned up some issues - with bzexport. See dependencies. We've reverted the switch, and are reopening this bug. Once the dependencies are fixed, we'll have another go and see if it sticks.

Gerv
No longer blocks: 1105122
Status: RESOLVED → REOPENED
Depends on: 1105122, 1104906
Resolution: FIXED → ---
For reference, the switch is to change the CNAME of api-dev.bugzilla.mozilla.org between:
api-dev.community.scl3.mozilla.com (BzAPI box)
and
bugzilla-apidev.vips.scl3.mozilla.com (redirect to BMO)

Gerv
Depends on: 1105141
Depends on: 1105257
tl;dr from bug 1105141 is that using a 301 redirect causes clients to turn POSTs into payload-less GETs, so we need a non-30x-redirect solution.

It would also likely be wise to switch bzexport to the new endpoint prior to this relanding, to confirm whether bug 1033394 comment 7 is still a problem.
Depends on: 1033394
Hmm. fubar: what options do we have? Can Zeus do some transparent proxying/URL rewriting thing?

edmorley: yes, clearly, switching all code over is what we really want to do. This redirect is a stopgap, not a permanent solution.

Gerv
Depends on: 1098291
Depends on: 1105433
Bug 1105433 unfortunately definitely blocks this from landing, bug 1098291 may also do so, but needs more investigation.
Depends on: 1089805, 1036802
No longer depends on: 1104906
(In reply to Ed Morley (moved to Treeherder) [:edmorley] from comment #9)
> tl;dr from bug 1105141 is that using a 301 redirect causes clients to turn
> POSTs into payload-less GETs, so we need a non-30x-redirect solution.

From reading Wikipedia:
http://www.wikiwand.com/en/List_of_HTTP_status_codes#/3xx_Redirection
it seems that HTTP status codes 307 and 308 are designed to engineer a redirect without a change of HTTP method. So perhaps we could start using 307 (seems better supported than 308) rather than 301?

Gerv
(In reply to Gervase Markham [:gerv] from comment #12)
> (In reply to Ed Morley (moved to Treeherder) [:edmorley] from comment #9)
> > tl;dr from bug 1105141 is that using a 301 redirect causes clients to turn
> > POSTs into payload-less GETs, so we need a non-30x-redirect solution.
> 
> From reading Wikipedia:
> http://www.wikiwand.com/en/List_of_HTTP_status_codes#/3xx_Redirection
> it seems that HTTP status codes 307 and 308 are designed to engineer a
> redirect without a change of HTTP method. So perhaps we could start using
> 307 (seems better supported than 308) rather than 301?

Yeah in theory that should work, but in practice I don't believe it will, given:
https://hg.python.org/cpython/file/8cd84e62c1fd/Lib/urllib2.py#l579

(We'll just get an HTTPError exception)
(In reply to Ed Morley (away until 3rd Jan) [:edmorley] from comment #13)
> Yeah in theory that should work, but in practice I don't believe it will,
> given:
> https://hg.python.org/cpython/file/8cd84e62c1fd/Lib/urllib2.py#l579

:-(

So does Zeus or some other middlebox have a transparent redirect function we can employ, then?

Gerv
The suggestion in bug 1105122 comment 19 sounds like the best way forwards - however at least for now, this bug is blocked on bzapi compatibility layer fixes (bug 1105433) and investigation/possible-fixes (bug 1105257, bug 1098291).
No longer depends on: 1105257
Given bug 1105433 comment 9, I think we needn't block on that bug any more.

As such, I think we're ready to proceed again here.

Note: We cannot do a 301 redirect due to bug 1105122 comment 17, so we'll need to do something like:

(In reply to David Lawrence [:dkl] from bug 1105122 comment #19)
> What if we too twiddle DNS to just make all requests to
> api-dev.bugzilla.mozilla.org and the use mod_rewrite in .htaccess to rewrite
> the path?
> 
> Example:
> RewriteRule ^(latest|1\.2|1\.3)/(.*)$ bzapi/$1 [NE]
Status: REOPENED → ASSIGNED
No longer depends on: 1105433
Depends on: 1126315
(In reply to Ed Morley [:edmorley] from comment #16)
> Given bug 1105433 comment 9, I think we needn't block on that bug any more.
> 
> As such, I think we're ready to proceed again here.
> 
> Note: We cannot do a 301 redirect due to bug 1105122 comment 17, so we'll
> need to do something like:
> 
> (In reply to David Lawrence [:dkl] from bug 1105122 comment #19)
> > What if we too twiddle DNS to just make all requests to
> > api-dev.bugzilla.mozilla.org and the use mod_rewrite in .htaccess to rewrite
> > the path?
> > 
> > Example:
> > RewriteRule ^(latest|1\.2|1\.3)/(.*)$ bzapi/$1 [NE]

Filed bug 1126315 to track the rewrite rule separately.

dkl
(In reply to David Lawrence [:dkl] from comment #17) 
> Filed bug 1126315 to track the rewrite rule separately.
> 
> dkl

fubar, bug 1126315 is committed now so when live, we should be able to just do a simple redirect without additional logic.

dkl
OK.
Summary: Redirect api-dev.bugzilla.mozilla.org requests to bmo's bzapi compatibility layer → Change api-dev.bugzilla.mozilla.org DNS to point to bmo
So the plan of record now is to change the DNS for api-dev.bugzilla.mozilla.org to point to the servers of bugzilla.mozilla.org, and use .htaccess-based rewriting on BMO to redirect requests to the compat layer. Can we confirm that this is going to work? Does BMO pay attention to the Host: header?

Once we know it's going to work, we can change the DNS (with a short TTL, because reverting it would be the backout plan) any time after the next BMO push day. Looks like the current TTL is 60 seconds.

Gerv
:fubar: are you able to have another go at this?

Gerv
Flags: needinfo?(klibby)
DNS is updated.

;; ANSWER SECTION:
api-dev.bugzilla.mozilla.org. 13 IN	CNAME	bugzilla-apidev.vips.scl3.mozilla.com.
bugzilla-apidev.vips.scl3.mozilla.com. 13 IN A	63.245.215.123
Flags: needinfo?(klibby)
i suspect this caused bug 1131036; can the redirect please be disabled until that issue is resolved?
Depends on: 1131036
(In reply to Byron Jones ‹:glob› from comment #23)
> i suspect this caused bug 1131036; can the redirect please be disabled until
> that issue is resolved?

Reverted in inventory; DNS will update in ~180s.
the fix for that bug has been pushed to prod; can we re-enable the redirect again please?
Updated in inventory; DNS should be live in ~180s.
This seems to have stuck :-)) Decommissioning the old server is tracked in bug 1082028.

Gerv
Status: ASSIGNED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
No longer depends on: 1366390
You need to log in before you can comment on or make changes to this bug.