1205867 - Migrate Pulse/PulseGuardian from phx1 to CloudAMQP/Heroku

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Mark Côté [:mcote]]

The time has come to move Pulse (the RabbitMQ cluster) and PulseGuardian (the Pulse management web app) out of phx1 and into CloudAMQP and Heroku, respectively.

The migration plan is at https://docs.google.com/document/d/1F207nMJUXXxyDNuJuoPDfFzK39RSy0gOqrYMR-21AcQ/edit#

On the day of, I will need someone on hand who can close ports on the Pulse zlb, can dump the PulseGuardian database and deliver it to me, can dump the RabbitMQ definitions and deliver them to me, and can update the pulse.mozilla.org DNS entry.

I've been testing everything I can ahead of time to make sure it goes smoothly.  Jgriffin has agreed to be around to help with testing and such.

Comment 1

[hwine]

:mcote - this will also require new network flows to be added for the new IP addresses to be accessible by buildbot machines. That work doesn't need a TCW afaik, but will block this work.

Can you open a bug against releng to add the flows, and include the IP addresses there? (And block this bug, of course)

h/t :arr for thinking of this!

Comment 2

[hwine]

:mcote also, could you add the information requested in https://wiki.mozilla.org/IT/ChangeControl#Submitting_a_Change_Request then set the "cab-review" flag please? Holler if you need a hand.

Comment 3

[Mark Côté [:mcote]]

Thanks!  Filed request for netflows and CAB, blocking this bug.

Comment 4

[Mark Côté [:mcote]]

As discussed we'll be doing this outside of a TCW.  Current plan is to migrate at least Pulse itself next Wednesday, October 7th, around 5 pm PDT.  PulseGuardian may be moved earlier.

Comment 5

[Mark Côté [:mcote]]

PulseGuardian was successfully migrated to Heroku on 2015/10/06.

We attempted to migrate Pulse to CloudAMQP today, but we neglected to get a proper SSL certificate for pulse.mozilla.org onto the CloudAMQP cluster, so clients with strict hostname checking were failing to connect.  We've contacted CloudAMQP support to figure out the best way to handle this, whether that be providing them with our cert or some other mechanism.  We should be good to retry next week.

Comment 6

[Mark Côté [:mcote]]

The migration today succeeded.  The VIPs for the old cluster have been turned off and all traffic is flowing through the CloudAMQP instance.  The old cluster is still running but will be decommissioned in a day or two; see bug 1214636.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The move of the RabbitMQ server totally broke our Mozmill CI because of IP (and maybe other) change. As result our machines can no longer connect to th broker. We should really announce those changes to all the customers and not break them unexpectedly. I will handle our regression via bug 1215464.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Hal Wine [:hwine] (use NI) from comment #1)
> :mcote - this will also require new network flows to be added for the new IP
> addresses to be accessible by buildbot machines. That work doesn't need a
> TCW afaik, but will block this work.

Exactly this was not done for our machines so that we are stranded now. Sadly I cannot open bug 1205889 at all due to permission issues, so I will most likely reference this bug in the one I will file now. I assume we need an identical setup.

Comment 9

[Axel Grude [:realRaven]]

Not sure if this is related but since this morning I cannot access any mozilla.org sites anymore from my Irish ISP (meteor.ie) same thing happened to at least one German user. Fx is complaining about not being able to test authenticity of pages

Secure Connection Failed

The connection to the server was reset while the page was loading.

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. I can ping the servers and here is a traceroute:

Tracing route to addons.dynect.mozilla.net [63.245.216.132]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  TINY.mydomain.example [192.168.0.1]
  2    86 ms    89 ms    89 ms  10.144.48.34
  3   138 ms   139 ms   109 ms  10.144.240.18
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7   163 ms   152 ms   126 ms  er1.ams1.nl.above.net [80.249.208.122]
  8   124 ms   130 ms   133 ms  ae14.cr1.ams10.nl.zip.zayo.com [64.125.21.77]
  9   199 ms   189 ms   199 ms  ae7.cr2.dca2.us.zip.zayo.com [64.125.27.33]
 10   200 ms   209 ms   209 ms  v22.ae29.cr2.iah1.us.zip.zayo.com [64.125.31.222]
 11   342 ms   263 ms   295 ms  ae6.mpr4.phx2.us.zip.zayo.com [64.125.31.66]
 12   270 ms   259 ms   249 ms  64.124.201.182.mozilla.com [64.124.201.182]
 13   267 ms   239 ms   239 ms  v-1058.core1.market.phx1.mozilla.net [63.245.216.54]
 14   239 ms   248 ms   239 ms  addons-mozilla-org.mktns.services.phx1.mozilla.net [63.245.216.132]

could you look into this?

Comment 10

[Stefan Plewako [:stef]]

Similar situation here (UPC, Poland).

$ traceroute www.mozilla.org
traceroute to bedrock-prod-zlb.vips.scl3.mozilla.com (63.245.215.20), 64 hops max, 52 byte packets
 1  192.168.0.1 (192.168.0.1)  1.811 ms  1.337 ms  1.716 ms
 2  * * *
 3  89-75-22-129.infra.chello.pl (89.75.22.129)  16.391 ms  9.289 ms  10.357 ms
 4  pl-waw04a-rd2-ae-8-2124.aorta.net (84.116.252.97)  32.940 ms  36.045 ms  33.299 ms
 5  84.116.137.61 (84.116.137.61)  36.828 ms  36.354 ms  34.548 ms
 6  * * *
 7  84-116-130-130.aorta.net (84.116.130.130)  32.931 ms  34.248 ms  35.859 ms
 8  nl-ams04a-ri2-xe-1-0-0.aorta.net (84.116.134.69)  34.694 ms
    84.116.134.113 (84.116.134.113)  35.643 ms
    84.116.136.30 (84.116.136.30)  34.301 ms
 9  er1.ams1.nl.above.net (80.249.208.122)  39.777 ms  42.851 ms  63.132 ms
10  ae14.cr1.ams10.nl.zip.zayo.com (64.125.21.77)  43.635 ms  39.642 ms  38.241 ms
11  v142.ae29.cr2.ord2.us.zip.zayo.com (64.125.30.170)  136.386 ms  134.221 ms  136.577 ms
12  ae13.mpr2.den1.us.zip.zayo.com (64.125.21.254)  163.721 ms  163.975 ms  159.682 ms
13  ae11.mpr1.den1.us.zip.zayo.com (64.125.31.153)  165.336 ms  162.623 ms  161.595 ms
14  ae5.cr1.sjc2.us.zip.zayo.com (64.125.25.53)  188.194 ms  190.466 ms  194.792 ms
15  ae8.cr2.sjc2.us.zip.zayo.com (64.125.20.254)  190.186 ms  190.771 ms  189.400 ms
16  ae7.mpr2.pao1.us.zip.zayo.com (64.125.25.138)  192.016 ms  194.475 ms  190.034 ms
17  64.125.170.34.t00539-05.above.net (64.125.170.34)  193.219 ms  189.102 ms  196.938 ms
18  xe-0-0-1.border1.scl3.mozilla.net (63.245.219.130)  194.952 ms  187.942 ms  194.658 ms
19  v-1026.core1.scl3.mozilla.net (63.245.214.70)  190.806 ms  195.734 ms  199.839 ms
20  * * *
21  * * *

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Both last two comments are not related to this topic. I would suggest you to file a new on that.