1297316 - New Relic Unable to ping pontoon.mozilla.org, snippets.mozilla.com-python, oneanddone.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations :: Marketing Infrastructure, task)

Description

[Matjaz Horvat [:mathjazz]]

New Relic is sending a lot of "Unable to ping" messages in the last 10 hours for at least three sites:
- pontoon.mozilla.org
- snippets.mozilla.com-python
- oneanddone.mozilla.org

This is an example of such message:
https://rpm.newrelic.com/accounts/263620/applications/7307215/downtime/24129345

"https://pontoon.mozilla.org/ failed 13 times over 17 minutes, the most recent at 23:33:26 today (1 minute ago). The last failure was a DNS resolution error for pontoon.mozilla.org."

From the end user perspective, everything seems to be working fine.

Comment 1

[Peter Radcliffe [:pir]]

These sites are all in AWS and not on IT infrastructure. Passing to cloudops.

Comment 2

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

None of these belong to cloudops.

Comment 3

[Matjaz Horvat [:mathjazz]]

As of 2016-08-23 19:28:00 PDT, the errors no longer appear.

Comment 4

[Matjaz Horvat [:mathjazz]]

It's back (since yesterday).

Comment 5

[Francesco Lodolo [:flod]]

:giorgos on IRC pointed me to bug 1265674 (don't have access, not sure if this is a dupe of that).

Comment 6

[Giorgos Logiotatidis [:giorgos]]

FTR these came from back-from-the-dead snippets server instances managed by WebOps and have been shut down again. As :flod already mentioned the shut down process has been tracked down in bug 1265674

Snippets servers are now in Engagement Engeering's Infra and we follow bugs under Infrastructure & Operation::Engagement: Infrastructure

Comment 7

[Ben (:bensternthal)]

Closing.

If this happens again please open a new bug. Note pontoon and oneanddone are not hosted on marketing infra and bugs related to that should be in webops (i think) components.