Closed Bug 1351153 Opened 7 years ago Closed 7 years ago

releases.mozilla.org (54.192.18.20) is DOWN :PING CRITICAL - Packet loss = 100%

Categories

(Cloud Services :: Operations: Product Delivery, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ryanc, Unassigned)

Details

      No description provided.
Hello,

Had this come in,

nagios-scl3> Tue 02:47:26 UTC [5645] releases.mozilla.org (54.192.18.20) is DOWN :PING CRITICAL - Packet loss = 100%

Site looks accessible, and curl returns 200 according according to docs,

https://mana.mozilla.org/wiki/display/SVCOPS/Product+Delivery#ProductDelivery-Troubleshooting

$ curl -v -o /dev/null https://archive.mozilla.org/pub/firefox/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 52.84.124.6...
* TCP_NODELAY set
* Connected to archive.mozilla.org (52.84.124.6) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: ftp.mozilla.org
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /pub/firefox/ HTTP/1.1
> Host: archive.mozilla.org
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK


Traceroute from nagios3.private.scl3 looks good,

 Host                                                                                                                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. fw1.private.scl3.mozilla.net                                                                                                               0.0%    10    0.7   0.7   0.7   0.8   0.0
 2. 63.245.214.53                                                                                                                              0.0%    10    2.7   2.7   2.3   2.9   0.0
 3. 63.245.214.46                                                                                                                              0.0%    10    1.1   1.1   1.0   1.2   0.0
 4. xe-1-2-3.border1.sjc2.mozilla.net                                                                                                          0.0%    10    1.9   1.9   1.8   2.0   0.0
 5. xe-1-2-0.border1.pao1.mozilla.net                                                                                                          0.0%    10    2.2   2.1   2.0   2.3   0.0
 6. ip67-92-171-49.z171-92-67.customer.algx.net                                                                                                0.0%    10    2.6   4.5   2.5  21.7   6.0
 7. 207.88.13.224.ptr.us.xo.net                                                                                                                0.0%    10   41.7  41.6  41.4  42.1   0.0
 8. 207.88.12.143.ptr.us.xo.net                                                                                                                0.0%    10   41.6  43.4  41.5  52.9   4.0
 9. 207.88.12.140.ptr.us.xo.net                                                                                                                0.0%    10   41.7  44.3  41.4  54.4   5.0
10. 207.88.12.180.ptr.us.xo.net                                                                                                                0.0%    10   41.5  44.4  41.3  71.2   9.4
11. 207.88.14.189.ptr.us.xo.net                                                                                                                0.0%    10   41.4  41.7  41.3  42.7   0.3
12. 52.95.217.244                                                                                                                              0.0%    10   41.4  41.4  41.3  41.5   0.0
13. 176.32.125.152                                                                                                                             0.0%     9   94.4  91.8  78.9  98.9   5.9
14. 176.32.125.155                                                                                                                             0.0%     9   75.8  75.3  74.4  76.0   0.4
15. 54.239.43.198                                                                                                                              0.0%     9   75.3  75.3  75.1  75.8   0.0
16. 54.239.104.151                                                                                                                            25.0%     9   84.0  78.2  76.6  84.0   2.8
17. ???
18. server-54-192-18-20.iad12.r.cloudfront.net                                                                                                 0.0%     9   75.6  75.7  75.6  75.8   0.0
Update,

Site is back up (according to nagios in scl3),

19:58:10 <@nagios-scl3> Tue 02:58:10 UTC [5652] releases.mozilla.org (54.192.18.20) is UP :PING OK - Packet loss = 0%, RTA = 75.50 ms
Summary: nagios-scl3> Tue 02:47:26 UTC [5645] releases.mozilla.org (54.192.18.20) is DOWN :PING CRITICAL - Packet loss = 100% → releases.mozilla.org (54.192.18.20) is DOWN :PING CRITICAL - Packet loss = 100%
releases/archive is just Cloudfront -> s3. I'm not sure Cloudfront officially support ICMP, so a ping check probably isn't needed.

It's possible that whatever Cloudfront node nagios was talking to went down or just doesn't respond to icmp. All other monitors are green, so I will close this out.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
I added a simple https uptime check in pingdom for this, to check every 5 minutes and Ryan will yank the nagios ping check.
You need to log in before you can comment on or make changes to this bug.