972589 - One and Done prod paas 404ing after a bit of uptime

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Osmose [:osmose, :mkelly]]

After getting One and Done set up successfully on the prod paas, we started seeing constant 404s across the site after maybe an hour of uptime. The 404s seemingly started without any change by me.

Once again, my ops-idiodicy is blocking me. Any tips on how to proceed?

Comment 1

[Chris Turra [:cturra]]

i restarted your app, which moved it to a new dea server in the paas cluster. right now, we're sorting out some name server stuff as tracked in bug 971983, which i suspect is related here.

additionally, we've seen another app lose connection in the past when the stackato core routing service marked the app as down due to connection attempts. that permanent solution to that is unrelated to the bug mentioned above, but is in the patches i will be applying to the prod cluster tomorrow evening.

  https://mail.mozilla.org/pipermail/paas-users/2014-February/000194.html

Comment 2

[Chris Turra [:cturra]]

it looks like since the maintenance we did on monday evening, applications are no longer being marked as down by the stackato router (causing the 404's we were seeing here). i am going to mark this bug as r/fixed, but please re-open if this issue creeps back up on us again o/

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Thx; this host is returning a 200 OK: https://oneanddone.mozilla.org/en-US/