831363 - https://treestatus.mozilla.org/ is down after PHX issues

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Ed Morley [:emorley]]

(We need bug 822317 to avoid this)

Comment 1

[Ed Morley [:emorley]]

s/avoid this/avoid this in the future/

Comment 2

[Brandon Burton [:solarce]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #0)
> (We need bug 822317 to avoid this)

Unfortunately the issues aren't resolved yet, I'll update this bug once they are

Comment 3

[Brandon Burton [:solarce]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #0)
> (We need bug 822317 to avoid this)

I am not sure if Bug 822317 is related, as that is handling a backend MySQL master failover, where as our current issue is a frontend load balancer failure, not just a failover situation.

Comment 4

[Ed Morley [:emorley]]

(In reply to Brandon Burton [:solarce] from comment #3)
> I am not sure if Bug 822317 is related, as that is handling a backend MySQL
> master failover, where as our current issue is a frontend load balancer
> failure, not just a failover situation.

Ah true

Comment 5

[Brandon Burton [:solarce]]

Things are currently stable. Root cause investigation is still ongoing in https://bugzilla.mozilla.org/show_bug.cgi?id=831353 but my testing and our monitoring shows treestatus.mo as working now

Comment 6

[Ed Morley [:emorley]]

I'm still getting intermittent 500 ISEs from treestatus - particularly when submitting forms. Any ideas?

Comment 7

[Chris AtLee [:catlee]]

What's in the logs?

Comment 8

[Brandon Burton [:solarce]]

I believe some of this issue has been resolved in code and we know now to restart treestatus if there is a failover event