Lando and Phabricator are slow, frequently returning 502 and 504 responses respectively
Categories
(Conduit :: Lando, defect)
Tracking
(Not tracked)
People
(Reporter: emilio, Unassigned)
Details
https://lando.services.mozilla.com/D177398/ often returns a 502 for me. Phabricator also returns a 504. It sometimes works, even though excruciatingly slowly.
|
||
Updated•2 years ago
|
|
||
Comment 1•2 years ago
|
||
2 out of the 3 active phabricator instances had incredibly high load averages (40+) and were slow to connect to over ssh.
I manually scaled up to 5 instances while also initiating an instance refresh to reset the failing instances. This resulted in the site being accessible but with very high latency.
After some further investigation, I saw very high CPU and IO usage on our RDS instance which also happened to be during our nightly backups.
The CPU and IO load decreased after the backup was completed, but it looks like both CPU and IO usage have been higher than normal over the last week. We will need to look into increasing our compute and disk io resources on the phabricator RDS instance if this higher usage is expected to continue.
Looks like this issue has been resolved.
Description
•