1386273 - balrog admin sometimes gets sluggish when l10n is running

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), enhancement, P3)

Description

[[e-mail] (:bhearsum)]

I noticed this yesterday while trying to use the admin interface while a bunch of respun l10n were submitting to Balrog. Relud pointed out that the write latency to the database was high (https://screenshots.firefox.com/u1XuLrcNKxLi5PNh/app.datadoghq.com). We get spikes like this daily, though their peaks seem different. Probably when multiple sets of nightlies are running at the same time, it gets worse (eg: date+central).

The reason write latency is so bad is probably because the UPDATEs we make are so massive (and we make them to the primary tables + history tables). We may make multiple attempts at them as well, because of update races. https://bugzilla.mozilla.org/show_bug.cgi?id=1294493 might help here, if we decide to go forward with it.

I don't think we need to do anything about this immediately, other than be aware of it.

Comment 1

[Daniel Thorn [:relud]]

We could just increase the admin instance size and see if it gets better.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Daniel Thorn [:relud] from comment #1)
> We could just increase the admin instance size and see if it gets better.

That sounds like a good first step to me. If things stay the same with a larger admin instance, it starts to point in the direction of the db or network.

Comment 3

[[e-mail] (:bhearsum)]

I don't think this has been an issue since the GCP migration.