729786 - Triggering PGO builds on a branch that doesn't do PGO should trigger them, rather than crashing self-serve

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

The ideal outcome of going to https://build.mozilla.org/buildapi/self-serve/build-system/rev/b7582d84aa15 and sticking b7582d84aa15 in the "Create new PGO builds on build-system revision" box would be to get PGO builds on that revision.

A barely survivable outcome would be nothing happening at all, or an understandable error message.

The current outcome is apparently "I see an UPDATE and then a COMMIT, then nothing else from that worker, and a spontaneous restart," as in, trying to get PGO builds on the build-system branch completely breaks self-serve.

Comment 1

[Chris AtLee [:catlee]]

(In reply to Phil Ringnalda (:philor) from comment #0)
> The current outcome is apparently "I see an UPDATE and then a COMMIT, then
> nothing else from that worker, and a spontaneous restart," as in, trying to
> get PGO builds on the build-system branch completely breaks self-serve.

can you be more specific here?

Comment 2

[Phil Ringnalda (:philor)]

May have been a red herring, self-serve wasn't exactly well at the time, but it would have been considerably more specific with the "<nthomas>:" at the start of that quote :)

I'm at least partway through (like, one of two reconfigs through) adding PGO on build-system and services-central, but if you want to see whether it really does break self-serve, we could trigger some on a twig - none of them do PGO at all do they?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I think this is me getting confused when cruncher was getting overwhelmed by low memory. The UPDATE/COMMIT are into the local db on cruncher for the self-serve operations, rather than the schedulers db remotely. The underlying issue was that buildapi was getting restarted a lot before bear & arr increased the memory on cruncher. I saw a couple of OOM kills in the kernel log but otherwise didn't spot any reason for the respawning.

I just retried the PGO forcing in comment #0 and tailed the buildapi log on cruncher, no problems seen. But I hadn't seen comment #2 when I did that, so the test may not be valid.