Triggering PGO builds on a branch that doesn't do PGO should trigger them, rather than crashing self-serve

RESOLVED INCOMPLETE

Status

Release Engineering
General
--
critical
RESOLVED INCOMPLETE
6 years ago
a year ago

People

(Reporter: philor, Unassigned)

Tracking

({buildapi})

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
The ideal outcome of going to https://build.mozilla.org/buildapi/self-serve/build-system/rev/b7582d84aa15 and sticking b7582d84aa15 in the "Create new PGO builds on build-system revision" box would be to get PGO builds on that revision.

A barely survivable outcome would be nothing happening at all, or an understandable error message.

The current outcome is apparently "I see an UPDATE and then a COMMIT, then nothing else from that worker, and a spontaneous restart," as in, trying to get PGO builds on the build-system branch completely breaks self-serve.
(In reply to Phil Ringnalda (:philor) from comment #0)
> The current outcome is apparently "I see an UPDATE and then a COMMIT, then
> nothing else from that worker, and a spontaneous restart," as in, trying to
> get PGO builds on the build-system branch completely breaks self-serve.

can you be more specific here?
(Reporter)

Comment 2

6 years ago
May have been a red herring, self-serve wasn't exactly well at the time, but it would have been considerably more specific with the "<nthomas>:" at the start of that quote :)

I'm at least partway through (like, one of two reconfigs through) adding PGO on build-system and services-central, but if you want to see whether it really does break self-serve, we could trigger some on a twig - none of them do PGO at all do they?
I think this is me getting confused when cruncher was getting overwhelmed by low memory. The UPDATE/COMMIT are into the local db on cruncher for the self-serve operations, rather than the schedulers db remotely. The underlying issue was that buildapi was getting restarted a lot before bear & arr increased the memory on cruncher. I saw a couple of OOM kills in the kernel log but otherwise didn't spot any reason for the respawning.

I just retried the PGO forcing in comment #0 and tailed the buildapi log on cruncher, no problems seen. But I hadn't seen comment #2 when I did that, so the test may not be valid.

Updated

6 years ago
Component: Release Engineering → Release Engineering: Developer Tools
QA Contact: release → lsblakk
Whiteboard: [buildapi][selfserve]
QA Contact: lsblakk → hwine

Updated

6 years ago
Keywords: buildapi
Whiteboard: [buildapi][selfserve]
(Reporter)

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → INCOMPLETE
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
(Assignee)

Updated

a year ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.