The ideal outcome of going to https://build.mozilla.org/buildapi/self-serve/build-system/rev/b7582d84aa15 and sticking b7582d84aa15 in the "Create new PGO builds on build-system revision" box would be to get PGO builds on that revision. A barely survivable outcome would be nothing happening at all, or an understandable error message. The current outcome is apparently "I see an UPDATE and then a COMMIT, then nothing else from that worker, and a spontaneous restart," as in, trying to get PGO builds on the build-system branch completely breaks self-serve.
(In reply to Phil Ringnalda (:philor) from comment #0) > The current outcome is apparently "I see an UPDATE and then a COMMIT, then > nothing else from that worker, and a spontaneous restart," as in, trying to > get PGO builds on the build-system branch completely breaks self-serve. can you be more specific here?
May have been a red herring, self-serve wasn't exactly well at the time, but it would have been considerably more specific with the "<nthomas>:" at the start of that quote :) I'm at least partway through (like, one of two reconfigs through) adding PGO on build-system and services-central, but if you want to see whether it really does break self-serve, we could trigger some on a twig - none of them do PGO at all do they?
I think this is me getting confused when cruncher was getting overwhelmed by low memory. The UPDATE/COMMIT are into the local db on cruncher for the self-serve operations, rather than the schedulers db remotely. The underlying issue was that buildapi was getting restarted a lot before bear & arr increased the memory on cruncher. I saw a couple of OOM kills in the kernel log but otherwise didn't spot any reason for the respawning. I just retried the PGO forcing in comment #0 and tailed the buildapi log on cruncher, no problems seen. But I hadn't seen comment #2 when I did that, so the test may not be valid.
Component: Release Engineering → Release Engineering: Developer Tools
QA Contact: release → lsblakk
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → INCOMPLETE
Product: mozilla.org → Release Engineering
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.