809756 - Linux PGO builds timing out

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=16833333&tree=Mozilla-Inbound

MOZ_PGO_INSTRUMENTED=1 OBJDIR=obj-firefox JARLOG_DIR=obj-firefox/jarlog/en-US python obj-firefox/_profile/pgo/profileserver.py 10
args: ['/builds/slave/m-in-lnx-pgo/build/obj-firefox/dist/firefox/firefox-bin', '-no-remote', '-profile', '/builds/slave/m-in-lnx-pgo/build/obj-firefox/_profile/pgo/pgoprofile/', 'http://localhost:8888/index.html']
INFO | automation.py | Application pid: 23164
Xlib:  extension "RANDR" missing on display ":2.0".
(that seems like An Clue to blame, but it is not, it's present in every run, green or not)
command timed out: 10800 seconds without output, attempting to kill

inbound regression range: https://tbpl.mozilla.org/php/getParsedLog.php?id=16833333&tree=Mozilla-Inbound

central regression range (since it merged that range): https://hg.mozilla.org/mozilla-central/rev/c652c39732f8

inbound and central are closed.

Comment 1

[Phil Ringnalda (:philor)]

Nice work with the clipboard there.

inbound regression range: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=0fb51f488b03&tochange=fd836a1f49c9

Comment 2

[Phil Ringnalda (:philor)]

First failure on inbound was on the 9am periodic, the noon periodic on central was green, which should let infra changes off the hook.

Comment 3

[Mike Hommey [:glandium]]

A screenshot would have been useful.

Comment 4

[Phil Ringnalda (:philor)]

Rather a lot of PGO builds triggered in https://tbpl.mozilla.org/?tree=Mozilla-Inbound&fromchange=0fb51f488b03&tochange=557e4cdad7c5

Comment 5

[Mike Hommey [:glandium]]

Looks like this one is the culprit.
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=eb636c95b1c0

I think it's bug 803669, but it's safer to just backout everything.

Comment 6

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/82c59210f11f
https://hg.mozilla.org/integration/mozilla-inbound/rev/e0d7b394462b

Comment 7

[Ed Morley [:emorley]]

(In reply to Mike Hommey [:glandium] from comment #5)
> Looks like this one is the culprit.
> https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=eb636c95b1c0
> 
> I think it's bug 803669, but it's safer to just backout everything.

Whilst the job hasn't yet completed, the PGO run on that push is definitely broken (both from runtime & stdio of that job) - and the push before was green, so I'm confident the backout will work. As such (and seeing as mozilla-central is already closed, so not much to lose), I'm going to merge to m-c before the tip retrigger (eta 1 hour) comes back.

Comment 8

[Ed Morley [:emorley]]

That backout worked, trees reopened.