711666 - investigate issues with w32-ix-slave43 and w32-ix-slave44

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

pulling this offline as per philor's request in #build due to several failures and exceptions.  did a graceful shutdown, disabled in slavealloc and opened this bug.

Comment 1

[Phil Ringnalda (:philor)]

And then after requesting, he went AWOL.

43 and 44 came back out of bug 702390. 44 was burning jobs from lacking the Python ssl module, not sure what its current state is; 43 apparently survived one opt build, but exceptions out of debug builds by failing to be able to do graphserver posts because it lacks simplejson.

Comment 2

[Phil Ringnalda (:philor)]

I think we can all agree that this test of the "Give philor a hair-trigger for badslaves that were resuscitated on a Friday afternoon" system was a rousing success.

Apparently the simplejson thing was fallout from bug 705403, but since w32-ix-slave43 did three inbound debug builds during the time that anything which did them would have failed, it OMG FRIDAY AFTERNOON KILL IT QUICK BEFORE EVERYBODY LEAVES.

Comment 3

[Phil Ringnalda (:philor)]

Sigh, or maybe there are issues with both of them, since https://tbpl.mozilla.org/php/getParsedLog.php?id=8004242&tree=Firefox and https://tbpl.mozilla.org/php/getParsedLog.php?id=8004253&tree=Firefox are 44 blowing a pair of builds due to a lack of simplejson long after the 705403 revert.

Comment 4

[Matt Brubeck (:mbrubeck)]

44: https://tbpl.mozilla.org/php/getParsedLog.php?id=8007185&tree=Mozilla-Inbound

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to Phil Ringnalda (:philor) from comment #3)
> Sigh, or maybe there are issues with both of them, since
> https://tbpl.mozilla.org/php/getParsedLog.php?id=8004242&tree=Firefox and
> https://tbpl.mozilla.org/php/getParsedLog.php?id=8004253&tree=Firefox are 44
> blowing a pair of builds due to a lack of simplejson long after the 705403
> revert.

Bug 705403 is what allows newly-imaged slaves to find simplejson, so this doesn't surprise me.

(In reply to Phil Ringnalda (:philor) from comment #1) 
> 43 and 44 came back out of bug 702390. 44 was burning jobs from lacking the
> Python ssl module, not sure what its current state is; 43 apparently
> survived one opt build, but exceptions out of debug builds by failing to be
> able to do graphserver posts because it lacks simplejson.

That's the real bug here. Were there any other slaves failing to graphserver post except for w32-ix-slave43 and w32-ix-slave44?

/me goes to track down logs.

Comment 6

[Chris Cooper [:coop] (he/him)]

So it would seem that these two machines came back from re-imaging, connected to OPSI, but didn't grab any packages that had been deployed since the ref image was taken. 

I can mark the slaves to reinstall *everything*, since at this point that seems safest, but I would certainly prefer a better solution here than the one OPSI is currently (not) providing.

We'll still be blocked on getting the graphserver post steps to use the new PATH (bug 705403). Not sure why they're not inheriting that properly yet.

Comment 7

[Chris Cooper [:coop] (he/him)]

These are back in production now. Fuzzer env issues are being dealt with in bug 705403.