740505 - (talos-r3-fed-070) decommission talos-r3-fed-070

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[Jake Watkins [:dividehex]]

no vga sig | *REIMAGED*

This slave had a corrupted filesystem after it was rebooted.  I defaulted to reimaging rather than repair the fs.

Comment 2

[Mike Taylor [:bear]]

re-enabled in slavealloc

Comment 3

[[e-mail] (:bhearsum)]

Needed to have some puppet mucking before it got into production again.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Please reboot.

Comment 5

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

File system is busted again:

https://tbpl.mozilla.org/php/getParsedLog.php?id=14675369&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=14675551&tree=Mozilla-Inbound

Comment 6

[Chris Cooper [:coop] (he/him)]

It's been disabled in slavealloc now, so bumping down to normal.

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14675286&tree=Mozilla-Inbound was when it went bad, sort of oddly timed during shutdown of the successful test run.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

ssh'd in and killed the buildbot process, since the slavealloc disable doesn't take effect if it's not rebooting. Attempted to reboot it.

Comment 9

[[e-mail] (:bhearsum)]

Back in production.

Comment 10

[Armen [:armenzg]]

Producing green jobs.

Comment 11

[[e-mail] (:bhearsum)]

needs a reboot

Comment 12

[Phil Ringnalda (:philor)]

Filesystem is corrupt (comment 1 comment 5), or at least it's read-only and burning jobs, so now it needs to be disabled, but slavealloc won't be enough (comment 6 comment 8) so you'll need to ssh in (comment 8).

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

I did 'sudo reboot' on this, and now it's unresponsive to ssh. Disabled in slavealloc just in case.

Lets reimage again, it seems like it helps for a few months each time.

Comment 14

[Rail Aliiev [:rail]]

Hmm, if you ssh to this host you can notice DNS vs hostname mismatch:

$ ssh talos-r3-fed-070.build.mozilla.org hostname
talos-r3-fed64-070.build.mozilla.org
            ^^

It is not a fed64 slave:

$ uname -m
i686

Comment 15

[Rail Aliiev [:rail]]

Bad post-imaging maybe?

Comment 16

[Kim Moir [:kmoir] ET]

I'll fix this rail, my typo

Comment 17

[Kim Moir [:kmoir] ET]

back in production

Comment 18

[Justin Wood (:Callek)]

Ok, since I don't have any sign this is anything bug a slave-issue so far, we had a handful of tooltool errors on a single job with this slave

https://tbpl.mozilla.org/php/getParsedLog.php?id=19145041&tree=Mozilla-Inbound&full=1#error0

e.g.
15:23:49    ERROR -  ERROR - transfer from http://runtime-binaries.pvt.build.mozilla.org/tooltool//sha512/cea07d65a39a244961f183711b14d054c90690b69a79d89a3783f9a634f9ace7f6e70033e963a4f58ca8482b3aec8d4c5d3227cc7a0bc61e6afeccf2acc1a789 to emulator.zip failed due to a difference of 586906717 bytes

If it happens again, but only for this slave we'll pull this. If it happens for another slave anytime soon we'll hand off to IT for why runtime-binaries is acting up in terms of packets.

Comment 19

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19149223&tree=Mozilla-Inbound - a timeout in a test which has never timed out before.

Comment 20

[Justin Wood (:Callek)]

Ok, reopening -- disabling in slavealloc and leaving for coop/Tomcat to diagnose on monday.

Comment 21

[Aki Sasaki (not active)]

Puppetized, enabled, running in production.

Comment 22

[Phil Ringnalda (:philor)]

That boy still ain't right.

https://tbpl.mozilla.org/php/getParsedLog.php?id=21160623&tree=Mozilla-Inbound is, um, some sort of untar failure.

https://tbpl.mozilla.org/php/getParsedLog.php?id=21154002&tree=Firefox is an unzip timeout.

https://tbpl.mozilla.org/php/getParsedLog.php?id=21157524&tree=Mozilla-Inbound is tooltool not liking what it downloaded.

Comment 23

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=21181678&tree=Mozilla-Inbound - read-only filesystem

Comment 24

[[e-mail] (:bhearsum)]

Back in production.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Might want to take it back out...
https://tbpl.mozilla.org/php/getParsedLog.php?id=21348135&tree=Mozilla-B2g18 (and a ton more like it)

Comment 26

[[e-mail] (:bhearsum)]

It's been disabled, and not even online at the moment...

Comment 27

[Kim Moir [:kmoir] ET]

puppetized and back in production.

Comment 28

[Phil Ringnalda (:philor)]

You forgot the part where usually the person putting it back in production says "ran two green jobs" as though that was a sign of health. Though in this case, it would be "ran zero green jobs."

https://tbpl.mozilla.org/php/getParsedLog.php?id=22213793&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=22216536&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=22221969&tree=Mozilla-Central

Please disable it, and put it out of our collective misery, it's decomm time.

Comment 29

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=22225261&tree=Mozilla-Inbound - now it's back to having a read-only filesystem, which typically results in eating up dozens or hundreds of jobs since they only take a minute each.

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

buildbot stopped, disabled in slavealloc.

Comment 31

[Phil Ringnalda (:philor)]

I just love the dazed look in their eyes after you club them while they're in the middle of a frantic speed-RETRY run.

Comment 32

[Armen [:armenzg]]

We're fine without it as we don't have that much load on the rev3 fedoras anymore.

Comment 33

[Chris Cooper [:coop] (he/him)]

Reopening to officially decommission.

Comment 34

[[e-mail] (:bhearsum)]

Removed from buildbot-configs in https://hg.mozilla.org/build/buildbot-configs/rev/bb86e30e979b

It's not in Puppet anymore an it's not in slavealloc, so I think that's all there is to do.