927261 - foopy112 has a read-only filesystem

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Phil Ringnalda (:philor)]

Been a while, so I don't remember whether these tegras will continue to fail and burn jobs until it is fixed or disabled, or just once and then fail to come back for another shot.

https://tbpl.mozilla.org/php/getParsedLog.php?id=29172517&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=29172319&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=29172276&tree=Mozilla-Central

Comment 1

[Armen [:armenzg]]

did you mean foopy112?

Comment 2

[Armen [:armenzg]]

The slavealloc note is out of date.

Comment 3

[Phil Ringnalda (:philor)]

Cute, I read bottom-up rather than top-down through

foopy112
pdu2.df201-4.build.mtv1.mozilla.com Outlet: AA4
foopy08:/builds/tegra-047

in https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=tegra-047.

Comment 4

[Chris Cooper [:coop] (he/him)]

The console log visible on foopy112-mgmt is showing a kernel panic.

Comment 5

[Justin Wood (:Callek)]

To be clear, if this needs a full reimage, thats ok. Also ok is if it needs a hardware replacement [from something we have in-stock]. All data on the machine is recoverable with puppet+minimal human intervention.

Comment 6

[Vinh Hua [:vinh]]

Re-imaging host.

Comment 7

[Vinh Hua [:vinh]]

Re-image completed. Please reopen ticket if you need anything else.

Comment 8

[Justin Wood (:Callek)]

(In reply to Vinh Hua [:vinh] from comment #7)
> Re-image completed. Please reopen ticket if you need anything else.

Sorry, no its not (just ran now):

[root@foopy112 ~]# date
Mon Oct 21 11:06:04 PDT 2013

It has the puppet kickstart PW for root login, no auth keys. Also has the (aiui proper) deploypass but puppetizing fails due to the date mismatch.

Please correct.

Comment 9

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #8)
> (In reply to Vinh Hua [:vinh] from comment #7)
> > Re-image completed. Please reopen ticket if you need anything else.
> 
> Sorry, no its not (just ran now):

Also of note, /root/puppetize.sh is 0 bytes.

Comment 10

[Vinh Hua [:vinh]]

Reimage completes but puppetize fails or so.
Getting resolving proxy.dmz.mtv1.mozilla.com...failed

wget: unable to resolve host address 'proxy.dmz.mtv1.mozilla.com'

Dustin - Amy said you can help?  I've attached a snapshot.

Comment 11

[Vinh Hua [:vinh]]

Attachment: foopy12-fail.png

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, I guess the proxies aren't as widely deployed as I'd hoped.  I modified the script:

 # set up the puppetize script to run at boot
 hgrepo="http://hg.mozilla.org/build/puppet"
-# use the proxy so that this works in QA
-http_proxy=http://proxy.dmz.<%= @datacenter %>.mozilla.com:3128/ wget -O/root/puppetize.sh "$hgrepo/raw-file/default/modules/puppet/files/puppetize.sh" || fail
+# try to use the proxy, falling back to a direct fetch (e.g., if the DC doesn't have a proxy)
+http_proxy=http://proxy.dmz.<%= @datacenter %>.mozilla.com:3128/ wget -O/root/puppetize.sh "$hgrepo/raw-file/default/modules/puppet/files/puppetize.sh" \
+    || wget -O/root/puppetize.sh "$hgrepo/raw-file/default/modules/puppet/files/puppetize.sh" \
+    || fail
 chmod +x /root/puppetize.sh

so, please try again in ~1h?

Comment 13

[Vinh Hua [:vinh]]

foopy112 finished reimaging. I did not see the proxy error this time around.

Comment 14

[Vinh Hua [:vinh]]

< vinh> Callek|Buildduty: I've just reimaged foopy112 again. Can you verify?
12:05 < Callek|Buildduty> vinh: looks up for me now, thanks