540688 - n810 maintenance 20090119

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

The requisite number of n810s have fallen over and/or gone read-only since the last maintenance.

Also, p-m-m fell over on Jan 14 and I want to make sure everything's running smoothly.

So:

* n810 audit
* n810 reimaging
* rebalance staging-mobile-master and production-mobile-master n810 pools

Comment 1

[Aki Sasaki (not active)]

Kicked off talos runs of latest 1.9.2/trunk builds.
Ran a check to see uptime/buildbot master/existence of standalone.txt per n810.

Reimage list:

maemo-n810-06 -- fs corruption
maemo-n810-09 -- down
maemo-n810-15 -- fs corruption
maemo-n810-16 -- down
maemo-n810-17 -- down
maemo-n810-18 -- down
maemo-n810-21 -- fs corruption
maemo-n810-26 -- down
maemo-n810-27 -- down
maemo-n810-42 -- fs corruption
maemo-n810-47 -- down
maemo-n810-58 -- down
maemo-n810-62 -- down
maemo-n810-63 -- fs corruption
maemo-n810-67 -- fs corruption
maemo-n810-71 -- fs corruption
maemo-n810-72 -- fs corruption
maemo-n810-74 -- down
maemo-n810-77 -- down
maemo-n810-78 -- down
maemo-n810-80 -- fs corruption

I've brought up n810s that were down before, only to find they were down because of fs corruption + idleness.  Tends to save time to just reimage 'em.

Kicked over quite a number of n810s from baking on staging over to production.

Note to jhford: this is all work I'm doing over ssh; if sshd isn't running on the n900s I'll probably assign this sort of bug to you. ;-)

Comment 2

[Aki Sasaki (not active)]

Commands for later:

  all_maemo_ssh.sh "uptime; grep master /builds/buildbot/host.py; ls -l /builds/standalone.txt && cat /builds/standalone.txt" 2>&1 | tee foo

(manually edit this list; move staging-mobile-master slaves over to production-mobile-master)

(reimage n810s)

Once "foo" is only the list of reimaged n810s,

  for i in `cat foo` ; do   echo $i;   scp *.deb root@$i:;   ssh root@$i dpkg -i \*.deb;   ssh root@$i "echo $i > /etc/hostname";   ssh root@$i "reboot; exit"; done

in a directory with foo, xrestop_0.4-1osso1_armel.deb, libxres1_1.0.3-1osso2_armel.deb

That can definitely be improved, but hasn't been a priority.
Still on the "reimage n810s" step.

Comment 3

[Aki Sasaki (not active)]

58 isn't coming back up but I'm done with the rest. I'll leave that one for next time.

Comment 4

[Aki Sasaki (not active)]

Noticed maemo-n810-20 had a hostname of "maemo-n810-23" which was borking buildbot connectivity for both 20 and 23; fixed.  Still investigating mail issues.  I'll reopen.