1162607 - switch3.r602-2.ops.releng.scl3 down

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Van Le [:van]]

bad hard drive on the switch. will need to replace and configure new switch.

Comment 1

[Van Le [:van]]

error: mount of main filesystems (mount -a) failed

An emergency shell can be started to repair any damage and
perform the required operation before exiting and allowing
this script to continue operation.

continue, shell, abort, retry, or reboot ? continue
Verified jbase-ex-11.4R5.5 signed by PackageProduction_11_4_0
/packages/fips-mode-arm-11.4R5.5 already mounted
/packages already mounted
/packages already mounted
/packages already mounted
/packages already mounted
/packages already mounted
/packages already mounted
/packages already mounted
rm: /var/etc/filters/*: Not a directory
rm: /var/run/*: Not a directory
mkdir: /var/run: Bad file descriptor
Creating configuration.....
rm: /var/db/schema.db: Not a directory
rm: /var/db/juniper.db: Not a directory
rm: /var/db/juniper.data: Not a directory
rm: /var/db/juniper.save: Not a directory
rm: /var/db/*[~+-]: Not a directory
rm: /var/db/*.old: Not a directory
rm: /var/db/mounttab: Not a directory
mkdir: /var/run: Bad file descriptor
Loading configuration ...
mgd: error: could not open database: /var/run/db/schema.db: Not a directory
mgd: error: Database open failed for file '/var/run/db/schema.db': Not a directory
mgd: error: could not open database schema: /var/run/db/schema.db
mgd: error: could not open database schema
Warning: Commit failed, activating partial configuration.
Warning: Edit the router configuration to fix these errors.
cp: /var/run/db/juniper.data: Not a directory
cd: can't cd to /var/etc
mv: rename *+ to var/*+: No such file or directory
pid 434 (mv), uid 0 inumber 2 on /var: out of inodes

/var: create/symlink failed, no inodes free
mv: var/COPYRIGHT: No space left on device
mv: var/altconfig: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/altroot: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: rename bin to var/bin: Not a directory
mv: var/boot: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: cannot rename a mount point
mv: var/data: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: cannot rename a mount point
mv: rename etc to var/etc: Not a directory
mv: var/jail: Bad file descriptor
mv: /bin/cp: terminated with 1 (non-zero) status
mv: symlink: /packages/jkernel-ex-2200: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/libexec: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/mfs: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/mnt: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/modules: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/opt: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/packages: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/proc: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/root: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/sbin: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: var/tftpboot: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
mv: rename tmp to var/tmp: Not a directory
mv: var/usr: No space left on device
mv: /bin/cp: terminated with 1 (non-zero) status
machdep.bootsuccess: 0 -> 0

Performing initialization of management services ...
mgd: error: could not open database: /var/run/db/schema.db: Not a directory
mgd: error: Database open failed for file '/var/run/db/schema.db': Not a directory
mgd: error: could not open database schema: /var/run/db/schema.db
mgd: error: could not open database schema
mgd: error: database schema is out of date, rebuilding it
mgd: error: could not open database: /var/run/db/juniper.data: Not a directory
mgd: error: Database open failed for file '/var/run/db/juniper.data': Not a directory
mgd: error: Cannot read configuration: Could not open configuration database
mgd: error: daemon MGD detects existing daemon using lock file '/var/run/mgd.pid'

error: initialization of management services (mgd -S) failed

An emergency shell can be started to repair any damage and
perform the required operation before exiting and allowing
this script to continue operation.

continue, shell, abort, retry, or reboot ?

Comment 2

[Van Le [:van]]

we replaced switch3.r602-2 with retired switch3.r602-7.

vans-MacBook-Pro:~ vle$ fping switch3.r602-2.ops.releng.scl3.mozilla.net
switch3.r602-2.ops.releng.scl3.mozilla.net is alive

panda relay that resides on this switch is back online.

vans-MacBook-Pro:~ vle$ fping !$
fping panda-relay-007.p1.releng.scl3.mozilla.com
panda-relay-007.p1.releng.scl3.mozilla.com is alive

Comment 3

[Van Le [:van]]

bad drive error msg:

01:56 <@van> (da0:umass-sim0:0:0:0): READ(10). CDB: 28 0 0 1b 51 40 0 0 4 0
01:56 <@van> (da0:umass-sim0:0:0:0): CAM Status: SCSI Status Error
01:56 <@van> (da0:umass-sim0:0:0:0): SCSI Status: Check Condition
01:56 <@van> (da0:umass-sim0:0:0:0): MEDIUM ERROR asc:11,0
01:56 <@van> (da0:umass-sim0:0:0:0): Unrecovered read error
01:56 <@van> (da0:umass-sim0:0:0:0): Retrying Command (per Sense Data)
01:56 <@van> cg 0: bad magic number
01:56 <@van> *** The installer exited with status 38 ***
01:56 <@van> *** The installation is unsuccessful!!! ***