413342 - VI server is down again

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

I'm getting the "No connection could be made because the target machine actively refused it" error.

I can still connect to the machine with RDP still.

Comment 1

[matthew zeier [:mrz]]

grr.  investigating again.

Comment 2

[matthew zeier [:mrz]]

Log for VMware VirtualCenter, pid=4460, version=2.0.2, build=build-62327, option=Release, section=2
[2008-01-21 09:59:37.828 'App' 5444 info] Current working directory: C:\WINDOWS\system32
[2008-01-21 09:59:37.828 'App' 5444 info] Initializing SSL context
[2008-01-21 09:59:38.484 'App' 5444 info] Vmacore::InitSSL: doVersionCheck = true, handshakeTimeoutUs = 120000000
[2008-01-21 09:59:38.484 'BaseLibs' 5444 info] NFC connection accept timeout: 180000 milliseconds

[2008-01-21 09:59:38.484 'BaseLibs' 5444 info] NFC request timeout: 180000 milliseconds

[2008-01-21 09:59:38.484 'BaseLibs' 5444 info] NFC read timeout: 60000 milliseconds

[2008-01-21 09:59:38.484 'BaseLibs' 5444 info] NFC write timeout: 600000 milliseconds

[2008-01-21 09:59:38.484 'App' 5444 info] Starting VMware VirtualCenter 2.0.2 build-62327
[2008-01-21 09:59:38.484 'App' 5444 info] Log directory: C:\WINDOWS\TEMP\vpx.
[2008-01-21 09:59:38.484 'App' 5444 info] Account name: SYSTEM
[2008-01-21 09:59:38.484 'App' 5444 info] [VpxOsLayer] Enabled low-frag process heap.

[2008-01-21 09:59:38.484 'App' 5444 info] [VpxOsLayer] Enabled low-frag crt heap.

[2008-01-21 09:59:38.484 'App' 5444 info] [VpxLRO] 34 max LROs
[2008-01-21 09:59:38.484 'App' 5444 info] [VpxLRO] 6 reserved internal LROs
[2008-01-21 09:59:38.484 'App' 5444 info] [VpxLRO] 6 reserved blocker LROs
[2008-01-21 09:59:38.484 'App' 5444 info] [VpxLRO] 6 reserved short LROs
[2008-01-21 09:59:38.500 'App' 5444 info] [VpxLRO] 4 reserved long LROs
[2008-01-21 09:59:38.500 'App' 5444 info] [VpxLRO] 600-second task lifetime
[2008-01-21 09:59:44.312 'Locale' 5444 info] Locale subsystem initialized from C:\Program Files\VMware\VMware VirtualCenter 2.0\locale/ with default locale en.
[2008-01-21 09:59:44.656 'App' 5444 error] VM has invalid name 
[2008-01-21 09:59:44.656 'App' 5444 error] Invalid VM. Removing /vpx/vm/#1233/
[2008-01-21 09:59:44.656 'App' 5444 error] [VpxdInvt] Delete failed for /vpx/vm/#1233/
[2008-01-21 09:59:44.671 'App' 5444 error] [VpxdMain] Failed to initialize: vim.fault.DatabaseError
[2008-01-21 09:59:44.671 'App' 5444 error] Failed to intialize VMware VirtualCenter. Shutting down...

Comment 3

[matthew zeier [:mrz]]

For those playing along,

ello Matthew, 

Based on these 3 lines:
[2008-01-21 09:59:44.656 'App' 5444 error] VM has invalid name 
[2008-01-21 09:59:44.656 'App' 5444 error] Invalid VM. Removing
/vpx/vm/#1233/
[2008-01-21 09:59:44.656 'App' 5444 error] [VpxdInvt] Delete failed for
/vpx/vm/#1233/

And an internal KB article, which says this.
VC database was trying to delete the vm 1775 which status was invalid.

Logged into the VC center database. Located the table VPX_VM. Opened the
table and located the ID 1775 and maually deleted the whole row/record for
the ID 1775.
After deletion, restarted the VC service. VC service started running
properly.

Comment 4

[matthew zeier [:mrz]]

Fails with some other error - 

[2008-01-21 13:22:22.796 'Locale' 3880 info] Locale subsystem initialized from C:\Program Files\VMware\VMware VirtualCenter 2.0\locale/ with default locale en.
[2008-01-21 13:22:23.031 'BaseLibs' 3880 warning] VmdbCtxGet: Failed to get /vpx/vm/#1233/_private/managedObjImpl/ (Null cross reference)

[2008-01-21 13:22:23.031 'App' 3880 error] [VpxdMain] Failed to initialize: vmodl.fault.ManagedObjectNotFound
[2008-01-21 13:22:23.031 'App' 3880 error] Failed to intialize VMware VirtualCenter. Shutting down...


Note - the VM it was complaining about (1233) is sm-bes01.

Comment 5

[matthew zeier [:mrz]]

Fixed.  Holding off on adding dev-vmware01 until I understand things better.  

Took a backup of the database so I shouldn't have to deal with this as much next time (losing the datacenter tree is just extra effort).