596735 - After uninstalling OPSI mw64-ix-slave01 does not boot

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

Attachment: blue screen shot

I was trying to re-install OPSI on the machine and after I rebooted I can't connect anymore to the machine.

The only thing I am capable of seeing is a blue screen with the operating system's logo at the bottom.

Anyone that can help me debug this?

Meanwhile I will get back to mw64-ix-ref and install VNC and ssh in case I hit the same problem.

Comment 1

[Justin Dow [:jabba]]

Dropping to normal sev, so it doesn't page oncall. I informed Armen of the IPMI interface, so that releng can look at these issues directly as well.

Comment 2

[Armen [:armenzg]]

Thanks Justin (both) I will make sure to use the IPMI.

I will poke at this myself as now I have access to the IPMI service.

Comment 3

[Armen [:armenzg]]

Justin wrote in:
https://bugzilla.mozilla.org/show_bug.cgi?id=595328#c5
> Tried to poke at slave01 to see what was happening.  The machine would go
> through its startup sequence, then get to the stage where it is supposed to
> show a login window, but it never shows, just hangs.  After about a minute, a
> window titled "OPSI login blocker" Service Request failed. 
> 
> I booted into safe mode and tried to see if I could disable any startup
> services (buildbot) but it did not work.

Comment 4

[Corey Shields [:cshields]]

Try dropping the opsiclientd service and see if it boots?

I still don't think your install is working properly.  We might want to compare version numbers again (since you were looking at the wrong opsi server last time)

Comment 5

[Corey Shields [:cshields]]

I said that but misread the bug as "install" not "uninstall"..  Seems that there is still something leftover, like the preloginloader (that interrupts the boot-to-login screen chain).  Still, check the service list from safe mode and see if opsiclientd is still listed somehow.

Comment 6

[Armen [:armenzg]]

Attachment: Opsi login blocker - service request failed screen shot

OK I am getting closer.
I am using iKVM Java Viewer to use the IPMI interface and this is the screen shot I get to see:
 "Opsi login blocker"

When I reboot I still have not managed to enter safe mode as my up and down keystrokes are not being detected.

I wonder if "opsiclientd" service is really disabled.

Comment 7

[Armen [:armenzg]]

Attachment: delpreloginloader.ins

For reference, this is what gets executed for deinstallation of OPSI.

Comment 8

[Armen [:armenzg]]

I wrote on their forums:
https://forum.opsi.org/viewtopic.php?f=8&t=1815&sid=1c5b10b5499df2fb005b9d9e7048c76f&p=9037#p9037

They recommended me to upgrade to opsi 3.4 and remove a couple of keys:
> [Registry_vista_del_loginblocker]
> deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Providers\{d2028e19-82fe-44c6-ad64-51497c97a02a}]
> deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Provider Filters\{d2028e19-82fe-44c6-ad64-51497c97a02a    }]

Let's see if I will be able to reach safe mode through my freshly installed Windows VM and using IPMI.

Comment 9

[Corey Shields [:cshields]]

Yeah, I have to agree with the updating..  Not having any kind of change log or bug tracker it is hard to tell what changed between 3.4.0.0 and 3.4.0.2 but that could be the key (and obviously up to 3.4.0.14)

Is that a possibility for staging-opsi?

Comment 10

[Armen [:armenzg]]

(In reply to comment #9)
> 
> Is that a possibility for staging-opsi?

bhearsum would you be fine with this if ended up having to do this? we would probably follow up with production as well.

Comment 11

[Armen [:armenzg]]

In https://forum.uib.de/viewtopic.php?f=8&t=939#p4730 it is mentioned that editing c:\program files\opsi.org\preloginloader\opsiclientd\opsiclientd.conf and add under the section called [event_gui_startup] the option block_login of this section to false:
> block_login = false

The problem is that I am not being able to reach the machine with the web IPMI access.

I have to try it to make sure that is the problem.

Comment 12

[Armen [:armenzg]]

(In reply to comment #8)
> I wrote on their forums:
> https://forum.opsi.org/viewtopic.php?f=8&t=1815&sid=1c5b10b5499df2fb005b9d9e7048c76f&p=9037#p9037
> 
> They recommended me to upgrade to opsi 3.4 and remove a couple of keys:
> > [Registry_vista_del_loginblocker]
> > deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Providers\{d2028e19-82fe-44c6-ad64-51497c97a02a}]
> > deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Provider Filters\{d2028e19-82fe-44c6-ad64-51497c97a02a    }]
> 
After doing the machine is now able to login (even though I am now having issues with RDP).

> Let's see if I will be able to reach safe mode through my freshly installed
> Windows VM and using IPMI.
>
I was able to connect through Java iKVM Viewer once I downgrade to Java runtime 6.14 from 6.21.

Besides slave01, this morning I found win64-ix-ref in the same condition and it is *not* from uninstalling OPSI but from having forgotten to resume last night opsiclientd service (which is mentioned here https://forum.uib.de/viewtopic.php?f=8&t=939#p4694). 

As of now the remaining issues:
* win64-ix-slave01 in the same condition as mw64-ix-slave01 was during these 2-3 days
* I have to use Java iKVM viewer to have access to the machine but I can't access the web interface. Not sure if IPMI was enabled on the machine or not. (TODO file a bug for IT to fix)
* Fix mw64-ix-slave01 RDP issue (TODO try once more and then file an IT bug if can't figure it out)

Once the machines are working:
* Install opsi from the staging-opsi mount (which seems to mount after trying few times and/or waiting long enough)
* Write a dummy opsi package to test if packages get installed properly or not

Comment 13

[Armen [:armenzg]]

I just kept this bug open to force me write a blog post trying to summarize all that we went through:
http://armenzg.blogspot.com/2010/10/opsi-nightmares-on-windows-2008-r2-64.html

We fixed the machine so there is nothing left to be done in here.