807763 - A lot of Windows 64-bit machines have lost their AutoAdminLogon value from the registry

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Armen [:armenzg]]

We're not sure if this is related to the recent password update but a lot of them were out of the pool.

METHOD 1
To recuperate them you can run this from cmd (not SSH even with runas /user:Administrator).
runas /user:Administrator "reg add \"HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\" /v DefaultPassword /d password /t REG_SZ /f"

Anyone know what could have changed recently?
Could the machines had had a recent re-imaging?

From looking at bugzilla I can see that philor filed 11 of those slaves on Oct. 25th (a day after the password reset on bug 802911 - I don't think it is related).

I have tried to reproduce the scenario where I would go and update the password but the registry would be intact.

coop fix the following slaves (used method 2):
w64-ix-slave45
w64-ix-slave46
w64-ix-slave49
w64-ix-slave50
w64-ix-slave51
w64-ix-slave52
w64-ix-slave54
w64-ix-slave56
w64-ix-slave57
w64-ix-slave60
w64-ix-slave64 2012-10-17 07:46:20
w64-ix-slave65 2012-10-16 22:15:31
w64-ix-slave66
w64-ix-slave67
w64-ix-slave68
w64-ix-slave69 2012-10-17 12:04:00
w64-ix-slave71
w64-ix-slave72 2012-10-19 16:57:00
w64-ix-slave73
w64-ix-slave74
w64-ix-slave75
w64-ix-slave76
w64-ix-slave77
w64-ix-slave81 2012-10-18 03:06:19
w64-ix-slave82 2012-10-21 05:39:00
w64-ix-slave83
w64-ix-slave84

The dates/times I'm putting are from last job per slave which was generated at 2012-11-01 12:30:02. I'm not sure if the slaves that coop touched are back online or not.

I debugged and used method 1:
w64-ix-slave79 	2012-10-15 09:58:31

METHOD 2:
There is also this other way:
* login as Administrator via VNC or RDP, whichever will get you access
* from the Start Menu, select "Run..." and type "control userpasswords2"
* select the cltbld user
* uncheck the box for "Users must enter a user name and password to
use this computer." If it is already unchecked, check it, then uncheck
it again. This will force the settings to refresh.
* click "Apply"
* enter the current password (twice) for cltbld in the "Automatically
Log On" popup dialog
* click "OK"
* restart the machine. Use your name as the reason for restart.

Comment 1

[Armen [:armenzg]]

I needed to use as well this with w64-ix-slave78:
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogon /d 1 /t REG_SZ /f

I'm happy to close this and let it go for now.

We should keep an eye to see if this happens again.

We might want to add a registry monitoring tool to catch issues like this:
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
Or enable auditing of the registry with GPO (once we have it):
http://searchwinit.techtarget.com/tip/Auditing-changes-to-the-registry

Comment 2

[Armen [:armenzg]]

We might also write a pre-flight script to check on problems like this bug 712206.
Or see if nagios could check for certain packages.
Or even puppet for Windows!

A world of infinite solutions :D

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Puppet for Windows will put the setting in place.  A pre-flight script wouldn't run if the key wasn't in place, so that won't help :)

Comment 4

[Armen [:armenzg]]

13 days ago we did a pass through the slaves but it seems that some of those slaves stayed in that state and some new ones have joined the list.
w64-ix-slave61 21 days, 10:53:39   broken  bm14-try1   2012-10-24 18:46:32 1:34:51     Yes
w64-ix-slave47 18 days, 16:28:46   broken  bm14-try1   2012-10-27 13:12:32 1:33:44     Yes
w64-ix-slave48 18 days, 7:35:21    broken  bm14-try1   2012-10-27 21:29:32 2:10:09     Yes
w64-ix-slave62 13 days, 18:20:33   broken  bm14-try1   2012-11-01 11:28:32 1:25:57     Yes
w64-ix-slave53 12 days, 16:39:21   broken  bm14-try1   2012-11-02 12:35:27 2:00:14     Yes
w64-ix-slave55 12 days, 14:33:02   broken  bm14-try1   2012-11-02 15:07:32 1:34:28     Yes
w64-ix-slave29 11 days, 21:23:55   broken  bm14-try1   2012-11-03 08:59:32 0:51:35     Yes
w64-ix-slave58 11 days, 18:20:40   broken  bm14-try1   2012-11-03 11:25:32 1:28:50     Yes
w64-ix-slave44 9 days, 22:20:10    broken  bm14-try1   2012-11-05 08:14:32 0:40:20     Yes
w64-ix-slave63 8 days, 12:21:26    broken  bm14-try1   2012-11-06 17:55:32 0:58:04     Yes
w64-ix-slave59 8 days, 10:27:03    broken  bm14-try1   2012-11-06 19:17:32 1:30:27     Yes

Comment 5

[Armen [:armenzg]]

Here's how to see that the reg entries are missing (RDP or VNC only):
reg query "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultPassword
reg query "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogon
Through SSH you can determine if the value exists but it does not show the actual value even if you use runas.

To fix the machines RDP as Administrator and run these:
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultPassword /d password /t REG_SZ /f"
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogon /d 1 /t REG_SZ /f
shutdown -f -r -t 0

I've gone through all that list of slaves and added the missing registry values.

What we have not yet determined is why would those slave have their keys modified in between jobs?

Comment 6

[Armen [:armenzg]]

It seems that on the first time we *only* looked at the build slaves since all of the slaves in comment 4 were try slaves.
This seems an on-going issue and it happens out of nowhere (no re-imaging of the slave, no package deployment).

Comment 7

[Armen [:armenzg]]

To my surprise *all* slaves *don't* have "DefaultPassword" and they manage to reboot and login.

I'm out of ideas on what to check.

In our instructions it says that we use this reg file:
https://bug645024.bugzilla.mozilla.org/attachment.cgi?id=544857
which should both DefaultPassword and AutoAdminLogon.
We also remove AutoLogonCount.

relops, what does our sequence task say we actually do?

#####################

Not sure if this is what is happening to us (I don't see a solution):
"Recently, I was made aware that when the network workgroup is changed, the
auto logon is disabled. It also seems to happen intermittently when the
network cable is plugged in during bootup. I checked the registry, and under
HKLM\Software\Microsoft\WindowsNT\CurrentVersion\Winlogon
DefaultPassword key is deleted, and AutoAdminLogon key is set to zero. After
I add DefaultPassword key back and change AutoAdminLogon to 1, Auto logon
works normally again. I am almost certain that this just starting happening
recently, does this sound like a setup problem, or possibly a missing
component issue. Any help would be appreciated.
--
Ryan"

http://www.pcreview.co.uk/forums/auto-logon-disabled-t3050861.html

Comment 8

[Armen [:armenzg]]

Changing the summary.
AutoAdminLogon gets deleted.
DefaultPassword seems that does not exist even on working slaves.
This is an ongoing problem.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Does it still look like it's being lost on previously-working slaves, or is the concern that MDT isn't setting it at install time?

Comment 10

[Armen [:armenzg]]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> Does it still look like it's being lost on previously-working slaves, or is
> the concern that MDT isn't setting it at install time?

It is happening on previously-working slaves.

On another note, I'm surprised that working slaves do not have DefaultPassword which should have been set at install time. 

C:\Users\Administrator>reg query "HKLM\Software\Microsoft\Windows NT\CurrentVers
ion\Winlogon" /v DefaultPassword
ERROR: The system was unable to find the specified registry key or value.

C:\Users\Administrator>reg query "HKLM\Software\Microsoft\Windows NT\CurrentVers
ion\Winlogon" /v AutoAdminLogon
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon
    AutoAdminLogon    REG_SZ    1

Comment 11

[Armen [:armenzg]]

We're concerned that some Windows 7 machines are seeing this as well:
bug 795470 and bug 733723.

Comment 12

[Armen [:armenzg]]

We noticed that we could not login to the Administrator account on those two win7 slaves.

Comment 13

[Armen [:armenzg]]

The win7 slaves could be red herrings.
They were not capable of logging in but they had the registry values needed to auto-login.