895914 - Disable the "shutdown event tracker" on non-gpo-managed Win64 build machines

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

As reported in bug 893888, our win64 build slaves are displaying the "shutdown event tracker" dialog after an unclean shutdown.  This prevents cltbld from logging in and the buildbot slave process from starting.

RelEng may want to update the current win64 build slaves to not show this dialog.  Reminder that newly reimaged machines will not get that setting automatically.

Note that we should weigh the priority of this manual work against us having gpo-managed win64 build slaves in a matter of weeks.  If we choose not to do this work, PDU reboots done to win64 build slaves (manually or by kittenherder) will require a follow-up step to close the shutdown event tracker dialog.

Comment 1

[Armen [:armenzg]]

coop tried this but I don't think it is working:

[reply] [-] Description Armen Zambrano [:armenzg] (Use needinfo flag) (Release Enginerring) (EDT/UTC-4) 1 month ago

'''
The shutdown event tracker was disabled to avoid manual intervention after power outages and other improper shutdowns. This was done with this OPSI package, which sets the following registry key as follows:
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability\ShutdownReasonOn\REG_DWORD:0x0 
'''

OR

http://mxr.mozilla.org/build/source/mozharness/external_tools/count_and_reboot.py#25
os.system("shutdown -f -r -t 0")

jhopkins, where do we reboot Win64 machines inside of briarpatch?

I found these:
https://github.com/mozilla/briar-patch/search?q=shutdown%28%29&ref=cmdform

Armen Zambrano [:armenzg] (Use needinfo flag) (Release Enginerring) (EDT/UTC-4) 1 month ago
Flags: needinfo?([e-mail])
[reply] [-] Comment 1 John Hopkins (:jhopkins) 1 month ago

A clean shutdown won't trigger the dialog.  It's when the computer crashes or the power is removed that we see this dialog.
Kittenherder (or a human) can PDU-reboot a Windows build slave which results in an "unclean" shutdown.  The kittenherder code is here: https://github.com/mozilla/briar-patch/blob/master/releng/remote.py#L357 (in rebootPDU())

Some instructions on how to disable the shutdown event tracker:

http://blogs.technet.com/b/chenley/archive/2011/03/05/how-to-disable-the-shutdown-tracker-in-windows-server-2008-r2.aspx
http://www.askvg.com/how-to-disable-remove-annoying-shutdown-event-tracker-in-windows-server-2003-2008/

We need to disable it on the existing win64 build slaves and the new win64 build slaves.

Flags: needinfo?([e-mail])
John Hopkins (:jhopkins) 1 month ago
Component: Release Engineering: Platform Support → RelOps; Blocks: 781277; Assignee: [e-mail] → [e-mail]; Product: mozilla.org → Infrastructure & Operations; Summary: Please disable the "shutdown event tracker" on the Win64 build machines and/or use /c to give a comment to every reboot we do → Please disable the "shutdown event tracker" on the gpo-managed Win64 build machines; QA Contact: [e-mail] → [e-mail]
[reply] [-] Comment 2 Amy Rich [:arich] [:arr] 1 month ago

Mark can work on automating this for the new version of the builders that will be joined to the domain.

Assignee: [e-mail] → [e-mail]
[reply] [-] Comment 3 Mark Cornmesser [:markco] 1 month ago

This is now handled by the 2008_shutdown_tracker GPO, and the GPO has been linked to the builder OU. 

 Computer Configuration-> Policies-> Administrative Templates-> System-> Policy->
Display Shutdown Event Tracker Disabled.

Status: NEW → RESOLVED
Resolution: --- → FIXED
Last Resolved: 2013-07-22 15:04:44
[reply] [-] Comment 4 Chris Cooper [:coop] 1 week ago

I'm using the following sequence of commands to deploy (via ssh) and verify the fix on the win64 builders:

reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability /v ShutdownReasonOn /t REG_DWORD /d 0
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability /v ShutdownReasonUI /t REG_DWORD /d 0
reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability

Comment 2

[Armen [:armenzg]]

What kind of copy/paste was this? This is what I tried to paste:

coop tried this but I don't think it is working:
reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability /v ShutdownReasonOn /t REG_DWORD /d 0
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability /v ShutdownReasonUI /t REG_DWORD /d 0
reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability

Comment 3

[Armen [:armenzg]]

I've found that this seems to work.
Changing the pref through gpedit.msc would change "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability".
gpedit.msc will still show the setting as "not configured" rather than "disabled".

reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability"
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability" /v ShutdownReasonOn /t REG_DWORD /d 0
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability" /v ShutdownReasonUI /t REG_DWORD /d 0
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Reliability"

Comment 4

[Armen [:armenzg]]

I deployed the change to all of our win64 machines (up to 157).

w64-ix-slave39 is the exception.

Comment 5

[Armen [:armenzg]]

The machine came back up and I deployed the change.