1397129 - Lock file not clearing after first run of OCC on Moonshot VMs

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Mark Cornmesser [:markco]]

First run of OCC is trigger by a schedule task on start up on the boot following the MDT deploy. It is starting and making C:\DSC\in-progree.lock. When the dsc manifest is applied generic worker is installed. The installation, I believe, reboots. This reboots occurs before the first OCC run is completed. On the next boot the task user logs in the rundsc schedule task kicks off but exits because the lock file is in place, so the ttask-claim-state.valid file is never created. This then causes the generic worker start bat to continuous check for the task-claim-state.valid. This causes the Vm to sit, not pick up tasks, and not reboot.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

we'll need to see what reboot is being applied and what process is triggering it.

whenever rundsc.ps1 triggers a reboot, it is also supposed to clear the lock file (eg: https://github.com/mozilla-releng/OpenCloudConfig/blob/2d4ef1bf/userdata/rundsc.ps1#L317-L318)

it should also leave a message in the event log citing a reason for the reboot (eg: https://papertrailapp.com/groups/2488493/events?q=program%3Auser32)

i'd check the event log (or papertrail, if that's working on hw) and try to identify what reboot reason is given and then use that to identify which code path is triggering the reboot so that we can then work out why the lock file gets left behind.

Comment 2

[Mark Cornmesser [:markco]]

Lock file timestamp 9/6/20147 12:09 am

12:09:35 System event
The process C:\windows\system32\shutdown.exe (T-W864-IX-000) has initiated the restart of computer T-W864-IX-000 on behalf of user NT AUTHORITY\SYSTEM for the following reason: No title for this reason could be found

(No prior Application events for 29 seconds)
12:09:33 Application event Source igfxCUI
The description for Event ID 0 from source igfxCUIService2.0.0.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer
12:09:33 Application event Source NSSM
Service Generic Worker received START control, which will be handled.

Genric Worker log
2017/09/06 00:09:35 Exiting worker so it can reboot...

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

just an observation that generic worker shouldn't be running if occ is running. the log message suggests that the reboot is triggered by g-w. if the occ lock file still exists, it means that something allowed g-w to start before occ had finished. that will need to be fixed first.

Comment 4

[Mark Cornmesser [:markco]]

I was not able to replicate this by running the commands manual. However, I hit a case where the rundsc script spun on waiting for WinRM service to stop. 

For now I have added to start up schedule tasks to the install. One to remove the lock the file and one to stop the WinRM service. It seems to continuous pick up tasks once these are in place. I am performing one more test on a fresh install to see if it will continuously pick up tasks with out any human interaction. 

Once I have additional hardware to work with I will dive back into this.

Comment 5

[Mark Cornmesser [:markco]]

I have moved away from the schedule task and add the following to the generic worker bat in my testing repo:

C:\generic-worker\generic-worker.exe run --config C:\generic-worker\gen_worker.config >> C:\generic-worker\generic-worker.log 2>&1
+if %errorlevel% equ 67 goto RmLock

<nul (set/p z=) >C:\dsc\task-claim-state.valid
shutdown /r /t 0 /f /c "Rebooting as generic worker ran successfully"

+:RmLock
+net stop winrm
+del C:DSC\in-progress.lock
+shutdown /r /t 0 /f /c "Rebooting as generic worker exit with code 67"