Closed Bug 1485628 Opened 2 years ago Closed 2 years ago

Intermittent [taskcluster:error] exit status 1073807364

Categories

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task, P5)

Product:
Infrastructure & Operations
Component:
RelOps: OpenCloudConfig
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: grenade)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

GitHub Pull Request
58 bytes, text/x-github-pull-request
markco
: review+
Details | Review 
Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=195437795&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/CmX9SKQCSYK-lKJwufHCeA/runs/0/artifacts/public/logs/live_backing.log

[taskcluster 2018-08-23T04:07:42.746Z] Worker Type (gecko-3-b-win2012) settings:
[taskcluster 2018-08-23T04:07:42.746Z]   {
[taskcluster 2018-08-23T04:07:42.746Z]     "aws": {
[taskcluster 2018-08-23T04:07:42.746Z]       "ami-id": "ami-ed88668e",
[taskcluster 2018-08-23T04:07:42.746Z]       "availability-zone": "us-west-1b",
[taskcluster 2018-08-23T04:07:42.746Z]       "instance-id": "i-0339ee416a0c486c2",
[taskcluster 2018-08-23T04:07:42.746Z]       "instance-type": "c4.4xlarge",
[taskcluster 2018-08-23T04:07:42.746Z]       "local-ipv4": "10.143.16.53",
[taskcluster 2018-08-23T04:07:42.746Z]       "public-hostname": "ec2-54-183-247-60.us-west-1.compute.amazonaws.com",
[taskcluster 2018-08-23T04:07:42.746Z]       "public-ipv4": "54.183.247.60"
[taskcluster 2018-08-23T04:07:42.746Z]     },
[taskcluster 2018-08-23T04:07:42.746Z]     "config": {
[taskcluster 2018-08-23T04:07:42.746Z]       "deploymentId": "0ee0ac5d029f",
[taskcluster 2018-08-23T04:07:42.746Z]       "runTasksAsCurrentUser": false
[taskcluster 2018-08-23T04:07:42.746Z]     },
[taskcluster 2018-08-23T04:07:42.746Z]     "generic-worker": {
[taskcluster 2018-08-23T04:07:42.746Z]       "go-arch": "amd64",
[taskcluster 2018-08-23T04:07:42.746Z]       "go-os": "windows",
[taskcluster 2018-08-23T04:07:42.746Z]       "go-version": "go1.10.3",
[taskcluster 2018-08-23T04:07:42.746Z]       "release": "https://github.com/taskcluster/generic-worker/releases/tag/v10.11.2",
[taskcluster 2018-08-23T04:07:42.746Z]       "revision": "4b47d5d9f45eabc283947da0a6b1c02eddd0a7fe",
[taskcluster 2018-08-23T04:07:42.746Z]       "source": "https://github.com/taskcluster/generic-worker/commits/4b47d5d9f45eabc283947da0a6b1c02eddd0a7fe",
[taskcluster 2018-08-23T04:07:42.746Z]       "version": "10.11.2"
[taskcluster 2018-08-23T04:07:42.746Z]     },
[taskcluster 2018-08-23T04:07:42.746Z]     "machine-setup": {
[taskcluster 2018-08-23T04:07:42.746Z]       "ami-created": "2018-08-09 11:06:49.946Z",
[taskcluster 2018-08-23T04:07:42.746Z]       "manifest": "https://github.com/mozilla-releng/OpenCloudConfig/blob/0ee0ac5d029f6f409960d3bf372c9e2aa103cc97/userdata/Manifest/gecko-3-b-win2012.json"
[taskcluster 2018-08-23T04:07:42.746Z]     }
[taskcluster 2018-08-23T04:07:42.746Z]   }
[taskcluster 2018-08-23T04:07:42.746Z] Task ID: CmX9SKQCSYK-lKJwufHCeA
[taskcluster 2018-08-23T04:07:42.746Z] === Task Starting ===
[taskcluster 2018-08-23T04:07:43.676Z] Uploading redirect artifact public/logs/live.log to URL https://g237opaaaaawkzy4xt2fhl2x64fwahxsnhqfvo2gsuy3y6et.taskcluster-worker.net:60023/log/CO0fqGUvRum-PkGzo1HGeA with mime type "text/plain; charset=utf-8" and expiry 2018-08-23T14:07:41.300Z
[taskcluster 2018-08-23T04:07:43.932Z] Executing command 0: "c:\Program Files\Mercurial\hg.exe" robustcheckout --sharebase y:\hg-shared --purge --upstream https://hg.mozilla.org/mozilla-unified --revision 96c84ecff48708165d27b56260f11d60c04166d2 https://hg.mozilla.org/integration/mozilla-inbound .\src

Z:\task_1534997022>"c:\Program Files\Mercurial\hg.exe" robustcheckout --sharebase y:\hg-shared --purge --upstream https://hg.mozilla.org/mozilla-unified --revision 96c84ecff48708165d27b56260f11d60c04166d2 https://hg.mozilla.org/integration/mozilla-inbound .\src 
(using Mercurial 4.5.3)

clone [                                         ]    8799774/2664470783 10m04s
clone [                                          ]   14474986/2664470783 9m40s
clone [                                          ]   19532947/2664470783 9m47s
clone [                                         ]   22288653/2664470783 10m41s
clone [                                          ]   31366639/2664470783 9m13s
clone [                                          ]   45864021/2664470783 7m26s
clone [                                          ]   52620271/2664470783 7m27s
clone [                                          ]   54983303/2664470783 7m57s
clone [                                          ]   58571603/2664470783 8m15s
clone [>                                         ]   63724606/2664470783 8m18s
clone [>                                         ]   66785326/2664470783 8m36s
clone [>                                         ]   70170374/2664470783 8m48s
clone [>                                         ]   85800199/2664470783 7m44s
clone [>                                         ]   95720744/2664470783 7m24s
clone [>                                         ]  117410774/2664470783 6m23s[taskcluster 2018-08-23T09:08:31.486Z]    Exit Code: 1073807364
[taskcluster 2018-08-23T09:08:31.486Z]    User Time: 0s
[taskcluster 2018-08-23T09:08:31.486Z]  Kernel Time: 0s
[taskcluster 2018-08-23T09:08:31.486Z]    Wall Time: 5h0m47.5542556s
[taskcluster 2018-08-23T09:08:31.486Z]       Result: FAILED
[taskcluster 2018-08-23T09:08:31.486Z] === Task Finished ===
[taskcluster 2018-08-23T09:08:31.486Z] Task Duration: 5h0m47.5542556s
[taskcluster:error] Uploading error artifact public/build from file public/build with message "Could not read directory 'Z:\\task_1534997022\\public\\build'", reason "file-missing-on-worker" and expiry 2019-08-23T03:50:44.472Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'Z:\task_1534997022\public\build'
[taskcluster 2018-08-23T09:08:32.740Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/CmX9SKQCSYK-lKJwufHCeA/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-08-23T03:50:44.472Z
[taskcluster:error] exit status 1073807364
[taskcluster:error] file-missing-on-worker: Could not read directory 'Z:\task_1534997022\public\build'
Summary: Intermittent [taskcluster:error] exit status 1073807364 when cloning repos fails. → Intermittent [taskcluster:error] exit status 1073807364 when cloning repos fails. (task timeout during cloning)
Exit code 1073807364 is 0x40010004 in hexadecimal.

According to https://msdn.microsoft.com/en-us/library/cc704588 this exit code is "DBG_TERMINATE_PROCESS" and means "Debugger terminated the process."

I suspect the process was killed by something else, rather than completing naturally.

This is reminiscent of bug 1444168 comment 38.

Mark/Rob, can you check if it is something in OCC is killing these processes?

Many thanks.
Flags: needinfo?(rthijssen)
Flags: needinfo?(mcornmesser)
I managed to take a look at logs myself, and found a smoking gun. See e.g. 

This task failure:

  https://tools.taskcluster.net/groups/AjYZ2PsYTl-7ncsmyaj3Jw/tasks/TeSp94wJQcaJOCVRUD-YlA/runs/0/logs/public%2Flogs%2Flive.log#L7049

  [taskcluster 2018-11-26T01:13:16.835Z]    Exit Code: 1073807364

Seems to be caused by:

  https://papertrailapp.com/systems/2650328882/events?focus=1003519006302461992&selected=1003519006302461992

  Nov 26 02:13:18 i-03c909367bcb17268.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-03C909367BCB1) has initiated the restart of computer I-03C909367BCB1 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  


So it looks like a service pack installation interrupted the task ("reason: Operating System: Service pack (Planned)"), and that caused the machine to reboot mid-task, sending a kill signal to all processes, hence the exit code denoting that the process was forcefully killed.

I suspect service pack updates need to be disabled on win10 workers. Rob/Mark, can you take a look? Thanks!
Component: Mercurial: robustcheckout → RelOps: OpenCloudConfig
Product: Developer Services → Infrastructure & Operations
QA Contact: rthijssen
Summary: Intermittent [taskcluster:error] exit status 1073807364 when cloning repos fails. (task timeout during cloning) → Intermittent [taskcluster:error] exit status 1073807364
See https://papertrailapp.com/groups/853883/events?q=Operating%20System%3A%20Service%20pack%20(Planned)&focus=1004165691403841539

I would recommend setting up an alert based on this query, so we can see if this ever regresses.

Some recent examples:


Nov 28 01:13:08 i-03191d2dbba299bf7.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-03191D2DBBA29) has initiated the restart of computer I-03191D2DBBA29 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:13:31 i-0384279c0f809cc0c.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0384279C0F809) has initiated the restart of computer I-0384279C0F809 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:20:03 i-000b918b4612cc5d5.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-000B918B4612C) has initiated the restart of computer I-000B918B4612C on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:20:03 i-000b918b4612cc5d5.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-000B918B4612C) has initiated the restart of computer I-000B918B4612C on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:21:02 i-057c6186bcaac45e3.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-057C6186BCAAC) has initiated the restart of computer I-057C6186BCAAC on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:29:01 i-01c924af55d5b7bf8.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-01C924AF55D5B) has initiated the restart of computer I-01C924AF55D5B on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:29:01 i-01c924af55d5b7bf8.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-01C924AF55D5B) has initiated the restart of computer I-01C924AF55D5B on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:29:01 i-01c924af55d5b7bf8.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-01C924AF55D5B) has initiated the restart of computer I-01C924AF55D5B on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 01:29:01 i-01c924af55d5b7bf8.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-01C924AF55D5B) has initiated the restart of computer I-01C924AF55D5B on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:01:26 i-0d3fe7b1e941a44a5.gecko-t-win10-64-gpu.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D3FE7B1E941A) has initiated the restart of computer I-0D3FE7B1E941A on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:02:19 i-0e086060bb80ad3e9.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0E086060BB80A) has initiated the restart of computer I-0E086060BB80A on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:04:44 i-0c346c58482dbe218.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0C346C58482DB) has initiated the restart of computer I-0C346C58482DB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:07:36 i-0f3981a3508c4e344.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F3981A3508C4) has initiated the restart of computer I-0F3981A3508C4 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:09:33 i-018c6afcf92a91f0e.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-018C6AFCF92A9) has initiated the restart of computer I-018C6AFCF92A9 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:10:05 i-08c1293cfcb0d7426.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-08C1293CFCB0D) has initiated the restart of computer I-08C1293CFCB0D on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:11:47 i-05438639fb135aa3e.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-05438639FB135) has initiated the restart of computer I-05438639FB135 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:14:30 i-0e266026539c69bf9.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0E266026539C6) has initiated the restart of computer I-0E266026539C6 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 02:26:15 i-0f3981a3508c4e344.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F3981A3508C4) has initiated the restart of computer I-0F3981A3508C4 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:01:03 i-0f432773d79f3f00d.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F432773D79F3) has initiated the restart of computer I-0F432773D79F3 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:02:07 i-0fba71fb99bc8e753.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0FBA71FB99BC8) has initiated the restart of computer I-0FBA71FB99BC8 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:02:11 i-0470254268e9fc498.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0470254268E9F) has initiated the restart of computer I-0470254268E9F on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:02:18 i-050ebb956b126d895.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-050EBB956B126) has initiated the restart of computer I-050EBB956B126 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:03:06 i-0f82f1324312ce988.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F82F1324312C) has initiated the restart of computer I-0F82F1324312C on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:03:08 i-0725519e32788655a.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0725519E32788) has initiated the restart of computer I-0725519E32788 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:05:30 i-06757ec6f3654cbb7.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-06757EC6F3654) has initiated the restart of computer I-06757EC6F3654 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:07:23 i-0d75c391aec928d33.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D75C391AEC92) has initiated the restart of computer I-0D75C391AEC92 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:07:26 i-0a87f114f7497d61b.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0A87F114F7497) has initiated the restart of computer I-0A87F114F7497 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:07:29 i-0a77d50c68910f2d4.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0A77D50C68910) has initiated the restart of computer I-0A77D50C68910 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:07:49 i-0613071be268a88a4.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0613071BE268A) has initiated the restart of computer I-0613071BE268A on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:08:20 i-0314c308977d27fb9.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0314C308977D2) has initiated the restart of computer I-0314C308977D2 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:08:29 i-03f0f6d5b00699823.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-03F0F6D5B0069) has initiated the restart of computer I-03F0F6D5B0069 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:09:00 i-0f211163cedf5de1a.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F211163CEDF5) has initiated the restart of computer I-0F211163CEDF5 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:10:00 i-00cecf09555526c8f.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-00CECF0955552) has initiated the restart of computer I-00CECF0955552 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:10:05 i-0812ecd109cf22825.gecko-t-win10-64-gpu.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0812ECD109CF2) has initiated the restart of computer I-0812ECD109CF2 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:10:30 i-031ef7e7b7c82559f.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-031EF7E7B7C82) has initiated the restart of computer I-031EF7E7B7C82 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:11:18 i-09ad17f9f975ac7e7.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-09AD17F9F975A) has initiated the restart of computer I-09AD17F9F975A on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:13:21 i-03abe7c276c02915f.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-03ABE7C276C02) has initiated the restart of computer I-03ABE7C276C02 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:14:19 i-086397053bad211ff.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-086397053BAD2) has initiated the restart of computer I-086397053BAD2 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:14:22 i-0bc2c5d23d1afa30d.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0BC2C5D23D1AF) has initiated the restart of computer I-0BC2C5D23D1AF on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 03:31:20 i-0725519e32788655a.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0725519E32788) has initiated the restart of computer I-0725519E32788 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:00:14 i-0bbe054c2df96565b.gecko-t-win10-64-gpu.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0BBE054C2DF96) has initiated the restart of computer I-0BBE054C2DF96 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:01:06 i-057c6186bcaac45e3.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-057C6186BCAAC) has initiated the restart of computer I-057C6186BCAAC on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:02:58 i-0f79750ce91629cd0.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F79750CE9162) has initiated the restart of computer I-0F79750CE9162 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:06:37 i-03033318d10101f8e.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-03033318D1010) has initiated the restart of computer I-03033318D1010 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:08:41 i-0e20e9bef7dc7ff88.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0E20E9BEF7DC7) has initiated the restart of computer I-0E20E9BEF7DC7 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:09:25 i-05872906cf1a8ac4c.gecko-t-win10-64-gpu.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-05872906CF1A8) has initiated the restart of computer I-05872906CF1A8 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:11:33 i-0d0a980e9403979e4.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D0A980E94039) has initiated the restart of computer I-0D0A980E94039 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:14:07 i-000ce4625b1accead.gecko-t-win10-64-gpu.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-000CE4625B1AC) has initiated the restart of computer I-000CE4625B1AC on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:14:52 i-07ef77d06e570f94b.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-07EF77D06E570) has initiated the restart of computer I-07EF77D06E570 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:15:03 i-0ddea70fb9ec6ff20.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0DDEA70FB9EC6) has initiated the restart of computer I-0DDEA70FB9EC6 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:19:48 i-057c6186bcaac45e3.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-057C6186BCAAC) has initiated the restart of computer I-057C6186BCAAC on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:19:48 i-057c6186bcaac45e3.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-057C6186BCAAC) has initiated the restart of computer I-057C6186BCAAC on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:30:15 i-0d0a980e9403979e4.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D0A980E94039) has initiated the restart of computer I-0D0A980E94039 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:30:15 i-0d0a980e9403979e4.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D0A980E94039) has initiated the restart of computer I-0D0A980E94039 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:30:15 i-0d0a980e9403979e4.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D0A980E94039) has initiated the restart of computer I-0D0A980E94039 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 04:44:32 i-07ef77d06e570f94b.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-07EF77D06E570) has initiated the restart of computer I-07EF77D06E570 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:00:21 i-01f34302e121d310e.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-01F34302E121D) has initiated the restart of computer I-01F34302E121D on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:00:40 i-0d051cdaec8f2cf55.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D051CDAEC8F2) has initiated the restart of computer I-0D051CDAEC8F2 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:02:07 i-0553dcc0aa725b956.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0553DCC0AA725) has initiated the restart of computer I-0553DCC0AA725 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:02:34 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:04:01 i-0b7bde4ec61a55bdd.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0B7BDE4EC61A5) has initiated the restart of computer I-0B7BDE4EC61A5 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:04:11 i-03f0f6d5b00699823.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-03F0F6D5B0069) has initiated the restart of computer I-03F0F6D5B0069 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:04:41 i-08d9a457fce2d8bb6.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-08D9A457FCE2D) has initiated the restart of computer I-08D9A457FCE2D on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:05:28 i-050657c91ba669cd3.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-050657C91BA66) has initiated the restart of computer I-050657C91BA66 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:06:10 i-0391716265a73bce0.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0391716265A73) has initiated the restart of computer I-0391716265A73 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:08:58 i-0d373bed2ba283dec.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0D373BED2BA28) has initiated the restart of computer I-0D373BED2BA28 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:09:24 i-0391c4174d2358424.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0391C4174D235) has initiated the restart of computer I-0391C4174D235 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:09:30 i-0a447e97d793d271d.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0A447E97D793D) has initiated the restart of computer I-0A447E97D793D on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:09:44 i-0a1246e328bba100d.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0A1246E328BBA) has initiated the restart of computer I-0A1246E328BBA on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:11:01 i-04048faeb96146ad4.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-04048FAEB9614) has initiated the restart of computer I-04048FAEB9614 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:11:27 i-046d09b1ea8cb952b.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-046D09B1EA8CB) has initiated the restart of computer I-046D09B1EA8CB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:11:55 i-040f830683e0c8261.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-040F830683E0C) has initiated the restart of computer I-040F830683E0C on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:14:41 i-0ee9de784268f355c.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EE9DE784268F) has initiated the restart of computer I-0EE9DE784268F on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:21:32 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:21:33 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:27:40 i-0a1246e328bba100d.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0A1246E328BBA) has initiated the restart of computer I-0A1246E328BBA on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:27:40 i-0a1246e328bba100d.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0A1246E328BBA) has initiated the restart of computer I-0A1246E328BBA on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:31:17 i-0391716265a73bce0.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0391716265A73) has initiated the restart of computer I-0391716265A73 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:31:17 i-0391716265a73bce0.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0391716265A73) has initiated the restart of computer I-0391716265A73 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:31:18 i-0391716265a73bce0.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0391716265A73) has initiated the restart of computer I-0391716265A73 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 05:31:23 i-0391c4174d2358424.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0391C4174D235) has initiated the restart of computer I-0391C4174D235 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:02:08 i-0ebf34806fc58e77b.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EBF34806FC58) has initiated the restart of computer I-0EBF34806FC58 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:05:03 i-04f043be62f4df181.gecko-t-win10-64-gpu.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-04F043BE62F4D) has initiated the restart of computer I-04F043BE62F4D on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:05:48 i-02185030820f47b82.gecko-t-win10-64-gpu.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-02185030820F4) has initiated the restart of computer I-02185030820F4 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:06:42 i-089d0649ff49016dd.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-089D0649FF490) has initiated the restart of computer I-089D0649FF490 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:07:25 i-0dacd1d4d5e299c26.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0DACD1D4D5E29) has initiated the restart of computer I-0DACD1D4D5E29 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:09:45 i-072513bec3f11459f.gecko-t-win10-64.usw1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-072513BEC3F11) has initiated the restart of computer I-072513BEC3F11 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:11:56 i-0315fa983d387efa3.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0315FA983D387) has initiated the restart of computer I-0315FA983D387 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:12:56 i-057c6186bcaac45e3.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-057C6186BCAAC) has initiated the restart of computer I-057C6186BCAAC on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:13:36 i-0f2fb96f47fc42c6f.gecko-t-win10-64-gpu.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0F2FB96F47FC4) has initiated the restart of computer I-0F2FB96F47FC4 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 06:14:06 i-050657c91ba669cd3.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-050657C91BA66) has initiated the restart of computer I-050657C91BA66 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:02:36 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:11:57 i-0315fa983d387efa3.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0315FA983D387) has initiated the restart of computer I-0315FA983D387 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:13:04 i-0c273889fa40ee230.gecko-t-win10-64.euc1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0C273889FA40E) has initiated the restart of computer I-0C273889FA40E on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:15:04 i-011ab0e079f592bc6.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-011AB0E079F59) has initiated the restart of computer I-011AB0E079F59 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:21:24 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:21:24 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 07:21:25 i-0eae58ee84cabfa72.gecko-t-win10-64.usw2.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-0EAE58EE84CAB) has initiated the restart of computer I-0EAE58EE84CAB on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:  
Nov 28 08:08:12 i-08c6782abbdf5c005.gecko-t-win10-64.use1.mozilla.com User32: The process c:\windows\system32\svchost.exe (I-08C6782ABBDF5) has initiated the restart of computer I-08C6782ABBDF5 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010   Shutdown Type: restart   Comment:
yes.

searching papertrail for 1073807364 shows that this exit code is usually preceded by a system restart:
https://papertrailapp.com/groups/2488493/events?q=1073807364%20OR%200x80020010

the reason given for the restart is:
The process c:\windows\system32\svchost.exe (i-xxx) has initiated the restart of computer i-xxx on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Service pack (Planned)   Reason Code: 0x80020010

so it appears that we need to figure out how to disable Service pack installs or see if something is wrong with the disabling of wuauserv.

i'll take this one.
Flags: needinfo?(rthijssen)
Assignee: nobody → rthijssen
Status: NEW → ASSIGNED
Flags: needinfo?(mcornmesser)
Attached file GitHub Pull Request — 
this change sets a number of registry keys relating to windows update configuration. additionally, it synchronises a couple prod/beta manifests which had slightly diverged:
- gecko-t-win10-64 is synced with gecko-t-win10-64-beta 
- gecko-t-win10-64-gpu is synced with gecko-t-win10-64-gpu-b

the sync changes are just to keep beta and prod manifests the same, so that change tests on beta are valid when promoted to prod.

one of the sync changes is to remove the nvidia 520 driver from prod where we recently upgraded the base ami to use g3 instances which use a different nvidia driver. the 520 driver was used to support g2 instances whch are no longer used. on g3 we bake the nvidia grid driver into the ami during iso-to-ami conversion (see: https://github.com/mozilla-platform-ops/relops-image-builder/blob/f5c47c181647949c2a6fd84c47a2d486023f63a9/manifest.json#L110-L116)

another sync change is to add registry keys to beta which were added to prod yesterday (see: https://github.com/mozilla-releng/OpenCloudConfig/commit/c421eb84b254a74a84916ad5bd1d957bea060f33, https://bugzilla.mozilla.org/show_bug.cgi?id=1510220):

- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\DeferUpgrade: 1
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\DeferUpgradePeriod: 8
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\DeferUpdatePeriod: 4

the new registry key changes are:

- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\NoAutoRebootWithLoggedOnUsers: 1
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\NoAutoUpdate: 1
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\AUOptions: 1
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\ScheduledInstallDay: 1
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\ScheduledInstallTime: 1
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\AutomaticMaintenanceEnabled: 0
- HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU\AllowMUUpdateService: 0
- HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\Maintenance\MaintenanceDisabled: 1

Some of these keys will be used now to fix the immediate problem with service pack updates and reboots (NoAutoRebootWithLoggedOnUsers, NoAutoUpdate, AUOptions).

The remaining keys are set so that we have a source controlled record of change if/when we enable automatic updates in future. The settings and their associated values are documented here: https://support.microsoft.com/en-us/help/328010/how-to-configure-automatic-updates-by-using-group-policy-or-registry-s
Attachment #9028268 - Flags: review?(mcornmesser)
Attachment #9028268 - Flags: review?(mcornmesser) → review+
Blocks: 1509761
papertrail logs indicate that the patch has worked. instances that still had updating enabled have mostly died off and we see occurrences of "0x80020010" in the logs disappearing.

https://papertrailapp.com/groups/2488493/events?q=0x80020010
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
No longer blocks: 1509761
See Also: → 1527134

This happened again:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=228300503&repo=autoland&lineNumber=16265

02:00:57 INFO - TEST-START | /html/rendering/bindings/the-select-element-0/option-label.html
02:00:57 INFO - PID 8200 | 1550109657648 Marionette INFO Testing http://web-platform.test:8000/html/rendering/bindings/the-select-element-0/option-label.html == http://web-platform.test:8000/html/rendering/bindings/the-select-element-0/option-label-ref.html
02:00:57 INFO - PID 8200 | 1550109657755 Marionette INFO No differences allowed
02:00:57 INFO - PID 8200 | 1550109657755 Marionette INFO Found 2681 pixels different, maximum difference per channel 255
02:00:57 INFO - TEST-FAIL | /html/rendering/bindings/the-select-element-0/option-label.html | took 119ms
02:00:57 INFO - PID 8200 | [Parent 3088, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | [Child 6216, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | [Child 6216, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | 1550109657837 Marionette INFO Stopped listening on port 2828
02:00:57 INFO - PID 8200 | [Parent 3088, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | [Child 3568, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | [Child 3568, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | [Child 5360, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:57 INFO - PID 8200 | [Child 5360, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:58 INFO - PID 8200 | [Parent 3088, Gecko_IOThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:58 INFO - PID 8200 | [Child 6176, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:58 INFO - PID 8200 | [Child 6176, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
02:00:58 INFO - PID 8200 | [GPU 10808, Chrome_ChildThread] WARNING: pipe erro
02:00:58 INFO - PID 8200 | ###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
02:00:58 INFO - PID 8200 | ###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
02:00:58 INFO - Browser exited with return code 0
02:00:58 INFO - PROCESS LEAKS None
02:00:58 INFO - Closing logging queue
02:00:58 INFO - queue closed
02:00:58 INFO - Setting up ssl
02:00:58 INFO - certutil |
02:00:58 INFO - certutil |
02:00:58 INFO - certutil |
02:00:58 INFO - Certificate Nickname Trust Attributes
02:00:58 INFO - SSL,S/MIME,JAR/XPI
02:00:58 INFO -
02:00:58 INFO - web-platform-tests CT,,
02:00:58 INFO -
02:00:58 INFO - Application command: Z:\task_1550106353\build\application\firefox\firefox.exe -marionette about:blank --wait-for-browser -profile c:\users\task_1550106353\appdata\local\temp\tmpzpqb28.mozrunner
02:00:58 INFO - Starting runner
02:00:59 INFO - PID 10732 | 1550109659480 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid extension permission: mozillaAddons
02:00:59 INFO - PID 10732 | 1550109659480 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid extension permission: resource://pdf.js/
02:00:59 INFO - PID 10732 | 1550109659480 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid extension permission: about:reader*
02:01:01 INFO - PID 10732 | WARN: compileToBinary(235):
02:01:01 INFO - PID 10732 | C:\fakepath(203,14-129): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them
02:01:01 INFO - PID 10732 | C:\fakepath(209,58-112): warning X3571: pow(f, e) will not work for negative f, use abs(f) or conditionally handle negative values if you expect them
[taskcluster 2019-02-14T02:01:01.540Z] Exit Code: 1073807364
[taskcluster 2019-02-14T02:01:01.540Z] User Time: 0s
[taskcluster 2019-02-14T02:01:01.540Z] Kernel Time: 15.625ms
[taskcluster 2019-02-14T02:01:01.540Z] Wall Time: 13m4.686918s
[taskcluster 2019-02-14T02:01:01.540Z] Result: FAILED
[taskcluster 2019-02-14T02:01:01.540Z] === Task Finished ===
[taskcluster 2019-02-14T02:01:01.541Z] Task Duration: 13m4.6889218s
[taskcluster 2019-02-14T02:01:02.197Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:02.678Z] Uploading artifact public/logs/log_critical.log from file logs\log_critical.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:03.135Z] Uploading artifact public/logs/log_error.log from file logs\log_error.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:03.507Z] Uploading artifact public/logs/log_fatal.log from file logs\log_fatal.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:03.912Z] Uploading artifact public/logs/log_info.log from file logs\log_info.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:04.378Z] Uploading artifact public/logs/log_raw.log from file logs\log_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:04.872Z] Uploading artifact public/logs/log_warning.log from file logs\log_warning.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:05.222Z] Uploading artifact public/test_info/wpt_errorsummary.log from file build\blobber_upload_dir\wpt_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:05.602Z] Uploading artifact public/test_info/wpt_raw.log from file build\blobber_upload_dir\wpt_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:06.257Z] Uploading artifact public/test_info/wptreport.json from file build\blobber_upload_dir\wptreport.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-02-14T00:46:40.237Z
[taskcluster 2019-02-14T02:01:06.804Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/AMKa0_klQ8K_iU0gNT3wtQ/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-02-14T00:46:40.237Z
[taskcluster:error] exit status 1073807364

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

this just means that an instance was shut down mid task.

this bug was created when it was happening frequently (because occ or something was shutting down machines due to flawed logic or race conditions).

there's not much we can do about these very low occurrence incidents (2 - 6 out of 3k) as this is likely just due to losing instances due to circumstances beyond our control or a person clicking terminate in the aws provisioner ui.

Status: REOPENED → RESOLVED
Closed: 2 years ago → 2 years ago
Resolution: --- → FIXED

This issue can be seen again: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=234990780&repo=mozilla-inbound&lineNumber=3304

14:35:14 INFO - PID 2708 | Cycle 1(14): loaded http://127.0.0.1:50101/tests/svg_static/composite-scale-opacity.svg (next: http://127.0.0.1:50101/tests/svg_static/composite-scale-rotate.svg)
14:35:15 INFO - PID 2708 | Cycle 1(15): loaded http://127.0.0.1:50101/tests/svg_static/composite-scale-opacity.svg (next: http://127.0.0.1:50101/tests/svg_static/composite-scale-rotate.svg)
[taskcluster 2019-03-20T14:35:15.374Z] Exit Code: 1073807364
[taskcluster 2019-03-20T14:35:15.374Z] User Time: 0s
[taskcluster 2019-03-20T14:35:15.374Z] Kernel Time: 0s
[taskcluster 2019-03-20T14:35:15.374Z] Wall Time: 13m59.5459579s
[taskcluster 2019-03-20T14:35:15.374Z] Result: FAILED
[taskcluster 2019-03-20T14:35:15.374Z] === Task Finished ===
[taskcluster 2019-03-20T14:35:15.374Z] Task Duration: 13m59.5464884s
[taskcluster 2019-03-20T14:35:15.788Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-03-19T12:42:17.319Z
[taskcluster 2019-03-20T14:35:16.592Z] Uploading artifact public/test_info/svgr-e10s_errorsummary.log from file build\blobber_upload_dir\svgr-e10s_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2020-03-19T12:42:17.319Z
[taskcluster 2019-03-20T14:35:17.071Z] Uploading artifact public/test_info/svgr-e10s_raw.log from file build\blobber_upload_dir\svgr-e10s_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2020-03-19T12:42:17.319Z
[taskcluster 2019-03-20T14:35:18.269Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/feNpeOeJTYWVH2A4taBBpA/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-03-19T12:42:17.319Z
[taskcluster:error] exit status 1073807364

You need to log in before you can comment on or make changes to this bug.