1559157 - Test/upgrade all OpenCloudConfig workers to generic-worker 15.1.0

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Pete Moore [:pmoore][:pete]]

These are the current versions:

releng-hardware/gecko-t-win10-64-ux:                                       generic-worker 10.11.2
aws-provisioner-v1/gecko-1-b-win2012:                                      generic-worker 14.1.0
aws-provisioner-v1/gecko-2-b-win2012:                                      generic-worker 14.1.0
aws-provisioner-v1/gecko-3-b-win2012-c4:                                   generic-worker 14.1.0
aws-provisioner-v1/gecko-3-b-win2012-c5:                                   generic-worker 14.1.0
aws-provisioner-v1/gecko-3-b-win2012:                                      generic-worker 14.1.0
aws-provisioner-v1/gecko-t-win10-64-alpha:                                 generic-worker 14.1.0
aws-provisioner-v1/gecko-t-win10-64-gpu-a:                                 generic-worker 14.1.0
aws-provisioner-v1/gecko-t-win10-64-gpu:                                   generic-worker 14.1.0
aws-provisioner-v1/gecko-t-win10-64:                                       generic-worker 14.1.0
aws-provisioner-v1/gecko-t-win7-32-gpu:                                    generic-worker 14.1.0
aws-provisioner-v1/gecko-t-win7-32:                                        generic-worker 14.1.0
aws-provisioner-v1/relops-image-builder:                                   generic-worker 14.1.0
releng-hardware/gecko-t-win10-64-hw:                                       generic-worker 14.1.2
aws-provisioner-v1/gecko-1-b-win2012-beta:                                 generic-worker 15.0.0
aws-provisioner-v1/gecko-t-win10-64-beta:                                  generic-worker 15.0.0
aws-provisioner-v1/gecko-t-win10-64-gpu-b:                                 generic-worker 15.0.0
aws-provisioner-v1/gecko-t-win7-32-beta:                                   generic-worker 15.0.0
aws-provisioner-v1/gecko-t-win7-32-cu:                                     generic-worker 15.0.0
aws-provisioner-v1/gecko-t-win7-32-gpu-b:                                  generic-worker 15.0.0
aws-provisioner-v1/gecko-t-win10-64-cu:                                    not yet determined - task Sk0Op_74TmmdqP4HVn0qvQ still pending...

Current try job running to test 15.0.0 on the beta worker types.

Not quite sure why task Sk0Op_74TmmdqP4HVn0qvQ is still pending...

Comment 1

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #0)

Not quite sure why task Sk0Op_74TmmdqP4HVn0qvQ is still pending...

From papertrail, we seem to be stuck waiting for registry key HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 to exist on gecko-t-win10-64-cu workers...

Jun 13 17:51:31 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>ping -n 2 127.0.0.1  1>/nul  
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>goto CheckForStuckRects3    
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>echo Wait for registry setting to exist before changing it...   Wait for registry setting to exist before changing it...   
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 /ve   ERROR: The system was unable to find the specified registry key or value.   
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>if 1 EQU 0 goto HideTaskBar    
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>echo HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 does not yet exist   HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 does not yet exist   
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>ping -n 2 127.0.0.1  1>/nul  
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>goto CheckForStuckRects3    
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>echo Wait for registry setting to exist before changing it...   Wait for registry setting to exist before changing it...   
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 /ve   ERROR: The system was unable to find the specified registry key or value.   
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>if 1 EQU 0 goto HideTaskBar    
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>echo HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 does not yet exist   HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 does not yet exist   
Jun 13 17:51:33 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>ping -n 2 127.0.0.1  1>/nul  
Jun 13 17:51:34 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>goto CheckForStuckRects3    
Jun 13 17:51:34 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>echo Wait for registry setting to exist before changing it...   Wait for registry setting to exist before changing it...   
Jun 13 17:51:34 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 /ve   ERROR: The system was unable to find the specified registry key or value.   
Jun 13 17:51:34 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>if 1 EQU 0 goto HideTaskBar    
Jun 13 17:51:34 i-0cf03c76f83104129.gecko-t-win10-64-cu.usw2.mozilla.com generic-worker: Z:\task_1560435540>echo HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 does not yet exist   HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\StuckRects3 does not yet exist

Rob, any ideas what could be causing this?

Comment 2

[Pete Moore [:pmoore][:pete]]

Attachment: regedit screenshot

From this screenshot, where I am logged in as Administrator, indeed the StuckRects3 key seems to be missing.

I'm logged into:

Region: us-west-2
i-0cf03c76f83104129: rdp Administrator@34.222.220.56

Comment 3

[Pete Moore [:pmoore][:pete]]

Attachment: regedit screenshot 2

Whoops, it is there, I was looking for a file instead of a folder in the UI, but I see it now...

Comment 4

[Pete Moore [:pmoore][:pete]]

Attachment: task_1560435540 regedit screenshot 1

Logging in as task user task_1560435540 I can also see the StuckRects3 registry entry...

Comment 5

[Pete Moore [:pmoore][:pete]]

Attachment: Screenshot 2019-06-13 at 18.37.43.png

Note, when first logging in as the task user, there was a window open, but closing it didn't fix the problem, so I don't think it is related.

Comment 6

[Pete Moore [:pmoore][:pete]]

Attachment: cmd.exe for task user showing reg key exists

Executing the same registry query in a cmd.exe shell in the task user's desktop shows the key to exist.

The plot thickens!

Comment 7

[Pete Moore [:pmoore][:pete]]

I wonder if the issue is that the process is not running as the task user, but is running as LocalSystem (which would be a generic-worker bug).

Created PR 206 to test this theory. I added a test in the CI for this, let's wait for the CI to run and see if it fails or not.

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: Process Explorer showing process user

Using ProcessExplorer, it indeed looks like the cmd.exe is running as LocalSystem, which it should not be.

This is strange, because the test from comment 7 seems to show that the script does run as the task user, at least on Windows Server 2012 R2, which runs the same version of generic-worker as Windows 10.

Unfortunately, the worker type that is broken is actually the one that runs the Windows 10 CI, so we can't see the test results on that platform.

But now I think about it - maybe this is the problem. Maybe the problem is for worker types that have "runTasksAsCurrentUser": true (i.e. the CI worker types), that run the "runAfterUserCreation" script as the current user, rather than the task user. Perhaps previously they ran the "runAfterUserCreation" script as the task user. I can check this.

So in the end, maybe this (minor) bug only affects the generic-worker CI worker types. I say it is a minor bug, because the combination of setting "runTasksAsCurrentUser": true and having "runAfterUserCreation" set to a non empty string is extremely rare, and generally not useful, and probably only occurs on gecko-t-win10-64-cu.

Comment 9

[Rob Thijssen [:grenade (EET/UTC+0300)]]

i don't know

Comment 10

[Pete Moore [:pmoore][:pete]]

(In reply to Rob Thijssen [:grenade (EET/UTC+0300)] from comment #9)

i don't know

It was bug 1559210.

Comment 11

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #0)

Current try job running to test 15.0.0 on the beta worker types.

New try job running to test 15.0.1 on the beta worker types.

Comment 12

[Pete Moore [:pmoore][:pete]]

Attachment: GitHub Pull Request for OpenCloudConfig (gw 14.1.0 -> 15.1.0)

Comment 13

[Pete Moore [:pmoore][:pete]]

With Whistler last week, I didn't get round to landing this, and in the meantime we've had a new release, so I've tested it here and it also passes, so I'll go with 15.1.0 instead of 15.0.1.

Will make the PR now...

Comment 14

[Pete Moore [:pmoore][:pete]]

Deploying to production now!