Bugzilla

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of generic-worker, like 14.1.2
3) The reason tasks run under generic-worker 14.1.2 despite the automatic login not working, are that:
    a) it does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one
    b) presumably, the testdroid user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed this is indeed the case.
4) After adding debug to the generic-worker logs, I can see that generic-worker successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with regedit, or using `reg query` on the command line, the keys that generic-worker wrote appear not to be in the registry. It is like generic-worker is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, 14.1.2 runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that generic-worker 14.1.2 and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that generic-worker calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of generic-worker, like 14.1.2
3) The reason tasks run under generic-worker 14.1.2 despite the automatic login not working, are that:
    a) generic-worker 14.1.2 does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one, unlike generic-worker 16.5.5
    b) presumably, the `testdroid` user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed this is indeed the case.
4) After adding debug to the generic-worker logs, I can see that generic-worker successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with regedit, or using `reg query` on the command line, the keys that generic-worker wrote appear not to be in the registry. It is like generic-worker is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, 14.1.2 runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that generic-worker 14.1.2 and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that generic-worker calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of generic-worker, like 14.1.2
3) The reason tasks run under generic-worker 14.1.2 despite the automatic login not working, are that:
    a) generic-worker 14.1.2 does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one, unlike generic-worker 16.5.5
    b) presumably, the `testdroid` user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed that indeed all tasks are running as the `testdroid` OS user rather than the task user they should use.
4) After adding debug to the generic-worker logs, I can see that generic-worker successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with regedit, or using `reg query` on the command line, the keys that generic-worker wrote appear not to be in the registry. It is like generic-worker is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, 14.1.2 runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that generic-worker 14.1.2 and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that generic-worker calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of generic-worker, like 14.1.2
3) The reason tasks run under generic-worker 14.1.2 despite the automatic login not working, are that:
    a) generic-worker 14.1.2 does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one, unlike generic-worker 16.5.5 which asserts that the user is the correct one.
    b) Presumably, the `testdroid` user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed that indeed all tasks are running as the `testdroid` OS user rather than the task user they should use.
4) After adding debug to the generic-worker logs, I can see that generic-worker successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with regedit, or using `reg query` on the command line, the keys that generic-worker wrote appear not to be in the registry. It is like generic-worker is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, 14.1.2 runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that generic-worker 14.1.2 and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that generic-worker calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of generic-worker, like 14.1.2
3) The reason tasks run under generic-worker 14.1.2 despite the automatic login not working, are that:
    a) generic-worker 14.1.2 does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one, unlike generic-worker 16.5.5 which asserts that the user is the correct one.
    b) Presumably, the `testdroid` user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed that indeed all tasks are running as the `testdroid` OS user rather than the task user they should use (see log lines 22/23).
4) After adding debug to the generic-worker logs, I can see that generic-worker successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with regedit, or using `reg query` on the command line, the keys that generic-worker wrote appear not to be in the registry. It is like generic-worker is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, 14.1.2 runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that generic-worker 14.1.2 and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that generic-worker calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of `generic-worker`, like 14.1.2
3) The reason tasks run under `generic-worker` 14.1.2 despite the automatic login not working, are that:
    a) `generic-worker` 14.1.2 does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one, unlike `generic-worker` 16.5.5 which asserts that the user is the correct one.
    b) Presumably, the `testdroid` user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed that indeed all tasks are running as the `testdroid` OS user rather than the task user they should use (see log lines 22/23).
4) After adding debug to the `generic-worker` logs, I can see that `generic-worker` successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with `regedit`, or using `reg query` on the command line, the keys that `generic-worker` wrote appear not to be in the registry. It is like `generic-worker` is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, 14.1.2 runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that `generic-worker` 14.1.2 and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that `generic-worker` calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...

I've managed to reproduce the issue on an aarch64 windows laptop here, and discovered some interesting things:

1) Like Rob says, win10 on aarch64 seems not to have a `Remote Desktop Users` group, but that can be manually created
2) The auto logon is not working, also for older versions of `generic-worker`, like 14.1.2
3) The reason tasks run under `generic-worker 14.1.2` despite the automatic login not working, are that:
    a) `generic-worker 14.1.2` does not check that the logon session of the interactive user is for the task user it created - it just waits until the interactive desktop session is active, and assumes the user is the correct one, unlike `generic-worker 16.5.5` which asserts that the user is the correct one.
    b) Presumably, the `testdroid` user that was set up on the bitbar laptops was set up so that there was an interactive desktop session available for running tasks

    To test this theory, I submitted [a task](https://firefox-ci-tc.services.mozilla.com/tasks/YGivxSDsQTGGaAJGxjDWOQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FYGivxSDsQTGGaAJGxjDWOQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L22-23), which confirmed that indeed all tasks are running as the `testdroid` OS user rather than the task user they should use (see log lines 22/23).
4) After adding debug to the `generic-worker` logs, I can see that `generic-worker` successfully writes winlogon registry keys, and after rebooting and reading the registry keys back, it finds the same values it wrote to the registry before the reboot. However, for some reason, when I look in the registry with `regedit`, or using `reg query` on the command line, the keys that `generic-worker` wrote appear not to be in the registry. It is like `generic-worker` is writing them to a shadow registry file, not the one that the system is using. I have no explanation for this at the moment, since there should only be a single HKEY_LOCAL_MACHINE registry, and I run `regedit`/`reg query` from the same command shell as I run the go program that reads/writes to HKEY_LOCAL_MACHINE, yet they show different results. I intend to troubleshoot this further by patching the go standard library with additional debug output to log all the syscalls it makes.

In summary, `generic-worker 14.1.2` runs, but isn't operating correctly since it runs all tasks as the same OS user (`testdroid`). Version 16 doesn't run, because it is stricter that `generic-worker 14.1.2` and checks that the user it is running tasks as is the user that it created for the purpose. The reason the user isn't auto-logging in is because the updates to the winlogon registry settings from the go code that `generic-worker` calls, appear to be operating against a different registry to the actual registry that the system is using. This only happens on our aarch64 windows infrastructure - all our amd64 and i386 windows workers, for Windows 7, Windows 10, Windows Server 2012 R2, all operate as expected, and don't seem to have the same "shadow registry" issue that the aarch64 Windows 10 workers seem to have.

The investigation continues...