Closed Bug 1526982 Opened 6 years ago Closed 3 years ago

aarch64 needs to be setup for app update testing

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Platform:
ARM64
Windows
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: robert.strong.bugs, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

maintenance_service_regkey.reg
924 bytes, text/plain
Details

The new aarch64 platform will need to be setup for app update testing.

A few of the related bugs where the setup has been performed in the past.

bug 1203990 - test certs installed on system
bug 1353889 - write access to a registry location
bug 1067756 comment #2 - installation of the maintenance service
bug 1302928 - write access to the maintenance service install directory

:grenade, is this something you could help out with? The machine at Bitbar that is hooked up to try is setup with your OCC script hacks, I wonder if this is easy to see failures in the log, or if we need to do other technical challenges to solve this.

Flags: needinfo?(rthijssen)

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #0)

The new aarch64 platform will need to be setup for app update testing.

A few of the related bugs where the setup has been performed in the past.

bug 1203990 - test certs installed on system
bug 1353889 - write access to a registry location
bug 1067756 comment #2 - installation of the maintenance service
bug 1302928 - write access to the maintenance service install directory

i came across problems with one of these on aarch64 the other day and patched the aarch64 branch to fix the problem

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)

:grenade, is this something you could help out with? The machine at Bitbar that is hooked up to try is setup with your OCC script hacks, I wonder if this is easy to see failures in the log, or if we need to do other technical challenges to solve this.

yes, the bitbar machine is logging here: https://papertrailapp.com/systems/2927679092
when new aarch64 instances are added, they should show up here: https://papertrailapp.com/groups/12878032/events

it's probably a good idea to rerun the occ setup on that instance (it may have been last run before the patch mentioned above was applied). eg (from an elevated powershell prompt):

Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/aarch64/userdata/rundsc.ps1?{0}' -f [Guid]::NewGuid())
Flags: needinfo?(rthijssen)

The laptop has had OCC rerun, unfortunately the remaining tests are still failing. :grenade, can you look at papertrail to see what the latest occ run looks like to see if there are broken pieces.

Assuming it ran fine, possibly we need to revisit those bugs and adjust the work done there to support both the existing windows environment as well as the new arm64 environment.

Flags: needinfo?(rthijssen)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)

The laptop has had OCC rerun, unfortunately the remaining tests are still failing. :grenade, can you look at papertrail to see what the latest occ run looks like to see if there are broken pieces.

Assuming it ran fine, possibly we need to revisit those bugs and adjust the work done there to support both the existing windows environment as well as the new arm64 environment.

the occ run in the logs looked to be going well until 19:05:15 utc, yesterday when it abruptly stopped without throwing an error. it's possible that the instance lost network connectivity or something, but it's odd for the process to stop abruptly in the way that it did without logging some sort of exception. my suggestion would be to run it again and watch both the windows event log (Application/OpenCloudConfig & Application/dsc-run) and the output from the powershell console.

Flags: needinfo?(rthijssen)

:grenade, does the current OCC script do the actions in comment #1?

Flags: needinfo?(rthijssen)

yes occ should handle all the maintenance service registry settings and permissions.
see: https://github.com/mozilla-releng/OpenCloudConfig/blob/9e723b5/userdata/Manifest/gecko-t-win10-a64-beta.json#L896-L1165

Flags: needinfo?(rthijssen)

Thanks :grenade, :rstrong, given that we still see test failures, is there something else we can do? Possibly on arm64 there are other tweaks we need to make?

Flags: needinfo?(robert.strong.bugs)

Hi Joel, could you provide a link to a run with the remaining failures? Thanks

Flags: needinfo?(robert.strong.bugs) → needinfo?(jmaher)

here is a link from march 12th (a week ago):
https://taskcluster-artifacts.net/WWxa24aORlG8zpzVU2jnfg/0/public/logs/live_backing.log

Flags: needinfo?(jmaher)

Does the following registry key exist?
SOFTWARE\Mozilla\MaintenanceService\3932ecacee736d366d6436db0f55bce4

Flags: needinfo?(jmaher)

:edwin, can you help out here as I don't have a device. Maybe we could ask the team at Bitbar to look on a machine for us as well.

Flags: needinfo?(jmaher) → needinfo?(egao)

:rstrong - on my local windows10-aarch64 hardware I have the following:

HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\35562fadc262dec332219264bffef2fb

Inside there are two further 'folders':

35562fadc262dec332219264bffef2fb
  - 0
  - 1

Each contains the following:

0
  - issuer: DigiCert SHA2 Assured ID Code Signing CA
  - name: Mozilla Corporation

1
  - issuer: DigiCert Assured ID Code Signing CA-1
  - name: Mozilla Corporation
Flags: needinfo?(egao) → needinfo?(robert.strong.bugs)

I suspect that the maintenance service installer being used hasn't been updated to support aarch64. Try importing the attached registry file and running the tests.

Flags: needinfo?(robert.strong.bugs)

:rstrong - I have the results from the test run after registry was merged.

There were still numerous failures (not all of which are updater tests). The original error message logged at 1532801 is no longer observed, but is replaced with a new error:

21:51:16  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateStageSuccessSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2054] the updater.exe binary should be signed (if not, build system configuration bug?) - false == true
21:51:16     INFO -  xpcshellUtilsAUS.js:shouldRunServiceTest:2054
21:51:16     INFO -  xpcshellUtilsAUS.js:setupTestCommon:764
21:51:16     INFO -  c:/mozilla-build/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateStageSuccessSvc.js:run_test:11
21:51:16     INFO -  c:\mozilla-build\build\tests\xpcshell\head.js:_execute_test:520
21:51:16     INFO -  -e:null:1
21:51:16     INFO -  exiting test
21:51:16     INFO -  (xpcshell/head.js) | test MAIN run_test finished (1)
21:51:16     INFO -  exiting test
21:51:16  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateStageSuccessSvc.js | assertNoUncaughtRejections - [assertNoUncaughtRejections : 257] A promise chain failed to handle a rejection: [Exception... "Abort"  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js :: _abort_failed_test :: line 739"  data: no] - stack: _abort_failed_test@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:739:20
21:51:16     INFO -  do_report_result@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:846:5
21:51:16     INFO -  Assert<@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:54:5
21:51:16     INFO -  proto.report@resource://testing-common/Assert.jsm:213:10
21:51:16     INFO -  proto.ok@resource://testing-common/Assert.jsm:233:10
21:51:16     INFO -  shouldRunServiceTest@xpcshellUtilsAUS.js:2054:12
21:51:16     INFO -  setupTestCommon@xpcshellUtilsAUS.js:764:26
21:51:16     INFO -  run_test@c:/mozilla-build/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateStageSuccessSvc.js:11:8
21:51:16     INFO -  _execute_test@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:520:7
21:51:16     INFO -  @-e:1:1
21:51:16     INFO -  Rejection date: Tue Mar 19 2019 14:51:16 GMT-0700 (Pacific Daylight Time) - false == true
21:51:16     INFO -  resource://testing-common/PromiseTestUtils.jsm:assertNoUncaughtRejections:257
21:51:16     INFO -  c:\mozilla-build\build\tests\xpcshell\head.js:_execute_test:618
21:51:16     INFO -  -e:null:1
21:51:16     INFO -  exiting test
21:51:16     INFO -  PID 15936 | JavaScript error: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js, line 739: NS_ERROR_ABORT:
21:51:16     INFO -  PID 15936 | JavaScript error: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js, line 739: NS_ERROR_ABORT
21:51:16     INFO -  <<<<<<<

The build used for this test was https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=8c574fd0aa5d1437dfff6046fd6c5e13095b8bfb which had the signing task complete successfully.

Let me know if you need any other pieces of information from the test run.

Flags: needinfo?(robert.strong.bugs)

Can you verify that the updater.exe is signed and that the test certs are installed on the system (e.g. bug 1203990).

Flags: needinfo?(robert.strong.bugs) → needinfo?(egao)

You should be able to run the following and it should return 0 if it is signed / trusted properly. It will return 1 if it isn't.
TestAUSHelper.exe check-signature <path to updater.exe>

:rstrong - The output of the command you've requested is 0.

I did notice that these files were all located deep inside C:\mozilla-build\ directory. For windows10-aarch64 the newest version of mozilla-build that installs and runs is 2.2.0 - could this have something to do with why the updater tests fail? I do not claim to understand how the tests work, but if the test scripts call these binaries to perform the update then perhaps it is plausible the old binary is at fault.

Flags: needinfo?(egao) → needinfo?(robert.strong.bugs)

Could you provide more of the test log? Either all of the tests or

starting from
TEST-START | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js

include everything from there up to
TEST-START | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js

starting from
TEST-START | toolkit/mozapps/update/tests/unit_service_updater/marSuccessCompleteSvc.js

include everything from there up to
TEST-START | toolkit/mozapps/update/tests/unit_service_updater/marSuccessPartialSvc.js

starting from
TEST-START | toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateSuccessSvc.js

include everything from there up to
TEST-START | toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateStageSuccessSvc.js

Flags: needinfo?(robert.strong.bugs) → needinfo?(egao)

What is confusing is the log from
https://taskcluster-artifacts.net/WWxa24aORlG8zpzVU2jnfg/0/public/logs/live_backing.log
18:39:03 WARNING - TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2033] the updater.exe binary should not be signed when the test registry key doesn't exist (if it is, build system configuration bug?) - false == true

shows that the binary is signed yet on the system you are running on it is showing that it is not signed.

from comment #21
21:51:16 WARNING - TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/marAppApplyUpdateStageSuccessSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2054] the updater.exe binary should be signed (if not, build system configuration bug?) - false == true

Were you checking the ERRORLEVEL returned from TestAUSHelper.exe?

It might be clearer if the test was run on taskcluster,

:rstrong - I have the logs below:

Local run: https://drive.google.com/file/d/1MRQQymf0uP3ES1dIMetWnJeE1GLl7E1V/view?usp=sharing

Note the entire try push is available here: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=233944154&revision=8c574fd0aa5d1437dfff6046fd6c5e13095b8bfb

The above was the last time I ran the xpcshell test with the updater tests enabled; since then, I've turned it off for windows10-aarch64.

Flags: needinfo?(egao) → needinfo?(robert.strong.bugs)

I copy-pasted the wrong logs for the local run - correcting in a minute.

The taskcluster run isn't reporting that the signing check failed and the local run is reporting that the signing check is failing.

From the local build
21:51:01 INFO - TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2011] the file or directory should exist, leafName: updater.exe - true == true
21:51:01 INFO - "14:51:01:213 | TEST-INFO | xpcshellUtilsAUS.js | [runTestHelperSync : 1842] Running c:\mozilla-build\build\tests\xpcshell\tests\toolkit\mozapps\update\tests\data\TestAUSHelper.exe check-signature c:\mozilla-build\build\application\firefox\updater.exe"
21:51:01 INFO - "14:51:01:279 | TEST-INFO | xpcshellUtilsAUS.js | [isBinarySigned : 2076] binary is not signed. TestAUSHelper.exe returned 1 for file c:\mozilla-build\build\application\firefox\updater.exe"

"binary is not signed" is not in the taskcluster log

This is the check where it will print to the log if the binary isn't signed
https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#2075

This is the check that has been failing on taskcluster when the registry isn't present
https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#2033

This is the check that is failing on the local run when the registry is present and the binary isn't passing the signing check
https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#2054

Since the signing checks are failing for you locally either the environment is weird (I doubt this is why but it is possible), the binary isn't signed, or "the test certs are installed on the system (e.g. bug 1203990)". I suspect it is the last one.

If you could run it on taskcluster with the registry key and the tests re-enabled I highly suspect that the log will provide more detail.

Flags: needinfo?(robert.strong.bugs)

(In reply to Robert Strong (Robert he/him) [:rstrong] (use needinfo to contact me) from comment #23)

The taskcluster run isn't reporting that the signing check failed and the local run is reporting that the signing check is failing.

From the local build
21:51:01 INFO - TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2011] the file or directory should exist, leafName: updater.exe - true == true
21:51:01 INFO - "14:51:01:213 | TEST-INFO | xpcshellUtilsAUS.js | [runTestHelperSync : 1842] Running c:\mozilla-build\build\tests\xpcshell\tests\toolkit\mozapps\update\tests\data\TestAUSHelper.exe check-signature c:\mozilla-build\build\application\firefox\updater.exe"
21:51:01 INFO - "14:51:01:279 | TEST-INFO | xpcshellUtilsAUS.js | [isBinarySigned : 2076] binary is not signed. TestAUSHelper.exe returned 1 for file c:\mozilla-build\build\application\firefox\updater.exe"

"binary is not signed" is not in the taskcluster log

This is the check where it will print to the log if the binary isn't signed
https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#2075

This is the check that has been failing on taskcluster when the registry isn't present
https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#2033

This is the check that is failing on the local run when the registry is present and the binary isn't passing the signing check
https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/tests/data/xpcshellUtilsAUS.js#2054

Since the signing checks are failing for you locally either the environment is weird (I doubt this is why but it is possible), the binary isn't signed, or "the test certs are installed on the system (e.g. bug 1203990)". I suspect it is the last one.

If you could run it on taskcluster with the registry key and the tests re-enabled I highly suspect that the log will provide more detail.

I re-ran the xpcshell suite with the updater tests re-enabled, and the registry file attached to this bug merged into the machines running these tasks from Taskcluster. You can view the try push here: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=retry%2Ctestfailed%2Cbusted%2Cexception%2Csuccess%2Crunning%2Cpending%2Crunnable&group_state=expanded&selectedJob=234931629&revision=e75dd2938a5275dadd9fed1d33e5963ec81bedbf

The logs are here: https://taskcluster-artifacts.net/ANiONvPoSeC_VRrZmkJxVw/0/public/logs/live_backing.log

Flags: needinfo?(robert.strong.bugs)

That made it quite a bit further. It looks like the old maintenance service installer might be causing issues on aarch64 so I am building a new one on try. I don't know how the build system installs the maintenance service and it might be necessary that it includes the maintenanceservice.exe for Windows aarch64 instead of the original one for Windows 32 bit that was added when the build system first supported the maintenance service.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=690b4a13643ee92907a421dd09f4f93138d935c6

Flags: needinfo?(robert.strong.bugs)

A maintenanceservice_installer.exe that should be compatible with aarch64 is available in the following installer
https://queue.taskcluster.net/v1/task/LttIxEqEQMGcYIrptdTIHw/runs/0/artifacts/public/build/install/sea/target.installer.exe

You should be able to extract that installer using 7-zip.
After it is extracted you should be able to use this instead of the maintenanceservice_installer.exe that is used by bug 1067756 comment #4
You might need to use a maintenanceservice.exe that was created for aarch64 when installing the maintenance service but I don't know for certain. After it is installed the tests replace the existing maintenanceservice.exe with the build's maintenanceservice.exe which is possible because of bug 1302928

Note: the maintenanceservice_installer.exe in the link I posted above adds the registry keys as well.

Actually, I think write access so the tests can copy the build's maintenance service into its install directory happened in bug 1067756.

(In reply to Robert Strong (Robert they/them) [:rstrong] (use needinfo to contact me) from comment #26)

A maintenanceservice_installer.exe that should be compatible with aarch64 is available in the following installer
https://queue.taskcluster.net/v1/task/LttIxEqEQMGcYIrptdTIHw/runs/0/artifacts/public/build/install/sea/target.installer.exe

You should be able to extract that installer using 7-zip.
After it is extracted you should be able to use this instead of the maintenanceservice_installer.exe that is used by bug 1067756 comment #4
You might need to use a maintenanceservice.exe that was created for aarch64 when installing the maintenance service but I don't know for certain. After it is installed the tests replace the existing maintenanceservice.exe with the build's maintenanceservice.exe which is possible because of bug 1302928

:rstrong - I re-ran the xpcshell test twice on the windows10-aarch64 hardware I have on hand:

target.installer.exe generated from aarch64 build

  • downloaded, then extracted the target.installer.exe from link
  • replaced existing maintenanceservice_installer.exe in the objdir (2 instances)
  • ran test

Results: https://drive.google.com/open?id=1PJnhLzxpxaH6c9LShvUjItaCGIncVEL4

target.installer.exe generated from comment 26

Results: https://drive.google.com/open?id=14m1sOnk8pEstaet2Po8HH-Bjg5-QJfzd

Both test runs were unsuccessful. If I need to use a newer revision of mozilla-central, please let me know. I believe I did the procedure correctly, but it could be that I have replaced the wrong files, or followed an incorrect procedure.

Flags: needinfo?(robert.strong.bugs)

Part of the build system setup process runs the maintenanceservice_installer.exe. I don't recall the bug number but here is a reference to that script - bug 1067756 comment #4 . The maintenanceservice_installer.exe that is used by the build setup script is actually quite old.

I don't think I said that the maintenanceservice_installer.exe needed to be placed in the location of the build's maintenanceservice_installer.exe and anyways, that won't help. In the end it will need to be part of the build system setup.

You can either replace the existing maintenanceservice_installer.exe used by the script referenced by bug 1067756 comment #4 or you should be able to manually uninstall the Mozilla Maintenance Service (might need admin) and then run maintenanceservice_installer.exe as admin to install the maintenance service on one of the task cluster systems.

Flags: needinfo?(robert.strong.bugs) → needinfo?(egao)

I just received access to the logs you posted and they look like they were run locally. If they aren't treeherder runs you will need to add the certs to your system as mentioned in comment #23 if you haven't to fix the previous issue that was fixed in comment #24.

Maybe Q can help out since I don't know anything about the scripts, etc. that setup the build systems.

Flags: needinfo?(q)

:rstrong - sorry for the lack of movement on this item over the past month or so, I had forgotten about this item.

Just to establish the current baseline, I pushed a try run with the updater test enabled which can be seen here.

The test continues to fail with generic-worker 14.0.2 installed using OCC. So it looks like some manual intervention and investigation is necessary to investigate this test.

I will try to have this test run on the local hardware by making the necessary adjustments as noted in comment #31.

Flags: needinfo?(egao)
OS: Unspecified → Windows
Hardware: Unspecified → ARM64

You also might ask Q for assistance since Q configured most of the other Windows systems and my trying to guide you without my having performed the steps on the systems side is pretty much trying to shoot a target in the dark.

I have run the steps as outlined the previous comments and have the results.

Changes made to test environment

  1. add cert to the windows10-aarch64 system from bug #1203990

Test procedures

bootstrapsvc.js test result

20:05:25     INFO -  TEST-START | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_overrideblocklist.js
20:05:26     INFO -  TEST-PASS | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_overrideblocklist.js | took 891ms
20:05:26     INFO -  TEST-START | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_upgrade.js
20:05:27     INFO -  TEST-PASS | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_upgrade.js | took 1113ms
20:05:27     INFO -  TEST-START | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js
20:05:28  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | xpcshell return code: 0
20:05:28     INFO -  TEST-INFO took 282ms
20:05:28     INFO -  >>>>>>>
20:05:28     INFO -  PID 13156 | Unable to load \\untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
20:05:28     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
20:05:28     INFO -  "20:05:27:869 | TEST-INFO | xpcshellUtilsAUS.js | [setupTestCommon : 741] start - general test setup"
20:05:28     INFO -  TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | setupTestCommon - [setupTestCommon : 743] gTestID should be 'undefined' (setupTestCommon should only be called once) - "undefined" === "undefined"
20:05:28     INFO -  TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2016] the file or directory should exist, leafName: updater.exe - true == true
20:05:28     INFO -  "20:05:27:873 | TEST-INFO | xpcshellUtilsAUS.js | [runTestHelperSync : 1847] Running c:\\mozilla-build\\build\\tests\\xpcshell\\tests\\toolkit\\mozapps\\update\\tests\\data\\TestAUSHelper.exe check-signature c:\\mozilla-build\\build\\application\\firefox\\updater.exe"
20:05:28     INFO -  "20:05:27:931 | TEST-INFO | xpcshellUtilsAUS.js | [isBinarySigned : 2081] binary is not signed. TestAUSHelper.exe returned 1 for file c:\\mozilla-build\\build\\application\\firefox\\updater.exe"
20:05:28     INFO -  "20:05:27:932 | TEST-INFO | xpcshellUtilsAUS.js | [runTestHelperSync : 1847] Running c:\\mozilla-build\\build\\tests\\xpcshell\\tests\\toolkit\\mozapps\\update\\tests\\data\\TestAUSHelper.exe wait-for-service-stop MozillaMaintenance 10"
20:05:28     INFO -  TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2052] the maintenance service should be installed (if not, build system configuration bug?) - 0 != 238
20:05:28  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2059] the updater.exe binary should be signed (if not, build system configuration bug?) - false == true
20:05:28     INFO -  xpcshellUtilsAUS.js:shouldRunServiceTest:2059
20:05:28     INFO -  xpcshellUtilsAUS.js:setupTestCommon:767
20:05:28     INFO -  c:/mozilla-build/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js:run_test:8
20:05:28     INFO -  c:\mozilla-build\build\tests\xpcshell\head.js:_execute_test:523
20:05:28     INFO -  -e:null:1
20:05:28     INFO -  exiting test
20:05:28     INFO -  (xpcshell/head.js) | test MAIN run_test finished (1)
20:05:28     INFO -  exiting test
20:05:28  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js | assertNoUncaughtRejections - [assertNoUncaughtRejections : 257] A promise chain failed to handle a rejection: [Exception... "Abort"  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js :: _abort_failed_test :: line 742"  data: no] - stack: _abort_failed_test@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:742:20
20:05:28     INFO -  do_report_result@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:849:5
20:05:28     INFO -  Assert<@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:57:5
20:05:28     INFO -  proto.report@resource://testing-common/Assert.jsm:213:10
20:05:28     INFO -  proto.ok@resource://testing-common/Assert.jsm:233:10
20:05:28     INFO -  shouldRunServiceTest@xpcshellUtilsAUS.js:2059:12
20:05:28     INFO -  setupTestCommon@xpcshellUtilsAUS.js:767:26
20:05:28     INFO -  run_test@c:/mozilla-build/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/bootstrapSvc.js:8:8
20:05:28     INFO -  _execute_test@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:523:7
20:05:28     INFO -  @-e:1:1
20:05:28     INFO -  Rejection date: Fri May 03 2019 20:05:27 GMT+0000 (Greenwich Mean Time) - false == true
20:05:28     INFO -  resource://testing-common/PromiseTestUtils.jsm:assertNoUncaughtRejections:257
20:05:28     INFO -  c:\mozilla-build\build\tests\xpcshell\head.js:_execute_test:621
20:05:28     INFO -  -e:null:1
20:05:28     INFO -  exiting test
20:05:28     INFO -  PID 13156 | JavaScript error: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js, line 742: NS_ERROR_ABORT:
20:05:28     INFO -  PID 13156 | JavaScript error: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js, line 742: NS_ERROR_ABORT
20:05:28     INFO -  <<<<<<<

It appears that this particular test continues to fail even after installing the certs. Granted, it was not tested in a CI environment but the necessary procedures should have been followed to ensure a compatible environment.

other tests

Various other tests in the same subdirectory also fail, usually with the same error about gtestId should be 'undefined':

20:05:28  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js | xpcshell return code: 0
20:05:28     INFO -  TEST-INFO took 280ms
20:05:28     INFO -  >>>>>>>
20:05:28     INFO -  PID 5188 | Unable to load \\untrusted-startup-test-dll.dll; LoadLibraryW failed: 126
20:05:28     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
20:05:28     INFO -  "20:05:28:201 | TEST-INFO | xpcshellUtilsAUS.js | [setupTestCommon : 741] start - general test setup"
20:05:28     INFO -  TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js | setupTestCommon - [setupTestCommon : 743] gTestID should be 'undefined' (setupTestCommon should only be called once) - "undefined" === "undefined"
20:05:28     INFO -  TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2016] the file or directory should exist, leafName: updater.exe - true == true
20:05:28     INFO -  "20:05:28:204 | TEST-INFO | xpcshellUtilsAUS.js | [runTestHelperSync : 1847] Running c:\\mozilla-build\\build\\tests\\xpcshell\\tests\\toolkit\\mozapps\\update\\tests\\data\\TestAUSHelper.exe check-signature c:\\mozilla-build\\build\\application\\firefox\\updater.exe"
20:05:28     INFO -  "20:05:28:261 | TEST-INFO | xpcshellUtilsAUS.js | [isBinarySigned : 2081] binary is not signed. TestAUSHelper.exe returned 1 for file c:\\mozilla-build\\build\\application\\firefox\\updater.exe"
20:05:28     INFO -  "20:05:28:262 | TEST-INFO | xpcshellUtilsAUS.js | [runTestHelperSync : 1847] Running c:\\mozilla-build\\build\\tests\\xpcshell\\tests\\toolkit\\mozapps\\update\\tests\\data\\TestAUSHelper.exe wait-for-service-stop MozillaMaintenance 10"
20:05:28     INFO -  TEST-PASS | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2052] the maintenance service should be installed (if not, build system configuration bug?) - 0 != 238
20:05:28  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js | shouldRunServiceTest - [shouldRunServiceTest : 2059] the updater.exe binary should be signed (if not, build system configuration bug?) - false == true
20:05:28     INFO -  xpcshellUtilsAUS.js:shouldRunServiceTest:2059
20:05:28     INFO -  xpcshellUtilsAUS.js:setupTestCommon:767
20:05:28     INFO -  c:/mozilla-build/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js:run_test:9
20:05:28     INFO -  c:\mozilla-build\build\tests\xpcshell\head.js:_execute_test:523
20:05:28     INFO -  -e:null:1
20:05:28     INFO -  exiting test
20:05:28     INFO -  (xpcshell/head.js) | test MAIN run_test finished (1)
20:05:28     INFO -  exiting test
20:05:28  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js | assertNoUncaughtRejections - [assertNoUncaughtRejections : 257] A promise chain failed to handle a rejection: [Exception... "Abort"  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js :: _abort_failed_test :: line 742"  data: no] - stack: _abort_failed_test@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:742:20
20:05:28     INFO -  do_report_result@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:849:5
20:05:28     INFO -  Assert<@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:57:5
20:05:28     INFO -  proto.report@resource://testing-common/Assert.jsm:213:10
20:05:28     INFO -  proto.ok@resource://testing-common/Assert.jsm:233:10
20:05:28     INFO -  shouldRunServiceTest@xpcshellUtilsAUS.js:2059:12
20:05:28     INFO -  setupTestCommon@xpcshellUtilsAUS.js:767:26
20:05:28     INFO -  run_test@c:/mozilla-build/build/tests/xpcshell/tests/toolkit/mozapps/update/tests/unit_service_updater/invalidArgInstallDirPathTooLongFailureSvc.js:9:8
20:05:28     INFO -  _execute_test@c:\\mozilla-build\\build\\tests\\xpcshell\\head.js:523:7
20:05:28     INFO -  @-e:1:1
20:05:28     INFO -  Rejection date: Fri May 03 2019 20:05:28 GMT+0000 (Greenwich Mean Time) - false == true
20:05:28     INFO -  resource://testing-common/PromiseTestUtils.jsm:assertNoUncaughtRejections:257
20:05:28     INFO -  c:\mozilla-build\build\tests\xpcshell\head.js:_execute_test:621
20:05:28     INFO -  -e:null:1
20:05:28     INFO -  exiting test
20:05:28     INFO -  PID 5188 | JavaScript error: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js, line 742: NS_ERROR_ABORT:
20:05:28     INFO -  PID 5188 | JavaScript error: c:\\mozilla-build\\build\\tests\\xpcshell\\head.js, line 742: NS_ERROR_ABORT
20:05:28     INFO -  <<<<<<<

next steps

Some things I could do:

  • ask the folks at Bitbar to install the cert from bug #1203990 to all of the machines currently on hand, then attempt this on CI

Otherwise, I'm not sure what I could do here.

Swapping the NI on Q to Rob, as Rob's much more familiar with both OCC and the Aarch64 work.

Flags: needinfo?(q) → needinfo?(rthijssen)

i will be working on this as part of the migration from occ to ronin

Depends on: 1572076
Flags: needinfo?(rthijssen)

this should have resolved itself recently when we got occ working again on these machines. i need to figure out how to check that the errors no longer occur. reinstating ni so that i have a monday morning reminder.

Flags: needinfo?(rthijssen)

i've had a look and i believe the certs are being installed and registry keys set:
https://github.com/mozilla-releng/OpenCloudConfig/blob/master/userdata/Manifest/gecko-t-win10-a64-beta.json#L1163-L1342

Flags: needinfo?(rthijssen)

backlog cleanup

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: