1526109 - windows/aarch64 - Invalid test mar file - toolkit/mozapps/update/tests/*

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: egao [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=226675668&repo=try

https://queue.taskcluster.net/v1/task/KTsONR17T9Sdo_lB4qrkLA/runs/0/artifacts/public/logs/live_backing.log

Issue observed as part of win10/aarch64 run.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Same error also observed for various files such as:

• toolkit/mozapps/update/tests/browser/browser_aboutDialog_fc_downloadOptIn_staging.js
• toolkit/mozapps/update/tests/browser/browser_aboutDialog_fc_patch_completeBadSize_complete.js
• toolkit/mozapps/update/tests/browser/browser_aboutDialog_fc_patch_partialBadSize_complete.js

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

here is more context:
03:43:22 INFO - Console message: AUS:SVC Downloader:onStopRequest - attempting to stage update: App Update Test
03:43:22 INFO - Console message: AUS:SVC readStatusFile - status: failed: 39, path: C:\ProgramData\Mozilla\updates\B4A3EEF2ED7784F0\updates\0\update.status
03:43:22 INFO - Console message: AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\B4A3EEF2ED7784F0\updates\0\bt.result
03:43:22 INFO - Console message: AUS:SVC readBinaryTransparencyResult - result: null, path: C:\ProgramData\Mozilla\updates\B4A3EEF2ED7784F0\updates\0\bt.result
03:43:22 INFO - Console message: AUS:SVC handleFallbackToCompleteUpdate - install of partial patch failed, downloading complete patch
03:43:22 INFO - Console message: AUS:SVC Creating Downloader
03:43:22 INFO - Console message: AUS:SVC UpdateService:_downloadUpdate
03:43:22 INFO - Console message: AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\B4A3EEF2ED7784F0\updates\0\update.status
03:43:22 INFO - Console message: AUS:SVC readStatusFile - status: null, path: C:\ProgramData\Mozilla\updates\B4A3EEF2ED7784F0\updates\0\update.status
03:43:22 INFO - Console message: AUS:SVC Downloader:_selectPatch - found existing patch with state: null
03:43:22 INFO - Console message: AUS:SVC Downloader:downloadUpdate - url: http://example.com/browser/toolkit/mozapps/update/tests/browser/simple.mar, path: C:\ProgramData\Mozilla\updates\B4A3EEF2ED7784F0\updates\0\update.mar, interval: 0
03:43:22 INFO - Console message: AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. topic: update-staged, status: downloading
03:43:22 INFO - Console message: AUS:SVC Downloader:onStartRequest - original URI spec: http://example.com/browser/toolkit/mozapps/update/tests/browser/simple.mar, final URI spec: http://example.com/browser/toolkit/mozapps/update/tests/browser/simple.mar
03:43:22 INFO - Console message: AUS:SVC Downloader:onProgress - progress: 1404/1404
03:43:22 INFO - Console message: AUS:SVC Downloader:onProgress - maxProgress: 1404 is not equal to expected patch size: 14041
03:43:22 INFO - Console message: AUS:SVC Downloader: cancel
03:43:22 INFO - Console message: AUS:SVC Downloader:onProgress - progress: 1404/1404
03:43:22 INFO - Console message: AUS:SVC Downloader:onProgress - maxProgress: 1404 is not equal to expected patch size: 14041
03:43:22 INFO - Console message: AUS:SVC Downloader: cancel
03:43:22 INFO - Console message: AUS:SVC Downloader:onStopRequest - original URI spec: http://example.com/browser/toolkit/mozapps/update/tests/browser/simple.mar, final URI spec: http://example.com/browser/toolkit/mozapps/update/tests/browser/simple.mar, status: 2147549183
03:43:22 INFO - Console message: AUS:SVC Downloader:onStopRequest - status: 2147549183, current fail: 0, max fail: 10, retryTimeout: 2000
03:43:22 INFO - Console message: AUS:SVC Downloader:onStopRequest - non-verification failure
03:43:22 INFO - Console message: AUS:SVC getStatusTextFromCode - transfer error: Failed (unknown reason), default code: 2152398849
03:43:22 INFO - Console message: AUS:SVC Downloader:onStopRequest - setting state to: download-failed
03:43:22 INFO - Console message: AUS:SVC Downloader:onStopRequest - notifying observers of error. topic: update-error, status: download-attempt-failed

overall I am not sure if we have updates on the normal server?

:rstrong, do you know if windows on arm64 has updates available?

Comment 3

[Robert Strong (they/them - no direct email)]

We use in tree mar files and the test http server for tests. The problem is that they are compressed using xz --x86 which doesn't work with arm64 ( bug 1514124 ). I'll take care of this in the next few days.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

thanks :rstrong!

Comment 5

[Robert Strong (they/them - no direct email)]

This doesn't actually depend on the releng bug 1514124. It just shows the same underlying issue which had to be fixed for the mar files we advertise to actual clients so removing dependency.

Comment 6

[Robert Strong (they/them - no direct email)]

Attachment: patch

Comment 7

[Robert Strong (they/them - no direct email)]

Joel, if you have a chance please try out this patch on aarch64. Thanks!

Comment 8

[Robert Strong (they/them - no direct email)]

Attachment: Bug 1526109 - Compress files in the test mar files without specifying --x86 so the test mar files will work with aarch64. r?mhowell

Changed simple.mar from a complete mar to a partial mar to prevent it from accidentally removing application files via an application precomplete file.
Changed the name of the precomplete to update_precomplete in simple.mar so it won't accidentally replace an application's precomplete file.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this appears to fix everything:
https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=aarch%2Cbc2&revision=91211f33951ccef418fb98586870b7358a286ac4

Comment 10

[Robert Strong (they/them - no direct email)]

Joel, I triggered the xpcshell X8 tests (runs the app update xpcshell tests) and mochitest-chrome tests and as far as app update goes all is well. It appears that the system running the tests isn't setup 100% or the build system isn't authenticode signing the updater which is less likely to be the issue. A test was added for this since the app update tests were being incorrectly blamed for the build system.

00:29:49 INFO - TEST-START | toolkit/mozapps/update/tests/unit_service_updater/checkUpdaterSigSvc.js
00:29:49 WARNING - TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/unit_service_updater/checkUpdaterSigSvc.js | xpcshell return code: 0

Here are a few of the related bugs
Bug 1203990
Bug 1353889
Bug 1067756

Comment 11

[Intermittent Failures Robot]

1 failures in 2658 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• windows10-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526109&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

given the above bugs, should I be working on fixing the image we use for testing? I am unclear if there is more work I can do, or if we need to update the browser/installer/tests?

Comment 14

[Robert Strong (they/them - no direct email)]

I filed bug 1526982 for the work that needs to be performed.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5f4edb1fe66f
Compress files in the test mar files without specifying --x86 so the test mar files will work with aarch64. r=mhowell

Comment 16

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/5f4edb1fe66f

Comment 17

[Edwin Takahashi (:egao | infrequent contributor)]

Confirmed to be fixed as bc7 was run with :rstrong's updated mar file.

New bc7: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227817689&repo=try&lineNumber=4246
Old bc7: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=226675668&repo=try&lineNumber=5996

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

if we look at all updater tests, the xpcshell tests still fail.

Comment 19

[Robert Strong (they/them - no direct email)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #18)

if we look at all updater tests, the xpcshell tests still fail.
Can you provide a link? Those failures could very well be due to bug 1526982

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

I am sure they are related to bug 1526982:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=91211f33951ccef418fb98586870b7358a286ac4&searchStr=xpcshell

Comment 21

[Robert Strong (they/them - no direct email)]

It looks like the only failure is the one mentioned in comment #10 which is due to bug 1526982