1836851 - Perma UV OS X TEST-UNEXPECTED-FAIL: [99.0 en-US complete] check_updates returned unknown error for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 99.0b8.dmg vs. downloads/Firefox 115.0b1.dmg: 3

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=418166103&repo=mozilla-beta
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Rouyo3aGTXSStfuXJT_5Fg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-06-05T19:51:14.060Z] FINISH ADD Contents/MacOS/XUL
[task 2023-06-05T19:51:14.060Z] FINISH ADD Contents/Library/LaunchServices/org.mozilla.updater
[task 2023-06-05T19:51:14.060Z] FINISH ADD Contents/Info.plist
[task 2023-06-05T19:51:14.060Z] succeeded
[task 2023-06-05T19:51:14.060Z] calling QuitProgressUI
[task 2023-06-05T19:51:14.092Z] Traceback (most recent call last):
[task 2023-06-05T19:51:14.092Z]   File "../compare-directories.py", line 257, in <module>
[task 2023-06-05T19:51:14.092Z]     file_list_diff = compare_listings(
[task 2023-06-05T19:51:14.092Z]   File "../compare-directories.py", line 130, in compare_listings
[task 2023-06-05T19:51:14.092Z]     if left_diff - set(ignore_missing):
[task 2023-06-05T19:51:14.092Z] TypeError: 'NoneType' object is not iterable
[task 2023-06-05T19:51:14.096Z] Comparing source/Firefox.app with target/Firefox.app...
[task 2023-06-05T19:51:14.097Z] TEST-UNEXPECTED-FAIL: unknown error from diff: 1
[task 2023-06-05T19:51:14.097Z] TEST-UNEXPECTED-FAIL: [99.0 en-US complete] check_updates returned unknown error for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 99.0b8.dmg vs. downloads/Firefox 115.0b1.dmg: 3
[task 2023-06-05T19:51:14.099Z] Using  https://aus5.mozilla.org/update/3/Firefox/99.0/20220324185704/Darwin_x86_64-gcc3-u-i386-x86_64/ru/beta-localtest/default/default/default/update.xml?force=1
[task 2023-06-05T19:51:14.100Z] Retrieving 'https://aus5.mozilla.org/update/3/Firefox/99.0/20220324185704/Darwin_x86_64-gcc3-u-i386-x86_64/ru/beta-localtest/default/default/default/update.xml?force=1' from cache...
[task 2023-06-05T19:51:14.105Z] Got this response:
[task 2023-06-05T19:51:14.106Z] <?xml version="1.0"?>
[task 2023-06-05T19:51:14.106Z] <updates>
[task 2023-06-05T19:51:14.106Z]     <update actions="showURL" appVersion="115.0" buildID="20230605155701" detailsURL="https://www.mozilla.org/ru/firefox/115.0/releasenotes/" displayVersion="115.0 Beta 1" openURL="https://www.mozilla.org/ru/firefox/115.0beta/whatsnew/?oldversion=%OLD_VERSION%" type="minor">
[task 2023-06-05T19:51:14.106Z]         <patch type="complete" URL="https://archive.mozilla.org/pub/firefox/candidates/115.0b1-candidates/build1/update/mac/ru/firefox-115.0b1.complete.mar" hashFunction="sha512" hashValue="3f8ad40540f48b41413dd18d230b9ba6c27130096ec27f9ca329da1d9ae601ecfa18a6809b616f93bfaad232d693ce2410ef01b42d7b82944dd8201c97c01cf2" size="100720933"/>
[task 2023-06-05T19:51:14.106Z]     </update>
[task 2023-06-05T19:51:14.108Z] </updates>
[task 2023-06-05T19:51:14.108Z] 
[task 2023-06-05T19:51:14.119Z] Retrieving 'https://archive.mozilla.org/pub/firefox/candidates/115.0b1-candidates/build1/update/mac/ru/firefox-115.0b1.complete.mar' from cache...
[task 2023-06-05T19:51:14.760Z] Retrieving 'https://archive.mozilla.org/pub/firefox/releases/99.0b8/linux-x86_64/ru/firefox-99.0b8.tar.bz2' from cache...
[task 2023-06-05T19:51:24.857Z] Found updater at updater
[task 2023-06-05T19:51:24.857Z] override_certs is '', not replacing any certificates
[task 2023-06-05T19:51:24.891Z] Retrieving 'https://archive.mozilla.org/pub/firefox/releases/99.0b8/mac/ru/Firefox 99.0b8.dmg' from cache...
[task 2023-06-05T19:51:24.983Z] Retrieving 'https://archive.mozilla.org/pub/firefox/candidates/115.0b1-candidates/build1/mac/ru/Firefox 115.0b1.dmg' from cache...
[task 2023-06-05T19:51:44.372Z] + /builds/worker/checkouts/gecko/tools/update-verify/release/updates/updater/firefox/updater /builds/worker/checkouts/gecko/tools/update-verify/release/updates/update /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app 0
[task 2023-06-05T19:51:44.384Z] Unable to init server: Could not connect: Connection refused
[task 2023-06-05T19:51:53.580Z] + set +x
[task 2023-06-05T19:51:53.581Z] PATCH DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/update
[task 2023-06-05T19:51:53.581Z] INSTALLATION DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app
[task 2023-06-05T19:51:53.581Z] WORKING DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app
[task 2023-06-05T19:51:53.581Z] UPDATE TYPE complete

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1836851 - [update-verify] Fix regression in compare-directories, r?#releng-reviewers!

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1836851 - [update-verify] Fix regression in compare-directories, r?#releng-reviewers!

Original Revision: https://phabricator.services.mozilla.com/D179997

Comment 3

[Andrew Halberstadt [:ahal]]

Comment on attachment 9337531 [details]
Bug 1836851 - [update-verify] Fix regression in compare-directories, r?#releng-reviewers!

Beta/Release Uplift Approval Request

• User impact if declined: update-verify tasks are busted
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed:
• Is Android affected?: Yes

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7faa48a3ad29
[update-verify] Fix regression in compare-directories, r=releng-reviewers,hneiva

Comment 5

[Donal Meehan [:dmeehan]]

Comment on attachment 9337529 [details]
Bug 1836851 - [update-verify] Fix regression in compare-directories, r?#releng-reviewers!

Approved for 115.0b1 build2

Comment 6

[Donal Meehan [:dmeehan]]

Comment on attachment 9337531 [details]
Bug 1836851 - [update-verify] Fix regression in compare-directories, r?#releng-reviewers!

Removing uplift request, grafted the patch for central

Comment 7

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/aaa8e29ba34e

Comment 8

[Cosmin Sabou [:CosminS]]

Unfortunately these are still failing after the uplift to beta. Added leave-open so it won't be closed when comment 4 reaches central, feel free to drop that when it's deemed as fixed.

Comment 9

[Donal Meehan [:dmeehan]]

Setting 115 back to affected

Comment 10

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/7faa48a3ad29

Comment 11

[Julien Cristau [:jcristau]]

Followup: https://github.com/mozilla-releng/scriptworker-scripts/pull/761

Comment 12

[Intermittent Failures Robot]

1 failures in 3885 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1836851&startday=2023-06-05&endday=2023-06-11&tree=all