Closed Bug 1577664 Opened 5 months ago Closed 5 months ago

Complete updates to DevEdition 70.0b2 failing

Categories

(Release Engineering :: Release Automation: Updates, defect, P1)

defect

Tracking

(firefox-esr60 unaffected, firefox-esr68 unaffected, firefox68 unaffected, firefox69 unaffected, firefox70 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr60 --- unaffected
firefox-esr68 --- unaffected
firefox68 --- unaffected
firefox69 --- unaffected
firefox70 --- fixed

People

(Reporter: nthomas, Assigned: nthomas)

References

Details

Attachments

(1 file, 1 obsolete file)

Eg https://taskcluster-artifacts.net/cJnuTObASSWiaafUIqImLg/0/public/logs/live_backing.log

[task 2019-08-29T20:43:27.546Z] + /builds/worker/checkouts/gecko/tools/update-verify/release/updates/updater/firefox/updater /builds/worker/checkouts/gecko/tools/update-verify/release/updates/update /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/firefox /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/firefox 0
[task 2019-08-29T20:43:27.556Z] Unable to init server: Could not connect: Connection refused
[task 2019-08-29T20:43:27.741Z] + set +x
[task 2019-08-29T20:43:27.742Z] PATCH DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/update
[task 2019-08-29T20:43:27.742Z] INSTALLATION DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/firefox
[task 2019-08-29T20:43:27.742Z] WORKING DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/firefox
[task 2019-08-29T20:43:27.742Z] failed: 23
[task 2019-08-29T20:43:27.742Z] calling QuitProgressUI
[task 2019-08-29T20:43:27.743Z] TEST-UNEXPECTED-FAIL: update status was not successful: failed: 23
[task 2019-08-29T20:43:27.743Z] TEST-UNEXPECTED-FAIL: [70.0 ach complete] check_updates returned failure for Linux_x86-gcc3 downloads/firefox-70.0b1.tar.bz2 vs. downloads/firefox-70.0b2.tar.bz2: 1

Error 23 is a VERSION_DOWNGRADE_ERROR. This is all platforms, all locales, completes but not partials (bug 1577634).

The complete mar files have this encoded metadata:

$ mar -T firefox-70.0b2.complete.mar  | head
Signature block found with 1 signature
1 additional block found:
  - Product Information Block:
    - MAR channel name: firefox-mozilla-aurora
    - Product version: 70.0a1

So 70.0a1 < 70.0 used in 70.0b1, so that's why we get the downgrade error. This only occurs on the complete updates, the partials have 70.0 set as expected.

It turns out the repackage tasks (which create the complete mar files) depend on an autoland job (KPYGHHbpQKC-oZaThL01FQ) to generate the mar utility. That has the milestone built into it as a default version. I think we should rebuild the mar-tools toolchain based on milestone.txt.

There's a second issue where we're not passing the actual version we want when calling mar, due to some mismatched variables.

The mar utility stores MOZ_APP_VERSION as the default Product Version for mar creation. So mar should be rebuilt whenever the milestone changes.

MOZ_FULL_PRODUCT_VERSION is only used here, while tools/update-packaging/make_full_update.sh expects MOZ_PRODUCT_VERSION.

Depends on D44089

Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/464a2ec59a58
mar-tools toolchain should depend on milestone.txt, r=Callek
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ca2208744242
pass product version to the complete mar generation script, r=Callek

And pushed to env var fix to beta too:
https://hg.mozilla.org/releases/mozilla-beta/rev/fc4326204da468558cfd3ad27bf420a0012b5c70

The backout of the toolchain fix seems to be related to resources being able to depend on directories but not files.

Flags: needinfo?(nthomas)
Assignee: nobody → nthomas

Not sure why the decision task doesn't like the toolchain patch, but it's not blocking deved 70.0b2 now that the MOZ_PRODUCT_VERSION change has landed. We'll use mar from task KPYGHHbpQKC-oZaThL01FQ, which is autoland fb03ac72bc47c9b4bd97e3277ab0cee02026aa7f. There is no diff between that rev and beta in toolkit/mozapps/updater.

Flags: needinfo?(nthomas)
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

If we aren't using the compiled-in version anywhere, is there any reason to make mar-tools depend on the version?

It might make sense to remove compiling in the version (and the update-channel) into the tool, though.

Regressions: 1458385
Blocks: 1577760
Attachment #9089252 - Attachment is obsolete: true

(In reply to Tom Prince [:tomprince] from comment #11)

If we aren't using the compiled-in version anywhere, is there any reason to make mar-tools depend on the version?

It might make sense to remove compiling in the version (and the update-channel) into the tool, though.

If rstrong is happy with that in bug 1577760 then that sounds like a good solution.

How does this regress bug 1458385 ?

There's no need to uplift this to release or esr, because the mar-tools toolchain is new in 70 (bug 1458385 again).

Status: REOPENED → RESOLVED
Closed: 5 months ago5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.