Closed Bug 1745203 Opened 2 years ago Closed 2 years ago

Add pushmsixscript scriptworker-script

Categories

(Release Engineering :: General, task)

Tracking

(firefox97 fixed, firefox101 fixed)

RESOLVED FIXED
Tracking Status
firefox97 --- fixed
firefox101 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(8 files)

I have been adding a new scriptworker-script, to automate publishing Firefox (in .msix format) to the Microsoft Store. For posterity / documentation purposes, here's a bug to tie together the various PRs.

Added pushmsix.yaml to relengworker sops:

  • cd <relengworker-sops>/projects/relengworker/k8s/values
  • sops -d pushflatpak.yaml > pushmsix-unenc.yaml
  • modify pushmsix-unenc.yaml:
    • s/pushflatpak/pushmsix/
    • populate taskclusterAccessToken by resetting the access token for each client in the taskcluster UI (https://firefox-ci-tc.services.mozilla.com/auth/clients?search=pushmsix, open each, use ... menu in lower right, "reset access token", then scroll up to see alert)
    • keep ed25519PrivateKey (same value for all scripts)
    • remove flatpak secrets, add msix secrets, encoded with 'base64 -w0'
  • sops --encrypt --gcp-kms "projects/moz-fx-relengworker-prod-a67d/locations/global/keyRings/sops-default/cryptoKeys/default" pushmsix-unenc.yaml > pushmsix.yaml
  • sops updatekeys pushmsix.yml
  • git add/commit/push

With a special config file, I have been able to run pushmsixscript to upload nightly builds to the Firefox Nightly app on the Microsoft Store -- but those submissions fail, on the final, commit phase. I found I hit the same errors (name mismatches) if I upload manually via the Partner Centre; apparently this is a known issue, specific to our nightlies (which is currently our preferred test playground). nalexander will follow-up; see bug 1746678.

Depends on: 1746678

Minor updates to the script, based on real-world testing against the Store:
https://github.com/mozilla-releng/scriptworker-scripts/pull/454

I pushed to the dev branch yesterday, but k8s-image-pushmsixscript-python39 kept hanging (3600 s timeout). Resulting logs were a little scrambled but suggested problems pushing to docker hub. We noticed we were missing the pushmsixscript repo on hub.docker.com, but did not have permission to create a new repo there. jbuck (cloudops/sre services) created the repo and made these notes:

SRE:
 - Create a new repository on docker hub https://hub.docker.com/repository/create?namespace=mozilla 
 - Add the jenkinsv2 webhook to your new repository on the Webhooks tab: https://deployment-proxy.jenkinsv2.prod.mozaws.net/dockerhub 
 - Grant read/write permission to the mozillarelengservices team https://hub.docker.com/orgs/mozilla/teams/mozillarelengservices/permissions 
 - Run the Jenkins job to route webhooks to the correct Jenkins job: https://ops-master.jenkinsv2.prod.mozaws.net/job/add-dockerhub-relengworker-hooks/ 
 - You may need to approve running this script on https://ops-master.jenkinsv2.prod.mozaws.net/scriptApproval/ 

After this, a rerun of k8s-image-pushmsixscript-python39 succeeded -- all is well on the dev branch now.

(In reply to Geoff Brown [:gbrown] from comment #5)

With a special config file, I have been able to run pushmsixscript to upload nightly builds to the Firefox Nightly app on the Microsoft Store -- but those submissions fail, on the final, commit phase. I found I hit the same errors (name mismatches) if I upload manually via the Partner Centre; apparently this is a known issue, specific to our nightlies (which is currently our preferred test playground). nalexander will follow-up; see bug 1746678.

I built my own msix nightlies on try, with the name "corrected", went through considerable configuration in the Partner Centre, and eventually got that working, via the api - I'm no longer blocked on 1746678 (but leaving as a dependency, since it really should be resolved with Microsoft).

Testing against the Store revealed that we need to upload all architectures (32-bit, 64-bit) as part of one submission, leading to:

https://github.com/mozilla-releng/scriptworker-scripts/pull/457

Yesterday's scriptworker-scripts deployment put pushmsix on the production branch for the first time.
Then I landed https://github.com/mozilla-releng/k8s-autoscale/pull/123, automatically triggering a push to the k8s-autoscale dev branch.

Add taskcluster config for publishing msix archives on the Microsoft Store.

k8s-autoscale pushed to production today - no apparent issues.

https://scriptworker-scripts.readthedocs.io/en/latest/scriptworkers-autoscaling.html notes "for the change to have effect in the desired scriptworker(s), a new image for the latter needs to be pushed out to Docker". I pushed to production-pushmsixscript, that completed successfully, and I verified that https://hub.docker.com/r/mozilla/releng-pushmsixscript was updated.

Ryan, I may be able to enable automated publishing to the Microsoft Store next week. OK to proceed with that?
(And if so, that's for both Release and Beta, right?)

Flags: needinfo?(ryanvm)

I'm thinking we should start with Beta and see how it goes before we enable for Release.

Flags: needinfo?(ryanvm)

OK, will do.

Attachment #9260024 - Attachment description: Bug 1745203 - Add release-msix-push and release-secondary-msix-push tasks; r= → Bug 1745203 - Add release-msix-push task, run on beta only; r=
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b236557131cd
Add release-msix-push task, run on beta only; r=aki

leave-open until this is running on release.

Keywords: leave-open

Comment on attachment 9260024 [details]
Bug 1745203 - Add release-msix-push task, run on beta only; r=

Beta/Release Uplift Approval Request

  • User impact if declined: This patch is required on beta to enable automated pushes of beta releases to the Microsoft Store. There is no particular rush, but getting it in use sooner will help me complete this project, reducing manual effort for relman.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This change adds a taskcluster task, which nothing else depends on: If it fails, I think it can be safely ignored. Uploads to the Store will be restricted to the Beta application, and should be easy to un-do from the Partner Center.
  • String changes made/needed: none
Attachment #9260024 - Flags: approval-mozilla-beta?

Comment on attachment 9260024 [details]
Bug 1745203 - Add release-msix-push task, run on beta only; r=

Approved for 97.0b8, looking forward to seeing this in action!

Attachment #9260024 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Adding scopes like existing ones for flathub:firefox, for microsoftstore (pushmsix scriptworker).

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/ci/ci-configuration/rev/f9864f3db013
Add scopes for pushmsix / microsoftstore; r=hneiva

I somehow forgot to grant scopes, resulting in:
https://firefoxci.taskcluster-artifacts.net/Ce9KFE8uTza-y29N94dWMQ/0/public/logs/live_backing.log

[task 2022-01-25T21:07:39.720Z] Client ID task-client/Ce9KFE8uTza-y29N94dWMQ/0/on/us-east-1/i-0d6c077db8b3f0b25/until/1643145908.026 does not have sufficient scopes and is missing the following scopes:
[task 2022-01-25T21:07:39.720Z] 
[task 2022-01-25T21:07:39.720Z] ```
[task 2022-01-25T21:07:39.720Z] project:releng:microsoftstore:beta
[task 2022-01-25T21:07:39.720Z] ```
[task 2022-01-25T21:07:39.720Z] 
[task 2022-01-25T21:07:39.720Z] This request requires the client to satisfy the following scope expression:
[task 2022-01-25T21:07:39.720Z] 
[task 2022-01-25T21:07:39.720Z] ```
[task 2022-01-25T21:07:39.720Z] {
[task 2022-01-25T21:07:39.720Z]   "AllOf": [
[task 2022-01-25T21:07:39.720Z]     "project:releng:microsoftstore:beta",
[task 2022-01-25T21:07:39.720Z]     "queue:route:tc-treeherder.v2.mozilla-beta.63bf391388c3fbec63907b7442348d5fb778ceb4",
[task 2022-01-25T21:07:39.720Z]     "queue:create-task:project:none",
[task 2022-01-25T21:07:39.720Z]     "queue:scheduler-id:gecko-level-3",
[task 2022-01-25T21:07:39.720Z]     {
[task 2022-01-25T21:07:39.720Z]       "AnyOf": [
[task 2022-01-25T21:07:39.720Z]         "queue:create-task:highest:scriptworker-k8s/gecko-3-pushmsix",
[task 2022-01-25T21:07:39.720Z]         "queue:create-task:very-high:scriptworker-k8s/gecko-3-pushmsix",
[task 2022-01-25T21:07:39.720Z]         "queue:create-task:high:scriptworker-k8s/gecko-3-pushmsix"
[task 2022-01-25T21:07:39.720Z]       ]
[task 2022-01-25T21:07:39.720Z]     }
[task 2022-01-25T21:07:39.720Z]   ]
[task 2022-01-25T21:07:39.720Z] }
[task 2022-01-25T21:07:39.720Z] ```
[task 2022-01-25T21:07:39.721Z] 
[task 2022-01-25T21:07:39.721Z] ---
[task 2022-01-25T21:07:39.721Z] 
[task 2022-01-25T21:07:39.721Z] * method:     createTask
[task 2022-01-25T21:07:39.721Z] * errorCode:  InsufficientScopes
[task 2022-01-25T21:07:39.721Z] * statusCode: 403
[task 2022-01-25T21:07:39.721Z] * time:       2022-01-25T21:07:39.867Z

Leading to the urgent push above (comment 23, comment 24).

I initially added scopes for microsoftstore mirroring those for flathub.
However, Ryan noted that we would never ship msix RCs to beta:
The Store expects certain package names, so we can only ever push from
mozilla-beta to the Store beta channel, or from mozilla-release to the
Store release channel.

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/ci/ci-configuration/rev/808a44ada430
Modify scopes for pushmsix / microsoftstore; r=releng-reviewers,jcristau

The Beta submission took less than 3 minutes -- much, much faster than my testing against nightly (home internet uploads). Whew!

No longer depends on: 1746678
See Also: → 1746678

For msix pushes, set the payload's publishMode to Immediate for the
Beta channel, and Manual for all other channels (no other channels
are currently enabled, and Manual publishing is desired on Release).

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/51e1a655f92f
Set MSIX(push) task payload's publishMode; r=releng-reviewers,hneiva

Comment on attachment 9261780 [details]
Bug 1745203 - Set MSIX(push) task payload's publishMode; r=

Beta/Release Uplift Approval Request

  • User impact if declined: This patch adds the publishMode attribute to the MSIX(push) task payload and sets the mode to Immediate (publish immediately after upload and certification) for mozilla-beta. Currently MSIX(push) uses the setting previously used for the application, which is also currently Immediate, so there's no change in behavior, but this patch provides the requested flexibility of a task-based override.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Low risk: only affects the MSIX(push) task, only adds an optional parameter to the task payload.
  • String changes made/needed: none
Attachment #9261780 - Flags: approval-mozilla-beta?

Comment on attachment 9261780 [details]
Bug 1745203 - Set MSIX(push) task payload's publishMode; r=

We won't be shipping any more 97 betas this cycle, so this can just ride 98 to beta with next week's merge.

Attachment #9261780 - Flags: approval-mozilla-beta? → approval-mozilla-beta-

(In reply to Ryan VanderMeulen [:RyanVM] from comment #35)

We won't be shipping any more 97 betas this cycle, so this can just ride 98 to beta with next week's merge.

Which it did, and all's well: https://firefoxci.taskcluster-artifacts.net/L_4v6Qs7RzWysxmljwDYiw/0/public/logs/live_backing.log

However, Ryan noticed that, in the Partner Center, our Beta channel, after a successful update, continued to show the old version number, from the previous manual update. I investigated with tests on the Nightly channel, and it appears that the submission - although in all other ways apparently successful - does not update the package (the actual msix files).

I filed https://github.com/mozilla-releng/scriptworker-scripts/issues/474 and contacted our Microsoft Store support - they are investigating.

There's been no progress on https://github.com/mozilla-releng/scriptworker-scripts/issues/474; several attempts have been made to get help from Microsoft, but with no resolution and no response in recent weeks.

Until https://github.com/mozilla-releng/scriptworker-scripts/issues/474 is
resolved, I think it best to disable automatic pushes to the Microsoft Store.
Currently, if there's a manual submission in progress, the automated one will fail;
otherwise, the automated push will appear to succeed, but actually do nothing.

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/22dc1718983e
Stop running release-msix-push on beta; r=jcristau
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c337ab34689b
Re-enable release-msix-push on beta; r=releng-reviewers,aki

Push msix releases to the release app on the Microsoft Store. With recent
changes, the last several beta releases have been successful; let's expand
to the release channel.

:ryanvm - OK to enable msix pushes to the Store on the release channel soon? The last few betas have been successful. Also recall that Beta publishes immediately but Release is configured for manual publishing.

Flags: needinfo?(ryanvm)

Let's do it!

Flags: needinfo?(ryanvm)

:ahal - I'm having trouble implementing shipping-phase: by-.... See the errors in the tgdiff log at https://treeherder.mozilla.org/jobs?repo=try&revision=679282e00d5f4b1bcc325fe3a5ac234b163b29fa. Suggestions?

Flags: needinfo?(ahal)

If it's messy to do in the yaml (because the earlier release_deps transform relies on shipping-phase being set), maybe this in the release-msix-push transform?

+        # Override shipping-phase for release
+        if job["worker"]["publish-mode"] == "Manual":
+            job["shipping-phase"] = "promote"
Flags: needinfo?(ahal)
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/eca03cb43203
Run release-msix-push on release; r=releng-reviewers,gabriel
Attachment #9274335 - Attachment is obsolete: true
Attachment #9274335 - Attachment is obsolete: false
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/79f74f3462a0
For release, trigger on promote shipping-phase; r=jcristau
Keywords: leave-open
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Depends on: 1773380
Depends on: 1776696
Depends on: 1775165
Depends on: 1779435
See Also: → 1817658
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: