Closed Bug 1823542 Opened 3 years ago Closed 3 years ago

Intermittent Exception TaskVerificationError: push to Store aborted: pending submission found

Categories

(Release Engineering :: Release Automation, defect)

Product:
Release Engineering
Component:
Release Automation
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: NarcisB, Assigned: gbrown)

References

Details

live_backing.log
Push
Log snippet:

2023-03-20 21:30:25,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): manage.devcenter.microsoft.com:443
2023-03-20 21:30:31,661 - urllib3.connectionpool - DEBUG - https://manage.devcenter.microsoft.com:443 "GET /v1.0/my/applications/9NZVDKPMR9RD HTTP/1.1" 200 567
2023-03-20 21:30:31,662 - pushmsixscript.microsoft_store - INFO - response code: 200
2023-03-20 21:30:31,662 - pushmsixscript.microsoft_store - INFO - response body: {"id":"9NZVDKPMR9RD","primaryName":"Firefox","PackageFamilyName":"Mozilla.Firefox_n80bbvh6b1yt2","packageIdentityName":"Mozilla.Firefox","publisherName":"CN=082E9164-EE6C-4EC8-B62C-441FAE7BEFA1","firstPublishedDate":"2021-10-05T21:38:45.1973879Z","lastPublishedApplicationSubmission":{"id":"1152921505696041084","resourceLocation":"applications/9NZVDKPMR9RD/submissions/1152921505696041084"},"pendingApplicationSubmission":{"id":"1152921505696065913","resourceLocation":"applications/9NZVDKPMR9RD/submissions/1152921505696065913"},"hasAdvancedListingPermission":true}
2023-03-20 21:30:31,662 - pushmsixscript.microsoft_store - ERROR - There is a pending submission for this application on the Microsoft Store. Wait for the pending submission to complete, or delete the pending submission. Then retry this task.
2023-03-20 21:30:31,664 - scriptworker_client.client - ERROR - Failed to run async_main; exiting 3
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/scriptworker_client/client.py", line 178, in _handle_asyncio_loop
    await async_main(config, task)
  File "/app/lib/python3.9/site-packages/pushmsixscript/script.py", line 25, in async_main
    microsoft_store.push(config, msix_file_paths, channel, publish_mode)
  File "/app/lib/python3.9/site-packages/pushmsixscript/microsoft_store.py", line 64, in push
    _push_to_store(config, channel, msix_file_paths, publish_mode, access_token)
  File "/app/lib/python3.9/site-packages/pushmsixscript/microsoft_store.py", line 106, in _push_to_store
    _check_for_pending_submission(config, channel, session, headers)
  File "/app/lib/python3.9/site-packages/pushmsixscript/microsoft_store.py", line 142, in _check_for_pending_submission
    raise TaskVerificationError("push to Store aborted: pending submission found")
scriptworker_client.exceptions.TaskVerificationError: push to Store aborted: pending submission found
exit code: 3

Discussing this on #release-duty today. dmeehan found a pending submission on the store, deleted it, and the push succeeded on retry.

Curiously, the pending submission was from 2023-03-16 and we don't see any push tasks on that day on mozilla-release.

gcloud logs show pushmsix running around 2023-03-16T22:56:38; it pushes firefox with buildid 20230316192655.

Is it possibly https://firefox-ci-tc.services.mozilla.com/tasks/fOZ5SV3ZQnWBUWuik3-IIQ ? (but that's on beta).

Assignee: nobody → gbrown

(In reply to Geoff Brown [:gbrown] from comment #2)

Is it possibly https://firefox-ci-tc.services.mozilla.com/tasks/fOZ5SV3ZQnWBUWuik3-IIQ ? (but that's on beta).

That's the task I am seeing in gcloud. It is running on mozilla-beta, with parent SNqZpwgcRymrvMGcs0nWEw. The task definition confirms that the channel is beta, and the log, https://firefoxci.taskcluster-artifacts.net/fOZ5SV3ZQnWBUWuik3-IIQ/0/public/logs/live_backing.log confirms that the upload is going to the beta application.

2023-03-16 22:56:33,267 - pushmsixscript.script - INFO - found msix at /app/workdir/cot/cLLY6Jd1S5KBZow-98wIZw/public/build/target.store.msix
2023-03-16 22:56:35,779 - pushmsixscript.manifest - INFO - Archive version: 112.0.3.0
2023-03-16 22:56:35,780 - pushmsixscript.manifest - INFO - Firefox version: 112.0
2023-03-16 22:56:35,780 - pushmsixscript.manifest - INFO - Firefox build id: 20230316192655
...
2023-03-16 22:57:07,512 - urllib3.connectionpool - DEBUG - https://manage.devcenter.microsoft.com:443 "PUT /v1.0/my/applications/9NZW26FRNDLN/submissions/1152921505696067672 HTTP/1.1" 200 4508

Comment 0 confirms that the conflict / pending submission which triggered this bug report was on the release application -- so that should be independent of the beta submission found in comment 3.

On the Store, newer submissions have now been made to both beta and release applications, and I don't know of a way to look at submission history from the Store perspective.

I can't think of anything else to check here. There is a mystery as to where the pending submission on the release application originated. Manual submission, Store hiccup, ... ?

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX

(In reply to Intermittent Failures Robot from comment #8)

4 failures in 4077 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

In this incident, failures were on Beta. An earlier submission had been uploaded to the Store correctly, but failed certification so remained Pending.

Release Management manually deleted the pending submission in the Store and re-ran the failed push(MSIX) task.

Component: Release Automation: Uploading → Release Automation
You need to log in before you can comment on or make changes to this bug.