Closed Bug 1557461 Opened 5 years ago Closed 5 years ago

Partner repack problems with 68.0b8 and 67.0.2 - Intermittent Halting on failure "Can't determine what taskID to retrieve target.zip from"

Categories

(Release Engineering :: Release Automation: Other, defect, P1)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
defect

Tracking

(firefox-esr60 unaffected, firefox67 unaffected, firefox67.0.1 fixed, firefox68 fixed, firefox69 unaffected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox67 --- unaffected
firefox67.0.1 --- fixed
firefox68 --- fixed
firefox69 --- unaffected

People

(Reporter: intermittent-bug-filer, Assigned: nthomas)

References

Details

(Keywords: intermittent-failure, regression)

Attachments

(1 file)

Bug 1557461 - limit locales for partner/EME-free repacks to overlap of requested and enabled locales, r=sfraser
47 bytes, text/x-phabricator-request
Details | Review

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=250382208&repo=mozilla-beta
Full log: https://queue.taskcluster.net/v1/task/dy3O3QWpQHuItyyGrfwFeg/runs/0/artifacts/public/logs/live_backing.log

[task 2019-06-06T15:15:23.896Z] 15:15:23 INFO - 2019-06-06 15:15:23,896 - DEBUG - in /builds/worker/artifacts/mozilla-EME-free/mozilla-EME-free/an/working
[task 2019-06-06T15:15:23.952Z] 15:15:23 INFO - adding: firefox/ (stored 0%)
[task 2019-06-06T15:15:23.952Z] 15:15:23 INFO - adding: firefox/distribution/ (stored 0%)
[task 2019-06-06T15:15:23.952Z] 15:15:23 INFO - adding: firefox/distribution/distribution.ini (deflated 28%)
[task 2019-06-06T15:15:23.962Z] 15:15:23 INFO - 2019-06-06 15:15:23,962 - INFO - Done repacking win64 an build target.zip
[task 2019-06-06T15:15:23.962Z] 15:15:23 INFO - 2019-06-06 15:15:23,962 - INFO - Downloading from https://queue.taskcluster.net/v1/task/Pemd6H_yTWqtR_6XUzwxUA/artifacts/public/build/ar/target.zip
[task 2019-06-06T15:15:23.962Z] 15:15:23 INFO - 2019-06-06 15:15:23,962 - INFO - To: target.zip
[task 2019-06-06T15:15:23.962Z] 15:15:23 INFO - 2019-06-06 15:15:23,962 - INFO - CWD: /builds/worker/workspace/build/scripts/original_builds/68.0b8/build1/win64/ar
[task 2019-06-06T15:15:25.186Z] 15:15:25 INFO - 2019-06-06 15:15:25,186 - INFO - Downloading from https://queue.taskcluster.net/v1/task/Pemd6H_yTWqtR_6XUzwxUA/artifacts/public/build/ar/setup.exe
[task 2019-06-06T15:15:25.186Z] 15:15:25 INFO - 2019-06-06 15:15:25,186 - INFO - To: setup.exe
[task 2019-06-06T15:15:25.186Z] 15:15:25 INFO - 2019-06-06 15:15:25,186 - INFO - CWD: /builds/worker/workspace/build/scripts/original_builds/68.0b8/build1/win64/ar
[task 2019-06-06T15:15:25.761Z] 15:15:25 INFO - 2019-06-06 15:15:25,761 - INFO - Repacking win64 ar build target.zip
[task 2019-06-06T15:15:25.814Z] 15:15:25 INFO - 2019-06-06 15:15:25,814 - INFO - Copying files into firefox
[task 2019-06-06T15:15:25.815Z] 15:15:25 INFO - 2019-06-06 15:15:25,815 - DEBUG - Executing zip -r target.zip firefox
[task 2019-06-06T15:15:25.815Z] 15:15:25 INFO - 2019-06-06 15:15:25,815 - DEBUG - in /builds/worker/artifacts/mozilla-EME-free/mozilla-EME-free/ar/working
[task 2019-06-06T15:15:25.870Z] 15:15:25 INFO - adding: firefox/ (stored 0%)
[task 2019-06-06T15:15:25.870Z] 15:15:25 INFO - adding: firefox/distribution/ (stored 0%)
[task 2019-06-06T15:15:25.870Z] 15:15:25 INFO - adding: firefox/distribution/distribution.ini (deflated 28%)
[task 2019-06-06T15:15:25.879Z] 15:15:25 INFO - 2019-06-06 15:15:25,879 - INFO - Done repacking win64 ar build target.zip
[task 2019-06-06T15:15:25.880Z] 15:15:25 INFO - 2019-06-06 15:15:25,880 - CRITICAL - Can't determine what taskID to retrieve public/build/as/target.zip from
[task 2019-06-06T15:15:25.885Z] 15:15:25 ERROR - Return code: 1
[task 2019-06-06T15:15:25.885Z] 15:15:25 ERROR - 1 not in success codes: [0]
[task 2019-06-06T15:15:25.885Z] 15:15:25 WARNING - setting return code to 2
[task 2019-06-06T15:15:25.885Z] 15:15:25 FATAL - Halting on failure while running ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/virtualenvs/init/bin/python', 'tc-partner-repacks.py', '-v', '68.0b8', '-n', '1', '--platform', 'win64', '--taskid', 'EGXWFFu3Q5WDNf2EKYKOZA', '--taskid', 'BHB4SP5XQnmnV3CPQOUr8w', '--taskid', 'dmcU0RmUQ9WsTTUkCIrPIQ', '--taskid', 'X473X-hRZONtg_URipRXA', '--taskid', 'dzfrllswSbWwKwWN-8n1ig', '--taskid', 'apstEYQ4S0e_nuwFsuNtCw', '--taskid', 'Ym1-hECNRG-tpm4cYcbnVQ', '--taskid', 'TD4bpqULSj-_nvRkbcGHxQ', '--taskid', 'KpMyETgNQWK9NeGTGQLcnw', '--taskid', 'UewsI5QaTHmHZ2zVfqnWkA', '--taskid', 'J8ZjTWoaR_etrcZGkbw7hQ', '--taskid', 'XUsntbk0TOO3kgC6xBOsHA', '--taskid', 'Pemd6H_yTWqtR_6XUzwxUA', '--taskid', 'dus38c2QTCiq0brGfYdwYg', '--taskid', 'E8fvycWjSW-LQW1xrC7e3g', '--taskid', 'I5cEX5r5ReScgYKZMp8Rrg', '--taskid', 'a8CE-IY4SvirgWi3FhQ_Nw', '--taskid', 'cd96TfbvTCuA0fBU1axafg', '--taskid', 'Vyjrwk-dQLqr4EMZdsf56Q', '--taskid', 'XQw7-tLXSTOre89GOIZc_A']
[task 2019-06-06T15:15:25.885Z] 15:15:25 FATAL - Running post_fatal callback...
[task 2019-06-06T15:15:25.886Z] 15:15:25 FATAL - Exiting 2
[task 2019-06-06T15:15:25.886Z] 15:15:25 INFO - [mozharness: 2019-06-06 15:15:25.886006Z] Finished repack step (failed)

In Firefox 68 we have some locale changes:

  • bn-BD & bn-IN replaced by bn
  • as, en-ZA, mai, ml, or have been deprecated

These locales are used by

  • EME-free - all listed locales
  • softonic/softonic-003 - bn-BD, en-ZA
  • softonic/softonic-004 - en-ZA

The partner repack manifests and configs don't support different locales for beta and release, so we ended up with

  • failures in 68.0b8 when the upstream build for bn-BD couldn't be found
  • to unblock 68.0b8 a config change to EME-free, and a decision to skip partner repacks this time
  • failures in 67.0.2 build2 where the bn upstream can't be found

My plan for fixing this is to

  • unblock 67.0.2 by reverting the EME-free config change and rerun the jobs
  • before 68.0b9 & 67.0.3
    • write config patches which make the locales a superset of beta and release
    • enable the limit-locales functionality we normally use for staging, so that the flow in the decision task will be
      • look up EME-free/partner config from github
      • filter away any locales which aren't present in the l10n-changesets.json file
      • in the partner repack job set --limit-locale=<ab-CD> for each valid locale
    • the repack jobs will then ignore any locales in the config
  • finally once 68.0 has gone to build on release we can disable the limit-locale again, and clean up the locale configs. We will also need to check MozillaOnline's ESR config.
Assignee: nobody → nthomas
Status: NEW → ASSIGNED
Component: General → Release Automation: Other
Priority: P5 → P1
QA Contact: catlee → sfraser
See Also: → 1535299, 1538733
Summary: Intermittent Halting on failure while running ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', 'tc-partner-repacks.py', '-v', '68.0b8', '-n', [...] after Can't determine what taskID to retrieve target.zip from → Partner repack problems with 68.0b8 and 67.0.2 - Intermittent Halting on failure "Can't determine what taskID to retrieve target.zip from"

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)

My plan for fixing this is to

  • unblock 67.0.2 by reverting the EME-free config change and rerun the jobs

Revert, jobs are running.

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #1)

  • before 68.0b9 & 67.0.3
    • write config patches which make the locales a superset of beta and release

https://github.com/mozilla-partners/mozilla-EME-free/pull/13
https://github.com/mozilla-partners/softonic/pull/9

  • enable the limit-locales functionality we normally use for staging, so that the flow in the decision task will be

https://phabricator.services.mozilla.com/D34304

(In reply to Nick Thomas [:nthomas] (UTC+12) from comment #4)

https://github.com/mozilla-partners/mozilla-EME-free/pull/13
https://github.com/mozilla-partners/softonic/pull/9

Merged.

https://phabricator.services.mozilla.com/D34304

https://hg.mozilla.org/releases/mozilla-beta/rev/039662b2b76c2d7c0ae4ca537d4501a107c4b2f7
https://hg.mozilla.org/releases/mozilla-release/rev/5e57a549906f9df302b0dd979ef80dd8829fa75d

Status

  • ready for 68.0b9, or any 67.0.2 build3
  • there will unwanted partner jobs on 67.0.2 build2 push and ship, due to the changed config (bug 1496530). Feel free to cancel the partner-repack tasks and ignore the downstreams.
status-firefox67: --- → unaffected
status-firefox67.0.1: --- → fixed
status-firefox68: --- → fixed
status-firefox69: --- → unaffected
status-firefox-esr60: --- → unaffected

Once 68.0 ships we can fix up the locale lists so they only have the current locales. No gecko changes needed because passing all the --limit-locales didn't land in 69.0.

Removing the deprecated locales:
https://github.com/mozilla-partners/mozilla-EME-free/pull/14
https://github.com/mozilla-partners/softonic/pull/10

Winston, FYI a couple of changes to softonic distributions for 68.0:

  • softonic-003 - bn-BD becomes bn, en-ZA deprecated
  • softonic-004 - en-ZA deprecated
Flags: needinfo?(wbowden)
Flags: needinfo?(wbowden)

Is this fixed?

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: