Closed Bug 1608794 Opened 2 months ago Closed 1 month ago

automated-betas shipitscript task is broken for Fennec

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(firefox-esr6873+ fixed)

RESOLVED FIXED
Tracking Status
firefox-esr68 73+ fixed

People

(Reporter: mtabara, Assigned: mtabara)

References

(Regression)

Details

Attachments

(3 files)

Today we were supposed to see automation run the first Fennec beta end to end. However, the job never started, nor we had any busted tasks on esr68.

Digging up I found the following:

  • digging up the cron job from 4AM UTC here
  • it successfully finds the cron for daily-release here
  • finds the daily-release so the cron.yml is good and then submits it PUT /queue/v1/task/JY-YY5MoRAO2ujTkSKlEig HTTP/1.1” 200 507
  • which runs successfully
[taskcluster 2020-01-13 04:03:23.626Z] === Task Finished ===
[taskcluster 2020-01-13 04:03:26.350Z] Successful task run with exit code: 0 completed in 50.607 seconds
  • the shipitscript maybe-release task is busted though here

Three main issues so far:

  • why didn't this show up on Treeherder? I need to check the symbols, I may have misconfigured it
  • how come it fails with requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://shipit-api.mozilla-releng.net/releases when in staging things worked like a charm

Add Treeherder support for automated-betas on esr68

Comment on attachment 9120430 [details]
Bug 1608794 - bugfixes for Fennec automated betas. r=jlund

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: We need this in order to add Treeherder symbols to the automated-beta for Fennec jobs. Otherwise, any bustage remains unnoticed.
  • User impact if declined: None
  • Fix Landed on Version: 68.5.0
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): No risk, I've tested via taskgraph diff and it only allows the Treeherder symbols to be added.
  • String or UUID changes made by this patch:
Attachment #9120430 - Flags: approval-mozilla-esr68?
Attachment #9120430 - Attachment description: Bug 1608794 - add treeherder support for maybe-release fennec. r=jlund → Bug 1608794 - bugfixes for Fennec automated betas. r=jlund

Comment on attachment 9120430 [details]
Bug 1608794 - bugfixes for Fennec automated betas. r=jlund

Fixes needed for automated Fennec betas. Approved for 68.5b3.

Attachment #9120430 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+

So we fixed automation but we're hitting yet again another issue.
We're triggering a release but not the correct one. I dug a bit and found the issue.

Currently, when we build the maybe-release shipitscript task, we're baking the version in the payload here taking it from the parameters. However, this task runs outside of the action tasks and it's not in the context of release promotion.

Whereas the decision task has these parameters:

target_tasks_method: daily_releases
tasks_for: cron
try_mode: null
try_options: null
try_task_config: null
version: 68.5.0esr

the action task has the following:

target_tasks_method: promote_fennec_beta
tasks_for: hg-push
try_mode: null
try_options: null
try_task_config: null
version: 68.5b2

This is because the AC comes from Ship-it so it already has all the right revisions baked in.
We'll have to bake some logic in-tree to compute and read from mobile/android/config/version_files/beta/version_display

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment on attachment 9120740 [details]
Bug 1608794 - pass proper version in automated Fennec betas. r=jlund

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fixing the version will allow us to properly trigger Fennec automated betas.
  • User impact if declined: None
  • Fix Landed on Version: 68.5b3
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): RelEng Automation change, no user impact.
  • String or UUID changes made by this patch:
Attachment #9120740 - Flags: approval-mozilla-esr68?

Comment on attachment 9120740 [details]
Bug 1608794 - pass proper version in automated Fennec betas. r=jlund

Approved for 68.5b4.

Attachment #9120740 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+

Thanks for landing the patch. I went on and disabled Fennec beta in Ship-it production and manually triggered the cron job (that usually will trigger a Fennec beta on Mondays at 4:00 AM).

2020-01-14 16:10:59,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): shipit-api.mozilla-releng.net:443
2020-01-14 16:11:00,003 - urllib3.connectionpool - DEBUG - https://shipit-api.mozilla-releng.net:443 "GET /disabled-products HTTP/1.1" 200 51
2020-01-14 16:11:00,006 - shipitscript.ship_actions - INFO - Releases are disabled. Silently exit ...

As expected the job silently failed because releases are disabled. Which is perfect!

Also the version looks good:

 "version": "68.5b4"

So next Monday we should have a good 68.5b4 Fennec beta triggered by automation.
Closing the bug, will re-open if weird behavior comes up.

Status: REOPENED → RESOLVED
Closed: 2 months ago1 month ago
Resolution: --- → FIXED
See Also: → 1610291
You need to log in before you can comment on or make changes to this bug.