1608794 - automated-betas shipitscript task is broken for Fennec

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Today we were supposed to see automation run the first Fennec beta end to end. However, the job never started, nor we had any busted tasks on esr68.

Digging up I found the following:

• digging up the cron job from 4AM UTC here
• it successfully finds the cron for daily-release here
• finds the daily-release so the cron.yml is good and then submits it PUT /queue/v1/task/JY-YY5MoRAO2ujTkSKlEig HTTP/1.1” 200 507
• which runs successfully

[taskcluster 2020-01-13 04:03:23.626Z] === Task Finished ===
[taskcluster 2020-01-13 04:03:26.350Z] Successful task run with exit code: 0 completed in 50.607 seconds

• the shipitscript maybe-release task is busted though here

Three main issues so far:

• why didn't this show up on Treeherder? I need to check the symbols, I may have misconfigured it
• how come it fails with requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://shipit-api.mozilla-releng.net/releases when in staging things worked like a charm

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1608794 - bugfixes for Fennec automated betas. r=jlund

Add Treeherder support for automated-betas on esr68

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9120430 [details]
Bug 1608794 - bugfixes for Fennec automated betas. r=jlund

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: We need this in order to add Treeherder symbols to the automated-beta for Fennec jobs. Otherwise, any bustage remains unnoticed.
• User impact if declined: None
• Fix Landed on Version: 68.5.0
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): No risk, I've tested via taskgraph diff and it only allows the Treeherder symbols to be added.
• String or UUID changes made by this patch:

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9120430 [details]
Bug 1608794 - bugfixes for Fennec automated betas. r=jlund

Fixes needed for automated Fennec betas. Approved for 68.5b3.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/ba35b4e926ed8da2b7796272948d9d86cfec8f8d

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1608794 - add manual hook for esr68 daily-releases. r=jlund

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

So we fixed automation but we're hitting yet again another issue.
We're triggering a release but not the correct one. I dug a bit and found the issue.

Currently, when we build the maybe-release shipitscript task, we're baking the version in the payload here taking it from the parameters. However, this task runs outside of the action tasks and it's not in the context of release promotion.

Whereas the decision task has these parameters:

target_tasks_method: daily_releases
tasks_for: cron
try_mode: null
try_options: null
try_task_config: null
version: 68.5.0esr

the action task has the following:

target_tasks_method: promote_fennec_beta
tasks_for: hg-push
try_mode: null
try_options: null
try_task_config: null
version: 68.5b2

This is because the AC comes from Ship-it so it already has all the right revisions baked in.
We'll have to bake some logic in-tree to compute and read from mobile/android/config/version_files/beta/version_display

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1608794 - pass proper version in automated Fennec betas. r=jlund

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9120740 [details]
Bug 1608794 - pass proper version in automated Fennec betas. r=jlund

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fixing the version will allow us to properly trigger Fennec automated betas.
• User impact if declined: None
• Fix Landed on Version: 68.5b3
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): RelEng Automation change, no user impact.
• String or UUID changes made by this patch:

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9120740 [details]
Bug 1608794 - pass proper version in automated Fennec betas. r=jlund

Approved for 68.5b4.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/84675375273f

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Thanks for landing the patch. I went on and disabled Fennec beta in Ship-it production and manually triggered the cron job (that usually will trigger a Fennec beta on Mondays at 4:00 AM).

2020-01-14 16:10:59,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): shipit-api.mozilla-releng.net:443
2020-01-14 16:11:00,003 - urllib3.connectionpool - DEBUG - https://shipit-api.mozilla-releng.net:443 "GET /disabled-products HTTP/1.1" 200 51
2020-01-14 16:11:00,006 - shipitscript.ship_actions - INFO - Releases are disabled. Silently exit ...

As expected the job silently failed because releases are disabled. Which is perfect!

Also the version looks good:

 "version": "68.5b4"

So next Monday we should have a good 68.5b4 Fennec beta triggered by automation.
Closing the bug, will re-open if weird behavior comes up.