fennec 53.0b1 issues (tc fennec relpro)

RESOLVED FIXED

Status

Release Engineering
Releases
P1
normal
RESOLVED FIXED
8 months ago
7 months ago

People

(Reporter: aki, Unassigned)

Tracking

({leave-open})

unspecified
leave-open

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

8 months ago
- mtabara fixed the hook: https://bugzilla.mozilla.org/show_bug.cgi?id=1343649#c28
- android l10n is busted; we probably need to rename release_mozilla-beta_android_api_15.py to mozilla-beta_android_api_15.py

probably more to come.
So in order to trigger the first graph via hook, we added the necessary hook[1] on Friday but apparently we forgot to also add the corresponding role. While attempting to trigger the hook today for Fennec 53.0b1 it failed with the following message:

Missing at least one of the following scopes:

    queue:create-task:aws-provisioner-v1/gecko-decision
    queue:define-task:aws-provisioner-v1/gecko-decision
    queue:task-group-id:gecko-level-3/H2yiqYp_RXqI1ZYcAfbQrA
    queue:schedule-task:gecko-level-3/H2yiqYp_RXqI1ZYcAfbQrA/H2yiqYp_RXqI1ZYcAfbQrA

Only scopes present are:
    have assume:hook-id:project-releng/candidates-fennec-beta

Not sure I got this right but in my understanding each hook has to have a corresponding role in order to be able to trigger tasks. Since we didn't, it default to a "fictional" role which it's only scope is itself "assume:role ...". I created the role[2] associated with the hook and granted it only one scope "assume:project:releng:release:mozilla-beta" which pretty much expands to our needs. I vimdiffed the differences against the dev one [3] and things look the same way (except the obvious jamun vs "mozilla-beta" differences).

Triggering the graph worked - https://tools.taskcluster.net/task-group-inspector/#/SejnPXCNRMSw1UdcP4VazA?_k=84xa9s

[1]: https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-beta
[2]: https://tools.taskcluster.net/auth/roles/#hook-id:project-releng%252fcandidates-fennec-beta
[3]: https://tools.taskcluster.net/auth/roles/#hook-id:project-releng%252fcandidates-fennec-dev
(Reporter)

Comment 2

8 months ago
Created attachment 8844602 [details] [diff] [review]
7c8025a7a6d5
Attachment #8844602 - Flags: review?(mtabara)
Attachment #8844602 - Flags: review?(mtabara) → review+
(Reporter)

Comment 3

8 months ago
Comment on attachment 8844602 [details] [diff] [review]
7c8025a7a6d5

This won't work, jamun single locale mozconfig is pretty different from m-b.
Attachment #8844602 - Attachment is obsolete: true
Attachment #8844602 - Flags: review+ → review?
(Reporter)

Updated

8 months ago
Attachment #8844602 - Flags: review?
(Reporter)

Comment 4

8 months ago
Created attachment 8844614 [details] [diff] [review]
fix fennec single locale releases?

This may be better, or may break.
Attachment #8844614 - Flags: review?(mtabara)
Attachment #8844614 - Flags: review?(mtabara) → review+
The latest graph is https://tools.taskcluster.net/task-group-inspector/#/Xl3872IGQr63maJWcb1YYg?_k=nqknzz
Aki landed another bustage fix at: https://hg.mozilla.org/releases/mozilla-beta/rev/a4a198383638
Triggered yet another graph https://tools.taskcluster.net/task-group-inspector/#/GQ13KLhkTqigqCh-bxtLFQ?_k=z6bd3g
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #6)
> Aki landed another bustage fix at:
> https://hg.mozilla.org/releases/mozilla-beta/rev/a4a198383638
> Triggered yet another graph
> https://tools.taskcluster.net/task-group-inspector/#/GQ13KLhkTqigqCh-
> bxtLFQ?_k=z6bd3g

We now have officially a 53.0b1-build3 under candidates shipped by Taskcluster! \o/

http://archive.mozilla.org/pub/mobile/candidates/53.0b1-candidates/build3/
(Reporter)

Comment 8

8 months ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/c55c0ec84a7275becab95d39edfd3a7b42119023
bug 1345232 - fix android single locale release mh configs. r=mtabara a=release
(Reporter)

Comment 9

8 months ago
l10n mozharness config changes landed on m-i and m-a, so we don't hit this again in 54b1 or 55b1.
Priority: -- → P1

Comment 10

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/c55c0ec84a72
Status: NEW → RESOLVED
Last Resolved: 8 months ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Keywords: leave-open

Comment 11

7 months ago
mihai: releasewarrior still has this release as open. I believe it's live on play store but I've lost context of what else is needed to be done or fixed in this new world.

1) can I close this in releasewarrior? https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-53.0b1.md#build-3

2) How should releaseduty handle fennec betas over next few releaseses? basically follow: How should releaseduty handle it that is different from the old buildbot way

3) should we expect any bugs with b2? Will we have b1/b2 overlap issues if relman gtb today/monday

4) how do you propose we handle the noop shipit entries that are stuck permapending under "reviewed"?

apologies for the question dump
(In reply to Jordan Lund (:jlund) from comment #11)
> mihai: releasewarrior still has this release as open. I believe it's live on
> play store but I've lost context of what else is needed to be done or fixed
> in this new world.
> 
> 1) can I close this in releasewarrior?
> https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-
> 53.0b1.md#build-3

kill it with fire - we just shipped it oficially! \o/
More comments https://bugzilla.mozilla.org/show_bug.cgi?id=1343990#c17

reminder to myself to answer the rest of the questions tomorrow.
Flags: needinfo?(mtabara)
Before I forget, I'll dump some thoughts now and will revisit later.

mtabara> on a side note, I think we need to fix the no-op Ship-it thing. we'll be running again on the same scenario with beta2: Ship-it no-op, release in "pending", trigger hook or via releasetasks, mark the release in ShipIt DB, schedule releasetasks graph2
00:57:33 <mtabara> I'm thinking that for betas at least we can chain the graphs together once I graph1 is tested too. and push_to_mirrors should be set to false so that we have a human decision task there. It's the most of the automation we could do at first glance. Therefore we'd have: Ship-It no-op, pending release, releasetasks schedule graph, wait for QE sign-off, resolve human decision task, profit! \o/.

Main drawbacks:
* no emails to keep QE updated to what's happening
* busfactor: we need :rail or :nthomas or :bhearsum with access to Ship-It DB to s,"pending","started" for that particular release
* might be other I cannot think of right now.

P.S. Need to investigate how notifications work, we might at least plug some parts of the graph at least. 
P.P.S. Need to document stuff in releasewarrior
Okay, I'll rewind and re-answer to :jlund's question here as a sum-up.

(In reply to Jordan Lund (:jlund) from comment #11)
> mihai: releasewarrior still has this release as open. I believe it's live on
> play store but I've lost context of what else is needed to be done or fixed
> in this new world.
> 
> 1) can I close this in releasewarrior?
> https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-
> 53.0b1.md#build-3

Yes, we've done the rest of the bits on Friday afternoon (sans the bouncer aliases and publishing to Balrog, neither mandatory for now)
 
> 2) How should releaseduty handle fennec betas over next few releaseses?
> basically follow: How should releaseduty handle it that is different from
> the old buildbot way

I still need to meld graph1 into releasetasks so that we have one big graph to rule them all that releaseduty can easily use. I'll do that this week so until then, unfortunately we still need to deal with some manual steps. I'll document all of that. I'm around too this week, so I'll help :jlorenzo and :aki with that.

> 3) should we expect any bugs with b2? Will we have b1/b2 overlap issues if
> relman gtb today/monday

I expect no issues. 53.0b1 is now shipped so hopefully no overlap issues
 
> 4) how do you propose we handle the noop shipit entries that are stuck
> permapending under "reviewed"?

This is indeed a problem. Current release-runner (relpro) logic isn't capable yet to handle the Fennec so that means we're short on a number of things:

* no emails to keep QE updated to what's happening
* busfactor: we need :rail or :nthomas or :bhearsum with access to Ship-It DB to s,"pending","started" for that particular release
* no notifications for tasks happening in the graphs generated by decision task / releasetasks / hook

We need to discuss all these aspects so I'll add some item in the trello board for Wednesday. 

> apologies for the question dump

No worries, these are all very good questions so ++ for raising these good points!
Flags: needinfo?(mtabara)
Note to self: I think we can tweak `default` branch in tools to handle Fennec as well, at least to the step where the releases get to the 'started' point. With the releasetasks graph(s) into place, we could automate the rest of it so that releaseduty's job gets easier.
See Also: → bug 1346892
You need to log in before you can comment on or make changes to this bug.