1345232 - fennec 53.0b1 issues (tc fennec relpro)

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, enhancement, P1)

Description

[Aki Sasaki (not active)]

- mtabara fixed the hook: https://bugzilla.mozilla.org/show_bug.cgi?id=1343649#c28
- android l10n is busted; we probably need to rename release_mozilla-beta_android_api_15.py to mozilla-beta_android_api_15.py

probably more to come.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

So in order to trigger the first graph via hook, we added the necessary hook[1] on Friday but apparently we forgot to also add the corresponding role. While attempting to trigger the hook today for Fennec 53.0b1 it failed with the following message:

Missing at least one of the following scopes:

    queue:create-task:aws-provisioner-v1/gecko-decision
    queue:define-task:aws-provisioner-v1/gecko-decision
    queue:task-group-id:gecko-level-3/H2yiqYp_RXqI1ZYcAfbQrA
    queue:schedule-task:gecko-level-3/H2yiqYp_RXqI1ZYcAfbQrA/H2yiqYp_RXqI1ZYcAfbQrA

Only scopes present are:
    have assume:hook-id:project-releng/candidates-fennec-beta

Not sure I got this right but in my understanding each hook has to have a corresponding role in order to be able to trigger tasks. Since we didn't, it default to a "fictional" role which it's only scope is itself "assume:role ...". I created the role[2] associated with the hook and granted it only one scope "assume:project:releng:release:mozilla-beta" which pretty much expands to our needs. I vimdiffed the differences against the dev one [3] and things look the same way (except the obvious jamun vs "mozilla-beta" differences).

Triggering the graph worked - https://tools.taskcluster.net/task-group-inspector/#/SejnPXCNRMSw1UdcP4VazA?_k=84xa9s

[1]: https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-beta
[2]: https://tools.taskcluster.net/auth/roles/#hook-id:project-releng%252fcandidates-fennec-beta
[3]: https://tools.taskcluster.net/auth/roles/#hook-id:project-releng%252fcandidates-fennec-dev

Comment 2

[Aki Sasaki (not active)]

Attachment: 7c8025a7a6d5

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 8844602 [details] [diff] [review]
7c8025a7a6d5

This won't work, jamun single locale mozconfig is pretty different from m-b.

Comment 4

[Aki Sasaki (not active)]

Attachment: fix fennec single locale releases?

This may be better, or may break.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

The latest graph is https://tools.taskcluster.net/task-group-inspector/#/Xl3872IGQr63maJWcb1YYg?_k=nqknzz

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Aki landed another bustage fix at: https://hg.mozilla.org/releases/mozilla-beta/rev/a4a198383638
Triggered yet another graph https://tools.taskcluster.net/task-group-inspector/#/GQ13KLhkTqigqCh-bxtLFQ?_k=z6bd3g

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #6)
> Aki landed another bustage fix at:
> https://hg.mozilla.org/releases/mozilla-beta/rev/a4a198383638
> Triggered yet another graph
> https://tools.taskcluster.net/task-group-inspector/#/GQ13KLhkTqigqCh-
> bxtLFQ?_k=z6bd3g

We now have officially a 53.0b1-build3 under candidates shipped by Taskcluster! \o/

http://archive.mozilla.org/pub/mobile/candidates/53.0b1-candidates/build3/

Comment 8

[Aki Sasaki (not active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c55c0ec84a7275becab95d39edfd3a7b42119023
bug 1345232 - fix android single locale release mh configs. r=mtabara a=release

Comment 9

[Aki Sasaki (not active)]

l10n mozharness config changes landed on m-i and m-a, so we don't hit this again in 54b1 or 55b1.

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c55c0ec84a72

Comment 11

[Jordan Lund (:jlund)]

mihai: releasewarrior still has this release as open. I believe it's live on play store but I've lost context of what else is needed to be done or fixed in this new world.

1) can I close this in releasewarrior? https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-53.0b1.md#build-3

2) How should releaseduty handle fennec betas over next few releaseses? basically follow: How should releaseduty handle it that is different from the old buildbot way

3) should we expect any bugs with b2? Will we have b1/b2 overlap issues if relman gtb today/monday

4) how do you propose we handle the noop shipit entries that are stuck permapending under "reviewed"?

apologies for the question dump

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Jordan Lund (:jlund) from comment #11)
> mihai: releasewarrior still has this release as open. I believe it's live on
> play store but I've lost context of what else is needed to be done or fixed
> in this new world.
> 
> 1) can I close this in releasewarrior?
> https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-
> 53.0b1.md#build-3

kill it with fire - we just shipped it oficially! \o/
More comments https://bugzilla.mozilla.org/show_bug.cgi?id=1343990#c17

reminder to myself to answer the rest of the questions tomorrow.

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Before I forget, I'll dump some thoughts now and will revisit later.

mtabara> on a side note, I think we need to fix the no-op Ship-it thing. we'll be running again on the same scenario with beta2: Ship-it no-op, release in "pending", trigger hook or via releasetasks, mark the release in ShipIt DB, schedule releasetasks graph2
00:57:33 <mtabara> I'm thinking that for betas at least we can chain the graphs together once I graph1 is tested too. and push_to_mirrors should be set to false so that we have a human decision task there. It's the most of the automation we could do at first glance. Therefore we'd have: Ship-It no-op, pending release, releasetasks schedule graph, wait for QE sign-off, resolve human decision task, profit! \o/.

Main drawbacks:
* no emails to keep QE updated to what's happening
* busfactor: we need :rail or :nthomas or :bhearsum with access to Ship-It DB to s,"pending","started" for that particular release
* might be other I cannot think of right now.

P.S. Need to investigate how notifications work, we might at least plug some parts of the graph at least. 
P.P.S. Need to document stuff in releasewarrior

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

Okay, I'll rewind and re-answer to :jlund's question here as a sum-up.

(In reply to Jordan Lund (:jlund) from comment #11)
> mihai: releasewarrior still has this release as open. I believe it's live on
> play store but I've lost context of what else is needed to be done or fixed
> in this new world.
> 
> 1) can I close this in releasewarrior?
> https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-
> 53.0b1.md#build-3

Yes, we've done the rest of the bits on Friday afternoon (sans the bouncer aliases and publishing to Balrog, neither mandatory for now)
 
> 2) How should releaseduty handle fennec betas over next few releaseses?
> basically follow: How should releaseduty handle it that is different from
> the old buildbot way

I still need to meld graph1 into releasetasks so that we have one big graph to rule them all that releaseduty can easily use. I'll do that this week so until then, unfortunately we still need to deal with some manual steps. I'll document all of that. I'm around too this week, so I'll help :jlorenzo and :aki with that.

> 3) should we expect any bugs with b2? Will we have b1/b2 overlap issues if
> relman gtb today/monday

I expect no issues. 53.0b1 is now shipped so hopefully no overlap issues
 
> 4) how do you propose we handle the noop shipit entries that are stuck
> permapending under "reviewed"?

This is indeed a problem. Current release-runner (relpro) logic isn't capable yet to handle the Fennec so that means we're short on a number of things:

* no emails to keep QE updated to what's happening
* busfactor: we need :rail or :nthomas or :bhearsum with access to Ship-It DB to s,"pending","started" for that particular release
* no notifications for tasks happening in the graphs generated by decision task / releasetasks / hook

We need to discuss all these aspects so I'll add some item in the trello board for Wednesday. 

> apologies for the question dump

No worries, these are all very good questions so ++ for raising these good points!

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: I think we can tweak `default` branch in tools to handle Fennec as well, at least to the step where the releases get to the 'started' point. With the releasetasks graph(s) into place, we could automate the rest of it so that releaseduty's job gets easier.

Comment 16

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.