integrate Fennec within release automation

RESOLVED FIXED

Status

Release Engineering
Release Automation
P1
normal
RESOLVED FIXED
7 months ago
28 days ago

People

(Reporter: mtabara, Assigned: mtabara)

Tracking

(Blocks: 4 bugs)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [releaseduty])

Attachments

(7 attachments, 4 obsolete attachments)

46 bytes, text/x-github-pull-request
mtabara
: review+
mtabara
: checked-in+
Details | Review | Splinter Review
55 bytes, text/x-github-pull-request
mtabara
: review+
mtabara
: checked-in+
Details | Review | Splinter Review
8.27 KB, patch
rail
: review+
mtabara
: checked-in+
Details | Diff | Splinter Review
17.69 KB, patch
rail
: review+
mtabara
: checked-in+
Details | Diff | Splinter Review
1.97 KB, patch
rail
: review+
mtabara
: checked-in+
Details | Diff | Splinter Review
56 bytes, text/x-github-pull-request
rail
: review+
mtabara
: checked-in+
Details | Review | Splinter Review
50 bytes, text/x-github-pull-request
mtabara
: review+
mtabara
: review+
mtabara
: checked-in+
Details | Review | Splinter Review
(Assignee)

Description

7 months ago
We discussed this during releaseduty today and decided to investigate how can we plug the existing releasetasks graph generation[1] within our relpro release runner. For that we need to:

1) make sure we update the Ship-it database via API to alter the "pending" state to "started" and mark it as completed from the submitted/reviewed state 

2) plug the fennec releasetasks scheduling part within either the release-runner bash script or within the release runner itself t ocall out the Fennec main graph

3) would be nice to make this work from within release-runner.py so that we also take into account the send-mail-to-drivers part.

I'll look more into details of how this is supposed to work and I'll try to come up with a solution.

[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fenenc-graph
(Assignee)

Updated

7 months ago
Blocks: 1347637

Comment 1

7 months ago
This probably involves hacking release-runner.py:main [1]

Not sure if we should keep most of it as a generic main() and then call a separate set of functions in the |for release in releases| loop [2], or in the try/except [3] inside the for loop, or something else.

We probably want to keep this block [4] that marks as completed and emails r-d.

[1] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l293
[2] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l380
[3] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l413
[4] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l513
Priority: -- → P1

Comment 2

7 months ago
:mtabara, I was planning on taking a look at this last week, but never got a chance to.  This week I'll likely be focused on bug 1346422, but I can try to help with this.
(Assignee)

Comment 3

7 months ago
(In reply to Aki Sasaki [:aki] from comment #2)
> :mtabara, I was planning on taking a look at this last week, but never got a
> chance to.  This week I'll likely be focused on bug 1346422, but I can try
> to help with this.

No worries about it, sounds like last week was eclectic from what I read from the email.
I still need to deploy balrogscript stuff from bug 1341352 which is something I owe, but other than that most of the Fennec graph stuff is done so I can focus on this. 

Thanks!
(Assignee)

Comment 4

7 months ago
Am picking this up today with ETA set for EOW.
(Assignee)

Comment 5

6 months ago
Testing and deploying this has been delayed until 54.0b2 given the current release week we're dealing with.
(Assignee)

Updated

4 months ago
See Also: → bug 1374458
(Assignee)

Comment 6

3 months ago
Interesting, found in triage that :sfraser was also working on this in bug 1338150. Need to check out his patches, see if there's any way I can reuse some of that work to declare victory here.
See Also: → bug 1338150
(Assignee)

Updated

2 months ago
Summary: [tcmigration] bake new Fennec releasetasks graph scheduling within existing relpro release runner → [tcmigration] bake Fennec missing bbconfigs to fit within existing relpro release runner
(Assignee)

Comment 7

2 months ago
Pending a decision on this this week to see what is the best approach to fixing this.
(Assignee)

Comment 8

2 months ago
Created attachment 8903082 [details] [diff] [review]
WIP - adding a releaserunner2 to call the existing graph genaration for Fennec.
Attachment #8903082 - Flags: feedback?(rail)
(Assignee)

Updated

2 months ago
Summary: [tcmigration] bake Fennec missing bbconfigs to fit within existing relpro release runner → [tcmigration] add releaserunner2 to integrate Fennec within automation
Comment on attachment 8903082 [details] [diff] [review]
WIP - adding a releaserunner2 to call the existing graph genaration for Fennec.

Review of attachment 8903082 [details] [diff] [review]:
-----------------------------------------------------------------

Sweet! (code--)++

::: buildfarm/release/release-runner2.py
@@ +119,4 @@
>      rc = 0
>      for release in new_releases:
> +        try:
> +            relconfigs_root = '~/releasetasks/releasetasks/release_configs'

You can probably use the config for this variable. Hard coding is not cool :)

@@ +119,5 @@
>      rc = 0
>      for release in new_releases:
> +        try:
> +            relconfigs_root = '~/releasetasks/releasetasks/release_configs'
> +            relconfigs_tmpl = 'prod_{}_fennec_full_graph.yml'.format(release['branchShortName'])

The same here. "prod" means a lot! :)

@@ +123,5 @@
> +            relconfigs_tmpl = 'prod_{}_fennec_full_graph.yml'.format(release['branchShortName'])
> +            configs = '/'.join([relconfigs_root, relconfigs_tmpl])
> +
> +            cmd = [
> +                'python',

Maybe explicitly use the virtual environment set up by puppet? Could be done by moving "python" and some other args to release-runner.yml and generate them by puppet.

@@ +124,5 @@
> +            configs = '/'.join([relconfigs_root, relconfigs_tmpl])
> +
> +            cmd = [
> +                'python',
> +                'releasetasks_graph_gen.py',

Probably you either want to specify the full path or use cwd.

@@ +126,5 @@
> +            cmd = [
> +                'python',
> +                'releasetasks_graph_gen.py',
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',

Hmmmm. I don't think we update the "ini" config anymore. We should stop using it and even delete it. Also we should find a way to avoid hard coding the path (harder to run locally).

@@ +128,5 @@
> +                'releasetasks_graph_gen.py',
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),

missing "="?

@@ +129,5 @@
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),
> +                '--build-number {}'.format(release['buildNumber']),

and here

@@ +130,5 @@
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),
> +                '--build-number {}'.format(release['buildNumber']),
> +                '--mozilla-revision {}'.format(release["mozillaRevision"]),

and here. Or you can pass it as a separate list entry.
Attachment #8903082 - Flags: feedback?(rail) → feedback+
See Also: → bug 1395516
(Assignee)

Updated

2 months ago
Whiteboard: [releaseduty]
(Assignee)

Comment 10

2 months ago
Created attachment 8903528 [details] [diff] [review]
WIP tools diff.
(Assignee)

Comment 11

2 months ago
Created attachment 8903529 [details] [diff] [review]
WIP puppet diff
Attachment #8903082 - Attachment is obsolete: true
(Assignee)

Comment 12

2 months ago
Patches done. Leftovers:
* check if I need to tweak this as well relabs-puppet ??? via https://dxr.mozilla.org/build-central/search?q=releaserunner_root&redirect=true
* deploy this to my loaner to make sure puppet puts all the right pieces 
* once puppet is successful, chain this to bm83
* start testing staging releases
* start testing production releases.

I'm out for the day but am back tomorrow to compensate and continue.
Am getting closer.
(Assignee)

Comment 13

a month ago
Created attachment 8906963 [details] [review]
Add Fennec puppet changes for release runner integration.

@rail: 

Created a PR for ease of reviewing and automatic unit testing. Once green, I'll follow along the normal hg process.
(Assignee)

Comment 14

a month ago
Created attachment 8906967 [details] [review]
Add Fennec tools changes for release runner integration.

@rail: 

Created a PR for ease of reviewing and automatic unit testing. Once green, I'll follow along the normal hg process.
(Assignee)

Comment 15

a month ago
Comment on attachment 8906967 [details] [review]
Add Fennec tools changes for release runner integration.

Approved by :rail on Github.
Landed on https://hg.mozilla.org/build/tools/rev/5ebc21009566ccbe00fdc58fae1da9375abf068a
Attachment #8906967 - Flags: review+
Attachment #8906967 - Flags: checked-in+
(Assignee)

Comment 16

a month ago
Comment on attachment 8906963 [details] [review]
Add Fennec puppet changes for release runner integration.

Approved by :rail on Github.
Will pin bm83 (staging) to my environment and start testing there.
Attachment #8906963 - Flags: review+
(Assignee)

Comment 17

a month ago
Bustage fix: https://hg.mozilla.org/build/tools/rev/5f79def3870b
(Assignee)

Comment 18

a month ago
Yet another bustage fix: https://hg.mozilla.org/build/tools/rev/7bf0e0a599f550acfdc40e3eaee7a08719a9940c
(Assignee)

Comment 19

a month ago
Created attachment 8907165 [details] [review]
Add Fennec releasetasks changes for release runner integration.

WIP for releasetasks. Will call for r? once I'm done testing on staging.
(Assignee)

Comment 20

a month ago
Taking a step further, I had to do the following to move pass yet another error:

`ln -s /home/cltbld/releasetasks/releasetasks /builds/releaserunner2/lib/python2.7/site-packages/releasetasks`
(Assignee)

Comment 21

a month ago
This is getting really weird. We must have forgotten to update some of the packages in releasrunner, otherwise I can't explain differences in the virtual environments being created.

Both releaserunner and releaserunner2 have the same definition (with all deps pinned). Yet, releaserunner venv has more packages installed than releaserunner2:

mihaitabara@mozspace:[]$ diff releaserunner releaserunner2 
12d11
< cffi==0.8.6
19d17
< httplib2==0.7.4
21d18
< oauth2==1.5.211
23d19
< pycparser==2.10
38d33
< treeherder-client==2.0.1
40c35
< zope.interface==4.0.2
---
> zope.interface==4.0.2

This could be explained by someone logged in and added packages via pip install and forgot to amend the init.pp afterward. However, even so, there are some packages that get installed under releaserunner, that are missing for releaserunner2. cffi 0.8.6 is present for releaserunner but is missing from pypi internal mirrors. This could be explained by someone logging in and installing the package from a different pypi mirror or simply it has been removed in the meantime.

Puppet skipped already installed packages, hence we never hit this for releaserunner, I suppose.
(Assignee)

Comment 22

a month ago
Adding cffi 0.8.6 to internal pypi solved the problem (pgpy was failing because of it). I made sure no other modules would be impacted by adding this missing package (as in, having unpinned cffi).

With this change into place, Fennec integration into releaserunner worked via this graphs IDs[1][2].
Now I need to figure out how to send that email.

[1]: G5tTNymeR1uzhCDe-YVnaQ
[2]: DprwiN0VRQOMOHB6nDLG0w
(Assignee)

Updated

a month ago
Summary: [tcmigration] add releaserunner2 to integrate Fennec within automation → integrate Fennec within release automation
(Assignee)

Comment 23

a month ago
Leftovers: integrate part of releasetasks_graph_gen.py logic within releaserunner2.py so that I can send the emails out in the same script. This would simplify some of the configs from puppet too. Will revisit this tomorrow morning with fresh brains.
(Assignee)

Comment 24

a month ago
Comment on attachment 8907165 [details] [review]
Add Fennec releasetasks changes for release runner integration.

r+'ed by :rail in PR.
merged in releasetasks, landed - https://github.com/mozilla-releng/releasetasks/commit/9d511512e83b34b510488f528c8a8c68762ee147
Attachment #8907165 - Flags: review+
Attachment #8907165 - Flags: checked-in+
(Assignee)

Comment 25

a month ago
Created attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.
Attachment #8903528 - Attachment is obsolete: true
Attachment #8903529 - Attachment is obsolete: true
(Assignee)

Comment 26

a month ago
Created attachment 8907885 [details] [diff] [review]
Add Fennec puppet changes for release runner integration.
Attachment #8906963 - Attachment is obsolete: true
(Assignee)

Comment 27

a month ago
This work is now completed. Successfully submitted a bunch of staging releases along with their corresponding emails.
Three repos need changes to make this happen: tools, releasetasks and puppet.

Some patches already landed:
a) releasetasks:
https://github.com/mozilla-releng/releasetasks/pull/278

b) part I in tools:
https://hg.mozilla.org/build/tools/rev/5ebc21009566
https://hg.mozilla.org/build/tools/rev/5f79def3870b
https://hg.mozilla.org/build/tools/rev/7bf0e0a599f5


Leftovers to land after :rail is blessing them is:
c) part II in tools: 
Attachment 8907884 [details] [diff]

d) puppet
Attachment 8907885 [details] [diff]

e) perform one-time symbolic link between releasetasks package and releasetasks folder. Something like bug 1347635 comment 20

After all this reviewed and deployed, we need to:
* unpin bm83 from my environment
* tweak moco-nodes.pp to: 
> install releaserunner2 as dev-fennec on bm83
* install releaserunner2 as prod-fennec on bm85
* adjust docs[1] to reflect those scenarions for graph2 only during RC weeks.

[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fennec-graph
(Assignee)

Updated

a month ago
Attachment #8907884 - Flags: review?(rail)
(Assignee)

Updated

a month ago
Attachment #8907885 - Flags: review?(rail)
Attachment #8907885 - Flags: review?(rail) → review+
(Assignee)

Comment 28

a month ago
Comment on attachment 8907885 [details] [diff] [review]
Add Fennec puppet changes for release runner integration.

https://hg.mozilla.org/build/puppet/rev/e65f9bfce3ee
Attachment #8907885 - Flags: checked-in+
Comment on attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.

Review of attachment 8907884 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/release/release-runner2.py
@@ +163,4 @@
>  
>              rr.update_status(release, 'Generating task graph')
>  
> +            task_group_id = gen_main(release_runner_config, releasetasks_kwargs, tc_config, options=fake_options)

ooh!
Attachment #8907884 - Flags: review?(rail) → review+
(Assignee)

Comment 30

a month ago
Comment on attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.

https://hg.mozilla.org/build/tools/rev/e7efa4f81256

I noticed we could also reuse the load_branch_config() function from the *_gen.py script so I importe that as well to avoid code duplication. I carry-over rail's r+ after successfully testing the change on staging with yet another release.
Attachment #8907884 - Flags: checked-in+
(Assignee)

Comment 31

a month ago
Created attachment 8908716 [details] [diff] [review]
Enable dev and production releaserunner2 for Fennec in puppet
Attachment #8908716 - Flags: review?(rail)
Attachment #8908716 - Flags: review?(rail) → review+
(Assignee)

Comment 32

a month ago
Comment on attachment 8908716 [details] [diff] [review]
Enable dev and production releaserunner2 for Fennec in puppet

https://hg.mozilla.org/build/puppet/rev/f2555994b8f71c8732a162697020d9e1cc595f5e
https://hg.mozilla.org/build/puppet/rev/94dd78f55061927126f1d37b45a3de37c00728c1
Attachment #8908716 - Flags: checked-in+
(Assignee)

Comment 33

a month ago
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #27)
> This work is now completed. Successfully submitted a bunch of staging
> releases along with their corresponding emails.
> Three repos need changes to make this happen: tools, releasetasks and puppet.
> 
> Some patches already landed:
> a) releasetasks:
> https://github.com/mozilla-releng/releasetasks/pull/278
> 
> b) part I in tools:
> https://hg.mozilla.org/build/tools/rev/5ebc21009566
> https://hg.mozilla.org/build/tools/rev/5f79def3870b
> https://hg.mozilla.org/build/tools/rev/7bf0e0a599f5
> 
> 
> Leftovers to land after :rail is blessing them is:
> c) part II in tools: 
> Attachment 8907884 [details] [diff]
> 
> d) puppet
> Attachment 8907885 [details] [diff]
> 
> e) perform one-time symbolic link between releasetasks package and
> releasetasks folder. Something like bug 1347635 comment 20
> 
> After all this reviewed and deployed, we need to:
> * unpin bm83 from my environment
> * tweak moco-nodes.pp to: 
> > install releaserunner2 as dev-fennec on bm83
> * install releaserunner2 as prod-fennec on bm85

All done.

I'll adjust the docs once we have a successful run of Fennec next week.
(Assignee)

Updated

29 days ago
Depends on: 1390491
(Assignee)

Updated

29 days ago
Blocks: 1351280
(Assignee)

Updated

29 days ago
Blocks: 1401134
(Assignee)

Updated

29 days ago
Blocks: 1390756
(Assignee)

Comment 34

29 days ago
Created attachment 8909718 [details] [review]
Add releasetasks support for Fennec release graph3
Attachment #8909718 - Flags: review?(rail)
(Assignee)

Comment 35

29 days ago
Created attachment 8909719 [details] [review]
Update releasewrrior with Fennec status quo.
Attachment #8909719 - Flags: review?(kmoir)
Attachment #8909719 - Flags: review?(jlorenzo)
(Assignee)

Comment 36

29 days ago
Comment on attachment 8909719 [details] [review]
Update releasewrrior with Fennec status quo.

r+'ed by :jlorenzo and :kmoir in the PR.
Attachment #8909719 - Flags: review?(kmoir)
Attachment #8909719 - Flags: review?(jlorenzo)
Attachment #8909719 - Flags: review+
(Assignee)

Comment 37

29 days ago
Comment on attachment 8909719 [details] [review]
Update releasewrrior with Fennec status quo.

https://github.com/mozilla/releasewarrior/commit/412e0e0b52ddd524acf7555b86c3daa83c1f61b6
Attachment #8909719 - Flags: checked-in+
(Assignee)

Updated

29 days ago
Blocks: 1346465
Attachment #8909718 - Flags: review?(rail) → review+
(Assignee)

Comment 38

28 days ago
Comment on attachment 8909718 [details] [review]
Add releasetasks support for Fennec release graph3

https://github.com/mozilla-releng/releasetasks/commit/14d569fe1f456642a97b63666ee334c85fbc8eb6

updated bm83, bm85.
Attachment #8909718 - Flags: checked-in+
(Assignee)

Comment 39

28 days ago
It gives me great deal of satisfaction to finally close this bug.
All is good in the universe now.
Status: ASSIGNED → RESOLVED
Last Resolved: 28 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.