Closed Bug 1347635 Opened 5 years ago Closed 4 years ago

integrate Fennec within release automation

Categories

(Release Engineering :: Release Automation: Other, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: mtabara)

References

(Blocks 1 open bug)

Details

(Whiteboard: [releaseduty])

Attachments

(7 files, 4 obsolete files)

46 bytes, text/x-github-pull-request
mtabara
: review+
mtabara
: checked-in+
Details | Review
55 bytes, text/x-github-pull-request
mtabara
: review+
mtabara
: checked-in+
Details | Review
8.27 KB, patch
rail
: review+
mtabara
: checked-in+
Details | Diff | Splinter Review
17.69 KB, patch
rail
: review+
mtabara
: checked-in+
Details | Diff | Splinter Review
1.97 KB, patch
rail
: review+
mtabara
: checked-in+
Details | Diff | Splinter Review
56 bytes, text/x-github-pull-request
rail
: review+
mtabara
: checked-in+
Details | Review
50 bytes, text/x-github-pull-request
mtabara
: review+
mtabara
: review+
mtabara
: checked-in+
Details | Review
We discussed this during releaseduty today and decided to investigate how can we plug the existing releasetasks graph generation[1] within our relpro release runner. For that we need to:

1) make sure we update the Ship-it database via API to alter the "pending" state to "started" and mark it as completed from the submitted/reviewed state 

2) plug the fennec releasetasks scheduling part within either the release-runner bash script or within the release runner itself t ocall out the Fennec main graph

3) would be nice to make this work from within release-runner.py so that we also take into account the send-mail-to-drivers part.

I'll look more into details of how this is supposed to work and I'll try to come up with a solution.

[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fenenc-graph
Blocks: 1347637
This probably involves hacking release-runner.py:main [1]

Not sure if we should keep most of it as a generic main() and then call a separate set of functions in the |for release in releases| loop [2], or in the try/except [3] inside the for loop, or something else.

We probably want to keep this block [4] that marks as completed and emails r-d.

[1] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l293
[2] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l380
[3] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l413
[4] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l513
Priority: -- → P1
:mtabara, I was planning on taking a look at this last week, but never got a chance to.  This week I'll likely be focused on bug 1346422, but I can try to help with this.
(In reply to Aki Sasaki [:aki] from comment #2)
> :mtabara, I was planning on taking a look at this last week, but never got a
> chance to.  This week I'll likely be focused on bug 1346422, but I can try
> to help with this.

No worries about it, sounds like last week was eclectic from what I read from the email.
I still need to deploy balrogscript stuff from bug 1341352 which is something I owe, but other than that most of the Fennec graph stuff is done so I can focus on this. 

Thanks!
Am picking this up today with ETA set for EOW.
Testing and deploying this has been delayed until 54.0b2 given the current release week we're dealing with.
See Also: → 1374458
Interesting, found in triage that :sfraser was also working on this in bug 1338150. Need to check out his patches, see if there's any way I can reuse some of that work to declare victory here.
See Also: → 1338150
Summary: [tcmigration] bake new Fennec releasetasks graph scheduling within existing relpro release runner → [tcmigration] bake Fennec missing bbconfigs to fit within existing relpro release runner
Pending a decision on this this week to see what is the best approach to fixing this.
Summary: [tcmigration] bake Fennec missing bbconfigs to fit within existing relpro release runner → [tcmigration] add releaserunner2 to integrate Fennec within automation
Comment on attachment 8903082 [details] [diff] [review]
WIP - adding a releaserunner2 to call the existing graph genaration for Fennec.

Review of attachment 8903082 [details] [diff] [review]:
-----------------------------------------------------------------

Sweet! (code--)++

::: buildfarm/release/release-runner2.py
@@ +119,4 @@
>      rc = 0
>      for release in new_releases:
> +        try:
> +            relconfigs_root = '~/releasetasks/releasetasks/release_configs'

You can probably use the config for this variable. Hard coding is not cool :)

@@ +119,5 @@
>      rc = 0
>      for release in new_releases:
> +        try:
> +            relconfigs_root = '~/releasetasks/releasetasks/release_configs'
> +            relconfigs_tmpl = 'prod_{}_fennec_full_graph.yml'.format(release['branchShortName'])

The same here. "prod" means a lot! :)

@@ +123,5 @@
> +            relconfigs_tmpl = 'prod_{}_fennec_full_graph.yml'.format(release['branchShortName'])
> +            configs = '/'.join([relconfigs_root, relconfigs_tmpl])
> +
> +            cmd = [
> +                'python',

Maybe explicitly use the virtual environment set up by puppet? Could be done by moving "python" and some other args to release-runner.yml and generate them by puppet.

@@ +124,5 @@
> +            configs = '/'.join([relconfigs_root, relconfigs_tmpl])
> +
> +            cmd = [
> +                'python',
> +                'releasetasks_graph_gen.py',

Probably you either want to specify the full path or use cwd.

@@ +126,5 @@
> +            cmd = [
> +                'python',
> +                'releasetasks_graph_gen.py',
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',

Hmmmm. I don't think we update the "ini" config anymore. We should stop using it and even delete it. Also we should find a way to avoid hard coding the path (harder to run locally).

@@ +128,5 @@
> +                'releasetasks_graph_gen.py',
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),

missing "="?

@@ +129,5 @@
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),
> +                '--build-number {}'.format(release['buildNumber']),

and here

@@ +130,5 @@
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),
> +                '--build-number {}'.format(release['buildNumber']),
> +                '--mozilla-revision {}'.format(release["mozillaRevision"]),

and here. Or you can pass it as a separate list entry.
Attachment #8903082 - Flags: feedback?(rail) → feedback+
See Also: → 1395516
Whiteboard: [releaseduty]
Attached patch WIP tools diff. (obsolete) — Splinter Review
Attached patch WIP puppet diff (obsolete) — Splinter Review
Attachment #8903082 - Attachment is obsolete: true
Patches done. Leftovers:
* check if I need to tweak this as well relabs-puppet ??? via https://dxr.mozilla.org/build-central/search?q=releaserunner_root&redirect=true
* deploy this to my loaner to make sure puppet puts all the right pieces 
* once puppet is successful, chain this to bm83
* start testing staging releases
* start testing production releases.

I'm out for the day but am back tomorrow to compensate and continue.
Am getting closer.
@rail: 

Created a PR for ease of reviewing and automatic unit testing. Once green, I'll follow along the normal hg process.
@rail: 

Created a PR for ease of reviewing and automatic unit testing. Once green, I'll follow along the normal hg process.
Comment on attachment 8906967 [details] [review]
Add Fennec tools changes for release runner integration.

Approved by :rail on Github.
Landed on https://hg.mozilla.org/build/tools/rev/5ebc21009566ccbe00fdc58fae1da9375abf068a
Attachment #8906967 - Flags: review+
Attachment #8906967 - Flags: checked-in+
Comment on attachment 8906963 [details] [review]
Add Fennec puppet changes for release runner integration.

Approved by :rail on Github.
Will pin bm83 (staging) to my environment and start testing there.
Attachment #8906963 - Flags: review+
WIP for releasetasks. Will call for r? once I'm done testing on staging.
Taking a step further, I had to do the following to move pass yet another error:

`ln -s /home/cltbld/releasetasks/releasetasks /builds/releaserunner2/lib/python2.7/site-packages/releasetasks`
This is getting really weird. We must have forgotten to update some of the packages in releasrunner, otherwise I can't explain differences in the virtual environments being created.

Both releaserunner and releaserunner2 have the same definition (with all deps pinned). Yet, releaserunner venv has more packages installed than releaserunner2:

mihaitabara@mozspace:[]$ diff releaserunner releaserunner2 
12d11
< cffi==0.8.6
19d17
< httplib2==0.7.4
21d18
< oauth2==1.5.211
23d19
< pycparser==2.10
38d33
< treeherder-client==2.0.1
40c35
< zope.interface==4.0.2
---
> zope.interface==4.0.2

This could be explained by someone logged in and added packages via pip install and forgot to amend the init.pp afterward. However, even so, there are some packages that get installed under releaserunner, that are missing for releaserunner2. cffi 0.8.6 is present for releaserunner but is missing from pypi internal mirrors. This could be explained by someone logging in and installing the package from a different pypi mirror or simply it has been removed in the meantime.

Puppet skipped already installed packages, hence we never hit this for releaserunner, I suppose.
Adding cffi 0.8.6 to internal pypi solved the problem (pgpy was failing because of it). I made sure no other modules would be impacted by adding this missing package (as in, having unpinned cffi).

With this change into place, Fennec integration into releaserunner worked via this graphs IDs[1][2].
Now I need to figure out how to send that email.

[1]: G5tTNymeR1uzhCDe-YVnaQ
[2]: DprwiN0VRQOMOHB6nDLG0w
Summary: [tcmigration] add releaserunner2 to integrate Fennec within automation → integrate Fennec within release automation
Leftovers: integrate part of releasetasks_graph_gen.py logic within releaserunner2.py so that I can send the emails out in the same script. This would simplify some of the configs from puppet too. Will revisit this tomorrow morning with fresh brains.
Comment on attachment 8907165 [details] [review]
Add Fennec releasetasks changes for release runner integration.

r+'ed by :rail in PR.
merged in releasetasks, landed - https://github.com/mozilla-releng/releasetasks/commit/9d511512e83b34b510488f528c8a8c68762ee147
Attachment #8907165 - Flags: review+
Attachment #8907165 - Flags: checked-in+
This work is now completed. Successfully submitted a bunch of staging releases along with their corresponding emails.
Three repos need changes to make this happen: tools, releasetasks and puppet.

Some patches already landed:
a) releasetasks:
https://github.com/mozilla-releng/releasetasks/pull/278

b) part I in tools:
https://hg.mozilla.org/build/tools/rev/5ebc21009566
https://hg.mozilla.org/build/tools/rev/5f79def3870b
https://hg.mozilla.org/build/tools/rev/7bf0e0a599f5


Leftovers to land after :rail is blessing them is:
c) part II in tools: 
Attachment 8907884 [details] [diff]

d) puppet
Attachment 8907885 [details] [diff]

e) perform one-time symbolic link between releasetasks package and releasetasks folder. Something like bug 1347635 comment 20

After all this reviewed and deployed, we need to:
* unpin bm83 from my environment
* tweak moco-nodes.pp to: 
> install releaserunner2 as dev-fennec on bm83
* install releaserunner2 as prod-fennec on bm85
* adjust docs[1] to reflect those scenarions for graph2 only during RC weeks.

[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fennec-graph
Attachment #8907884 - Flags: review?(rail)
Attachment #8907885 - Flags: review?(rail)
Attachment #8907885 - Flags: review?(rail) → review+
Comment on attachment 8907885 [details] [diff] [review]
Add Fennec puppet changes for release runner integration.

https://hg.mozilla.org/build/puppet/rev/e65f9bfce3ee
Attachment #8907885 - Flags: checked-in+
Comment on attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.

Review of attachment 8907884 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/release/release-runner2.py
@@ +163,4 @@
>  
>              rr.update_status(release, 'Generating task graph')
>  
> +            task_group_id = gen_main(release_runner_config, releasetasks_kwargs, tc_config, options=fake_options)

ooh!
Attachment #8907884 - Flags: review?(rail) → review+
Comment on attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.

https://hg.mozilla.org/build/tools/rev/e7efa4f81256

I noticed we could also reuse the load_branch_config() function from the *_gen.py script so I importe that as well to avoid code duplication. I carry-over rail's r+ after successfully testing the change on staging with yet another release.
Attachment #8907884 - Flags: checked-in+
Attachment #8908716 - Flags: review?(rail) → review+
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #27)
> This work is now completed. Successfully submitted a bunch of staging
> releases along with their corresponding emails.
> Three repos need changes to make this happen: tools, releasetasks and puppet.
> 
> Some patches already landed:
> a) releasetasks:
> https://github.com/mozilla-releng/releasetasks/pull/278
> 
> b) part I in tools:
> https://hg.mozilla.org/build/tools/rev/5ebc21009566
> https://hg.mozilla.org/build/tools/rev/5f79def3870b
> https://hg.mozilla.org/build/tools/rev/7bf0e0a599f5
> 
> 
> Leftovers to land after :rail is blessing them is:
> c) part II in tools: 
> Attachment 8907884 [details] [diff]
> 
> d) puppet
> Attachment 8907885 [details] [diff]
> 
> e) perform one-time symbolic link between releasetasks package and
> releasetasks folder. Something like bug 1347635 comment 20
> 
> After all this reviewed and deployed, we need to:
> * unpin bm83 from my environment
> * tweak moco-nodes.pp to: 
> > install releaserunner2 as dev-fennec on bm83
> * install releaserunner2 as prod-fennec on bm85

All done.

I'll adjust the docs once we have a successful run of Fennec next week.
Depends on: 1390491
Blocks: 1401134
Blocks: 1390756
Attachment #8909719 - Flags: review?(kmoir)
Attachment #8909719 - Flags: review?(jlorenzo)
Comment on attachment 8909719 [details] [review]
Update releasewrrior with Fennec status quo.

r+'ed by :jlorenzo and :kmoir in the PR.
Attachment #8909719 - Flags: review?(kmoir)
Attachment #8909719 - Flags: review?(jlorenzo)
Attachment #8909719 - Flags: review+
Blocks: 1346465
Attachment #8909718 - Flags: review?(rail) → review+
Comment on attachment 8909718 [details] [review]
Add releasetasks support for Fennec release graph3

https://github.com/mozilla-releng/releasetasks/commit/14d569fe1f456642a97b63666ee334c85fbc8eb6

updated bm83, bm85.
Attachment #8909718 - Flags: checked-in+
It gives me great deal of satisfaction to finally close this bug.
All is good in the universe now.
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.