1347635 - integrate Fennec within release automation

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P1)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

We discussed this during releaseduty today and decided to investigate how can we plug the existing releasetasks graph generation[1] within our relpro release runner. For that we need to:

1) make sure we update the Ship-it database via API to alter the "pending" state to "started" and mark it as completed from the submitted/reviewed state 

2) plug the fennec releasetasks scheduling part within either the release-runner bash script or within the release runner itself t ocall out the Fennec main graph

3) would be nice to make this work from within release-runner.py so that we also take into account the send-mail-to-drivers part.

I'll look more into details of how this is supposed to work and I'll try to come up with a solution.

[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fenenc-graph

Comment 1

[Aki Sasaki (not active)]

This probably involves hacking release-runner.py:main [1]

Not sure if we should keep most of it as a generic main() and then call a separate set of functions in the |for release in releases| loop [2], or in the try/except [3] inside the for loop, or something else.

We probably want to keep this block [4] that marks as completed and emails r-d.

[1] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l293
[2] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l380
[3] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l413
[4] https://hg.mozilla.org/build/tools/file/42fb84926c3a/buildfarm/release/release-runner.py#l513

Comment 2

[Aki Sasaki (not active)]

:mtabara, I was planning on taking a look at this last week, but never got a chance to.  This week I'll likely be focused on bug 1346422, but I can try to help with this.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Aki Sasaki [:aki] from comment #2)
> :mtabara, I was planning on taking a look at this last week, but never got a
> chance to.  This week I'll likely be focused on bug 1346422, but I can try
> to help with this.

No worries about it, sounds like last week was eclectic from what I read from the email.
I still need to deploy balrogscript stuff from bug 1341352 which is something I owe, but other than that most of the Fennec graph stuff is done so I can focus on this. 

Thanks!

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Am picking this up today with ETA set for EOW.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Testing and deploying this has been delayed until 54.0b2 given the current release week we're dealing with.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Interesting, found in triage that :sfraser was also working on this in bug 1338150. Need to check out his patches, see if there's any way I can reuse some of that work to declare victory here.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Pending a decision on this this week to see what is the best approach to fixing this.

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: WIP - adding a releaserunner2 to call the existing graph genaration for Fennec.

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8903082 [details] [diff] [review]
WIP - adding a releaserunner2 to call the existing graph genaration for Fennec.

Review of attachment 8903082 [details] [diff] [review]:
-----------------------------------------------------------------

Sweet! (code--)++

::: buildfarm/release/release-runner2.py
@@ +119,4 @@
>      rc = 0
>      for release in new_releases:
> +        try:
> +            relconfigs_root = '~/releasetasks/releasetasks/release_configs'

You can probably use the config for this variable. Hard coding is not cool :)

@@ +119,5 @@
>      rc = 0
>      for release in new_releases:
> +        try:
> +            relconfigs_root = '~/releasetasks/releasetasks/release_configs'
> +            relconfigs_tmpl = 'prod_{}_fennec_full_graph.yml'.format(release['branchShortName'])

The same here. "prod" means a lot! :)

@@ +123,5 @@
> +            relconfigs_tmpl = 'prod_{}_fennec_full_graph.yml'.format(release['branchShortName'])
> +            configs = '/'.join([relconfigs_root, relconfigs_tmpl])
> +
> +            cmd = [
> +                'python',

Maybe explicitly use the virtual environment set up by puppet? Could be done by moving "python" and some other args to release-runner.yml and generate them by puppet.

@@ +124,5 @@
> +            configs = '/'.join([relconfigs_root, relconfigs_tmpl])
> +
> +            cmd = [
> +                'python',
> +                'releasetasks_graph_gen.py',

Probably you either want to specify the full path or use cwd.

@@ +126,5 @@
> +            cmd = [
> +                'python',
> +                'releasetasks_graph_gen.py',
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',

Hmmmm. I don't think we update the "ini" config anymore. We should stop using it and even delete it. Also we should find a way to avoid hard coding the path (harder to run locally).

@@ +128,5 @@
> +                'releasetasks_graph_gen.py',
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),

missing "="?

@@ +129,5 @@
> +                '--release-runner-ini',
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),
> +                '--build-number {}'.format(release['buildNumber']),

and here

@@ +130,5 @@
> +                '../../../release-runner.ini',
> +                '--branch-and-product-config={}'.format(configs),
> +                '--version {}'.format(release['version']),
> +                '--build-number {}'.format(release['buildNumber']),
> +                '--mozilla-revision {}'.format(release["mozillaRevision"]),

and here. Or you can pass it as a separate list entry.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: WIP tools diff.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: WIP puppet diff

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Patches done. Leftovers:
* check if I need to tweak this as well relabs-puppet ??? via https://dxr.mozilla.org/build-central/search?q=releaserunner_root&redirect=true
* deploy this to my loaner to make sure puppet puts all the right pieces 
* once puppet is successful, chain this to bm83
* start testing staging releases
* start testing production releases.

I'm out for the day but am back tomorrow to compensate and continue.
Am getting closer.

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add Fennec puppet changes for release runner integration.

@rail: 

Created a PR for ease of reviewing and automatic unit testing. Once green, I'll follow along the normal hg process.

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add Fennec tools changes for release runner integration.

@rail: 

Created a PR for ease of reviewing and automatic unit testing. Once green, I'll follow along the normal hg process.

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8906967 [details] [review]
Add Fennec tools changes for release runner integration.

Approved by :rail on Github.
Landed on https://hg.mozilla.org/build/tools/rev/5ebc21009566ccbe00fdc58fae1da9375abf068a

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8906963 [details] [review]
Add Fennec puppet changes for release runner integration.

Approved by :rail on Github.
Will pin bm83 (staging) to my environment and start testing there.

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Bustage fix: https://hg.mozilla.org/build/tools/rev/5f79def3870b

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Yet another bustage fix: https://hg.mozilla.org/build/tools/rev/7bf0e0a599f550acfdc40e3eaee7a08719a9940c

Comment 19

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add Fennec releasetasks changes for release runner integration.

WIP for releasetasks. Will call for r? once I'm done testing on staging.

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

Taking a step further, I had to do the following to move pass yet another error:

`ln -s /home/cltbld/releasetasks/releasetasks /builds/releaserunner2/lib/python2.7/site-packages/releasetasks`

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

This is getting really weird. We must have forgotten to update some of the packages in releasrunner, otherwise I can't explain differences in the virtual environments being created.

Both releaserunner and releaserunner2 have the same definition (with all deps pinned). Yet, releaserunner venv has more packages installed than releaserunner2:

mihaitabara@mozspace:[]$ diff releaserunner releaserunner2 
12d11
< cffi==0.8.6
19d17
< httplib2==0.7.4
21d18
< oauth2==1.5.211
23d19
< pycparser==2.10
38d33
< treeherder-client==2.0.1
40c35
< zope.interface==4.0.2
---
> zope.interface==4.0.2

This could be explained by someone logged in and added packages via pip install and forgot to amend the init.pp afterward. However, even so, there are some packages that get installed under releaserunner, that are missing for releaserunner2. cffi 0.8.6 is present for releaserunner but is missing from pypi internal mirrors. This could be explained by someone logging in and installing the package from a different pypi mirror or simply it has been removed in the meantime.

Puppet skipped already installed packages, hence we never hit this for releaserunner, I suppose.

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

Adding cffi 0.8.6 to internal pypi solved the problem (pgpy was failing because of it). I made sure no other modules would be impacted by adding this missing package (as in, having unpinned cffi).

With this change into place, Fennec integration into releaserunner worked via this graphs IDs[1][2].
Now I need to figure out how to send that email.

[1]: G5tTNymeR1uzhCDe-YVnaQ
[2]: DprwiN0VRQOMOHB6nDLG0w

Comment 23

[Mihai Tabara [:mtabara]⌚️GMT]

Leftovers: integrate part of releasetasks_graph_gen.py logic within releaserunner2.py so that I can send the emails out in the same script. This would simplify some of the configs from puppet too. Will revisit this tomorrow morning with fresh brains.

Comment 24

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8907165 [details] [review]
Add Fennec releasetasks changes for release runner integration.

r+'ed by :rail in PR.
merged in releasetasks, landed - https://github.com/mozilla-releng/releasetasks/commit/9d511512e83b34b510488f528c8a8c68762ee147

Comment 25

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add more tools changes to refine the Fennec release runner integration.

Comment 26

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add Fennec puppet changes for release runner integration.

Comment 27

[Mihai Tabara [:mtabara]⌚️GMT]

This work is now completed. Successfully submitted a bunch of staging releases along with their corresponding emails.
Three repos need changes to make this happen: tools, releasetasks and puppet.

Some patches already landed:
a) releasetasks:
https://github.com/mozilla-releng/releasetasks/pull/278

b) part I in tools:
https://hg.mozilla.org/build/tools/rev/5ebc21009566
https://hg.mozilla.org/build/tools/rev/5f79def3870b
https://hg.mozilla.org/build/tools/rev/7bf0e0a599f5


Leftovers to land after :rail is blessing them is:
c) part II in tools: 
Attachment 8907884 [details] [diff]

d) puppet
Attachment 8907885 [details] [diff]

e) perform one-time symbolic link between releasetasks package and releasetasks folder. Something like bug 1347635 comment 20

After all this reviewed and deployed, we need to:
* unpin bm83 from my environment
* tweak moco-nodes.pp to: 
> install releaserunner2 as dev-fennec on bm83
* install releaserunner2 as prod-fennec on bm85
* adjust docs[1] to reflect those scenarions for graph2 only during RC weeks.

[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fennec-graph

Comment 28

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8907885 [details] [diff] [review]
Add Fennec puppet changes for release runner integration.

https://hg.mozilla.org/build/puppet/rev/e65f9bfce3ee

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.

Review of attachment 8907884 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/release/release-runner2.py
@@ +163,4 @@
>  
>              rr.update_status(release, 'Generating task graph')
>  
> +            task_group_id = gen_main(release_runner_config, releasetasks_kwargs, tc_config, options=fake_options)

ooh!

Comment 30

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8907884 [details] [diff] [review]
Add more tools changes to refine the Fennec release runner integration.

https://hg.mozilla.org/build/tools/rev/e7efa4f81256

I noticed we could also reuse the load_branch_config() function from the *_gen.py script so I importe that as well to avoid code duplication. I carry-over rail's r+ after successfully testing the change on staging with yet another release.

Comment 31

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Enable dev and production releaserunner2 for Fennec in puppet

Comment 32

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8908716 [details] [diff] [review]
Enable dev and production releaserunner2 for Fennec in puppet

https://hg.mozilla.org/build/puppet/rev/f2555994b8f71c8732a162697020d9e1cc595f5e
https://hg.mozilla.org/build/puppet/rev/94dd78f55061927126f1d37b45a3de37c00728c1

Comment 33

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #27)
> This work is now completed. Successfully submitted a bunch of staging
> releases along with their corresponding emails.
> Three repos need changes to make this happen: tools, releasetasks and puppet.
> 
> Some patches already landed:
> a) releasetasks:
> https://github.com/mozilla-releng/releasetasks/pull/278
> 
> b) part I in tools:
> https://hg.mozilla.org/build/tools/rev/5ebc21009566
> https://hg.mozilla.org/build/tools/rev/5f79def3870b
> https://hg.mozilla.org/build/tools/rev/7bf0e0a599f5
> 
> 
> Leftovers to land after :rail is blessing them is:
> c) part II in tools: 
> Attachment 8907884 [details] [diff]
> 
> d) puppet
> Attachment 8907885 [details] [diff]
> 
> e) perform one-time symbolic link between releasetasks package and
> releasetasks folder. Something like bug 1347635 comment 20
> 
> After all this reviewed and deployed, we need to:
> * unpin bm83 from my environment
> * tweak moco-nodes.pp to: 
> > install releaserunner2 as dev-fennec on bm83
> * install releaserunner2 as prod-fennec on bm85

All done.

I'll adjust the docs once we have a successful run of Fennec next week.

Comment 34

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add releasetasks support for Fennec release graph3

Comment 35

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Update releasewrrior with Fennec status quo.

Comment 36

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8909719 [details] [review]
Update releasewrrior with Fennec status quo.

r+'ed by :jlorenzo and :kmoir in the PR.

Comment 37

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8909719 [details] [review]
Update releasewrrior with Fennec status quo.

https://github.com/mozilla/releasewarrior/commit/412e0e0b52ddd524acf7555b86c3daa83c1f61b6

Comment 38

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8909718 [details] [review]
Add releasetasks support for Fennec release graph3

https://github.com/mozilla-releng/releasetasks/commit/14d569fe1f456642a97b63666ee334c85fbc8eb6

updated bm83, bm85.

Comment 39

[Mihai Tabara [:mtabara]⌚️GMT]

It gives me great deal of satisfaction to finally close this bug.
All is good in the universe now.