releasetasks error while trying to generate 54.0 second graph (push-to-mirrors)

RESOLVED FIXED

Status

Release Engineering
Release Automation
RESOLVED FIXED
7 months ago
7 months ago

People

(Reporter: mtabara, Assigned: jlorenzo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

7 months ago
Traceback (most recent call last):
  File "releasetasks_graph_gen.py", line 228, in <module>
    main(release_runner_config, releasetasks_kwargs, tc_config)
  File "releasetasks_graph_gen.py", line 122, in main
    graph = make_task_graph_strict_kwargs(**kwargs)
  File "/builds/releaserunner/tools/lib/python/kickoff/__init__.py", line 326, in make_task_graph_strict_kwargs
    return make_task_graph(**kwargs)
  File "/home/cltbld/releasetasks/releasetasks/__init__.py", line 58, in make_task_graph
    return yaml.safe_load(template.render(**template_vars))
  File "/builds/releaserunner/lib/python2.7/site-packages/jinja2/environment.py", line 894, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/cltbld/releasetasks/releasetasks/templates/desktop/release_graph.yml.tmpl", line 172, in top-level template code
    {{ email_cdntest()|indent(4) }}
  File "/home/cltbld/releasetasks/releasetasks/templates/desktop/release_graph.yml.tmpl", line 170, in template
    {% include "emails/cdntest.yml.tmpl" %}
  File "/home/cltbld/releasetasks/releasetasks/templates/desktop/emails/cdntest.yml.tmpl", line 3, in top-level template code
    {% for channel in release_channels %}
TypeError: 'NoneType' object is not iterable

All these notifications have been added during 54.0 beta 2 so I suspect we actually never tested against the RC going to release. Investigating now ...
(Assignee)

Comment 1

7 months ago
Created attachment 8877186 [details] [review]
releasetasks PR

Good point! We're likely missing release_channels from the configs.
Assignee: mtabara → jlorenzo
Status: NEW → ASSIGNED
Attachment #8877186 - Flags: review?(mtabara)
(Assignee)

Comment 2

7 months ago
Fallout of bug 1348844
Blocks: 1348844
(Reporter)

Comment 3

7 months ago
So I was super surprised to see this error as in "how the hell did this work so far?"
Dug a bit and found the following:

1. second graphs reckon their `release_channels` by taking it from release configs, more specifically the `channels` field - https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/releasetasks_graph_gen.py#l80

2. that field is currently empty (hence the NoneType error) from all release configs - https://github.com/mozilla-releng/releasetasks/blob/master/releasetasks/release_configs/prod_mozilla-release_firefox_rc_graph_2.yml#L34

3. in releasetasks, all ocurrences of release_channels are as follows:

mihaitabara@mozspace:[master]~/work/mozilla/clones/git/releasetasks/releasetasks/templates$ grep -R 'release_channels' *
desktop/bb_update_verify.yml.tmpl:{% for channel in release_channels %}
desktop/emails/cdntest.yml.tmpl:{% for channel in release_channels %}
desktop/emails/final.yml.tmpl:{% for channel in release_channels %}
desktop/emails/localtest.yml.tmpl:{% for channel in release_channels %}
desktop/tc_update_verify.yml.tmpl:{% for channel in release_channels %}
desktop/updates.yml.tmpl:                channels: {{ release_channels | sort() | join(", ") }}

The emails stuff landed during 54.0b2, which means this is the first RC we are shipping to release channel after the emails have been added. All the other templates (namely bb_update_verify.yml.tmpl, tc_update_verify.yml.tmpl and updates.yml.tmpl) are locateed in the first graph which computes its `release_configs` value from by using branch configs + some logic (see https://hg.mozilla.org/build/tools/file/tip/buildfarm/release/release-runner.py#l481).

Therefore the emails are the first templates to use this variable in the second graph, hence the failing.

Conclusion:
* we need to update all `channels` under release_configs with their corresponding `final_verify_channels` value.

Johan already has a PR here - https://github.com/mozilla-releng/releasetasks/pull/248/files, we just need to update that value to existing `channels` instead of `release_channels`.

@rail: does this sound reasonable?
Flags: needinfo?(rail)
It makes sense, sure.
Flags: needinfo?(rail)
(Reporter)

Comment 6

7 months ago
Pulled the changes on both bm{83,85}. 
This worked for 54.0 and 52.2.0. 
All good.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.