1506319 - BouncerCheck failures do not result in a task failure for staging releases

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Bouncer, enhancement)

Description

[Justin Wood (:Callek)]

So on a try staging run I got a green bouncer check task for Firefox, (didn't reference .msi which I'll need to check again)

However these all failed due to the staging s3 bucket...

https://tools.taskcluster.net/groups/N1g_zvypTIeQS26e_Gw7TA/tasks/S-AmFF3tT8uI83DJ6rT-aw/details

The resulting task was green though, which makes me worried other production failures would also result in a green result.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Thanks for filing this - this is staging specfic error and it's not affecting production. 
However, we should definitely look into fixing this to ease the staging testing.

I'm adding this the [releaseduty] whiteboard item to keep this under the radar.

Task fails at this[1] line because the `https://ftp.stage.mozaws.net` prefixed locations are not defined slightly above here[2]. All of those locations are production specific, hence this task doesn't fail for production.

The easiest solution here is to move those locations from the script itself to the configs, which are per release / product / tree.

If we moved them under the configs[3], we could point all the `dev_bouncer_*` ones to the ftp.stage .. prefixes and enable this task run green in staging as well.

There's likely other ways to solve this but the prefix is defined in beetmoverscript configurations (here[4]) and the bouncer submission are done in the promotion phase via the bouncer submission. So when the bouncer check in the push phase queries staging bouncer, gets these values instead.

Another possiblity could be to move this outside of mozharness and bake it in the taskcluster configs which can be shared with all the other bouncer logic. 

[1]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/bouncer_check.py#l130
[2]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/bouncer_check.py#l110
[3]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases
[4]: https://github.com/mozilla-releng/build-puppet/blob/master/modules/beetmover_scriptworker/templates/script_config.json.erb#L83

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

At second glance and after some IRC chat, it might be worth understanding why we're simply warning there and not throwing an exception. IIRC this script is reused in both release-bouncer-check but also the cron-bouncer-check so there might be some arguments around that. 

@Simon: do you recall why we've set these[1][2] checks to warn out but not throw an error?

[1]: https://hg.mozilla.org/mozilla-central/rev/41506b1cb385#l3.36
[2]: https://hg.mozilla.org/mozilla-central/rev/41506b1cb385#l3.39

Comment 3

[Simon Fraser [:sfraser] ⌚️GMT]

I'd understood that treeherder would alert based on the string being produced, so I didn't consider too closely the difference between using python's warning() or error(). We should probably bump up the severity, there

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #3)
> I'd understood that treeherder would alert based on the string being
> produced, so I didn't consider too closely the difference between using
> python's warning() or error(). We should probably bump up the severity, there

Sounds good, makes sense. I'll keep this in the "[releaseduty]" radar to hopefully pick up when I grab the baton.

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1506319 Change log levels for bouncer check failures r=mtabara

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3ebe77971c61
Change log levels for bouncer check failures r=mtabara

Comment 7

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/3ebe77971c61

Comment 8

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/9496eb02b459