Closed Bug 1481198 Opened 2 years ago Closed 2 years ago

Thunderbird: Wrong bouncer entries are updated

Categories

(Thunderbird :: Build Config, defect)

defect
Not set
normal

Tracking

(thunderbird_esr6063+ fixed, thunderbird64 fixed, thunderbird65 fixed)

RESOLVED FIXED
Thunderbird 65.0
Tracking Status
thunderbird_esr60 63+ fixed
thunderbird64 --- fixed
thunderbird65 --- fixed

People

(Reporter: jlorenzo, Assigned: rjl)

References

Details

Attachments

(1 file, 1 obsolete file)

https://tools.taskcluster.net/groups/B5SChkanQf2ahu0Kr1POgQ/tasks/D9L1VHbUTLC_eNmaj-4bkA/runs/0/logs/public%2Flogs%2Flive_backing.log: 

> 2018-08-06 12:15:58,061 - bouncerscript.script - INFO - Sanity check versions and aliases before updating ...
> 2018-08-06 12:15:58,061 - scriptworker.client - ERROR - Failed to run async_main
> Traceback (most recent call last):
>   File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 203, in _handle_asyncio_loop
>     await async_main(context)
>   File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/script.py", line 99, in async_main
>     await action_map[context.action](context)
>   File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/script.py", line 70, in bouncer_aliases
>     check_product_names_match_aliases(context)
>   File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/task.py", line 88, in check_product_names_match_aliases
>     raise TaskVerificationError("Unrecognized alias:{}".format(alias))
> scriptworker.exceptions.TaskVerificationError: Unrecognized alias:thunderbird-esr-next-latest

This entry doesn't exist on the bouncer side. It should be "thunderbird-latest" instead. Moreover there is no SSL entry for thunderbird release referenced in bouncer. We may probably want to either create it, or just remove the in-tree definition.
In the meantime, I manually changed thunderbird-latest alias from "Thunderbird-52.9.1" to "Thunderbird-60.0"
Blocks: esr60
While shipping TB60, bouncerscript sanity caught some error that `thunderbird-esr-next-latest` didn't exist on bouncer. Which was true. We don't have that neither in bouncerscript regexes, nor in bouncer. I took a moment and double-checked we're up-to-date with entries in bouncer aliases.

Ran a quick script to double-check what we currently have under https://github.com/mozilla-releng/bouncerscript/blob/master/bouncerscript/constants.py#L1 against official bouncer entries and I got:

```
(Pdb) pp official_set.difference(our_set)
{'firefox-esr52-latest-ssl', 'firefox-esr52-latest'}
(Pdb) pp our_set.difference(official_set)
{'thunderbird-latest-ssl',
 'thunderbird-next-latest',
 'thunderbird-next-latest-ssl'}
```

So, to begin with
1.
i) We don't have the esr52 aliases included here. That's expected since that's not subject to bouncerscript these days.
ii) We have three additional aliases defined in our regexes that are nowhere to be found in bouncer, so most likely never used by other pieces of automation.

@tomprince: are we okay with removing these three aliases below from our sanity validation check? 
CCing @JohanLorenzo 
```
{'thunderbird-latest-ssl',
 'thunderbird-next-latest',
 'thunderbird-next-latest-ssl'}
```

2. Sounds like we need to fix the in-tree aliases for thunderbird to correctly submit the payload containing the `thunderbird-latest` instead.
:sancus, does thunderbird use any bouncer aliases, or is there any desire to? Firefox[1] uses URLs like https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64&lang=en-US that don't contain a version, and get updated to point to the most recent version as part of shipping the release.

I set this up as part of the taskcluster migration, but as indicated above, there was some issues with how this is setup. Either we can fix it, or just remove the code that updates the aliases.

[1] https://www.mozilla.org/en-US/firefox/download/
Flags: needinfo?(sancus)
We don't use these aliases right now, the website is aware of what version is the latest. I don't think these aliases would remove that need, since eg we still need to display the correct version numbers on release notes and elsewhere.

Maybe there is some potential future use for them but I don't think we need them right now. At least not on the website.
Flags: needinfo?(sancus)
:rjl, given the above, we should remove the bouncer-alias tasks from comm-*.
Assignee: nobody → rob
Component: Release Automation: Uploading → Release Automation: Other
QA Contact: mtabara → sfraser
Removed release-bouncer-aliases dependencies and removed the task
configuration.
Comment on attachment 9012368 [details]
Bug 1481198 - Remove bouncer-alias tasks from comm-*. r?tomprince

Tom Prince [:tomprince] has approved the revision.
Attachment #9012368 - Flags: review+
Duplicate of this bug: 1496980
These are referenced in http://ftp.mozilla.org/pub/thunderbird/releases/latest/README.txt so if the decision is taken to proceed with deletion rather than fixing, the README should also be updated.
Please reconsider removing this functionality.  Many of us (as USERS) use this ( as systems administrators, etc?) to try to automate downloading of NEW releases.  We've been led to believe that we can rely on these "bouncer"(?) URLs to point us to stable release versions.

e.g. using a lookup like:

    app_url="${moz_proto}://${moz_site}/?product=${app_name}&os=${app_os}&lang=${app_lang}"
    app_fname="$(curl ${curl_verbose} --silent --head "${app_url}" | sed -ne '/^Location:/{s/^.*\///;s/\r$//;p}')"

to test whether i already have the latest version downloaded in a staging area.

Trying to figure out how to get to the actual download locations (since ftp://ftp.mozilla.org is no longer around) going through all the "helpful" (not) automated web interfaces is really not friendly at all to enterprise management, so if you remove this mechanism, please replace it with some well-defined language on how we can get directly to the download URLs for current packages.
(e.g. i accidentally grabbed the Linux32 TB 60.2.1 via Web clicky on the penguin icon, and that sucked for a bit while i was trying to figure out why libraries were missing :-( )  PROGRAMMATIC FOR THE WIN!

thanks,
--stephen
(In reply to sdowdy from comment #10)
> Please reconsider removing this functionality.  Many of us (as USERS) use
> this ( as systems administrators, etc?) to try to automate downloading of
> NEW releases.  We've been led to believe that we can rely on these
> "bouncer"(?) URLs to point us to stable release versions.

FYI, the source of truth for Thunderbird versions(used by all thunderbird.net websites as well) is https://product-details.mozilla.org/1.0/thunderbird_versions.json -- this is automatically updated when a new version is marked shipped by release engineering.

We may yet keep the bouncer aliases though, that has certainly not been decided or this bug would be closed. :)
> FYI, the source of truth for Thunderbird versions(used by all
> thunderbird.net websites as well) is
> https://product-details.mozilla.org/1.0/thunderbird_versions.json

Thanks, that's handy to know. 
My concern with using the json file is mapping from the version number to a download URL. If the URLs follow a standard format that is fairly easy to compute and doesn't change (or rarely changes), that might be an acceptable solution for my case.
Duplicate of this bug: 1497574
(In reply to Paul Sparks from comment #13)
> If the URLs follow a standard format that is fairly easy to
> compute and doesn't change (or rarely changes), that might be an acceptable
> solution for my case.

They follow the format https://download.mozilla.org/?product=thunderbird-60.2.1-SSL&os=win&lang=en-US - while I can't guarantee that this will never, ever change, it hasn't changed in many years and any change would likely be accompanied by a rewrite for the old URLs anyway.
Attachment #9012368 - Attachment is obsolete: true
comm-esr60 is really a regular release branch, so it should be referring to
thunderbird-latest and thunderbird-latest-ssl.

Note that the comm-beta configuration in this file does work as expected.
Blocks: 1496980
Attachment 9020435 [details] is the fix. Plan is to get it into esr60 in time for the next release.
Comment on attachment 9020435 [details]
Bug 1481198 - update bouncer-aliases for comm-esr60 to correct mapping. r?darktrojan

[ESR Uplift Approval Request]

If this is not a sec:{high,crit} bug, please state case for ESR consideration: This fixes the thunderbird-latest bouncer at download.mozilla.org

User impact if declined: The bouncer continues to point to an old version for ESR releases. Beta works, but only because thunderbird-latest-beta points to a valid entry. Once that merges to esr60 it will be broken for that release.

Fix Landed on Version: not yet in c-c

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Change will only affect the release process if it fails. The bouncer is broken now for releases, it won't be any more broken if this fails.

String or UUID changes made by this patch: 

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: 

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: Yes

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): 

String changes made/needed:
Attachment #9020435 - Flags: approval-mozilla-esr60?
Attachment #9020435 - Flags: approval-mozilla-beta?
Component: Release Automation: Other → Build Config
Product: Release Engineering → Thunderbird
QA Contact: sfraser
Target Milestone: --- → Thunderbird 64.0
Version: unspecified → Trunk
Comment on attachment 9020435 [details]
Bug 1481198 - update bouncer-aliases for comm-esr60 to correct mapping. r?darktrojan

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):
Attachment #9020435 - Flags: approval-mozilla-esr60?
Attachment #9020435 - Flags: approval-mozilla-beta?
Attachment #9020435 - Flags: approval-comm-esr60?
Attachment #9020435 - Flags: approval-comm-beta?
Keywords: checkin-needed
Target Milestone: Thunderbird 64.0 → Thunderbird 65.0
Duplicate of this bug: 1502445
Duplicate of this bug: 1496980
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/50073ff4e729
update bouncer-aliases for comm-esr60 to correct mapping. r=darktrojan
Status: NEW → RESOLVED
Closed: 2 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Attachment #9020435 - Flags: approval-comm-esr60?
Attachment #9020435 - Flags: approval-comm-esr60+
Attachment #9020435 - Flags: approval-comm-beta?
Attachment #9020435 - Flags: approval-comm-beta+
You need to log in before you can comment on or make changes to this bug.