Thunderbird: Wrong bouncer entries are updated

RESOLVED FIXED in Thunderbird 65.0

Status

defect
RESOLVED FIXED
11 months ago
8 months ago

People

(Reporter: jlorenzo, Assigned: rjl)

Tracking

Trunk
Thunderbird 65.0
Dependency tree / graph

Thunderbird Tracking Flags

(thunderbird_esr6063+ fixed, thunderbird64 fixed, thunderbird65 fixed)

Details

Attachments

(1 attachment, 1 obsolete attachment)

Reporter

Description

11 months ago
https://tools.taskcluster.net/groups/B5SChkanQf2ahu0Kr1POgQ/tasks/D9L1VHbUTLC_eNmaj-4bkA/runs/0/logs/public%2Flogs%2Flive_backing.log: 

> 2018-08-06 12:15:58,061 - bouncerscript.script - INFO - Sanity check versions and aliases before updating ...
> 2018-08-06 12:15:58,061 - scriptworker.client - ERROR - Failed to run async_main
> Traceback (most recent call last):
>   File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 203, in _handle_asyncio_loop
>     await async_main(context)
>   File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/script.py", line 99, in async_main
>     await action_map[context.action](context)
>   File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/script.py", line 70, in bouncer_aliases
>     check_product_names_match_aliases(context)
>   File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/task.py", line 88, in check_product_names_match_aliases
>     raise TaskVerificationError("Unrecognized alias:{}".format(alias))
> scriptworker.exceptions.TaskVerificationError: Unrecognized alias:thunderbird-esr-next-latest

This entry doesn't exist on the bouncer side. It should be "thunderbird-latest" instead. Moreover there is no SSL entry for thunderbird release referenced in bouncer. We may probably want to either create it, or just remove the in-tree definition.
Reporter

Comment 1

11 months ago
In the meantime, I manually changed thunderbird-latest alias from "Thunderbird-52.9.1" to "Thunderbird-60.0"
Reporter

Updated

11 months ago
Blocks: esr60
While shipping TB60, bouncerscript sanity caught some error that `thunderbird-esr-next-latest` didn't exist on bouncer. Which was true. We don't have that neither in bouncerscript regexes, nor in bouncer. I took a moment and double-checked we're up-to-date with entries in bouncer aliases.

Ran a quick script to double-check what we currently have under https://github.com/mozilla-releng/bouncerscript/blob/master/bouncerscript/constants.py#L1 against official bouncer entries and I got:

```
(Pdb) pp official_set.difference(our_set)
{'firefox-esr52-latest-ssl', 'firefox-esr52-latest'}
(Pdb) pp our_set.difference(official_set)
{'thunderbird-latest-ssl',
 'thunderbird-next-latest',
 'thunderbird-next-latest-ssl'}
```

So, to begin with
1.
i) We don't have the esr52 aliases included here. That's expected since that's not subject to bouncerscript these days.
ii) We have three additional aliases defined in our regexes that are nowhere to be found in bouncer, so most likely never used by other pieces of automation.

@tomprince: are we okay with removing these three aliases below from our sanity validation check? 
CCing @JohanLorenzo 
```
{'thunderbird-latest-ssl',
 'thunderbird-next-latest',
 'thunderbird-next-latest-ssl'}
```

2. Sounds like we need to fix the in-tree aliases for thunderbird to correctly submit the payload containing the `thunderbird-latest` instead.
:sancus, does thunderbird use any bouncer aliases, or is there any desire to? Firefox[1] uses URLs like https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64&lang=en-US that don't contain a version, and get updated to point to the most recent version as part of shipping the release.

I set this up as part of the taskcluster migration, but as indicated above, there was some issues with how this is setup. Either we can fix it, or just remove the code that updates the aliases.

[1] https://www.mozilla.org/en-US/firefox/download/
Flags: needinfo?(sancus)
We don't use these aliases right now, the website is aware of what version is the latest. I don't think these aliases would remove that need, since eg we still need to display the correct version numbers on release notes and elsewhere.

Maybe there is some potential future use for them but I don't think we need them right now. At least not on the website.
Flags: needinfo?(sancus)
:rjl, given the above, we should remove the bouncer-alias tasks from comm-*.
Assignee: nobody → rob
Component: Release Automation: Uploading → Release Automation: Other
QA Contact: mtabara → sfraser
Assignee

Comment 6

9 months ago
Removed release-bouncer-aliases dependencies and removed the task
configuration.
Comment on attachment 9012368 [details]
Bug 1481198 - Remove bouncer-alias tasks from comm-*. r?tomprince

Tom Prince [:tomprince] has approved the revision.
Attachment #9012368 - Flags: review+
Duplicate of this bug: 1496980
These are referenced in http://ftp.mozilla.org/pub/thunderbird/releases/latest/README.txt so if the decision is taken to proceed with deletion rather than fixing, the README should also be updated.

Comment 10

9 months ago
Please reconsider removing this functionality.  Many of us (as USERS) use this ( as systems administrators, etc?) to try to automate downloading of NEW releases.  We've been led to believe that we can rely on these "bouncer"(?) URLs to point us to stable release versions.

e.g. using a lookup like:

    app_url="${moz_proto}://${moz_site}/?product=${app_name}&os=${app_os}&lang=${app_lang}"
    app_fname="$(curl ${curl_verbose} --silent --head "${app_url}" | sed -ne '/^Location:/{s/^.*\///;s/\r$//;p}')"

to test whether i already have the latest version downloaded in a staging area.

Trying to figure out how to get to the actual download locations (since ftp://ftp.mozilla.org is no longer around) going through all the "helpful" (not) automated web interfaces is really not friendly at all to enterprise management, so if you remove this mechanism, please replace it with some well-defined language on how we can get directly to the download URLs for current packages.
(e.g. i accidentally grabbed the Linux32 TB 60.2.1 via Web clicky on the penguin icon, and that sucked for a bit while i was trying to figure out why libraries were missing :-( )  PROGRAMMATIC FOR THE WIN!

thanks,
--stephen
(In reply to sdowdy from comment #10)
> Please reconsider removing this functionality.  Many of us (as USERS) use
> this ( as systems administrators, etc?) to try to automate downloading of
> NEW releases.  We've been led to believe that we can rely on these
> "bouncer"(?) URLs to point us to stable release versions.

FYI, the source of truth for Thunderbird versions(used by all thunderbird.net websites as well) is https://product-details.mozilla.org/1.0/thunderbird_versions.json -- this is automatically updated when a new version is marked shipped by release engineering.

We may yet keep the bouncer aliases though, that has certainly not been decided or this bug would be closed. :)

Comment 13

8 months ago
> FYI, the source of truth for Thunderbird versions(used by all
> thunderbird.net websites as well) is
> https://product-details.mozilla.org/1.0/thunderbird_versions.json

Thanks, that's handy to know. 
My concern with using the json file is mapping from the version number to a download URL. If the URLs follow a standard format that is fairly easy to compute and doesn't change (or rarely changes), that might be an acceptable solution for my case.
Duplicate of this bug: 1497574
(In reply to Paul Sparks from comment #13)
> If the URLs follow a standard format that is fairly easy to
> compute and doesn't change (or rarely changes), that might be an acceptable
> solution for my case.

They follow the format https://download.mozilla.org/?product=thunderbird-60.2.1-SSL&os=win&lang=en-US - while I can't guarantee that this will never, ever change, it hasn't changed in many years and any change would likely be accompanied by a rewrite for the old URLs anyway.
Attachment #9012368 - Attachment is obsolete: true
Assignee

Comment 16

8 months ago
comm-esr60 is really a regular release branch, so it should be referring to
thunderbird-latest and thunderbird-latest-ssl.

Note that the comm-beta configuration in this file does work as expected.

Updated

8 months ago
Blocks: 1496980
Assignee

Comment 17

8 months ago
Attachment 9020435 [details] is the fix. Plan is to get it into esr60 in time for the next release.
Assignee

Comment 18

8 months ago
Comment on attachment 9020435 [details]
Bug 1481198 - update bouncer-aliases for comm-esr60 to correct mapping. r?darktrojan

[ESR Uplift Approval Request]

If this is not a sec:{high,crit} bug, please state case for ESR consideration: This fixes the thunderbird-latest bouncer at download.mozilla.org

User impact if declined: The bouncer continues to point to an old version for ESR releases. Beta works, but only because thunderbird-latest-beta points to a valid entry. Once that merges to esr60 it will be broken for that release.

Fix Landed on Version: not yet in c-c

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Change will only affect the release process if it fails. The bouncer is broken now for releases, it won't be any more broken if this fails.

String or UUID changes made by this patch: 

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: 

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: Yes

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): 

String changes made/needed:
Attachment #9020435 - Flags: approval-mozilla-esr60?
Attachment #9020435 - Flags: approval-mozilla-beta?
Assignee

Updated

8 months ago
Component: Release Automation: Other → Build Config
Product: Release Engineering → Thunderbird
QA Contact: sfraser
Target Milestone: --- → Thunderbird 64.0
Version: unspecified → Trunk
Assignee

Comment 19

8 months ago
Comment on attachment 9020435 [details]
Bug 1481198 - update bouncer-aliases for comm-esr60 to correct mapping. r?darktrojan

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):
Attachment #9020435 - Flags: approval-mozilla-esr60?
Attachment #9020435 - Flags: approval-mozilla-beta?
Attachment #9020435 - Flags: approval-comm-esr60?
Attachment #9020435 - Flags: approval-comm-beta?
Assignee

Updated

8 months ago
Keywords: checkin-needed
Target Milestone: Thunderbird 64.0 → Thunderbird 65.0
Assignee

Updated

8 months ago
Duplicate of this bug: 1502445
Assignee

Updated

8 months ago
Duplicate of this bug: 1496980

Comment 22

8 months ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/50073ff4e729
update bouncer-aliases for comm-esr60 to correct mapping. r=darktrojan
Status: NEW → RESOLVED
Closed: 8 months ago
Keywords: checkin-needed
Resolution: --- → FIXED

Updated

8 months ago
Attachment #9020435 - Flags: approval-comm-esr60?
Attachment #9020435 - Flags: approval-comm-esr60+
Attachment #9020435 - Flags: approval-comm-beta?
Attachment #9020435 - Flags: approval-comm-beta+
You need to log in before you can comment on or make changes to this bug.