1910603 - "AppUpdatePin" is being ignored for Thunderbird - Port Bug 1762979 - Add Fx update pin to Balrog in release automation

Status: RESOLVED FIXED

(Thunderbird :: Upstream Synchronization, defect, P3)

Description

[Dominik Pflegerl [:dom6770]]

Steps to reproduce:

Creating ".\distribution\policies.json" with the following settings:

{
"policies": {
"AppUpdatePin": "115."
}
}

Actual results:

Thunderbird updated beyond 115 to 128 (with Mozilla Maintenance Service)

Expected results:

Thunderbird should NOT update automatically beyond 115.

Comment 1

[Magnus Melin [:mkmelin]]

If you set browser.policies.loglevel to All, what logs to you get in the Error Console?

Comment 2

[Dominik Pflegerl [:dom6770]]

Attachment: Error Console

If I set it to All, and launching Thunderbird 115.12.2 does logs do appear:

```

```

It's also weird that the Thunderbird Software Updater does appear, although the Mozilla Maintenance Service is installed.

Comment 3

[Stephen J. Cowley]

I have the same problem with 128.0.1esr. The output from setting browser.policies.loglevel to All (I had to generate the string) is:

Enterprise Policies: policies.json path = /store/DAMTP/sjc1/data/thunderbird-128/distribution/policies.json EnterprisePoliciesParent.sys.mjs:606
Enterprise Policies: policies.json content = {
"policies": {
"AppUpdatePin": "128.0."
}
}
EnterprisePoliciesParent.sys.mjs:607
1723540716733 addons.xpi WARN Checking /store/DAMTP/sjc1/data/thunderbird-128/distribution/extensions for addons

Comment 4

[YUKI "Piro" Hiroshi]

Attachment: App update log

(deleted)

Comment 5

[YUKI "Piro" Hiroshi]

Attachment: A log of application update on Thunderbird 115.14.0

(deleted)

Comment 6

[YUKI "Piro" Hiroshi]

Attachment: update log on Thunderbird 115.14.0

I've confirmed this with Thunderbird 115.14.0.
Tried with a policies.json:

{
  "policies": {
    "AppUpdatePin": "115.",
    "Preferences": {
      "browser.policies.loglevel": { "Value": "All", "Locked": true },
      "app.update.log": { "Value": true, "Locked": true }
    }
  }
}

In the log, the requested URL is:
https://aus.thunderbird.net/update/6/Thunderbird/115.14.0/20240801155430/WINNT_x86_64-msvc-x64/ja/esr/Windows_NT%2010.0.0.0.22631.4037%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?force=1&pin=115.

The response:

<?xml version="1.0"?>
<updates>
    <update actions="showURL" appVersion="128.1.1" buildID="20240820161839" detailsURL="https://live.thunderbird.net/thunderbird/releasenotes?locale=ja&amp;version=128.1.1&amp;channel=esr" displayVersion="128.1.1esr" openURL="https://live.thunderbird.net/thunderbird/whatsnew?locale=ja&amp;version=128.1.1&amp;channel=esr&amp;oldversion=%OLD_VERSION%" type="minor">
        <patch type="complete" URL="https://download.mozilla.org/?product=thunderbird-128.1.1esr-complete&amp;os=win64&amp;lang=ja" hashFunction="sha512" hashValue="073f2a22a86bb6881863aaaf1d8ea9c9f3da41b41d247eefd0fa01450427688432ab26c98afab2c885e09164b22e42912e3096812dcc74610de10d4fea4c95d9" size="71191969"/>
    </update>
</updates>

Thus the update information on the update server looks unexpected.

Comment 7

[cra3y]

I have this problem too (115.14 wants to update to 128.1.1esr)

My policies.json
{
"policies": {
"AppUpdatePin": "115."
}
}

PS: I've tried to use "115.14" instead of "115.", but result is the same - the background update process is doing update to 128.x

Comment 8

[cra3y]

UPDATE:

In my case the requested link was:
https://aus5.mozilla.org/update/6/Thunderbird/115.14.0/20240801155430/WINNT_x86_64-msvc-x64/pl/esr/Windows_NT%2010.0.0.0.22631.4037%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?force=1&pin=115.

and the answer:

<updates>
<update actions="showURL" appVersion="128.1.1" buildID="20240820161839" detailsURL="https://live.thunderbird.net/thunderbird/releasenotes?locale=pl&version=128.1.1&channel=esr" displayVersion="128.1.1esr" openURL="https://live.thunderbird.net/thunderbird/whatsnew?locale=pl&version=128.1.1&channel=esr&oldversion=%OLD_VERSION%" type="minor">
<patch type="complete" URL="https://download.mozilla.org/?product=thunderbird-128.1.1esr-complete&os=win64&lang=pl" hashFunction="sha512" hashValue="6f0fc5c7f8d1bf8f1dc798cbc01ef5669579b47f62542e9afd423b6292af117caa228d3a0274af7300805b497c638470ec928bc47303fdd1a995b01480f72527" size="72023065"/>
</update>
</updates>

then I fiddled with force parameter (set to 0) and the link is:
https://aus5.mozilla.org/update/6/Thunderbird/115.14.0/20240801155430/WINNT_x86_64-msvc-x64/pl/esr/Windows_NT%2010.0.0.0.22631.4037%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?force=0&pin=115.

and answer (correct - no update available):

<updates> </updates>

next I wanted to spoof my version (from 115.14 to 115.13 to check, if force=0 working, when the my version is 115.13 and the latest from "115" is 115.14)
the link is:
https://aus5.mozilla.org/update/6/Thunderbird/115.13.0/20240801155430/WINNT_x86_64-msvc-x64/pl/esr/Windows_NT%2010.0.0.0.22631.4037%20(x64)/ISET:SSE4_2,MEM:4095/default/default/update.xml?force=0&pin=115.

and the answer (correctly pointing that there is 115.14 version)

<updates>
<update appVersion="115.14.0" buildID="20240801155430" detailsURL="https://live.thunderbird.net/thunderbird/releasenotes?locale=pl&version=115.14.0&channel=esr" displayVersion="115.14.0" type="minor">
<patch type="complete" URL="https://download.mozilla.org/?product=thunderbird-115.14.0-complete&os=win64&lang=pl" hashFunction="sha512" hashValue="de515b3edc528b924d32dd1208fd80e49cac638bb576e2defdf5d70fc3123c174ad1802e35c0bcad3961d3bb3a9e3ddf7de3d29b09ddb534c9bf5355dcd7040a" size="65316155"/>
</update>
</updates>

Why Thunderbird update process using "force=1" parameter? How to prevent that?

Comment 9

[cra3y]

PS: I've no clue, why my previous post has been formatted in such weird way ....

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to cra3y from comment #9)

PS: I've no clue, why my previous post has been formatted in such weird way ....

It is because of the lines with dashes - I have removed them.

Comment 11

[cra3y]

Thanks :-)

Comment 12

[cra3y]

UPDATE2:

Googling and experimenting with "AppUpdateURL" I made this:

"AppUpdatePin": "115.",
"AppUpdateURL": "https://aus5.mozilla.org/update/6/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%SYSTEM_CAPABILITIES%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml?force=0"

And the update request (in the log) looks like:

AUS:SVC CheckerService:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Thunderbird/115.14.0/20240801155430/WINNT_x86_64-msvc-x64/pl/esr/Windows_NT%2010.0.0.0.19045.4780%20(x64)/ISET:SSE4_2,MEM:16339/default/default/update.xml?force=0&force=1&pin=115.

The second "force=1" canceled my "force=0" and the 128esr has been provided.

How to get rid the darn "force=1"?

Comment 13

[cra3y]

UPDATE3: (more tests)

I'm fiddling with query to aus5.mozilla.org, so I wanted to test, if the "pin" settings do something, so I've switched channel to release and make this query (the force parameter here doesn't matter - i've tested with force=0, force=1 and no force at all)

https://aus5.mozilla.org/update/6/Thunderbird/115.14.0/20240801155430/WINNT_x86_64-msvc-x64/pl/release/Windows_NT%2010.0.0.0.19045.4780%20(x64)/ISET:SSE4_2,MEM:16339/default/default/update.xml?force=1&pin=115.

the result is the 116.0, event that the pin states that the max version should be 115. :

<updates>
<update appVersion="116.0" buildID="20240415000000" displayVersion="116.0" type="minor">
<patch type="complete" URL="https://archive.mozilla.org/pub/thunderbird/releases/custom-updates/switch-to-esr116.0-eol-win.mar" hashFunction="sha512" > hashValue="23f0776fb72d13ef97b4463923847dbc96595c6b6c4e0a7c389c899b28eb6780b720cf06b0ad093da089126e4aaf296bf6f11043a2dd76e27dc5ae1bd37b9055" size="2612"/>
</update>
</updates>

My conclusion is that the pin parameter do nothing .... then I have to disable TB updated at all (I want to stick with 115.14 version due to IMAP problems on 128.x)

Comment 14

[cra3y]

UPDATE4: (the last one - sorry - I want to know, what is wrong)

I was curious, how mozilla's aus backend (barlog) will react, when I query for firefox update (using pin)

My query is (using Fox 105.0.0, pin 105):

https://aus5.mozilla.org/update/6/Firefox/105.0.0/20240801155430/WINNT_x86_64-msvc-x64/pl/release/Windows_NT(x64)/SSE3/default/default/update.xml?force=1&pin=105.

answer (sticks with 105.x):

<updates>
<update appVersion="105.0.3" buildID="20221007134813" detailsURL="https://www.mozilla.org/pl/firefox/105.0.3/releasenotes/" displayVersion="105.0.3" type="minor">
<patch type="complete" URL="https://download.mozilla.org/?product=firefox-105.0.3-complete&os=win64&lang=pl" hashFunction="sha512" hashValue="aaa2d3079af46cd34802da8ce10161cb512d1396ac1cdf721e8505e36099ac278dc9036f9829e38acb6d5918b2768e30a959c3eba7cc797d4cc3e51cea427ab0" size="61804807"/>
</update>
</updates>

Ok - change pin to 108.

query:

https://aus5.mozilla.org/update/6/Firefox/105.0.0/20240801155430/WINNT_x86_64-msvc-x64/pl/release/Windows_NT(x64)/SSE3/default/default/update.xml?force=1&pin=108.

answer - it still sticks with the max version, provided by pin parameter

<updates>
<update actions="showURL" appVersion="108.0.2" buildID="20230104165113" detailsURL="https://www.mozilla.org/pl/firefox/108.0.2/releasenotes/" displayVersion="108.0.2" openURL="https://www.mozilla.org/firefox/108.0.2/whatsnew/?oldversion=%OLD_VERSION%" type="minor">
<patch type="complete" URL="https://download.mozilla.org/?product=firefox-108.0.2-complete&os=win64&lang=pl" hashFunction="sha512" hashValue="69200aa64fee0debadc5a44445f9838ca0758d25eaab8458144ea3edfba7861ff1ab9c9f015e6513f7a96632cbeb54bcd9efd81b37df844f02dfc2db0b1b5741" size="62769499"/>
</update>
</updates>

My conclusion: the backend (barlog) has confirmed fault (pin parameter not working) in Thunderbird tree of updated (working well in Firefox tree).
Please change status to confirmed and fix it :-)

Comment 15

[Wayne Mery (:wsmwk)]

https://mzl.la/4cChcFI lists some relevant bug reports.

Did it ever work for you with Thunderbird?

Comment 16

[cra3y]

IDK, because in meantime I've written parser (in PHP), and all my Thunderbird are pointed into this parser.
The parser look into what is version of TB that asks for update, what are the settings ("pin" and "force" parameters) and it relays request to https://aus5.mozilla.org, but it looks for the data that is received from https://aus5.mozilla.org and depends mentioned parameters - it check if TB version matches (115!=128 for example) etc. and it decides to notify about update or not.

This is my workaround for this problem and it works well (no surprised updates with main version changes without my knowledge)

Comment 17

[mael.charlut]

I have the same problem: the AppUpdatePin policy is ignored.

• Tested via GPO and registry (HKLM\Software\Policies\Mozilla\Thunderbird)
• Tested with Thunderbird ESR 128.x and 140.x

Original question on support forum: https://support.mozilla.org/en-US/questions/1537467

=> This bug seems very "confirmed" to me...

Comment 18

[Magnus Melin [:mkmelin]]

Balrog (or other server side software) bug. Or feature.

Comment 19

[[e-mail] (:bhearsum)]

I've only followed this bug lightly, but I would be surprised if force=1 ignoring pins is intentional. That (poorly named) parameter controls whether or not backgroundRate is obeyed.

This should probably get refiled in https://github.com/mozilla-releng/balrog; moving to a RelEng component for now though.

As far as I can tell, this bug doesn't affect Firefox, so RelEng will be unlikely to be able to prioritize it soon. We're always happy to accept patches and guide people through developing them, though. If anyone wants a hand looking into fixing this in the server feel free to reach out to me by e-mail or in #balrog on Matrix.

Comment 20

[[e-mail] (:bhearsum)]

(If I've misunderstood, and this is actually a Thunderbird bug, my apologies - feel free to move it back!)

Comment 21

[BugBot [:suhaib / :marco]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:bhearsum, since the bug has recent activity, could you have a look please?

For more information, please visit BugBot documentation.

Comment 22

[mael.charlut]

Bump...
The problem is still here: same param works for Firefox and doesn't for Thunderbird.

I was hoping for a resolution before the next ESR version, so we could control the updating process, instead of the usual YOLO/prayers tactics.

Comment 23

[cm-schl]

I would like to add a thought: while reading this report and the comments of cra3y and [e-mail] (:bhearsum) I asked myself where in the codebase the decision is made to set force=1.

Looking at https://searchfox.org/firefox-main/source/toolkit/mozapps/update/UpdateService.sys.mjs#5485 I got the idea that only when a user checks the actual version of Thunderbird a FOREGROUND_CHECK is made and force=1 gets set.

This would also explain why many of our users did not get the version 153 of Thunderbird and only those who have to check the actual version on a regular basis (support etc.) got the update.

At the moment I can't say why force=1 lets Balrog ignore the app pinning (as cra3y showed and I can confirm using the urls documented in his comments) or if this is intentional. But I think Thunderbird should not set the parameter in a FOREGROUND_CHECK as long as this lets Balrog answer with a newer version than the one we try to pin.

Comment 24

[[e-mail] (:bhearsum)]

I have nothing to add to the latest comment. I agree this is probably incorrect behaviour, but unless there's implications for Firefox I don't have the cycles to look at it.

I would be happy to review a patch if someone wrote one. A good starting point might be to add some tests to https://github.com/mozilla-releng/balrog/blob/main/tests/web/test_client.py that cover both pin+force being passed, which we don't have at the moment.

Comment 25

[Magnus Melin [:mkmelin]]

I think the conclusions here were wrong. What's wrong is Thunderbird is missing a port of bug 1762979

Comment 26

[Magnus Melin [:mkmelin]]

Attachment: Bug 1910603 - "AppUpdatePin" is being ignored for Thunderbird - Port Bug 1762979 - Add Fx update pin to Balrog in release automation. r=#thunderbuild

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5298eed4522e
"AppUpdatePin" is being ignored for Thunderbird - Port Bug 1762979 - Add Fx update pin to Balrog in release automation. r=coreycb

Comment 28

[Magnus Melin [:mkmelin]]

I'm not 100% sure how this mechanism works. Am I correct to assume the actual code doesn't need uplift?

Comment 29

[[e-mail] (:bhearsum)]

There haven't been any changes to this server side in ages; if there's a client side fix it will need to get uplifted if you want it fixed sooner rather than later, I guess.

Comment 30

[Magnus Melin [:mkmelin]]

Comment on attachment 9630539 [details]
Bug 1910603 - "AppUpdatePin" is being ignored for Thunderbird - Port Bug 1762979 - Add Fx update pin to Balrog in release automation. r=#thunderbuild

Uplift Approval Request

• Please state case for uplift consideration and ensure bug severity is set: Can't pin update
• User impact if declined: (More of an admin impact)
• Is this code covered by automated tests?: No
• Has the fix been verified in Daily?: Yes
• Has the fix been verified in Beta?: No
• Needs manual test from QA?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Old porting patch
• Does the fix cause any migrations to be skipped?: No
• String changes made/needed: none

Comment 31

[Corey Bryant]

Comment on attachment 9630539 [details]
Bug 1910603 - "AppUpdatePin" is being ignored for Thunderbird - Port Bug 1762979 - Add Fx update pin to Balrog in release automation. r=#thunderbuild

[Triage Comment]
Sorry this missed 156 beta. Will be in 157 beta.

Comment 32

[Corey Bryant]

Comment on attachment 9630539 [details]
Bug 1910603 - "AppUpdatePin" is being ignored for Thunderbird - Port Bug 1762979 - Add Fx update pin to Balrog in release automation. r=#thunderbuild

[Triage Comment]
Approved for esr153