1893487 - Nightly thinks its up to date even though it is months old

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Wil Clouser [:clouserw]]

I'm running Nightly from 2024-02-29 on OS X (build id 20240229093832). I noticed it hasn't updated in a while so I clicked "About Nightly" and see that Nightly thinks it is up to date even though I have newer versions of Nightly on other computers.

I'm stuck in a leap year! How did I get in this weird place and how do I get out? Thanks.

Comment 1

[Wil Clouser [:clouserw]]

Attachment: about:support output

Comment 2

[Wil Clouser [:clouserw]]

Attachment: Screenshot 2024-04-25 at 8.53.13 AM.png

Comment 3

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:mak, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Robin Steuber (she/her) [:bytesized]]

Please attach a browser console log.

1. Navigate to about:config.
2. Set app.update.log to true.
3. Open the Browser Console either with the hotkey Control+Shift+J (Command+Shift+J on macOS), or via Hamburger Menu->More Tools->Browser Console
4. In the Filter textbox at the top, enter AUS: to filter out everything except the update messages.
5. Navigate to the "Update" section of about:preferences. It should automatically check for an update.
6. Once the update check has completed, copy the messages out of the Browser Console and attach them to this bug.

Comment 5

[Wil Clouser [:clouserw]]

Attachment: AUS browser log

Comment 6

[Robin Steuber (she/her) [:bytesized]]

The important lines of the log are definitely these ones:

AUS:SVC CheckerService:#updateCheck - sending request to: https://aus5.mozilla.org/update/6/Firefox/125.0a1/20240229093832/Darwin_aarch64-gcc3/en-US/nightly/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A24576/default/default/update.xml?force=1
AUS:SVC CheckerService:#updateCheck - request got 'load' event
AUS:SVC CheckerService:#updateCheck - request completed downloading document
AUS:SVC CheckerService:#parseUpdates - invalid <update/>, ignoring...
AUS:SVC CheckerService:#updateCheck - number of updates available: 0

It seems to be getting an update XML but it can't parse it. Odd. Well let's visit the update URL and see if it looks unexpected in some way.

<updates>
<update type="minor" displayVersion="125.0a1" appVersion="125.0a1" platformVersion="125.0a1" buildID="20240301094944"> </update>
</updates>

Uh, yeah. That looks pretty unexpected. The update element should contain at least one patch element but it doesn't.

I didn't actually check the exact build ID yet, but this version looks like it might be within the range that was affected by Bug 1882322. Fixing that required some special update server rules. Perhaps one of them isn't quite right?

@bhearsum Would you be able to shed any light on this unexpected Balrog response?

Comment 7

[[e-mail] (:bhearsum)]

It looks like this is related to the watershed that we set-up in https://bugzilla.mozilla.org/show_bug.cgi?id=1882729. That buildid is supposed to be receiving an update to 20240301094944, and indeed, the release in balrog is set up to serve one.

I do see what's happening here.

Balrog tries to identify client builds like the one in the aforementioned update URL by comparing them to the releases in its database, which contain a whole bunch of metadata. It will only serve a partial when it thinks that the client is running a version that has a partial (based on buildid + locale). We clean up nightly release metadata older than 14 days in Balrog, and once it disappears, we cannot serve partials to such versions.

Normally this all works fine. We also delete ordinary Nightly partial MARs older than 14 days, so there's no issue. But this watershed is special. It must be done with a partial, so once the metadata went away updates stopped working altogether because there was no complete to fall back to.

We still have the MARs we need to do these updates, but we'll need to rebuild the releases so Balrog can identify them, and serve updates again.

Comment 8

[[e-mail] (:bhearsum)]

I've also filed https://github.com/mozilla-releng/balrog/issues/3118 to improve the cleanup cronjob.

Comment 9

[[e-mail] (:bhearsum)]

As far fixing up the immediate issue, it looks like we're going to need to scrape archive to rebuild these releases, as the original script that built them used other releases that have been since deleted.

I don't have time to handle that today, but I'll come back to it tomorrow unless jcristau gets to it first.

Comment 10

[Julien Cristau [:jcristau]]

Attachment: script to create missing releases and update the references in Firefox-mac-partials-*

Comment 11

[Julien Cristau [:jcristau]]

After running the script from comment 10 and adding rules on stage to match prod, I do see the desired update:

$ curl -i https://stage.balrog.nonprod.cloudops.mozgcp.net/update/6/Firefox/125.0a1/20240229093832/Darwin_aarch64-gcc3/en-US/nightly/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A24576/default/default/update.xml?force=1; echo
HTTP/2 200 
server: nginx
date: Wed, 22 May 2024 11:48:45 GMT
content-type: text/xml; charset=utf-8
content-length: 581
vary: Accept-Encoding
rule-id: 3082
rule-data-version: 1
strict-transport-security: max-age=31536000;
x-content-type-options: nosniff
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-proxy-cache-status: MISS
via: 1.1 google
cache-control: public,max-age=90
alt-svc: clear

<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="125.0a1" appVersion="125.0a1" platformVersion="125.0a1" buildID="20240301094944">
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/03/2024-03-01-09-49-44-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240229093832-20240301094944.partial.mar" hashFunction="sha512" hashValue="ed1f79691dc4d2ab1a605a0038574c97364319b221f9737e937bc9d307de6ef27ad3e8bdf1eea65bd8011c39803bdd76a57f1adbca1592d9a9ed8f9be2451a92" size="19902127"/>
    </update>
</updates>

Also sanity checked other buildids in the affected range and they seem ok.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Julien Cristau [:jcristau] from comment #11)

After running the script from comment 10 and adding rules on stage to match prod, I do see the desired update:

$ curl -i https://stage.balrog.nonprod.cloudops.mozgcp.net/update/6/Firefox/125.0a1/20240229093832/Darwin_aarch64-gcc3/en-US/nightly/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A24576/default/default/update.xml?force=1; echo
HTTP/2 200 
server: nginx
date: Wed, 22 May 2024 11:48:45 GMT
content-type: text/xml; charset=utf-8
content-length: 581
vary: Accept-Encoding
rule-id: 3082
rule-data-version: 1
strict-transport-security: max-age=31536000;
x-content-type-options: nosniff
content-security-policy: default-src 'none'; frame-ancestors 'none'
x-proxy-cache-status: MISS
via: 1.1 google
cache-control: public,max-age=90
alt-svc: clear

<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="125.0a1" appVersion="125.0a1" platformVersion="125.0a1" buildID="20240301094944">
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/03/2024-03-01-09-49-44-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240229093832-20240301094944.partial.mar" hashFunction="sha512" hashValue="ed1f79691dc4d2ab1a605a0038574c97364319b221f9737e937bc9d307de6ef27ad3e8bdf1eea65bd8011c39803bdd76a57f1adbca1592d9a9ed8f9be2451a92" size="19902127"/>
    </update>
</updates>

Also sanity checked other buildids in the affected range and they seem ok.

This looks good to me. I was worried that we might re-cleanup these releases, but the new names for them (with the bug suffix) should get ignored by the cleanup job.

Comment 13

[Julien Cristau [:jcristau]]

Attachment: script to create missing releases and update the references in Firefox-mac-partials-* (for prod)

The previous script worked in stage because the Firefox-mac-partials didn't exist there. This is the script that actually ran against prod.

Comment 14

[[e-mail] (:bhearsum)]

I filed https://github.com/mozilla-releng/balrog/issues/3119 to help make issues like this more visibile in the future.

Comment 15

[Julien Cristau [:jcristau]]

Testing update URLs with build IDs in the range now I'm getting patch elements as expected:

$ { echo 20240201000000; curl -s 'https://aus-api.mozilla.org/api/v1/releases?name_prefix=Firefox-mac-partials&names_only=1' | jq -r '.names[] | split("-")[-1]'; } | while read buildid; do echo $buildid; curl -s https://aus5.mozilla.org/update/6/Firefox/125.0a1/$buildid/Darwin_aarch64-gcc3/en-US/nightly/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A16384/default/default/update.xml?force=1 | grep '<patch' ; echo; done
20240201000000
        <patch type="complete" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/05/2024-05-22-09-19-37-mozilla-central/firefox-128.0a1.en-US.mac.complete.mar" hashFunction="sha512" hashValue="75d997979456424a00aeaf4ec529fe3cea10762c650327bcdc38b5a731c0ea4d6208a20b773bbbfe2126d73c448d1afe311fc73d79dbb83f122dcd219ef124d2" size="123221790"/>

20240224093754
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-26-16-56-59-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240224093754-20240226165659.partial.mar" hashFunction="sha512" hashValue="1e6d9ef51df78ea5b004a766d87974bc48569d8e339bfe84874ff629972251a989c03a82c457546b3ff2eb7ffcbd3432f5c651ce47c95da128cd4bb3101c343f" size="18638974"/>

20240224211946
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-27-09-57-54-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240224211946-20240227095754.partial.mar" hashFunction="sha512" hashValue="f878d4cfbef6cb0e59c697dab606d61739038ed2f6b10674cd6682380be96429af842d605903a20fc61d0d6912d72abf93b095a75f15d10aa8ad97a315e29f11" size="21696666"/>

20240225092952
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-28-10-05-09-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240225092952-20240228100509.partial.mar" hashFunction="sha512" hashValue="7a2e1c184887c1f9fd33b8c64f3ad42b08734815a469d655292461360170f0bb41282160870be786802d8790a5a537a0cba0a138a8fdc63a845741f4b3b014bf" size="20472714"/>

20240226091922
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-28-21-23-27-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240226091922-20240228212327.partial.mar" hashFunction="sha512" hashValue="81ee5556108031fa369e1d430c1bfc4ccdcc76c5940eff22743d0947482c9e7844092e7dbc824094c1a5224e7b4d0be8c889cefb143cda9bb8b7aede3c375444" size="21480886"/>

20240226165659
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-29-09-38-32-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240226165659-20240229093832.partial.mar" hashFunction="sha512" hashValue="92a749a56b7ef3eb0d109c4081c28fd2f72b400b4723940cba6e05a2f0a9e9374fc66466d9b8284bdec5fd44f5314d60e1c1c7c7f73c2e7b7520b51aad4b62be" size="20845010"/>

20240227095754
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-29-21-06-42-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240227095754-20240229210642.partial.mar" hashFunction="sha512" hashValue="58b9a862a275222ce950b488aa31eb0c1d7db690cfd0821f38661b9fc0610ee594913660b737c7dc730ce2372996bf0da2bf85dca80e4c1aad12f781cc9eecc1" size="22586458"/>

20240228100509
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/03/2024-03-01-09-49-44-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240228100509-20240301094944.partial.mar" hashFunction="sha512" hashValue="3e0bf466c0b47ef9175eaa27a353054ee1ff5ee2c224396e6b250521d2f2847b44d2ecb41b2d6d6205ca4b16aef1faa153f5ec5802c1618e3f796fea1286075d" size="21845494"/>

20240228212327
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/03/2024-03-01-09-49-44-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240228212327-20240301094944.partial.mar" hashFunction="sha512" hashValue="f275ea4bf111f4b8cea261baf2248f44e69e8041e3e09b45a51a0127268f6d1ba44b6a3d4b5ec97c8155bc9ee55f77ba7b061d65b8f4542ff13ae7f84d2084f2" size="21288694"/>

20240229093832
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/03/2024-03-01-09-49-44-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240229093832-20240301094944.partial.mar" hashFunction="sha512" hashValue="ed1f79691dc4d2ab1a605a0038574c97364319b221f9737e937bc9d307de6ef27ad3e8bdf1eea65bd8011c39803bdd76a57f1adbca1592d9a9ed8f9be2451a92" size="19902127"/>

20240229210642
        <patch type="partial" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/03/2024-03-01-09-49-44-mozilla-central/partials/firefox-mozilla-central-125.0a1-mac-en-US-20240229210642-20240301094944.partial.mar" hashFunction="sha512" hashValue="2e8f2a52f7ce654574d1cf921a8abd4b902a06c52247dad5edd5fb53d909c0022e1db7dc7d594e16120b33915397aeacdb8bceebf721199dbbc999ee02e6e1d8" size="21290623"/>

20240301094944
        <patch type="complete" URL="https://archive.mozilla.org/pub/firefox/nightly/2024/05/2024-05-22-09-19-37-mozilla-central/firefox-128.0a1.en-US.mac.complete.mar" hashFunction="sha512" hashValue="75d997979456424a00aeaf4ec529fe3cea10762c650327bcdc38b5a731c0ea4d6208a20b773bbbfe2126d73c448d1afe311fc73d79dbb83f122dcd219ef124d2" size="123221790"/>

Comment 16

[Wil Clouser [:clouserw]]

Thanks! I can confirm I got the update.