Closed Bug 1809021 Opened 3 years ago Closed 2 years ago

Linux nightly files have stale signatures

Categories

(Cloud Services :: Operations: Product Delivery, defect)

Product:
Cloud Services
Component:
Operations: Product Delivery
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lilydjwg, Assigned: cvalaas)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:110.0) Gecko/20100101 Firefox/110.0

Steps to reproduce:

Since 2022-11-09, I get intermittent stale signatures for firefox nightly files. I have a script to repackage nightly tarballs to Arch Linux's native package format, and it started to fail for two out of three.

I have tried to wait for 300 seconds (from the cache-control header) and redownload the signature files but it still happens.

The URLs I'm using are like: https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/firefox-110.0a1.en-US.linux-x86_64.tar.bz2.asc

Actual results:

==> Verifying source file signatures with gpg...
    20230106.23-firefox-110.0a1.en-US.linux-x86_64.tar.bz2 ... FAILED (bad signature from public key EBE41E90F6F12F6D)
    20230106.23-firefox-110.0a1.zh-CN.linux-x86_64.tar.bz2 ... Passed
    20230106.23-firefox-110.0a1.ja.linux-x86_64.tar.bz2 ... Passed
    20230106.23-firefox-110.0a1.zh-TW.linux-x86_64.tar.bz2 ... Passed

The failed signatures are for previous nightlies (I manually examined some in the past).

Expected results:

When nightly tarballs get updated, the signatures should also be updated soon.

E.g. a firefox-110.0a1.en-US.linux-x86_64.tar.bz2 file downloaded at 2023-01-07 15:12:40.510017676 +0800 has sha1sum 025e9e607a7c5a1525ad9c6c7c930210c72025bb but even at 2023-01-07 18:22:59.985198820 +0800 I still got a signature made Fri 06 Jan 2023 07:32:20 PM CST (sha1sum e3d7a0ed9faf840af743f1f9e7318f28f2d63411). I get the correct signature file made Sat 07 Jan 2023 07:33:07 AM CST (sha1sum 2e46f6687bc638c538b57298a988e2871ae8b5cb) for that file only just now (at 2023-01-07 18:34:54.528795175 +0800).

PS: The CST timezone means UTC+8, so all the timestamps are in the same timezone.

Jon, is this something we can fix on the cdn or gcs side? The upload code sets cache-control: max-age=300 but it looks like that's not respected, e.g. I'm seeing responses with things like age: 71448.

Flags: needinfo?(jbuckley)
Flags: needinfo?(jbuckley)

Well, that shouldn't happen.
I've disabled serve-while-stale (which, as I understand it, should only serve stale content when the backend returns an error, which shouldn't be happening here).
Can you let me know if things seem better now, or the same?

It haven't happen since your comment, but I want to wait for a few days to see if it would happen again.

Sounds good!

It works fine. This issue didn't happen since.

Assignee: nobody → cvalaas
Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago
Component: Release Automation: Signing → Operations: Product Delivery
Product: Release Engineering → Cloud Services
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.