Open Bug 1860752 Opened 7 months ago Updated 2 months ago

The APT index of packages.mozilla.org is too large

Categories

(Release Engineering :: General, defect, P2)

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: jlorenzo, Assigned: gabriel)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

First reported by :sylvestre

Steps to reproduce

  1. sudo apt-get update

Expected results

Fetching the packages.mozilla.org index should just be a matter of downloading a few kilobytes.

Actual results

[...]
Get:5 https://packages.mozilla.org/apt mozilla InRelease [1356 B]
Get:6 https://packages.mozilla.org/apt mozilla/main all Packages [21.4 MB]
Fetched 21.4 MB in 2s (9199 kB/s)

Notes

I suspect we either need to tweak Google Artifact Registry or report a bug to them.

Flags: needinfo?(gabriel)

This is the repository index containing the platform-agnostic l10n packages.

There are roughly 200 l10n packages included in each nightly release.

The file that has grown excessively large is here:
https://packages.mozilla.org/apt/dists/mozilla/main/binary-all/Packages

Currently, it stands at 20M, as shown in the Actual results section above.

➜  ~ du -h Packages
 20M	Packages

Google Artifact Registry isn't providing compressed versions of the index, as can be seen here:
https://packages.mozilla.org/apt/dists/mozilla/main/binary-all/Packages.xz

According to the Debian repository documentation:
https://wiki.debian.org/DebianRepository/Format

Compression of indices

... [an] index may be compressed in one or multiple of the following formats:

    No compression (no extension)
    XZ (.xz extension)
    Gzip (.gz extension, usually for Contents files, and diffs)
    Bzip2 (.bz2 extension, usually for Translations)
    LZMA (.lzma extension) 

Clients must support xz compression, and must support gzip and bzip2 if they want to use the files that are listed as usual use cases of these formats. Support for all three formats is highly recommended, as gzip and bzip2 are historically more widespread.

Servers should offer only xz compressed files, except for the special cases listed above. Some historical clients may only understand gzip compression, if these need to be supported, gzip-compressed files may be offered as well.

When I compress the file locally I get the size down to 3M.

➜  ~ du -h Packages
 20M	Packages
➜  ~ xz Packages
➜  ~ du -h Packages.xz
3.0M	Packages.xz

By my count there are 38,683 packages indexed in this file (counted as the lines staring with Package)

Using the timestamp embedded in the nightly Version (and some napkin math) I derived the following from the index:

  • Nearly half the packages (19,668) are older than 90 days.
  • Approximately 65% of the packages are older than 60 days (25,585.)
  • On average, around 6,447 packages are published monthly.

I think we might need a cron job that can go in and clean-up 🧹 the repository as there are no clean-up policies for APT and YUM. Google Artifact Registry should serve compressed indexes... but maybe we could compress the file ourselves and redirect our CDN to the compressed version to hasten the process. I don't know how long it would take Google to start serving compressed indexes from Google Artifact Registry (after we highlight the issue.)

Flags: needinfo?(gabriel)

it should store only the last versions, no?
No need to keep previous one afaik

Oh. If that's the case then maybe there should be a task that cleans up the repo after publishing a new nightly. I guess someone might want to install an specific version of Firefox? Probably not

Specially not on nightly

Marking this as S2 because I don't think there's a work around and the repo is pretty bloated

Severity: -- → S2
Assignee: nobody → gabriel
Status: NEW → ASSIGNED
Priority: -- → P2

Some reasons we might want to keep some non-latest releases in the repository:

  • Sometimes older versions are needed by people dependent on systems or applications that are not compatible with the latest package
  • If there is a critical issue with a new release, having older packages available allows users to quickly rollback to a stable version

I filed the following two support tickets with Google for feature requests:

I think we can work around both of these missing features by adding a separate http handler for XZ compression, and removing old versions on a daily basis.

:jbuck could I get access to those support tickets? It would also be useful to support .gz compression in our request handler.

I've been poking at the APT registry API looking at this issue, I could re-use some of that and write a clean-up script to remove old versions (but I am not sure how we could deploy it and schedule it.)

A little off-topic, but there's also some discussion about the 404 page on Bug 1861929, perhaps we can deal with 404s in a handler too? Return a Mozilla 404 page?

:gabriel - you are CC'd on those tickets. You should just be able to reply. I don't know of an easy way to let you interact with the support portal itself, tho.

We could just deploy a cleanup script as a k8s cron.

We should be able to use nginx to serve up a custom 404, I think.

Depends on: 1863841

How can we deploy this as a k8 cron? Should I package it up in a docker image?

Flags: needinfo?(jbuckley)
Flags: needinfo?(cvalaas)

If you provide a Dockerfile in the repo then github actions can build and deploy it

Flags: needinfo?(jbuckley)
Flags: needinfo?(cvalaas)

(which we can plumb)

Depends on: 1865205

Thanks Chris. Could I have read-only access to the product delivery repository so I can call the read APIs locally while working on Bug 1863841?

Flags: needinfo?(cvalaas)

done, i think?

Flags: needinfo?(cvalaas)

It worked. I can look inside the repository. Thanks!

ping me when you're done so I can remove the bespoke access!

I don't need it anymore, you can remove it

done

(In reply to chris valaas [:cvalaas] from comment #12)

If you provide a Dockerfile in the repo then github actions can build and deploy it

I packaged the script into a docker image and deployed it.

https://github.com/mozilla-releng/mozilla-linux-pkg-manager#docker
https://hub.docker.com/r/mozillareleases/mozilla-linux-pkg-manager/tags

We're setting this up as a cron, yeah? How often? Daily? Weekly?

Flags: needinfo?(gabriel)

I think daily makes sense

Flags: needinfo?(gabriel)

We should trigger it when it is available to clean out the repository

I updated the script's image today btw (I had build an ARM image on accident, it is now an AMD64 image)

Can you make the image run-able as non-root? Our k8s clusters do not allow pods to run with root privs...
Error at the moment:

Creating virtualenv mozilla-linux-pkg-manager-VA82Wl8V-py3.11 in /.cache/pypoetry/virtualenvs
[...]
virtualenv: error: argument dest: the destination . is not write-able at /
Flags: needinfo?(gabriel)

Oh I didn't consider that. Yeah, it should be able to. Looking...

Flags: needinfo?(gabriel)

I modified the docker image to create and use a non-root user. I published it as 0.4.0.

Flags: needinfo?(cvalaas)

I'm not seeing the repo updated (but the new image is on dockerhub) -- what's the UID/GID of the user?

Flags: needinfo?(cvalaas) → needinfo?(gabriel)

Woops, I forgot to push.

pushes

It is 15731. Let me know if that causes issues / should be updated.

Flags: needinfo?(gabriel) → needinfo?(cvalaas)
Attached file logs.txt

Starts up fine now, but crashes. Error/trace attached

Flags: needinfo?(cvalaas)
Flags: needinfo?(gabriel)

(This is a run against the stage env)

There are VPN packages in the staging repo breaking some assumptions I made about the Firefox packages.

I updated the script to be a little more resilient and uploaded a 0.5.0 image.

Flags: needinfo?(gabriel)

Let me know if there are more issues. The output of this new version against the staging repo should be "successfully parsed the package data" and "there are no nightly packages."

Flags: needinfo?(cvalaas)

That's what I'm seeing now. I'll promote the dry-run to prod and see what happens.

Flags: needinfo?(cvalaas)
Attached file logs_prod.txt

Error seen in prod. Note the delete attempt(?) when --dry-run is set.

Flags: needinfo?(gabriel)

The attempt is expected. The script uses a validate_only flag available in the API, so it will send the request, but things will not be deleted. The errors looks like an IAM permissions issue, doesn't look like the script has the capability to send the delete requests

Flags: needinfo?(gabriel)

Sweet. I updated permissions and will run again.

Attached file logs_prod2.txt

Looks like you can only do 50 at a time

Flags: needinfo?(gabriel)

That's... interesting.

The docs say its 10,000 and so does the protocol buffer in Google's client... (maybe that is for docker images?)

Anyhow, I updated the script to do batches of 50

Flags: needinfo?(gabriel) → needinfo?(cvalaas)

Looks good now. Latest run ends with:

2023-12-06 22:04:58,716 - mozilla-linux-pkg-manager - INFO - Done cleaning up!

We good to run it for real?

Flags: needinfo?(cvalaas) → needinfo?(gabriel)

Yeah, that's the expected output, lgtm 👍

Flags: needinfo?(gabriel) → needinfo?(cvalaas)

I don't think it's working ... ?
I ran it twice and it claimed to be deleting tons of stuff each time. But I would expect nothing to happen the second run-through. I checked the repo itself and everything seems to still be there.

I wonder if these two lines:
https://github.com/mozilla-releng/mozilla-linux-pkg-manager/blob/a07f976a468d6f97957a277cc9c070418b64361c/src/mozilla_linux_pkg_manager/cli.py#L145-L146

need to be inside the for started on line 134?

IOW, I think it's only deleting the last "batch" from each "package"

Flags: needinfo?(cvalaas) → needinfo?(gabriel)

Yes. Good catch! It does looks like it only deleted a single batch. Fixing it...

Flags: needinfo?(gabriel)

I patched the indentation error and published the fix

Flags: needinfo?(cvalaas)

So far so good. I'll set this to run weekly (unless you've changed your mind?).

Flags: needinfo?(cvalaas)

It is 12.2M (to compare - vscode 3k - llvm 6k)

Sorry, you said daily. I'll set it to daily. The script finished successfully and we're all good!

Thanks Chris!

It is better now. The indexes with the Firefox packages are ~25k (that would be about ~4k if it was compressed)

The index with the l10n packages was around ~60M+ 😓 and now its at ~1.2M 😅

We could tweak the arguments to the script and delete a few hundred more outdated l10n packages.

I don't think that's going to get us all the way there, but we can get into the kilobyte realm using compressed indexes:

➜ wget https://packages.mozilla.org/apt/dists/mozilla/main/binary-all/Packages
--2023-12-07 12:06:40--  https://packages.mozilla.org/apt/dists/mozilla/main/binary-all/Packages
Resolving packages.mozilla.org (packages.mozilla.org)... 34.160.78.70
Connecting to packages.mozilla.org (packages.mozilla.org)|34.160.78.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1285769 (1.2M) [text/html]
Saving to: ‘Packages’

Packages            100%[===================>]   1.23M   770KB/s    in 1.6s

2023-12-07 12:06:42 (770 KB/s) - ‘Packages’ saved [1285769/1285769]

➜ xz Packages
➜ du -h Packages.xz
152K	Packages.xz
Depends on: 1875338

The updates to the script described on Bug 1875338 will land soon. Once I publish them we can start cleaning-up packages in a more flexible way.

For example to clean-up the devedition/beta packages we can do something like:

mozilla-linux-pkg-manager \
clean-up \
--package "^firefox-(devedition|beta)(-l10n-.+)?$" \
--retention-days 60 \
--repository mozilla \
--region us \
--dry-run \
2>&1 | tee clean-up-devedition-and-beta.log

The packages aren't super bloated with versions yet, but we should keep them in check so it doesn't get as bloated as the nightly.

I don't see a lot of activity on https://issuetracker.google.com/issues/308952923

Should I reach out to someone on our GCP account team? Start considering a workaround?

Flags: needinfo?(cvalaas)

I will ask for an update in our google slack channel

Flags: needinfo?(cvalaas)

The clean-up policies for apt repos seem to be working in preview now :) (tried them out on the dev account)

We could do:

  • A prefix rule matching "firefox-nightly" that keeps only the n most recent versions
  • A prefix rule matching "firefox-beta" and "firefox-devedition"

From slack 3/4/2024 3:11pm:

Hi @cvalaas regarding the first bug to reduce the size of package index, I checked with the product team and unfortunately this feature request has not been prioritized so there is no ETA. I am pushing for them to atleast evaluate it, and will keep you posted

😕

I think we can download and compress the indexes and store the compressed versions somewhere.
Then point at the files - i.e. https://packages.mozilla.org/apt/dists/mozilla/main/binary-all/Packages.{COMPRESSION_FORMAT}

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: