1865415 - consider dropping expiration for /try from 28 days to something lower

Status: RESOLVED WONTFIX

(Tecken :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

/try symbols are ephemeral and it's probably the case they're not used for very long. We currently have a 90-day expiration for those files. However, if they're only used for very long, then we shouldn't keep them for 90 days.

This covers adding some metrics to capture the age of symbols files from /try in weeks (0 weeks, 5 weeks, etc) so we can get a better idea for how long they're used for.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

If we can figure this out now and we can reduce the expiration, that could simplify the S3 -> GCS sync step for the GCP migration. Ergo we should do it soon.

Comment 2

[Sven Marnach :sven]

Attachment: [mozilla-services/tecken] bug-1865415: Emit file age metrics for downloaded symbols files. (#2851)

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

smarnach merged PR [mozilla-services/tecken]: bug-1865415: Emit file age metrics for downloaded symbols files. (#2851) in 4ce46fb.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

This went out just now in bug #1869351.

Sven: Can you add a panel to the Tecken App Metrics dashboard?

Comment 5

[Sven Marnach :sven]

Unfortunately, I can't add a proper histogram to Grafana. I'll try and figure out whether we can use Flux for the queries, as suggested at the linked page.

Alternatively, I could plot 50th, 95th and 99th percentile of the age distribution, which would probably be a good enough solution.

Comment 6

[Sven Marnach :sven]

I added the percentile graphs, and I think they are fine. All try storage downloads that happened so far had a file age of zero days.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Those graphs seem fine to me, too. Thank you!

Comment 8

[Sven Marnach :sven]

After collecting the data for one week, there have been around 20 downloads for files from try storage, and approximately 3 of them were one day old. The rest was zero days old. It looks like downloading files from try storage isn't very popular, and if it does happen the files are fresh. To get a decent amount of data, we should wait a couple of months. I'll keep this bug open.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Per a conversation on January 23rd, you said that objects in try/ are expired after 28 days.

Given that, we're not keeping try symbols for 90 days (I think I got this number from the Tecken GCP project plan after it had been copied/pasted for years of GCP project plans, but we changed it at some point) and we should update this bug and close it as WONTFIX.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

It's currently set to 28 days with Tecken in AWS. It'll be 28 days when we migrate Tecken to GCP. We don't want to drop it further. Given that, marking this as WONTFIX.