1831952 - migrate symbols server users to use symbols.mozilla.org

Status: RESOLVED FIXED

(Tecken :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

The public symbols bucket is an AWS S3 bucket that's available via HTTP. It used to be the case that we encouraged users to use the bucket url directly rather than using symbols.mozilla.org and then having symbols.mozilla.org redirect to the final URL.

Now that we are planning to move the public symbols to a new place, we should look for users (taskcluster, build systems, analysis systems, etc) that reference the AWS S3 bucket directly and migrate them to use symbols.mozilla.org.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Oops--I tied this to the wrong project.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I searched github for instances of https://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/ because that's the hard-coded url to the public symbols bucket.

python_mozetl / bhr_collection

• https://github.com/mozilla/python_mozetl/blob/main/mozetl/bhr_collection/bhr_collection.py#L1142-L1143

luser / breakpad-scrape-system-symbols

• https://github.com/luser/breakpad-scrape-system-symbols/blob/master/scrapesymbols/gathersymbols.py

mozilla / gecko-dev

• https://github.com/mozilla/gecko-dev/blob/master/tools/crashreporter/system-symbols/win/symsrv-fetch.py#L44-L46
• https://github.com/mozilla/gecko-dev/blob/master/tools/crashreporter/system-symbols/mac/scrapesymbols/gathersymbols.py#L27

squarewave / background-hang-reporter-job

• https://github.com/squarewave/background-hang-reporter-job/blob/master/background_hang_reporter_job/main.py#L688

luser / linux-symbol-scraping

• https://github.com/luser/linux-symbol-scraping/blob/25900f4afceacf9bc1bfbdb494dee977aa3a70a0/scrapedebs.py#L38

I think this is defunct.

kaiostech / gecko-b2g

• https://github.com/kaiostech/gecko-b2g/blob/gonk/tools/crashreporter/system-symbols/win/symsrv-fetch.py#L45

This is the KaiOS fork of B2G. I'll need to talk to someone about this.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I did PRs for these:

• python_mozetl / bhr_collection: https://github.com/mozilla/python_mozetl/pull/384
• luser / breakpad-scrape-system-symbols: https://github.com/luser/breakpad-scrape-system-symbols/pull/9
• squarewave / background-hang-reporter-job: https://github.com/squarewave/background-hang-reporter-job/pull/2
• luser / linux-symbol-scraping: https://github.com/luser/linux-symbol-scraping/pull/1

For the gecko-dev ones, I think I need to make the change in moz-central. That change might need to be uplifted to the various channels including ESR so that when we do the GCP migration, we don't have to keep the AWS S3 bucket for 2 years.

I don't know what to do with the KaiOS B2G fork.

Gabriele: Am I right about what I need to do for the mozilla/gecko-dev instances? What should I do about updating the kaiostech/gecko-b2g repository? And/or should I talk with someone else about this?

Comment 4

[Gabriele Svelto [:gsvelto]]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #3)

For the gecko-dev ones, I think I need to make the change in moz-central. That change might need to be uplifted to the various channels including ESR so that when we do the GCP migration, we don't have to keep the AWS S3 bucket for 2 years.
[...]
Gabriele: Am I right about what I need to do for the mozilla/gecko-dev instances?

Yes, that's correct.

I don't know what to do with the KaiOS B2G fork.
[...]
What should I do about updating the kaiostech/gecko-b2g repository? And/or should I talk with someone else about this?

Let's ask Fabrice. I don't think they're being used though as I never managed to wire it up to automation.

Comment 5

[[:fabrice] Fabrice Desré]

We don't use these symbols for KaiOS shipping builds since they have a different configuration and OEM specific additions. Feel free to do what's easiest from your side.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: Bug 1831952 - switch to use Mozilla Symbols Server. r?gsvelto

This switches two scripts to use the Mozilla Symbols Server for downloading
symbols rather than accessing the AWS S3 bucket directly so they don't stop
working correctly when we move symbols from AWS to GCP.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/647d2166134d
switch to use Mozilla Symbols Server. r=gsvelto

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/647d2166134d

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

I landed the change for bhr_collection. gsvelto said that there's a scheduled task that runs via taskcluster which looks at the scripts in the nightly channel, so we don't need to uplift those changes to other channels.

The other changes I made aren't critical, so we're good here.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

I also sent an email to stability and crash-reporting-wg mailing lists.