1422923 - Remove symbols upload from Socorro

Status: RESOLVED FIXED

(Socorro :: Webapp, task, P2)

Description

[Peter Bengtsson [:peterbe]]

As of https://bugzilla.mozilla.org/show_bug.cgi?id=1395647 all symbol uploads should go via symbols.mozilla.org and not via crash-stats.mozilla.com.

People who try to go there to upload symbols have been informed they should use the new place. There is no need to use the UI in Socorro to upload symbols or generally even look at the symbols uploaded in the admin interface. 

For the benefit of being careful we can tackle the UI around uploading symbols first and deal with the actual API of uploads in a later bug. 

We can either remove ALL UI elements related to symbols and let it 404/500 if someone still tries to use it. Or we can put up a redirect to symbols.mozilla.org for all things /symbols. Or a 404 page that specifically mentions to go to https://symbols.mozilla.org

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

There's a graph in the Socorro -prod dashboard that shows who's using the web API. By the time bug #1395647 is resolved, we should have accumulated enough data to determine who's using it and how often. If no one is using it, then we should just nix all the code.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Er, by "web API", I really meant "web ui for symbols uploads".

Comment 3

[Peter Bengtsson [:peterbe]]

I'm happy to help with a patch, but to save some time, can you say if you have a preference how you want this?
Completely 404 those old UI URLs? Or rewrite the template to say "Go to Mozilla Symbol Server instead"? Or a 301 redirect?

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think a 301 redirect sounds super!

If you wanted to do that, that'd be great! If not, then maybe someone will get to it next week.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to do now.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Just to toss some data in, I haven't seen a web-based upload in a while, so it's not being actively used anymore.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm tweaking the scope of this a bit to cover all of removing symbols upload from Socorro. That includes the following work:

1. removing web upload and API upload code
2. adding a 301 redirect for /symbols/* to https://symbols.mozilla.org/
3. removing symbols related configuration from .env files and consul

Peter: Is it ok to also delete the SymbolsUpload model which will delete all the data in the symbols_symbolsupload tables?

Comment 9

[Peter Bengtsson [:peterbe]]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #8)
> I'm tweaking the scope of this a bit to cover all of removing symbols upload
> from Socorro. That includes the following work:
> 
> 1. removing web upload and API upload code
> 2. adding a 301 redirect for /symbols/* to https://symbols.mozilla.org/
> 3. removing symbols related configuration from .env files and consul
> 
> Peter: Is it ok to also delete the SymbolsUpload model which will delete all
> the data in the symbols_symbolsupload tables?

Yes! But it might be a good idea to PG dump that data before it disappears. 
That way, if something - against all odds - comes up and we need to figure out who or what someone uploaded historically, it's good to have. 

We did something similar with the old PG data for old crash-per-user and we generated a big JSON blob or something and put it into Google Drive.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Ok. I'll dump the data and put it somewhere and then delete the tables. Thank you!

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Miles: I'm not sure how to get a dump of the symbols_symbolsupload table (which is really big). I need to run something like this:

    pg_dump --verbose -Z --data-only --table=symbols_symbolsupload \
        --username=<USERNAME> \
        --host=<HOST> \
        <DATABASE> > symbols_symbolsupload.dmp

I did a size query in psql on that table and it suggests it's 52 GB. I don't know how big the resulting dump file might be, but that's too big to run on the -prod admin node.

Should we spin up an AWS node just to do this? Any other ideas on how to do it?

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4323: bug 1422923 - removes symbols upload ui and code

Comment 13

[Miles Crabill [:miles]]

I've added a 1tb EBS volume to the prod admin node. It's mounted at /mnt/robert.

$ df /dev/xvdf
Filesystem      1K-blocks  Used Available Use% Mounted on
/dev/xvdf      1031987352 77848 979464320   1% /mnt/robert

I hope that's sufficient for your needs!

Comment 14

[Will Kahn-Greene [:willkg] ET needinfo? me]

Miles: That was perfect--thank you!

I dumped the table, gzipped it to make it smaller, uploaded it to Google Drive, and shared it with some peeps.

I'll add a migration to drop the table and clean up the Django app and all that.

Comment 15

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/86d829f2ebcb9be1809bf29f6e964390514166d9
bug 1422923 - removes symbols upload ui and code

This removes all of the code from the symbols app. It includes a db migration
to drop the symbols_symbolsupload table. It keeps the minimal bits for a
Django app so that the migration runs. We'll keep that for a while and then
remove that, too.

This also creates a redirect from /symbols/* to https://symbols.mozilla.org/
in case people try to go here for symbol uploads.

https://github.com/mozilla-services/socorro/commit/fead237b3bcd9bcb5f8579dada623b01928fea5c
Merge pull request #4323 from willkg/1422923-symbols-ui

 bug 1422923 - removes symbols upload ui and code

Comment 16

[Will Kahn-Greene [:willkg] ET needinfo? me]

That last PR removed symbols upload completely. Given that, I'm marking this FIXED.