1085530 - store uploaded symbols to S3

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Robert Helmer [:rhelmer]]

The Django app has an API and UI for uploading symbols, which are stored to an NFS mount. We are moving symbol storage to S3 so this needs to move too.

Comment 1

[Robert Helmer [:rhelmer]]

We should store some metadata about uploaded crashes into a (TBD) postgres table, for cleanup purposes (there are common symbol files across different builds, so we can't just use a naive S3 expiration policy.)

Also, we should expect releng to use this API instead of storing crashes straight to S3. I don't think we need to do anything specific to support this, just something to be aware of.

Comment 2

[(not currently active) Ted Mielczarek]

(In reply to Robert Helmer [:rhelmer] from comment #1)
> We should store some metadata about uploaded crashes into a (TBD) postgres
> table, for cleanup purposes (there are common symbol files across different
> builds, so we can't just use a naive S3 expiration policy.)

This is bug 914106.

Comment 3

[Robert Helmer [:rhelmer]]

We have two S3 buckets, for private and public symbols:

org.mozilla.crash-stats.symbols-public
org.mozilla.crash-stats.symbols-private

There's a "/v1/" pseudo-folder, and then it's the usual structure you'd find on NFS ("./symbols_os", "./symbols_ffx", and so on)

Comment 4

[Peter Bengtsson [:peterbe]]

Do we have a preference for how the files are supposed to be named/stored in the org.mozilla.crash-stats.symbols-public bucket? 

In the cron job we simply unpack every file in a certain directory into another directory. 
So, should I upload all uploads into an "incoming" prefix (or something)?

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #4)
> Do we have a preference for how the files are supposed to be named/stored in
> the org.mozilla.crash-stats.symbols-public bucket? 
> 
> In the cron job we simply unpack every file in a certain directory into
> another directory. 
> So, should I upload all uploads into an "incoming" prefix (or something)?

I don't think there's value in an "incoming" prefix, I'd just unpack straight into the /v1/ prefix.

Comment 6

[(not currently active) Ted Mielczarek]

Yes, this is what we want. It's fine if we overwrite existing files, files with the same name will have the same content (this is what we currently do). If you wanted to be smarter and check if files exist and simply leave the existing one in that case that would also be fine.

Comment 7

[Peter Bengtsson [:peterbe]]

PR https://github.com/mozilla/socorro/pull/2607

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/91db6f053cfc1ba8524417c2c58345c6338e9939
fixes bug 1085530 - store uploaded symbols to S3, r=@rhelmer

https://github.com/mozilla/socorro/commit/f9cdf40ae68ea52162337ddb691b01fa5460d3ec
Merge pull request #2607 from peterbe/bug-1085530-store-uploaded-symbols-to-s3

fixes bug 1085530 - store uploaded symbols to S3

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/7d096f6fffd18713e605ba768f9b4bbca0023947
Revert "fixes bug 1085530 - store uploaded symbols to S3, r=@rhelmer"

This reverts commit 91db6f053cfc1ba8524417c2c58345c6338e9939.

https://github.com/mozilla/socorro/commit/3bfc900ed49f5d50f5812cb83e2b9cdae6099206
Merge pull request #2657 from rhelmer/bug1085530-backout-s3-upload

Revert "fixes bug 1085530 - store uploaded symbols to S3, r=@rhelmer"

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/1a4e4fcc82f71d5ef3e372dc854677c135d46a9c
Reland "fixes bug 1085530 - store uploaded symbols to S3, r=@rhelmer"

This reverts commit 7d096f6fffd18713e605ba768f9b4bbca0023947.

https://github.com/mozilla/socorro/commit/0384757b15799cbdb5e100d17cde9986409ce506
Merge pull request #2658 from rhelmer/bug1085530-reland-s3-symbol-upload

Reland "fixes bug 1085530 - store uploaded symbols to S3, r=@rhelmer"