1422096 - Try build uploads

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Peter Bengtsson [:peterbe]]

Currently, there's a permission called "Upload Symbol Files". Let's add another permission called "Upload Try Symbol Files". When this permission is used, it will use a different S3 bucket+prefix configuration. One dedicated to Try builds. 

On the download side, we need to make downloading with Try builds opt-in. 
If you do `GET /foo.pdb/ABC123/foo.sym` it will only look in S3 bucket+prefix combos that are NOT for Try. If you want to include Try builds, you need to instead do `GET /foo.pdb/ABC123/foo.sym?try`.

Comment 1

[Peter Bengtsson [:peterbe]]

Ted, 
There's an opportunity here for big optimization, but I can't decide if it's worth it based on my lack of understanding of Try builds. 
If I make a try build for a niche feature I'm interested in, when it builds, will 99% (or some percentage) of the symbols have the exact same name and size as a symbol file from Nightly from whence my Try build is likely to be forked from?

Basically, when we iterate over the files in the zip and make a decision to upload or not, can we look in the general public S3 configuration too?

Comment 2

[(not currently active) Ted Mielczarek]

(In reply to Peter Bengtsson [:peterbe] from comment #1)
> Ted, 
> There's an opportunity here for big optimization, but I can't decide if it's
> worth it based on my lack of understanding of Try builds. 
> If I make a try build for a niche feature I'm interested in, when it builds,
> will 99% (or some percentage) of the symbols have the exact same name and
> size as a symbol file from Nightly from whence my Try build is likely to be
> forked from?

It's probably not likely, unfortunately, as the builds we ship to users are built differently from the default build configuration (they're PGO optimized on Windows/Linux).

> Basically, when we iterate over the files in the zip and make a decision to
> upload or not, can we look in the general public S3 configuration too?

We certainly could do this, it wouldn't be harmful, but I'm not sure if it would buy us much. It would be easy enough to check, I suppose, as try builds currently do produce symbols.zip files just like other builds, we just don't upload them to Socorro. We could modify that script I wrote you that fetches a list of symbols.zip files to get a list from try builds instead.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/tecken/pull/609

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/b41ffc8ebfe5cff23e15518b4310b2aadf8b01d6
Bug 1422096 try build uploads (#609)

* bug 1422096 - Try build uploads

* test passing

* better tests for try symbols

* bug 1422096 - Try build uploads

* prettier

* cleanup of broken tests

* fixes

* small fix to docs

Comment 6

[Peter Bengtsson [:peterbe]]

The code is in master (and Dev) and seems to work. 
Once this lands, if you don't explicitly set a DJANGO_UPLOAD_TRY_SYMBOLS_URL it becomes the same as DJANGO_UPLOAD_DEFAULT_URL but with a "/try" prefix (before any other prefixes like "v1"). 

https://bugzilla.mozilla.org/show_bug.cgi?id=1422978 is about configuring our production S3 bucket to have a different prefix per that specific prefix. Miles is waiting for Ted on a needinfo there.