1364515 - Symbol Upload in Tecken

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Peter Bengtsson [:peterbe]]

The basic functionality is documented in https://docs.google.com/document/d/1H7S2GPNL10ILAUSY1s8lf9FN34Gu2v5ogVTTIiqD7XQ/edit

In basics it's...

* Accept HTTP POST uploads of .zip files
* Open the .zip files, do some basic checks (https://github.com/mozilla/socorro/blob/6770b199a7cdb78056a6c31bac9289a4c4bab3f3/webapp-django/crashstats/symbols/views.py#L47)
* Figure out which bucket to upload to depend on config (https://github.com/mozilla/socorro/blob/6770b199a7cdb78056a6c31bac9289a4c4bab3f3/webapp-django/crashstats/symbols/views.py#L150)
* Do the actual upload to S3. Check each file first if the file exists and has the exact same size. Compress certain files based on extension. (https://github.com/mozilla/socorro/blob/6770b199a7cdb78056a6c31bac9289a4c4bab3f3/webapp-django/crashstats/symbols/views.py#L58)
* Write down (in the ORM) who uploaded what when and how each file within the .zip was either updated or skipped.

Comment 1

[Peter Bengtsson [:peterbe]]

This bug ignores and bypasses the concept of authorization. I.e. it won't look for either cookie or API token. However, every upload needs a user. Both for persistent logging and for ability to figure out which bucket it should upload to. I'll implement a temporary hack around that. 

The goal is to get parity with Socorro's symbol upload. However the first version of this will NOT be using a regular web form POST upload. Just curl. 

Also, unlike the Socorro symbol upload, the intention is to do a better job of storing which files were uploaded individually. In Socorro we used to just record the uploader, date and a multiline text blob of how each file upload did.

Comment 2

[Peter Bengtsson [:peterbe]]

Another major difference between how Socorro used to do it is that we now want to do the symbol uploads as background jobs. 
Meaning, the client uploads a .zip file full of files. We load the .zip file into memory (upon upload) and if the content looks sane we terminate the request with an error code 201. Then, immediately after, in a background process we start uploading the individual files to S3.

Comment 3

[Peter Bengtsson [:peterbe]]

PR https://github.com/mozilla-services/tecken/pull/142

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/tecken

https://github.com/mozilla-services/tecken/commit/1441f0ec5fbaf62dd0f2c3e1141c9379bf37884a
Bug 1364515 symbol upload (#142)

* fixes bug 1364515 - symbol upload

* small cleanups

* use post_migrate to create default groups

Comment 5

[Peter Bengtsson [:peterbe]]

Hmm... failed to include "fixes" in the first commit.