1187957 - Symbol upload API should check that .sym files in upload have the right path

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[(not currently active) Ted Mielczarek]

We've had a problem with Adobe uploading symbol files that aren't in the right directory structure, which means they can't get located for use in processing reports. .sym files actually contain the info needed to determine the path, so we should make the API sanity check that every .sym file in the upload is in the right place.

The first line of a .sym file will start with MODULE and look like this:
MODULE Linux x86_64 6EDC6ACDB282125843FD59DA9C81BD830 test

https://code.google.com/p/google-breakpad/wiki/SymbolFiles#MODULE_records

That's:
    MODULE operatingsystem architecture id name 

where "name" can contain spaces.

Given that line, the file should be at the relative path <name>/<id>/<name>.sym, with the only caveat being that if <name> ends with .pdb, the filename at the end drops the .pdb.

Examples:
http://symbols.mozilla.org/msvcr120.i386.pdb/16F5E2EF340A453ABC8B8F67DC6FD8082/msvcr120.i386.sym
MODULE windows x86 16F5E2EF340A453ABC8B8F67DC6FD8082 msvcr120.i386.pdb

http://symbols.mozilla.org/js/7D2FD70E34E4872AE22E2DBEA5A232370/js.sym
MODULE Linux arm 7D2FD70E34E4872AE22E2DBEA5A232370 js

Comment 1

[Peter Bengtsson [:peterbe]]

Pardon my blond brain today, but...

So for every .sym file in the archive, we look at its content and pluck out the name (which might contain spaces) the id.
Then we check that the .sym file we just opened was /<name>/<id>/<name>.sym?
So that checks the .sym file names. 

But where does .pdb come into this? If the first line is `MODULE windows x86 16F5E2EF340A453ABC8B8F67DC6FD8082 msvcr120.i386.pdb` do I simply drop the .pdb and apply the same check?

Comment 2

[Peter Bengtsson [:peterbe]]

I should have done that when I replied. I need more info from you Ted.

Comment 3

[(not currently active) Ted Mielczarek]

Very close! It's a little confusing, admittedly. If the name ends with .pdb then it should be:
<name>/<id>/<name minus .pdb>.sym

If the name does not end with .pdb then it should be:
<name>/<id>/<name>.sym

You can see the two files in comment 0 for concrete examples.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm bumping this to the Symbols component.

Peter: Does Tecken handle this and/or is this relevant?

Comment 5

[Peter Bengtsson [:peterbe]]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #4)
> I'm bumping this to the Symbols component.
> 
> Peter: Does Tecken handle this and/or is this relevant?

Yep. Tecken does a good job validating the structure of each file in the zip. Also, it returns the relevant message in the body in terms of a 400 Bad Request so it'd be easy to debug what's going on. Plus we now have a dedicated ReadTheDocs page about the upload from a users point of view.

So, as of https://github.com/mozilla-services/tecken/blob/ec2749ada20682b90262ed0ddd5082ce2470bb19/tecken/upload/views.py#L59-L77 and us soon switching to Tecken, shall we just close this issue once we do the migration? 

Actually, since there's nothing actionable to make out of this bug other than waiting for the Tecken migration, which will happen, I say we close this now.