1411399 - investigate using requests "files" argument for multipart/form-data payload generation

Status: RESOLVED WONTFIX

(Socorro :: General, task, P3)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Socorro uses poster (https://pypi.python.org/pypi/poster) for generating multipart/form-data payloads in the submitter to send to the Socorro collector as a breakpad crash report.

poster is no longer maintained and it doesn't support Python 3. When I was working on Antenna, I ended up writing my own mini-poster library that implemented part of poster:

https://github.com/mozilla-services/antenna/blob/master/testlib/mini_poster.py

Mike contends that we don't even need that and that we can use requests library "files" argument which will do the right thing:

https://github.com/mozilla-services/socorro/pull/4049#discussion_r146679705

I decided to push that off to a separate bug. Namely this one.

This bug covers removing the poster dependency and reworking the submitter socorro/submitter/breakpad_submitter_utilities.py:BreakpadPOSTDestination and external/http/crashstorage.py:HTTPPostCrashStorage accordingly (it's possible the latter can be removed altogether).

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Whatever we do here, we should do in Antenna, too.

My main concern is that the multipart/form-data output needs to be the same shape (same keys and values, same envelopes for key/value items, same handling for binaries, etc) as what poster produces. (This assumes that poster is producing multipart/form-data output that matches what we get from the breakpad crash reporter. Since we've been using poster for so long, I think we can assume that.)

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

We just nixed poster (bug #1457349), so I'm changing the summary.

We have multipart/form-data payload generating code in Antenna which we use for testing and we have it in the socorro-submitter which we use for submitting crashes to the stage environment.

Using requests might be helpful in both these cases as long as we can guarantee the payloads are similar to what the crash reporter sends and what the Socorro collector is expecting.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

We nixed the poster dependency when we nixed the submitter code and the http crashstorage, so this no longer blocks the switch to Python 3.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I looked into this. Using the "files" argument produces a different payload than the one we're building with the mini-poster code. For example, the raw_crash contains a bunch of metadata like "Product".

mini-poster serializes that like this:

Content-Disposition: form-data; name="Product"
Content-Type: text/plain; charset=utf-8

Firefox

Using the "files" argument in requests produces this:

Content-Disposition: form-data; name="Product"; filename="Product"

Firefox

That's different than what the crash reporter is producing. Since we're using this in scenarios where we're imitating the crash reporter, I think I don't want to go forward with this.