mozillavpn crash reports aren't being accepted after extract_payload rewrite
Categories
(Socorro :: Antenna, defect, P1)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
Details
Attachments
(4 files)
Looking at Crash Stats, we have no crash reports for MozillaVPN after March 2nd, 2022. That's when we did the Antenna deployment that rewrote extract_payload.
We need to figure out what's going on and why MozillaVPN crash reports aren't working anymore.
Assignee | ||
Comment 1•3 years ago
|
||
Marcus says he's seeing this when submitting crash reports to stage:
Discarded=malformed_no_annotations
I don't see any Sentry reports on stage in the last few days. Because it's getting discarded there aren't any collector_notes to look at. I checked the logs for stage for "extract payload exception" and don't see anything. I'm not sure what's going on.
I'll add some more logging and deploy that to stage to see if that helps.
Comment 2•3 years ago
|
||
I grabbed a capture of our request that would go to the socorro server.
Assignee | ||
Comment 3•3 years ago
|
||
Here are two annotation errors that increased after the deploy:
date | bad_json | no_annotations | comment |
---|---|---|---|
2/28 | 1 | 2 | |
3/1 | 37 | 13 | |
3/2 | 194 | 46 | deployed new code 16:00ish |
3/3 | 333 | 69 | |
3/4 | 694 | 103 | |
3/5 | 850 | 93 |
"bad_json" gets kicked up here:
That code looks pretty good. In either of those situations, it's not well-formed and the collector shouldn't accept it. The general shape of that code didn't change between the old extract_payload
and the new extract_payload
.
"no_annotations" gets kicked up here:
When there are MultipartParseErrors, they get logged and I'd see them in the logs. I see a slight increase in the number of "invalid text or charset: utf-8".
In bug #1757854, we discovered Fenix was missing an EOL when building the payload between the upload_file_minidump and the next part. When we deployed the new extract_payload
code, it was more strict about well-formedness and parsed the parts differently.
Marcus sent me a couple of captured payloads. After I stopped messing it up by fixing them in a text editor, I noticed that the upload_file_minidump part was missing a Content-Type
declaration. Falcon's multipart handling will assume that means it's text/plain
, but it's definitely not (it's binary) and that's why we're seeing "invalid text or charset: utf-8" in the logs.
Marcus is going to do a fix. I'll look into whether we can improve the error handling. "no_annotations" is the wrong error message here. Anything else would be better.
Assignee | ||
Comment 4•3 years ago
|
||
Comment 5•3 years ago
|
||
I think this will help.
Assignee | ||
Comment 6•3 years ago
|
||
Assignee | ||
Comment 7•3 years ago
|
||
Assignee | ||
Comment 8•3 years ago
|
||
Assignee | ||
Comment 9•3 years ago
|
||
I (finally) deployed this to prod in bug #1763206. I think this should fix Mozilla VPN.
I'm going to keep an eye on this over the next day and see if the numbers change.
Assignee | ||
Comment 10•3 years ago
|
||
This one came in just now: https://crash-stats.mozilla.org/report/index/cdac2060-10e9-49c7-b958-7440e0220405
I'm also no longer seeing the errors that we think were indications of the rejections happening (bad_json, no_annotations).
I'll check in again tomorrow.
Assignee | ||
Updated•3 years ago
|
Assignee | ||
Comment 11•3 years ago
|
||
Everything checks out. We're getting MozillaVPN reports now and there have been no new instances of bad_json and no_annotations. Marking as FIXED.
Description
•