[meta] Use crash-stats.mozilla.com for iOS crash reporting

RESOLVED WONTFIX

Status

()

Firefox for iOS
General
RESOLVED WONTFIX
3 years ago
2 years ago

People

(Reporter: sleroux, Assigned: sleroux)

Tracking

(Depends on: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Assignee)

Description

3 years ago
Given the lack of information we get from TestFlight and Apple when it comes to crashes, we need a crash reporting solution for iOS. Ideally we would integrate Breakpad so we can hook into the existing crash-stats/socorro infrastructure to be consistent with other platforms. Also, socorro's dashboard looks way more useful than anything from Apple.
Lars, Lonnen: what plates would we need to start spinning on your end to make this happen?
Flags: needinfo?(lars)
Flags: needinfo?(chris.lonnen)
(Assignee)

Comment 2

3 years ago
FYI I did some exploratory work this morning into integrating Breakpad and was able to get a version of it compiling into a dynamic framework so we can pull it in with Carthage. I had a talk in the #breakpad channel with ted/rhelmer about how to move forward on this. One thing I wanted to confirm before moving forward is if this will interpret swift crashes correctly as well.
Socorro, in some of its older modules, has hard coded OS names.  The amount of effort in getting Socorro to support iOS crashes hinges on what strings breakpad/stackwalker use identify the OS when processing a crash.  I don't recall the relationship between OSX and iOS.  If they share the same roots and an iOS crash identifies itself as being "Mac OSX", then Socorro problems will be minimal.  Otherwise we have three options:

1) sweep the Socorro codebase and eliminate the hard coded OS names [heavy]
2) add "iOS" to the hard coded legacy modules [less heavy]
3) use processor rewrite rules to rewrite "iOS" as "OSX" and allow those two OSes to be lumped together [trivial]

Lonnen will have more of an idea of how this fits into the grand scheme and schedule of Socorro development.
Flags: needinfo?(lars)

Comment 4

3 years ago
Next step is to get a sample crash or five from iOS. I'd like to support iOS and I think we can use transform rules to get there quickly. Need a better sense of scope.
Flags: needinfo?(chris.lonnen)
(Assignee)

Comment 5

3 years ago
What would you need to get this started from the iOS side? I've done some preliminary work in getting Breakpad into our iOS project. Would we need to fire away some crashes to crash-stats? Alternatively I could generate crash reports locally from Breakpad and send them over.
Flags: needinfo?(chris.lonnen)

Comment 6

3 years ago
Please generate a few and send them our way. We're flexible on the means for that... I'm not sure how big they are.
Flags: needinfo?(chris.lonnen)

Comment 7

3 years ago
(In reply to K Lars Lohn [:lars] [:klohn] from comment #3)
> 1) sweep the Socorro codebase and eliminate the hard coded OS names [heavy]

There's a bug around somewhere to do that in the long run, but I don't think we need to rush that atm.

> 3) use processor rewrite rules to rewrite "iOS" as "OSX" and allow those two
> OSes to be lumped together [trivial]

I think that's what we should do for now, just like we already are doing it for Android->Linux.
(Assignee)

Comment 8

3 years ago
Created attachment 8632257 [details]
iOS Crash.zip

I've integrated Breakpad into the iOS client and managed to grab a crash from the device. I've zipped up the .dmp and breakpad symbols along side of it. I'm in the middle of getting the app to start dumping crashes to the server so I'll let you know when that's setup,
(Assignee)

Comment 9

3 years ago
I have Breakpad working with Firefox for iOS but it seems the only thing that's missing is a proper server URL to start submitting our crash logs to. I've tried using https://crash-reports.mozilla.com and https://crash-reports.mozilla.com/submit but I'm getting an internal server error back. Any clue on what the URL should be?
(Assignee)

Comment 10

3 years ago
Created attachment 8636131 [details]
crash-reports-log.chls

I've attached a log of the request/response from Charles (http://www.charlesproxy.com/) if that helps with debugging.

Comment 11

3 years ago
Are you using the parameters that e.g. https://dxr.mozilla.org/mozilla-central/source/build/application.ini#50 has on there?
(Assignee)

Comment 12

3 years ago
Ah no - forgot the query params. I've added them in and still getting the 500 internal server response:

POST /submit?id=org.mozilla.ios.Fennec&version=1.0&buildid=26 HTTP/1.1
Host: crash-reports.mozilla.com
Proxy-Connection: keep-alive
Accept-Encoding: gzip, deflate
Content-Type: multipart/form-data; boundary=---------------------------000041A710D63AF1
Content-Length: 159851
Accept-Language: en-us
Accept: */*
Connection: keep-alive
User-Agent: Client/26 CFNetwork/711.0.6 Darwin/14.0.0
(Assignee)

Updated

3 years ago
Assignee: nobody → sleroux
(Assignee)

Updated

3 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 13

3 years ago
This might be a silly question but do I need an API key with crash-stats in order to submit crashes to crash-reports.mozilla.com/submit? If so, how would I get one? I also notice that the application ID we're using for FF is string formatted like so: {ec8030f7-c20a-464f-9b0e-13a3a9e97384} and I'm sending the bundle identifier. Is this another requirement?
(Assignee)

Comment 14

3 years ago
Ah nevermind - found the problem. Breakpad by default expects a Google server. I've updated it to use Socorro and it's reporting fine now. Here's a test exception I was able to send: https://crash-stats.mozilla.com/report/index/a9e2987a-8bbb-45eb-befb-ea0272150722
(Assignee)

Updated

3 years ago
Summary: Use crash-stats.mozilla.com for iOS crash reporting → [meta] Use crash-stats.mozilla.com for iOS crash reporting
(Assignee)

Comment 16

3 years ago
Comment on attachment 8637472 [details] [review]
PR https://github.com/mozilla/firefox-ios/pull/717

Going to move this PR into a sub bug and change this to meta
Attachment #8637472 - Attachment is obsolete: true
Attachment #8637472 - Flags: review?(rnewman)
(Assignee)

Updated

3 years ago
Depends on: 1186638

Comment 17

3 years ago
Looking at bp-a9e2987a-8bbb-45eb-befb-ea0272150722 - first, it's awesome that we have something working there!

Second, the "product" is right now set to "Client", which is quite suboptimal. It should be something like "FenneciOS" or similar. Also, we should send a "BuildID" and "ReleaseChannel", even if we generate something that just sounds right, as we have a lot of stuff on the crash-stats side that expect them. A BuildID that is based on a timestamp of when the build was compiled would be great, say 20150722033000 for a compile that started on 2015-07-22 at 03:30:00. The ReleaseChannel can be just "release" for release builds, possibly a "nightly" for daily test builds, basically anything that fits what desktop and Android Firefox have so we can run the same kind of analysis functionality.

Also, what's confusing is that there is a "Version" field set to 26 and a "version" field set to "1.0" - it would be best to have a "Version" field that is whatever we consider the official version of Firefox for iOS, as that's what we look at in analysis - if a "version" exists as well, it should be the same as "Version".

See the Metadata tabs of e.g. bp-cec18a9a-3a8e-438b-93a0-1f2bb2150722 (FennecAndroid nightly) or 8c94b771-c52d-4ca1-9874-7bf032150720 (Firefox release for Mac OS X) for what fields we send on other products.
(Assignee)

Comment 18

3 years ago
Hey Robert,

I've added the following parameters so it's in parity with Android:

ReleaseChannel
Vendor
ProductName

I can also add BuildID but was wondering if you knew where the script that generates this for Desktop/Android is on mozilla-central. I can just copy that over and add it to our build chain. At the moment I can send the app's bundle identifier as the ProductID but it looks like Android and Desktop have a ProductID formatted like so: {3c2e2abc-06d4-11e1-ac3b-374f68613e61}. Should I be sending a similar token for iOS?
Flags: needinfo?(kairo)

Comment 19

3 years ago
From all I see, https://dxr.mozilla.org/mozilla-central/source/toolkit/xre/make-platformini.py#27 is what's generating a build ID when none is fed to the build process, called by https://dxr.mozilla.org/mozilla-central/source/config/Makefile.in#52 - but for our nightly and release builds, an ID is actually fed into the process by the releng scripts.

Ben, can you help regarding where the releng process does that?

sleroux, I think you probably can use the simple variant that make-platformini.py is using.


Regarding the ProductID, for Gecko applications it's either a UUID in the form you pointed to above or a string like e.g. webapprt@mozilla.org - I think we should prefer the latter form.
Flags: needinfo?(kairo) → needinfo?(bhearsum)
(Assignee)

Comment 20

3 years ago
Thanks for the info Rob. I've attached a similar date generation script for the BuildID to execute on every build so I don't think I'll need any additional info on that. It runs as a pre-action before compiling the application so there shouldn't be any additional work needed.

As for the AppID, I'll send over fennecios@mozilla.org.

Comment 21

3 years ago
That sounds good!

Also clearing the request from Ben in this case.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 22

3 years ago
We're about to put out a build with breakpad/crash-stats included to our external testers and was wondering a couple of things:

1. Is there a simple way for us to get a list of crash reports for a given version/build number without additional API work? So far I've only been able to query by a specific ID.

2. For symbolication, who would be the right people to talk to attach the symbols to the test build? Is there additional work needed for this to happen on the Socorro side?

Thanks!
Flags: needinfo?(kairo)
(In reply to Stephan Leroux [:sleroux] from comment #22)
> 2. For symbolication, who would be the right people to talk to attach the
> symbols to the test build? Is there additional work needed for this to
> happen on the Socorro side?

You just need to get a token with symbol upload permission, and then you can upload symbols to Socorro's symbol upload API. Log in to https://crash-stats.mozilla.org with Persona, then file a bug (CC lonnen) asking for symbol upload permission for your account (like bug 1170280). Once you have that you can generate a token ( https://crash-stats.mozilla.com/api/tokens/ , make sure to select "Upload Symbols Files"), and then you can follow the directions at https://crash-stats.mozilla.com/symbols/upload/api/ or use a script like https://github.com/luser/breakpad-scrape-system-symbols/blob/master/uploadsymbols.py to upload a symbols.zip file.

Let me know if you have any other questions about that.

Comment 24

3 years ago
(In reply to Stephan Leroux [:sleroux] from comment #22)
> 1. Is there a simple way for us to get a list of crash reports for a given
> version/build number without additional API work? So far I've only been able
> to query by a specific ID.

Queries are the way to go right now, we'll need to set up a product in Socorro to make things easier and we'll need to feed release information to Socorro in some way. I'm on vacation until Monday, let's clear the details for this next week, it's all server-side so should not block the build itself.

That said, Rob, are you able to help here with what we need to get this up on Socorro?

For the symbols stuff, Ted is the better contact anyhow. :)
Flags: needinfo?(rhelmer)
(In reply to Stephan Leroux [:sleroux] from comment #22)
> We're about to put out a build with breakpad/crash-stats included to our
> external testers and was wondering a couple of things:
> 
> 1. Is there a simple way for us to get a list of crash reports for a given
> version/build number without additional API work? So far I've only been able
> to query by a specific ID.


You should be able to search by version and buildID on https://crash-stats.mozilla.com/search/ does that work?

crash-stats also has an API at https://crash-stats.mozilla.com/api/ but I think the search should do what you need.


> 2. For symbolication, who would be the right people to talk to attach the
> symbols to the test build? Is there additional work needed for this to
> happen on the Socorro side?
> 
> Thanks!


I think comment 23 and bug 1191309 cover this.
Flags: needinfo?(rhelmer)
(Assignee)

Comment 26

3 years ago
Yup - being able to filter by product/buildid is perfect for now.

As for uploading symbols, how would I upload symbols for different architectures (armv7/arm64) and should these files follow some sort of naming convention?

Thanks again for all the help on getting this setup - almost there!
Flags: needinfo?(rhelmer)
(In reply to Stephan Leroux [:sleroux] from comment #26)
> Yup - being able to filter by product/buildid is perfect for now.
> 
> As for uploading symbols, how would I upload symbols for different
> architectures (armv7/arm64) and should these files follow some sort of
> naming convention?


Redirecting this question to ted - I think we have different debug IDs for different architectures, but not sure.


> 
> Thanks again for all the help on getting this setup - almost there!
Flags: needinfo?(rhelmer) → needinfo?(ted)
(Assignee)

Comment 28

3 years ago
Hey Ted,

I wanted to follow up about how we can get our iOS logs symbolicated. I have symbols for some of the builds already on crash-stats but was wondering what needs to happen to get the logs that are already there symbolicated.
(In reply to Stephan Leroux [:sleroux] from comment #28)
> Hey Ted,
> 
> I wanted to follow up about how we can get our iOS logs symbolicated. I have
> symbols for some of the builds already on crash-stats but was wondering what
> needs to happen to get the logs that are already there symbolicated.

We should be able to just reprocess the crashes, now that the symbols are available. Generally crashes are processed right after being collected, and we re-process if something changes (symbols, processor code, etc.)

Should we re-process all crashes with "iOS" as OS name, and how far back?
(Assignee)

Comment 30

3 years ago
I have the symbols uploaded for version 1.0.0 buildid 30, 31, and 32. I'm not sure if there is a special format for this to automagically happen but I've prefixed each symbol file with the release channel like so:

aurora.1.0.32.armv7.sym

Where aurora = release
1.0 = version
32 = buildid
armv7 = architecture
Ah, it's not well-documented (and peterbe has been bugging me to write the docs I said I'd write), but the symbols need to be in a specific directory structure, specifically:
<filename>/<debug id>/<filename>.sym

If you look at the first line of the .sym file (starts with MODULE) you'll see the debug ID (the GUID there) and the filename is just the filename of the binary or dylib. There's a concrete example on the Breakpad wiki:
https://code.google.com/p/google-breakpad/wiki/LinuxStarterGuide#Producing_symbols_for_your_application

The stackwalker looks up symbol files by using the filename and debug ID from the minidump. Let me know if that helps or if you need more info.
Flags: needinfo?(ted)
(Assignee)

Updated

3 years ago
Depends on: 1195986

Updated

3 years ago
Depends on: 1196469
(Assignee)

Updated

3 years ago
Depends on: 1196782
(Assignee)

Updated

3 years ago
Depends on: 1196442

Updated

3 years ago
Flags: needinfo?(kairo)

Updated

3 years ago
Depends on: 1226242
(Assignee)

Comment 32

2 years ago
For the time being, I'm going to mark this as WONTFIX. Apple's Xcode crash reporter has been giving us all of the information we have needed so far (sybmbolicated crashes, crash rates, etc). When we want to investigate additional crash reporting options, we can reopen.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.