Closed Bug 521349 Opened 11 years ago Closed 10 years ago

crash reporter should submit to a URL including some client info for stat mining from access logs

Categories

(Toolkit :: Crash Reporting, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
Tracking Status
blocking1.9.2 --- -
status1.9.2 --- wanted
status1.9.1 --- wanted

People

(Reporter: ted, Assigned: sayrer)

References

Details

Attachments

(1 file)

Currently we submit all the useful info inside the POST. If we stuck some of it in the URL, it would make mining that info out of collector logs easier.

Morgamic: what data would be most useful there? Anything Socorro currently gets as part of the POST would be easy enough to put in the URL.
Depends on: 521351
I'd put all of it in and have a base redirect that points to the post handler.
The only catch is I'd highly recommend the first part of the path be a version, in case the URL changes over time.  We do that with other services.  AUS2, for example, has URLs that look like this:
https://aus2.mozilla.org/update/2/Firefox/2.0.0.13/2008031115/Darwin_Universal-gcc3/en-US/beta/Darwin%209.0.0/update.xml

But we should probably chat with Kovash about what fields we'd want to make sure to collect.
When you say "all of it" surely you don't mean all the stuff like "crash time" "install time" etc.

I can easily give you Vendor/Product/Version/BuildID. We don't currently have the ABI or locale available, but we could probably make those happen.
I'd recommend using the product uuid instead of name, it is much easier to handle.  If we could come close to what blocklist logs, we'd be doing very well.  I don't think the other crash metadata would be very useful for the logfile analysis.  We'll already be doing processing and statistics on the POST payload for that.

Here is the URL from blocklist:
extensions.blocklist.url = https://addons.mozilla.org/blocklist/3/%APP_ID%/%APP_VERSION%/%PRODUCT%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/
Oh, one more thing that could be extremely useful although there are privacy implications:

If we had a uuid that we could use to track number of crashes from a unique profile, it would be really valuable for metrics such as the number of crashes per user and such.  Ideally, the uuid would need to exist in both the payload and in the request so we can link them if needed.
(In reply to comment #5)
> If we had a uuid that we could use to track number of crashes from a unique
> profile, it would be really valuable for metrics such as the number of crashes
> per user and such.  Ideally, the uuid would need to exist in both the payload
> and in the request so we can link them if needed.

That was taken off the table in bug 444351. There are also some intranet posts on it, which make me sad, but such is life.
Ehh, if the developers making use of the service don't think the value of tracking number of crashes per installation outweighs the implication of a uuid, then it should be there.  The service is here to help the developers to whatever extent they want.
Blocks: 521362
Duplicate of this bug: 572206
Our crash spike investigations have highlighted how useful this would be.  Can we schedule this ASAP?
I have too many other things on my plate at the moment, but if you can find someone I can point them at the right places to fix this.
we're going to fix this tonight so that we can have the option of pushing a 3.6.4 beta update with trackable reporter URLs.
Pedro, please have your team be prepared to move as fast as possible on this if they do a beta push they'll want to be able to track stats right away.
The data would end up looking almost identical to blocklist I imagine, just a much smaller traffic volume.

If it is easier, we could look at using my ETL instead of webetl for it.
Roger that, I'll sync with you tomorrow on this
good news is that we seem to be able to process reports with query strings appended to the submission URL.

http://crash-stats.mozilla.com/report/index/2942d947-5b61-4a7a-ac5e-cfc862100616

was submitted to 

https://crash-reports.mozilla.com/submit?sayrer=test
for those following along at home, the default server URL lives in application.ini

you have to chase it down through like 4 levels of indirection :)
It gets set right here, which is probably also the best place to add this info:
http://mxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#2940

All the other info from application.ini is available right there in "appData" as well.
I don't know if we want to unilaterally append a query string there, but maybe we can just make the value in application.ini end with a '?', and have the startup code use that as the signal to append the query string.
Have we tested that appending a querystring to a POST request won't confuse the python collector?  Some servers don't like mizing those two things.  I'd suggest we try using a ; token instead.
As far as what values we put in, are we looking at copying the blocklist format like in comment 4?
Rob, can you give me a status update on this?
Attachment #452080 - Flags: review?(ted.mielczarek)
Attachment #452080 - Attachment is patch: true
Attachment #452080 - Attachment mime type: application/octet-stream → text/plain
Comment on attachment 452080 [details] [diff] [review]
use the uuid, version, buildid

ted said I could hardcode the stuff I did, and this seems to submit crash reports correctly.
Attachment #452080 - Flags: review?(vladimir)
http://hg.mozilla.org/mozilla-central/rev/1dfc9bafedd3
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
blocking1.9.2: --- → ?
aravind, can you check today's collector web access logs for submissions that match this URL:

https://crash-reports.mozilla.com/submit?id=ec8030f7-c20a-464f-9b0e-13a3a9e97384&version=3.7a6pre&buildid=20100617135309
Assignee: nobody → sayrer
Is this what you are looking for?

access_2010-06-17-14.gz:208.54.5.74 crash-reports.mozilla.com - [17/Jun/2010:14:15:53 -0700] "POST /submit?id=ec8030f7-c20a-464f-9b0e-13a3a9e97384&version=3.7a6pre&buildid=20100617135309 HTTP/1.1" 200 151080 "-" "Crash%20Reporter1.0 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBookPro4%2C1)" "s_vi=[CS]v1|2604D1AA05012845-4000010BC0002387[CE]"
(In reply to comment #24)
> Is this what you are looking for?

yes, thank you
Comment on attachment 452080 [details] [diff] [review]
use the uuid, version, buildid

Looks fine, FWIW.
Attachment #452080 - Flags: review?(ted.mielczarek) → review+
We'll take this fix on the branches but we wouldn't block on it. We don't intend to build again for 3.6.4, but if we do we'll want this on the relbranch. Otherwise, this should be landed *only* on default. It also probably wouldn't hurt to land this on 1.9.1 default
blocking1.9.2: ? → -
Blocks: 573462
Blocks: 578370
Comment on attachment 452080 [details] [diff] [review]
use the uuid, version, buildid

We should just take this on all our active branches. It's simple and useful.
Attachment #452080 - Flags: approval1.9.2.8?
Attachment #452080 - Flags: approval1.9.1.12?
Comment on attachment 452080 [details] [diff] [review]
use the uuid, version, buildid

a=LegNeato for 1.9.2.9 and 1.9.1.12.
Attachment #452080 - Flags: approval1.9.2.9?
Attachment #452080 - Flags: approval1.9.2.9+
Attachment #452080 - Flags: approval1.9.1.12?
Attachment #452080 - Flags: approval1.9.1.12+
Comment on attachment 452080 [details] [diff] [review]
use the uuid, version, buildid

Removing .9 approval as this missed landing before freeze. Feel free to nominate again, though the bar for approval will be higher.
Attachment #452080 - Flags: approval1.9.2.9-
Attachment #452080 - Flags: approval1.9.2.9+
Attachment #452080 - Flags: approval1.9.1.12-
Attachment #452080 - Flags: approval1.9.1.12+
You need to log in before you can comment on or make changes to this bug.