818701 - Process all crash reports with an email address

Status: VERIFIED FIXED

(Socorro :: Backend, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

As part of the work on automatic emails, we want to have access, in PostgreSQL, to all crash reports containing an email address. Those are currently not 100% processed, we need to add the following rule to our collector: 

from Lars in IRC:
("Email", lambda x: x, 100),

Should we be worried about the quantity of reports that this will be adding to our databases?

Comment 1

[Selena Deckelmann :selenamarie :selena]

For a sense of where we're at now, for crashes reported since monday: 

breakpad=# select count(distinct email) from reports where date_processed > (now()::date - '2 days'::interval); 
 count 
-------
 22078
(1 row)

breakpad=# select count(*) from reports where date_processed > (now()::date - '2 days'::interval); 
  count  
---------
 1279895
(1 row)

There are quite a few duplicate email addresses and non-email addresses in that field. Doing a very rough regex, I narrow it down to about 20,000 addresses.

Comment 2

[Laura Thomson :laura]

I could have sworn this was already the case.  Really annoying.

I'm not that concerned, fwiw.  It's about 1.5% that have emails attached.

Comment 3

[Robert Kaiser]

Just FYI, the B2G dogfooding program is putting a user-unique fully numeric ID into the email field, it would be good if we'd have a good way to extract a daily list of crash IDs and those numeric email fields - but that's probably some other bug. ;-)

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

PR: https://github.com/mozilla/socorro/pull/999

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/1bc7f1e63e75ce6457bb54dbc512cb96b05d2129
Fixes bug 818701 - Process all crash reports with an email address.

https://github.com/mozilla/socorro/commit/bd485219b3144ba5be62ab2b0a755167db71448e
Merge pull request #999 from AdrianGaudebert/818701-process-crashes-with-email

Fixes bug 818701 - Process all crash reports with an email address.

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

Not sure how to test this, I guess we should send a few crash reports with an email address and verify that they all get processed? Or maybe we could run HBase script to verify that all raw crashes with an email address also exist in processed form.

Comment 7

[Matt Brandt [:mbrandt]]

QA verified on stage. Thanks adrian - we used submitted a test crash and verified that the data in the ui looks correct and contains an email address.

Test crash borrowed from:
https://github.com/mozilla/socorro/blob/master/testcrash/7d381dc5-51e2-4887-956b-1ae9c2130109.json

Comment 8

[Matt Brandt [:mbrandt]]

Oops, I forgot to drop a link to the crashes we submitted.

https://crash-stats.allizom.org/report/index/f70eea32-423a-4954-b445-7887a2130116
https://crash-stats.allizom.org/report/index/5e47a408-a6e8-4999-8383-bf8bc2130116