706807 - Add productID to CSV output

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[Robert Kaiser]

As bug 706612 will add the app ID field to the crash reports, I think we'll want to have that in our CSVs we output as well (even though the native Fennec will have a different product name anyhow there).

Comment 1

[chris hofmann]

yeah, we will need this in the .csv files as soon as it starts appearing in crash reports so we can sort out native UI builds for Fennec.

Comment 2

[Robert Kaiser]

Chris, we will be able to sort out native UI builds by their different product name anyhow.

Comment 4

[[:jberkus] Josh Berkus]

This will require adding the field appid to the "reports" table, and for the processors to populate that field.

Comment 5

[Robert Kaiser]

(In reply to [:jberkus] Josh Berkus from comment #4)
> This will require adding the field appid to the "reports" table, and for the
> processors to populate that field.

Bug 706612 is filed for the processor stuff, FYI.

Comment 6

[[:jberkus] Josh Berkus]

Lars,

AppID has been added as a field to Reports on DevDB.

Comment 7

[[:jberkus] Josh Berkus]

Oops, changing it to productID.  Done.

Comment 8

[Laura Thomson :laura]

r? https://github.com/mozilla/socorro/pull/230

Comment 9

[[github robot]]

Commits pushed to https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/848a800be524e6f03f851926ee3076e3c04ee250
Fixes bug 706807 - Add productid to the columns in the csv dump

https://github.com/mozilla/socorro/commit/d8085641fb34cc09dbdef43589aeeaaaedca5df7
Merge pull request #230 from lauraxt/706807

Fixes bug 706807 - Add productid to the columns in the csv dump

Comment 10

[Robert Helmer [:rhelmer]]

QA verification info:

Per IRC discussion w/ laura and kairo, we are not going to actually populate this column until 2.4, since the "product" column will be rewritten to "FennecAndroid" in the CSV (as with everywhere else).

I've generated a public/private pair of CSVs for QA purposes, emailing to mbrandt. These should be compared to http://crash-analysis.mozilla.com/crash_analysis/ and should:

1) contain new productid column (right now unused)
2) have both FennecAndroid and Fennec rows present

Comment 11

[Robert Helmer [:rhelmer]]

Output from cron run on stage:

[[e-mail]1 ~]$ /data/socorro/application/scripts/startDailyUrl.py 
2011-12-21 14:35:03,138 INFO - current configuration:
2011-12-21 14:35:03,139 INFO - 	databaseHost=10.2.70.23
2011-12-21 14:35:03,140 INFO - 	databaseName=breakpad
2011-12-21 14:35:03,140 INFO - 	databasePassword=**********
2011-12-21 14:35:03,140 INFO - 	databaseUserName=breakpad_rw
2011-12-21 14:35:03,141 INFO - 	day=2011-12-21 00:00:00
2011-12-21 14:35:03,141 INFO - 	outputPath=.
2011-12-21 14:35:03,141 INFO - 	product=Firefox,Fennec,FennecAndroid
2011-12-21 14:35:03,142 INFO - 	publicOutputPath=.
2011-12-21 14:35:03,142 INFO - 	stderrErrorLoggingLevel=10
2011-12-21 14:35:03,142 INFO - 	stderrLineFormatString=%(asctime)s %(levelname)s - %(message)s
2011-12-21 14:35:03,143 INFO - 	syslogErrorLoggingLevel=10
2011-12-21 14:35:03,143 INFO - 	syslogFacilityString=local2
2011-12-21 14:35:03,143 INFO - 	syslogHost=localhost
2011-12-21 14:35:03,144 INFO - 	syslogLineFormatString=Socorro Daily URL (pid %(process)d): %(asctime)s %(levelname)s - %(threadName)s - %(message)s
2011-12-21 14:35:03,144 INFO - 	syslogPort=514
2011-12-21 14:35:03,144 INFO - 	version=
2011-12-21 14:35:03,145 INFO - 
2011-12-21 14:36:19,918 DEBUG - config.day = 2011-12-21 00:00:00; now = 2011-12-22 00:00:00; yesterday = 2011-12-21 00:00:00
2011-12-21 14:36:19,918 DEBUG - config.day = 2011-12-21 00:00:00; now = 2011-12-22; yesterday = 2011-12-21
2011-12-21 14:36:19,918 DEBUG - SQL is: 
      select
        r.signature,  -- 0
        r.url,        -- 1
        'http://crash-stats.mozilla.com/report/index/' || r.uuid as uuid_url, -- 2
        to_char(r.client_crash_date,'YYYYMMDDHH24MI') as client_crash_date,   -- 3
        to_char(r.date_processed,'YYYYMMDDHH24MI') as date_processed,         -- 4
        r.last_crash, -- 5
        r.product,    -- 6
        r.version,    -- 7
        r.build,      -- 8
        pd.branch,    -- 9
        r.os_name,    --10
        r.os_version, --11
        r.cpu_name || ' | ' || r.cpu_info as cpu_info,   --12
        r.address,    --13
        array(select ba.bug_id from bug_associations ba where ba.signature = r.signature) as bug_list, --14
        r.user_comments, --15
        r.uptime as uptime_seconds, --16
        case when (r.email is NULL OR r.email='') then '' else r.email end as email, --17
        (select sum(adu_count) from raw_adu adu
           where adu.date = '2011-12-22'
             and pd.product = adu.product_name and pd.version = adu.product_version
             and substring(r.os_name from 1 for 3) = substring(adu.product_os_platform from 1 for 3)
             and r.os_version LIKE '%'||adu.product_os_version||'%') as adu_count, --18
        r.topmost_filenames, --19
        case when (r.addons_checked is NULL) then '[unknown]'when (r.addons_checked) then 'checked' else 'not' end as addons_checked, --20
        r.flash_version, --21
        r.hangid, --22
        r.reason, --23
        r.process_type, --24
        r.app_notes, --25
        r.install_age, --26
        rd.duplicate_of, --27
        r.release_channel, --28
        r.productid --29
      from
        reports r left join productdims pd on r.product = pd.product and r.version = pd.version
            left join reports_duplicates rd on r.uuid = rd.uuid
      where
        '2011-12-21' <= r.date_processed and r.date_processed < '2011-12-22'
        and r.product in ('Firefox','Fennec','FennecAndroid') 
      order by 5 -- r.date_processed, munged
      
2011-12-21 14:36:38,169 DEBUG - MainThread - killing database connections
2011-12-21 14:36:38,170 DEBUG - MainThread - connection MainThread closed
2011-12-21 14:36:38,173 INFO - done.

Comment 12

[Matt Brandt [:mbrandt]]

(In reply to Robert Helmer [:rhelmer] from comment #10)
> QA verification info:
> I've generated a public/private pair of CSVs for QA purposes, emailing to
> mbrandt. These should be compared to
> http://crash-analysis.mozilla.com/crash_analysis/ and should:
> 
> 1) contain new productid column (right now unused)
> 2) have both FennecAndroid and Fennec rows present

QA verified the new reports, both the public and private reports contain:
- productid columns
- have FennecAndroid and Fennec rows present

Thanks rhelmer for being really helpful.