Closed Bug 1037855 Opened 10 years ago Closed 10 years ago

raw_adi has no entries for product_id of webapprt@mozilla.org

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kairo, Assigned: rhelmer)

References

Details

(Whiteboard: [DB Change]) 

In raw_adi, the "webapprt@mozilla.org" product ID is missing compared to raw_adu (but we have a product for that in Socorro):

breakpad=> SELECT product_guid FROM raw_adu WHERE date = '2014-07-09' GROUP BY product_guid; 
             product_guid             
--------------------------------------
 a23983c0-fd0e-11dc-95ff-0800200c9a66
 3550f703-e582-4d05-9a08-453d09bdfdc6
 ec8030f7-c20a-464f-9b0e-13a3a9e97384
 99bceaaa-e3c6-48c1-b981-ef9b46b67d60
 92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a
 aa3c5121-dab2-40e2-81ca-7ea25febc110
 webapprt@mozilla.org
(7 rows)

breakpad=> SELECT product_guid FROM raw_adi WHERE date = '2014-07-09' GROUP BY product_guid; 
              product_guid              
----------------------------------------
 {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 {a23983c0-fd0e-11dc-95ff-0800200c9a66}
 {99bceaaa-e3c6-48c1-b981-ef9b46b67d60}
 {3550f703-e582-4d05-9a08-453d09bdfdc6}
 {92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}
 {aa3c5121-dab2-40e2-81ca-7ea25febc110}
(6 rows)

Note that the format of the field also changed, maybe that's connected, esp. as the product_productid_map is defined that way:

breakpad=> SELECT product_name, productid FROM product_productid_map;
    product_name     |               productid                
---------------------+----------------------------------------
 WebAppRuntimeMobile | {webapprtmobile@mozilla.com}
 MetroFirefox        | {99bceaaa-e3c6-48c1-b981-ef9b46b67d60}
 B2G                 | {3c2e2abc-06d4-11e1-ac3b-374f68613e61}
 WebappRuntime       | {webapprt@mozilla.org}
 Fennec              | {a23983c0-fd0e-11dc-95ff-0800200c9a66}
 FennecAndroid       | {aa3c5121-dab2-40e2-81ca-7ea25febc110}
 Firefox             | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 Thunderbird         | {3550f703-e582-4d05-9a08-453d09bdfdc6}
 SeaMonkey           | {92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}
 Camino              | {camino@caminobrowser.org}
(10 rows)

The raw values for product IDs coming in are as follows:

breakpad=> SELECT product_guid FROM raw_adi_logs WHERE report_date='2014-07-09' GROUP BY product_guid;
              product_guid              
----------------------------------------
 avc-intellihouse-kiosk@guru.si
 org.index.blophome.1
 kmeleon@
 {e2fda1a4-762b-4020-b5ad-a41df1933103}
 support@samhk.com
 {99bceaaa-e3c6-48c1-b981-ef9b46b67d60}
 kiosk@wmedia.de
 geierlein@taxbird.de
 info@divider.net
 xulapp@xultest.org
 control-center@userful.com
 {59c81df5-4b7a-477b-912d-4e0fdf64e5f2}
 support@mobigator.com
 gaoguodong@sunniwell.net
 browser@samarius.ru
 webint@gmoresearch.com
 SQLiteManager@mrinalkant.blogspot.com
 {aa3c5121-dab2-40e2-81ca-7ea25febc110}
 firefox
 vkb_browser@tmm-software.com
 xulapp@zma.org
 a06cae4d-1ae6-4418-befc-2cf333187aba
 {b3cb1ea2-d039-476d-a3bc-03224fb82bd6}
 webapprt-mobile@mozilla.org
 {a23983c0-fd0e-11dc-95ff-0800200c9a66}
 displaylink@innovatrics.com
 {cf0bbf29-bbb7-4fc7-9fe3-a95698fadfee}
 lmp-nxp-orders
 webexec@rapidrollout.com
 {a79fe89b-6662-4ff4-8e88-09950ad4dfde}
 player@ki-wi.cz
 kasiosk@innovatrics.com
 prism@developer.mozilla.org
 IttMoz@ITTerritory
 {3550f703-e582-4d05-9a08-555d09bdfdc6}
 newman@newmanonline.org.uk
 {ec8030f7-c20a-464f-9b0e-13a3a9e910000
 {d162e408-f758-4bed-88eb-8e030803e64c}
 pencil@evolus.vn
 kiosk@ki-wi.cz
 {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
 {92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}
 0
 
 AVMAP
 term@eltersi.ru
 {847441a0-668c-4f1e-857f-7fb5fabebdb9}
 {3bdb0d0c-b8d1-4b2c-82ab-b5d13526a7e1}
 opensi@gplux.lu
 dev@patientway.com
 SetumApps@kds.ro
 {uedit@el-mundo.net}
 poczta.app@csw.pl
 webapprt@mozilla.org
 Attendance@alucast.cz
 webmaster@animetheme.com
 {3995563a-1d62-484c-823d-d5429f02da67}
 plate@xpud.org
 client@wavefax.com
 {3550f703-e582-4d05-9a08-453d09bdfdc6}
 ideskbrowser@haufe.de
 avmap.it
(62 rows)

webapprt@mozilla.org is listed there, so the raw data comes in from the logs.

We rarely look at WebapppRuntime data, so this isn't really urgent, but we should take a look, esp. as more work on desktop web apps is planned and we'll need to look at that data then.
KaiRo - how important is this bug, should it block the Hive and BAMO->AMO switch?
Flags: needinfo?(tmeyarivan)
Flags: needinfo?(scabral)
Flags: needinfo?(kairo)
Sorry clearing the needinfos here - as Kairo points out this is present in the raw logs, so it's something we're filtering out on the Socorro side.
Flags: needinfo?(tmeyarivan)
Flags: needinfo?(scabral)
Flags: needinfo?(kairo)
Kairo, yes it looks like the problem is missing {} around the webapprt@mozilla.org product_guid. I'll see what can be done.
OK I can special-case this in https://github.com/mozilla/socorro/pull/2289 when we do the INSERT into raw_adi - that's where we're doing other special-casing like collapsing all release% channels into "release".
Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/44405239202e63a4e615e3d6bc6e2100602daac0
fix bug 1037855 - special-case webapprt since the product_guid comes in without {}
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [DB Change]
Migration run on stage, backfilling..
BTW - this is taken care of in the migration as part of bug 1053632
Whiteboard: [DB Change]
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/683617115e80ea0d3aacf50be089addbfc8909e6
followup from bug 1037855 - this introduced an intermittent test
failure, specify ORDER BY to make results consistent

https://github.com/mozilla/socorro/commit/45f00c7839b0cc54ea1edf7498125d71a7941634
Merge pull request #2299 from rhelmer/nobug-intermittent-test-failure

followup from bug 1037855 - this introduced an intermittent test
Target Milestone: --- → 99
It looks to me like this would not be fixed on prod yet. Did this get pushed/enabled before today's pull for yesterday's data?


breakpad=> SELECT sum(adi_count),product_name,product_guid FROM raw_adi WHERE date='2014-08-19' AND product_version='34.0a1' GROUP BY product_name,product_guid ORDER BY product_name ASC,product_guid ASC;                                                              sum  |   product_name    |              product_guid              
-------+-------------------+----------------------------------------
     1 | 34central_x64     | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
  2537 | Fennec            | {aa3c5121-dab2-40e2-81ca-7ea25febc110}
 85554 | Firefox           | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
     1 | Firefox-Nightly   | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
   635 | Firefox-Trunk     | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
     1 | Iceweasel         | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
   706 | Thunderbird       | {3550f703-e582-4d05-9a08-453d09bdfdc6}
    44 | Thunderbird-Trunk | {3550f703-e582-4d05-9a08-453d09bdfdc6}
(8 Zeilen)

breakpad=> SELECT sum(adu_count),product_name,product_guid FROM raw_adu WHERE date='2014-08-19' AND product_version='34.0a1' GROUP BY product_name,product_guid ORDER BY product_name ASC,product_guid ASC;  
  sum  | product_name  |             product_guid             
-------+---------------+--------------------------------------
  2521 | Fennec        | aa3c5121-dab2-40e2-81ca-7ea25febc110
 84930 | Firefox       | ec8030f7-c20a-464f-9b0e-13a3a9e97384
   745 | Thunderbird   | 3550f703-e582-4d05-9a08-453d09bdfdc6
    13 | WebappRuntime | webapprt@mozilla.org
(4 Zeilen)
(In reply to Robert Kaiser (:kairo@mozilla.com, slow reaction due to vacation backlog) from comment #9)
> It looks to me like this would not be fixed on prod yet. Did this get
> pushed/enabled before today's pull for yesterday's data?

Yes it was run and raw_adi cleared out/backfilled for the past few days...


> breakpad=> SELECT sum(adi_count),product_name,product_guid FROM raw_adi
> WHERE date='2014-08-19' AND product_version='34.0a1' GROUP BY
> product_name,product_guid ORDER BY product_name ASC,product_guid ASC;       
> sum  |   product_name    |              product_guid              
> -------+-------------------+----------------------------------------
>      1 | 34central_x64     | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
>   2537 | Fennec            | {aa3c5121-dab2-40e2-81ca-7ea25febc110}
>  85554 | Firefox           | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
>      1 | Firefox-Nightly   | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
>    635 | Firefox-Trunk     | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
>      1 | Iceweasel         | {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
>    706 | Thunderbird       | {3550f703-e582-4d05-9a08-453d09bdfdc6}
>     44 | Thunderbird-Trunk | {3550f703-e582-4d05-9a08-453d09bdfdc6}
> (8 Zeilen)
> 
> breakpad=> SELECT sum(adu_count),product_name,product_guid FROM raw_adu
> WHERE date='2014-08-19' AND product_version='34.0a1' GROUP BY
> product_name,product_guid ORDER BY product_name ASC,product_guid ASC;  
>   sum  | product_name  |             product_guid             
> -------+---------------+--------------------------------------
>   2521 | Fennec        | aa3c5121-dab2-40e2-81ca-7ea25febc110
>  84930 | Firefox       | ec8030f7-c20a-464f-9b0e-13a3a9e97384
>    745 | Thunderbird   | 3550f703-e582-4d05-9a08-453d09bdfdc6
>     13 | WebappRuntime | webapprt@mozilla.org
> (4 Zeilen)
Ah looks like the raw_adi_logs has product_name "Webapp Runtime"... need to rewrite this as well.
Need to have a CASE in the JOIN as well.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Migration and backfill run on stage.
Whiteboard: [DB Change]
Migration run on prod:

INFO  [alembic.migration] Context impl PostgresqlImpl.
INFO  [alembic.migration] Will assume transactional DDL.
/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/alembic/util.py:189: UserWarning: Revision 321a815df1c7 referenced from 321a815df1c7 -> 5e14d46c725, bug 855423 change RAISE EXCEPTION to RAISE NOTICE is not present
  warnings.warn(msg)
INFO  [alembic.migration] Running upgrade 3a36327c2845 -> 52dbc7357409, bug 1056025 and bug 1037855 - rename Fennec->FennecAndroid and WebappRuntime correctly
You need to log in before you can comment on or make changes to this bug.