567923 - stage socorro new db table and cron job

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Austin King [:ozten]]

1) Please svn up scripts and socorro in stage.

2) Please create the new db table 'daily_crashes'. I think this is done via
scripts/setupDatabase.py

@aravind or @lars - please correct me

3) Run the new cron manually
scripts/startDailyCrash.py


Once this is finished, I will commit the frontend code.

I will also give you instructions for installing the cron later. We should run it once manually to backfill the tables, which may take several hours.

Comment 1

[Austin King [:ozten]]

(In reply to comment #0)
Instead of step #2, please execute this SQL:

CREATE TABLE daily_crashes
    (
        id serial NOT NULL PRIMARY KEY,
        COUNT INTEGER DEFAULT 0 NOT NULL,
        report_type CHAR(1) NOT NULL DEFAULT 'C',
        productdims_id INTEGER REFERENCES productdims(id),
        os_short_name CHAR(3),
        adu_day TIMESTAMP WITHOUT TIME ZONE NOT NULL,
        CONSTRAINT day_product_os_report_type_unique UNIQUE (adu_day, productdims_id, os_short_name, report_type)
    )

Comment 2

[Shyam Mani [:fox2mike]]

(In reply to comment #0)
> 1) Please svn up scripts and socorro in stage.

[root@mrapp-stage02 scripts]# svn up
A    startDailyCrash.py
U    config/processorconfig.py.dist
A    config/dailycrashconfig.py.dist
U    config/monitorconfig.py.dist
Updated to revision 2105.
[root@mrapp-stage02 scripts]# cd ../socorro/
[root@mrapp-stage02 socorro]# svn up
U    unittest/database/testSchema.py
U    unittest/collector/testCrashstorage.py
U    processor/processor.py
A    cron/daily_crash.py
U    hbase/hbaseClient.py
U    services/getCrash.py
U    services/aduByDay.py
U    database/schema.py
U    lib/psycopghelper.py
U    webapi/webapiService.py
U    monitor/monitor.py
Updated to revision 2105.


> 2) Please create the new db table 'daily_crashes'. I think this is done via

Don't think this really succeeded, suggestions? 

breakpad=# CREATE TABLE daily_crashes
breakpad-#     (
breakpad(#         id serial NOT NULL PRIMARY KEY,
breakpad(#         COUNT INTEGER DEFAULT 0 NOT NULL,
breakpad(#         report_type CHAR(1) NOT NULL DEFAULT 'C',
breakpad(#         productdims_id INTEGER REFERENCES productdims(id),
breakpad(#         os_short_name CHAR(3),
breakpad(#         adu_day TIMESTAMP WITHOUT TIME ZONE NOT NULL,
breakpad(#         CONSTRAINT day_product_os_report_type_unique UNIQUE (adu_day,
breakpad(# productdims_id, os_short_name, report_type)
breakpad(#     );
NOTICE:  CREATE TABLE will create implicit sequence "daily_crashes_id_seq" for serial column "daily_crashes.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "daily_crashes_pkey" for table "daily_crashes"
NOTICE:  CREATE TABLE / UNIQUE will create implicit index "day_product_os_report_type_unique" for table "daily_crashes"
CREATE TABLE

Comment 3

[Austin King [:ozten]]

Thanks fox2mike. Those notices look normal.

Comment 4

[Shyam Mani [:fox2mike]]

Ah okay, let me run the rest.

Comment 5

[Austin King [:ozten]]

Missed a step
3.1) cp config/dailycrashconfig.py.dist config/dailycrashconfig.py
3.2) Make sure you have the same env as a cron job (PYTHONPATH, etc)

Comment 6

[Shyam Mani [:fox2mike]]

I get this :

[processor@dm-breakpad-stage01 scripts]$ ./startDailyCrash.py 
Traceback (most recent call last):
  File "./startDailyCrash.py", line 10, in ?
    import dailycrashconfig as config
ImportError: No module named dailycrashconfig

Comment 7

[Aravind Gottipati [:aravind]]

Ran the script and posted the output to khan.

Comment 8

[Austin King [:ozten]]

The log is filled with ERROR - permission denied for relation daily_crashes.

I'm guessing there are now rows in the new table.

aravind - Can you look at the username/password and grant permissions? thanks.

Comment 9

[Aravind Gottipati [:aravind]]

Dropped and re-created the table.  Uploaded the log file to khan:/tmp/dailycrash.log

Comment 10

[Austin King [:ozten]]

Step 4 - restart the webservices.py code to pickup new AduByDay.py changes.
Step 5 - clear frontend web service call caches. This is probably memcached.

Comment 11

[Laura Thomson :laura]

aravind: I'm trying to work out why fox2mike had trouble staging this, sounds like something is missing from the docs.  Could you guys please work together to sort it out?

Comment 12

[Aravind Gottipati [:aravind]]

(In reply to comment #11)
> aravind: I'm trying to work out why fox2mike had trouble staging this, sounds
> like something is missing from the docs.  Could you guys please work together
> to sort it out?

My guess is that (from reading comment #1) is that only the scripts and processor layer were updated.  I guess the fix is to specifically say that you want all layers updated.

@ozten: Working on pushing the code out to the middleware layer.

Comment 13

[Aravind Gottipati [:aravind]]

middleware code updated.

Comment 14

[Austin King [:ozten]]

Step 3.3) Please update the webapiconfig.py and add

adubd.AduByDay200912 to the servicesList.default 

So it will look something like:
servicesList.default = [tcbst.TopCrashBySignatureTrends, sighist.SignatureHistory, adubd.AduByDay, adubd.AduByDay200912, crash.GetCrash]

Comment 15

[Aravind Gottipati [:aravind]]

Updated the config file - getting this error now http://breakpad.pastebin.mozilla.org/728540

Comment 16

[Austin King [:ozten]]

(In reply to comment #15)
Filed Bug#568322.

Comment 17

[Aravind Gottipati [:aravind]]

We know now that the middle-ware layer is functioning now, closing the bug.

Comment 18

[Austin King [:ozten]]

@aravind - Since this is not installed as a cron yet, can you run this script again:
./startDailyCrash.py

This will put data in for the last couple of days.

Thanks

Comment 19

[Aravind Gottipati [:aravind]]

Done.

Comment 20

[Austin King [:ozten]]

(In reply to comment #0)
This is looking good. Please re-run 
./startDailyCrash.py

Then also please do
Step 6) Install ./startDailyCrash.py as a daily cron job.
It should run after midnight, once per day.

Comment 21

[Aravind Gottipati [:aravind]]

Done, set cron job to run nightly at 00:05

Comment 22

[Austin King [:ozten]]

Notes for prod deployment:

Step 7) Ensure that webservices.py has read access to the daily_crashes table.
7.1) Restart apache/webservices if needed to pick up new source code (needed if mod_python)
7.2) curl "http://localhost:8085/201005/adu/byday/p/Firefox/v/3.6.4/rt/any/os/Windows;Mac;Linux/start/2010-05-22/end/2010-06-06"
Grep webapi.log for fatal, error, or warnings