797566 - crash-stats-dev database is out-of-date, suspect problem with mobeta schema changes

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Selena Deckelmann :selenamarie :selena]

Here are the errors from a recent refresh: http://pastebin.mozilla.org/1856196

stephend reports a lack of data here: https://crash-stats-dev.allizom.org/hangreport/byversion/Firefox/15.0.1

I'm running a backfill_matviews() and will report what happens when that's done.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Still busted after backfill_matviews() run.  This makes it hard to test changes that have to do with data.

Comment 2

[Matt Pressman [:mpressman]]

Attachment: output of loadMiniDBonDev.py

Here is the output of loadMiniDBonDev.py run as the postgres user

/data/socorro/application/scripts/staging/loadMiniDBonDev.py --file /pgdata/tmp/extractdb.tgz --database breakpad --postscript /data/socorro/application/scripts/staging/postsql/postsql.sh

Comment 3

[Selena Deckelmann :selenamarie :selena]

Requested review of https://github.com/mpressman/socorro/pull/2

Have run it mostly successfully for 1 week of data on test_20121004 database on DevDB, but building indexes takes a while. :)

Comment 4

[Selena Deckelmann :selenamarie :selena]

Hey Matt,

Have you had a chance to try this out? Thanks!

Comment 5

[Matt Pressman [:mpressman]]

This seemed to hang after printing out restore indexes and constraints - the connection disconnected and I verified the next section where it truncates relnames like 'frames%', I ran the select and it didn't return any results - it did not delete any of the *.dump files so it hung somewhere between:

print 'restore indexes and constraints'

runload('/usr/local/pgsql/bin/pg_restore -j 3 -Fc --post-data-only -U postgres minidb.schema.dump -d %s' % options.database_name)
#runload('/usr/local/pgsql/bin/pg_restore -j 3 -Fc --post-data-only -U postgres matview_schemas.dump -d %s' % options.database_name)

conn.disconnect()

conn = psycopg2.connect("dbname=%s user=postgres" % options.database_name)

conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)

cur = conn.cursor()

# truncate soon-to-be-dropped tables
cur.execute("""
            DO $f$
            DECLARE tab TEXT;
            BEGIN
                FOR tab IN SELECT relname
                    FROM pg_stat_user_tables
                    WHERE relname LIKE 'frames%' LOOP

                    EXECUTE 'TRUNCATE ' || tab;

                END LOOP;
            END; $f$;
        """)

# delete all the dump files
runload('rm *.dump')

I manually ran:
# load views which break on pg_restore, such as hang_report
loadPostSQL()

Comment 6

[Selena Deckelmann :selenamarie :selena]

Ok! New fix: https://github.com/mpressman/socorro/pull/3

disconnect should have been close()  :(

And I fixed a dependency problem in one of the postsql scripts.

Comment 7

[Selena Deckelmann :selenamarie :selena]

Matt - Was this able to be run last night?  thanks! -selena

Comment 9

[Matt Pressman [:mpressman]]

Still getting errors, but not sure if it's my environment, I've modified my env to match socorro1.dev.db.phx1 and am only running a load of 1 week, but this is the latest:

psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/default_versions.sql" ignored
psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/hang_report.sql" ignored
psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/home_page_graph_views.sql" ignored
psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/performance_check_1.sql" ignored
psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/product_crash_ratio.sql" ignored
psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/product_info.sql" ignored
psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/product_selector.sql" ignored
psql: warning: extra command-line argument "breakpad" ignored

Comment 10

[Matt Pressman [:mpressman]]

I think the problem is that I was using the wrong postql.sh - re-running now

Comment 11

[Matt Pressman [:mpressman]]

successfully ran when I used the correct postql.sh file - I'll run in dev next

Comment 12

[Selena Deckelmann :selenamarie :selena]

Thanks for the updates. 

How are things looking this morning?

Comment 13

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Selena Deckelmann :selena from comment #12)
> Thanks for the updates. 
> 
> How are things looking this morning?

From the Web QA side, our automation is now again passing, so that's at least one good datapoint (we don't have DB tests per se, though).

Comment 14

[Matt Pressman [:mpressman]]

I think everything is smooth now - I manually ran the auto-refresh just to verify and all went well just as it had prior. I think if tests are now passing, this is fixed.

Comment 15

[Matt Pressman [:mpressman]]

closing as this has been running successfully with the changes to extractMiniDB.py and loadMiniDBonDev.py