Here are the errors from a recent refresh: http://pastebin.mozilla.org/1856196 stephend reports a lack of data here: https://crash-stats-dev.allizom.org/hangreport/byversion/Firefox/15.0.1 I'm running a backfill_matviews() and will report what happens when that's done.
Still busted after backfill_matviews() run. This makes it hard to test changes that have to do with data.
Severity: normal → critical
Here is the output of loadMiniDBonDev.py run as the postgres user /data/socorro/application/scripts/staging/loadMiniDBonDev.py --file /pgdata/tmp/extractdb.tgz --database breakpad --postscript /data/socorro/application/scripts/staging/postsql/postsql.sh
Requested review of https://github.com/mpressman/socorro/pull/2 Have run it mostly successfully for 1 week of data on test_20121004 database on DevDB, but building indexes takes a while. :)
Assignee: nobody → mpressman
Hey Matt, Have you had a chance to try this out? Thanks!
Status: NEW → ASSIGNED
This seemed to hang after printing out restore indexes and constraints - the connection disconnected and I verified the next section where it truncates relnames like 'frames%', I ran the select and it didn't return any results - it did not delete any of the *.dump files so it hung somewhere between: print 'restore indexes and constraints' runload('/usr/local/pgsql/bin/pg_restore -j 3 -Fc --post-data-only -U postgres minidb.schema.dump -d %s' % options.database_name) #runload('/usr/local/pgsql/bin/pg_restore -j 3 -Fc --post-data-only -U postgres matview_schemas.dump -d %s' % options.database_name) conn.disconnect() conn = psycopg2.connect("dbname=%s user=postgres" % options.database_name) conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT) cur = conn.cursor() # truncate soon-to-be-dropped tables cur.execute(""" DO $f$ DECLARE tab TEXT; BEGIN FOR tab IN SELECT relname FROM pg_stat_user_tables WHERE relname LIKE 'frames%' LOOP EXECUTE 'TRUNCATE ' || tab; END LOOP; END; $f$; """) # delete all the dump files runload('rm *.dump') I manually ran: # load views which break on pg_restore, such as hang_report loadPostSQL()
Ok! New fix: https://github.com/mpressman/socorro/pull/3 disconnect should have been close() :( And I fixed a dependency problem in one of the postsql scripts.
Matt - Was this able to be run last night? thanks! -selena
Still getting errors, but not sure if it's my environment, I've modified my env to match socorro1.dev.db.phx1 and am only running a load of 1 week, but this is the latest: psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/default_versions.sql" ignored psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/hang_report.sql" ignored psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/home_page_graph_views.sql" ignored psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/performance_check_1.sql" ignored psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/product_crash_ratio.sql" ignored psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/product_info.sql" ignored psql: warning: extra command-line argument "/data/socorro/application/scripts/staging/postsql/product_selector.sql" ignored psql: warning: extra command-line argument "breakpad" ignored
I think the problem is that I was using the wrong postql.sh - re-running now
successfully ran when I used the correct postql.sh file - I'll run in dev next
Thanks for the updates. How are things looking this morning?
(In reply to Selena Deckelmann :selena from comment #12) > Thanks for the updates. > > How are things looking this morning? From the Web QA side, our automation is now again passing, so that's at least one good datapoint (we don't have DB tests per se, though).
I think everything is smooth now - I manually ran the auto-refresh just to verify and all went well just as it had prior. I think if tests are now passing, this is fixed.
closing as this has been running successfully with the changes to extractMiniDB.py and loadMiniDBonDev.py
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.