721018 - initialization routine needed for default rows in control tables.

Status: RESOLVED WONTFIX

(Socorro :: Database, task)

Description

[K Lars Lohn [:lars] [:klohn]]

In testing the new Python invocation of the Postgres stored procedure 'weekly_report_partitions', it appears that the stored procedure is incomplete.  It successfully creates the new 'reports' table weekly partitions, but it misses the subordinate tables, 'extensions' and 'plugins_reports'.

Comment 1

[[:jberkus] Josh Berkus]

Hmmm, I tested this in dev.  Are you seeing this in prod, or on stage, or somewhere else?

Comment 2

[K Lars Lohn [:lars] [:klohn]]

I'm seeing in a fresh install of a brand new empty database.  Specifically in the database in the vagrant virtual machine.

Comment 3

[[:jberkus] Josh Berkus]

This problem appears to be caused by the "reports_partition_info" table being empty on vagrant.  That table needs to be populated (hand).  Rhelmer?

Comment 4

[K Lars Lohn [:lars] [:klohn]]

So there are a set of tables in Postgres that are used to control how socorro works. It isn't sufficient to just create an empty database and run Socorro. We need a way to insert default rows into the control tables after creating the empty database. 

I'm assuming that we're getting away with this right now because when we load sample data, we're just happen to be getting the control data, too.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to [:jberkus] Josh Berkus from comment #3)
> This problem appears to be caused by the "reports_partition_info" table
> being empty on vagrant.  That table needs to be populated (hand).  Rhelmer?

If someone wants to help to add this sort of thing, each CSV in this file corresponds to a table:
https://github.com/mozilla/socorro/tree/master/tools/dataload

The dataload tool is documented here:
http://socorro.readthedocs.org/en/latest/populatepostgres.html

Also there is documentation about each table and how it's populated here:
http://socorro.readthedocs.org/en/latest/databasetablesbysource.html#databasetablesbysource-chapter

Ideally I'd like to be able to run the dataload tool on a fresh schema import and get 100% Socorro functionality. As it stands today, we know we're missing some documented tables, and I suspect that we have some undocumented dependencies too. Also, the dates on the test data and backfilling processes etc. need to be taken into account (this is something puppet can easily do for us, currently it calls backfill_matviews() for instance).

This probably deserves it's own bug, I'll file that momentarily.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to K Lars Lohn [:lars] [:klohn] from comment #4)
> So there are a set of tables in Postgres that are used to control how
> socorro works. It isn't sufficient to just create an empty database and run
> Socorro. We need a way to insert default rows into the control tables after
> creating the empty database. 
> 
> I'm assuming that we're getting away with this right now because when we
> load sample data, we're just happen to be getting the control data, too.

Yes if you load a dump derived from prod, you'll get all these important little details. I do believe that being able to create a DB "from scratch" is really important for development purposes, external users, and just for understanding how the system works though, so the fact that this does not work is a valid thing to bring up.

Comment 7

[[:jberkus] Josh Berkus]

So, need to do a few things here:

1. add reports_partition_info to the list of "manually populated tables". (Josh)

2. Add sensible error condition to the stored procedure if the table is empty (Josh)

3. Add table data to the empty-db load routine (rhelmer?)

Comment 8

[Robert Helmer [:rhelmer]]

Selena is working on something similar to this (feel free to dupe etc)

Comment 9

[K Lars Lohn [:lars] [:klohn]]

old and outdated - refile if this is still needed...