1387100 - need data for key lookup tables for dev environment for webapp

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

When building a dev environment that's usable, we need a bunch of lookup tables populated with data. It behooves us to use data that's real data from our -prod environment.

This bug covers:

1. making a list of lookup tables that need data
2. figuring out where to source that data (hard-coded, -prod API, make it up, and so on)
3. writing code to populate those tables with that data; we could use setupdb_app for this given that it's already there and does related things

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

setupdb_app already generates a bunch of lookup tables:

https://github.com/mozilla-services/socorro/blob/9655c07cb8d90683e86755eeb3f7ca41670b9e1c/socorro/external/postgresql/setupdb_app.py#L173

Here's what gets added:

https://github.com/mozilla-services/socorro/blob/9655c07cb8d90683e86755eeb3f7ca41670b9e1c/socorro/external/postgresql/staticdata.py

We should keep using that for those tables, but we should make sure the contents are updated with what we have in -prod.

For lookup tables that are missing, we can re-use the staticdata method where it makes sense.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

For the webapp, I'd like this url to work:

    http://localhost:8000/

That redirects to:

    http://localhost:8000/home/product/Firefox

After building a fresh -dev environment, that url shows a stunningly unhelpful 404 page.

I started working through the code and it seems like it's looking at product_versions.

So I wrote some code to populate product_versions using the API in -prod. It's a little dicey since the API doesn't provide all the data we have in the table and it changes some of the column names. :(

I tried populating with ftp_scraper, too, but no dice.

I need to spend some more time figuring out what data is needed for /home/product/Firefox, then can go back to populating the needed data.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

This took forever. There are two issues. First is that we need to populate product_versions, but doing that relies on a bunch of tables. Then we need to mark active versions. Then we need to account for the tables that the webapp home view uses.

All told, I added staticdata for:

* release_repositories
* products
* product_build_types
* product_release_channels (deprecated evil twin for product_build_types)
* special_product_platforms

I populated these tables with what we have in -prod. It's possible that some of that is not needed anymore. I didn't spend a lot of time trying to figure out what we could skip.

I also wrote a docker/run_update_release_data.sh script which runs several crontabber jobs to populate the product_versions table.

I added some key documentation, too.

The /home/product/Firefox page doesn't kick up a 404 anymore, so that achieves the goal.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3900 - add lookup tables

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/646045a3161254752ed51b1b2acc02423a1a663b
bug 1387100 - improve webapp in dev environment (#3900)

* bug 1387100 - improve webapp in dev environment

This adds lookup table data population and documented run lines that add release
data to the db. These make it possible to go to
http://localhost:8000/home/product/Firefox and have it mostly work.

Note that this only pulls release data for Firefox. Each additional product
takes another like 10 minutes to run. The thinking is that we're primarily
working on Firefox crashes, so if someone needs release data for something else,
they can adjust the scripts accordingly. If we need to make that more flexible,
we can make that change later.

* Add throttle data and exit-on-error to shell script

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

PR 3900 covered a bunch of lookup tables.

There's still data that's missing that parts of the webapp need to run (ADI, crashes, etc). That'll get covered in other bugs.