1193836 - Stop using the sql template file to generate the reference data db

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[Mauro Doglio [:mdoglio]]

We should let django handle the creation of the reference data database. This would reduce dramatically the effort required to do schema modifications and we could start using db migrations for real.

Comment 1

[Ed Morley [:emorley]]

+1000

Comment 2

[Mauro Doglio [:mdoglio]]

Added bug 1194221 as a blocker because there's no way to set the collation of a varchar using the django orm.

Comment 3

[Mauro Doglio [:mdoglio]]

Attachment: PR #968 - stop using refdata template

Comment 4

[William Lachance (:wlach)]

Comment on attachment 8678835 [details] [review]
PR #968 - stop using refdata template

This looks good to me! The unit test updates are pretty close to what I was trying to do a couple months ago when I was trying to fix the test database problem.

r+ with the issues Ed brought up fixed.

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 8678835 [details] [review]
PR #968 - stop using refdata template

Awesome! :-)

Comment 6

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/247cc3de64979efb3c968067bc0a4ab3a114bbb0
Bug 1193836 - Use a dedicated settings file for testing

https://github.com/mozilla/treeherder/commit/6134f26b6de9a4e02e152e964c91796b099d81d8
Bug 1193836 - Delete jobs test databases using fixture finalizer

This guarantees that jobs databases is dropped at the end of each
test. It also makes the jobs database life cycle easier to understand.

In general to keep the tests as fast as possible we shouldn't have much
code in the setup and tear down of each test.

https://github.com/mozilla/treeherder/commit/d5f847e559c3a074fef5c6c00bc920daf429db8e
Bug 1193836 - Decouple the jobs and reference databases for test

We were previously using the same database (test_treeherder) for both the
jobs and reference data model. I centralized the new db name in the test
settings file. All the test requiring the jobs db or its repository counterpart
can now access it using the `test_project` fixture, while utility functions use
directly the metioned setting. Where the project name is hardcoded in a static
file, I just replaced it with the new name `test_treeherder_jobs`

https://github.com/mozilla/treeherder/commit/74abf99e7f33bc23e73e4cd4066ba6dd760814de
Bug 1193836 - Stop relying on default active_status

The django orm doesn't enforce default values at the db level, so we
need to set explicitly the default value when we use the RefData model.
Once this is deployed we can remove the default value from the reference
data active_status columns.

https://github.com/mozilla/treeherder/commit/95dbd994105b4b5ff5bb61ea62017e63a4ae9a07
Bug 1193836 - Stop using init_master_db for tests setup

https://github.com/mozilla/treeherder/commit/5930b4f4c1a1c7d0d11204941a606e2427f89eff
Bug 1193836 - Use pytest-django to run tests

pytest-django doesn't setup a test database for every single test, but
only for those tests that actually require a db. Tests that require a db
need to either be marked with `@pytest.mark.django_db` or use a fixture
that has a dependency on `db` or `transactional_db`.
Using a non transactional db would make tests execution much faster, but
unfortunately it doesn't play well with the treeherder datasource
creation so I used a transactional_db.

pytest-django also allows you to specify a settings file to use for
tests in a pytest.ini file, which is nicer than monkeypatch the original
settings file in the pytest session start function :smiley:.

https://github.com/mozilla/treeherder/commit/829cf22bc482937406a22f1511db7a0db53a0648
Bug 1193836 - Update vm provisioning

This patch replaces the use of init_master_db with django `migrate`
and `load_initial_data` commands.

https://github.com/mozilla/treeherder/commit/0f3788036290146d99df76dea73edbebaa2b1aab
Bug 1193836 - Remove init_master_db command

Also remove the reference data sql template file.

https://github.com/mozilla/treeherder/commit/c876849a3cbbe9681f6ed60e336af3c05765e457
Bug 1193836 - Remove mention to init_master_db from docs

https://github.com/mozilla/treeherder/commit/160be6af717207e113a3c0701cbf6cda87623366
Bug 1193836 - let django create the test db on travis

Now that we have better integration between  pytest and django we don't
need to create the test db upfront. The test db will be created on the
first test that requiring it,

Comment 7

[Mauro Doglio [:mdoglio]]

Attachment: Followup

Comment 8

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/078f5902d86e5719d9a79c318ce07462631aea26
Bug 1193836 - use `--noinput` on migrate

This is a followup of 829cf22bc482937406a22f1511db7a0db53a0648