1134804 - Stage's datasource table entries are out of sync with the databases on disk

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

In bug 1134140 / bug 1134621 comment 4:

[2015-02-19 00:00:00,141: WARNING/Worker-12] datasource.hubs.MySQL: try_to_connect OperationalError encountered on attempt 0. Database:accessibility_jobs_1
[2015-02-19 00:00:00,141: WARNING/Worker-12] Error detected was:
(1049, "Unknown database 'accessibility_jobs_1'")

The problem is that stage doesn't have accessibility_jobs_1 since the accessibility repo is marked as active_status 'onhold', so init_datasources never created it on stage (stage was created after accessibility was marked onhold, unlike prod).

However, unlike init_datasource, cycle_data runs against _all_ repos, so on prod, we're failing because of the non-existant repos.

Now I guess it would still be nice to expire data from inactive repos, but I think the quickest fix here is to exclude them from cycle-data, and create separate handling for deleting them outright (per bug 1035294).

Comment 1

[Ed Morley [:emorley]]

Attachment: Only run cycle-data against active repositories

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8566732 [details] [review]
Only run cycle-data against active repositories

Makes sense, thanks!

Comment 3

[Ed Morley [:emorley]]

So after having something to eat & coming back to this, I can see I was talking nonsense.

cycle-data doesn't use the main list of repositories (on which we _would_ need to filter on active_status), it actually refers to the datasource table directly - which should in theory always have a 1:1 relationship with the databases on disk.

The reason we're getting this exception is that accessibility_jobs_1 does not exist on stage, but accessibility_objectstore_1 and their entries in datasource table do.

As part of the all-hands-on-deck "we've run out of space on stage situation last week, I dropped a few tables for inactive repos.. not realising (or checking) that cycle-data handles this case - so this is actually my fault oops.

A future patch should probably (a) make cycle-data more resilient against non-existent tables, and (b) make datasource return the correct exception (rather the one about not being able to close the connection) - but for now, I'll just remove the accessibility_jobs_1 entry from the datasource table, so cycle-data can succeed.

Comment 4

[Ed Morley [:emorley]]

That said, comparing some of the tasks, we use a mixture of "fetch all projects listed in the datasource table" and "fetch all repositories in the repositories fixture and filter by active_status". eg process_objects and cycle_data use the former, whereas fetch-pushlogs uses the latter.

I think we should switch them all to the latter - I'll file a new bug.

Comment 5

[Ed Morley [:emorley]]

cycle-data ran fine manually & this morning as the scheduled task (well, it timed out this morning, but due to the chunk size being too large, but that's pre-existing).