1328985 - Remove legacy datasource fields from treeherder/perfherder

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[William Lachance (:wlach)]

There are a number of fields in Treeherder / Perfherder which reference legacy datasource jobs which should go away, sooner or later:

* ds_job_id, result_set_id in "PerformanceDatum" (see treeherder/perf/models.py)
* project_specific_id in "Job" (see treeherder/treeherder/models.py)

Unfortunately we need to keep `project_specific_id` for at least a little while to handle legacy logviewer urls, but we should be able to fix up PerformanceDatum any time after bug 1318474 lands.

Comment 1

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) from comment #0)
> Unfortunately we need to keep `project_specific_id` for at least a little
> while to handle legacy logviewer urls

From what I can tell, the new IDs would have been used once bug 1318474 comment 24 was deployed (which was 29th Dec). The last jobs prior to that will expire 29th March, though we could always not wait until the full 4 months are up.

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3030

Comment 3

[William Lachance (:wlach)]

Comment on attachment 8935898 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3030

Datasource removal anniversary patch. :)

The only thing that's mildly scary is me deleting a column from mysql. Maybe with the newer version we're using this shouldn't be such a problem?

This is obviously not urgent at all.

Comment 4

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #3)
> The only thing that's mildly scary is me deleting a column from mysql. Maybe
> with the newer version we're using this shouldn't be such a problem?

Did some additional research and determined this is still problematic, so left the column in for now.

Comment 5

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #4)
> Did some additional research and determined this is still problematic, so
> left the column in for now.

Agree that's best for now. At some point in the future I'll remove both `project_specific_id` and `running_eta` at the same time.

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8935898 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3030

Many thanks for this! I spotted an issue with what Django generates for the migration unfortunately (getting tired of these Django MySQL migration bugs).

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8935898 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3030

Sorry about that! I have an updated PR which should uses runsql to make the changes to the db.

Comment 8

[Ed Morley [:emorley]]

Comment on attachment 8935898 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3030

Many thanks!

Comment 9

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/0c81ed8e76671019e4e09a1e96cd5cb6cce753c3
Bug 1328985 - Remove project_specific_id compatibility shim (#3030)

Comment 10

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3055

Comment 11

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/26ae45b0d61483fe2c9576e277ceea88e66491e4
Bug 1328985 - Make 0020_remove_project_specific_id a no-op (#3055)

To fix deployment, since it's currently failing with:
`django.db.utils.OperationalError: (1091, "Can't DROP 'job_repository_id_project_specific_id_7883a5e8_uniq'; check that column/key exists")`

We'll just drop the index by hand on dev/stage/prod instead - using:
`DROP INDEX `job_repository_id_1cfa81175c2709bc_uniq` ON treeherder.job;`

...since:
```
> show index from job where Column_name = 'project_specific_id'

******************** 1. row *********************
        Table: job
   Non_unique: 0
     Key_name: job_repository_id_1cfa81175c2709bc_uniq
 Seq_in_index: 2
  Column_name: project_specific_id
    Collation: A
  Cardinality: 11198
     Sub_part: 
       Packed: 
         Null: YES
   Index_type: BTREE
```

Comment 12

[Ed Morley [:emorley]]

(In reply to Treeherder Bugbot from comment #11)
> We'll just drop the index by hand on dev/stage/prod instead - using:
> `DROP INDEX `job_repository_id_1cfa81175c2709bc_uniq` ON treeherder.job;`

Done.

Comment 13

[Kyle Lahnakoski [:ekyle]]

Closing this because it is too general; we will make new bugs for specific removals, when we see them