1193923 - Add support for importing reference data from a production instance

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P4)

Description

[William Lachance (:wlach)]

For testing my performance schema, I started work on a script to allow importing reference data from an existing instance of treeherder. I'd like to get my WIP committed, after which maybe someone else can finish it?

Comment 1

[William Lachance (:wlach)]

Attachment: PR

Comment 2

[Mauro Doglio [:mdoglio]]

Comment on attachment 8647128 [details] [review]
PR

Left a couple of comments more, but this is a r+

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/b020dfc073114e4a4efcf8ee5cbe866b217e663e
Bug 1193923 - Initial ability to import reference data from another server

This an (incomplete) command for importing reference data from another server.
Needs more work before it's complete, but has already proven useful for my own
testing.

Comment 4

[William Lachance (:wlach)]

Ok, merged the initial script but we still need to fetch the other pieces of reference data to complete this. I think this is worth doing, I don't have time just at the moment but if someone wants to finish this, just go to this file:

https://github.com/mozilla/treeherder/blob/b020dfc073114e4a4efcf8ee5cbe866b217e663e/treeherder/model/management/commands/import_reference_data.py#L44

This lists out the set of things we still need to fetch and insert. So inside that file, fetch them with the treeherder client (you will probably need to add accessor methods for most/all of them to https://github.com/mozilla/treeherder/blob/master/treeherder/client/thclient/client.py), then create new database models for them following the pattern that I started for option collection and machine platform.

Comment 5

[Gabriel Machado [:goma]]

Hey guys. I've been working on this bug for a while. I have some question:
A) In Job Type:
       Should the jobgroup field be filled just with a string containing the id of the jobgroup?
       I did something like:
           'job_group': str(job_type['job_group'])
       Is it correct?
B) In Repository:
       The field repository_group asks for an integer(when I run the program using ./manage.py), however, when I get the repository from c.get_repositories, it gives me a dictionary containing 2 strings. What should I do on this situation? It seems I'm not understanding the whole structure of classes. :(

Comment 6

[William Lachance (:wlach)]

(In reply to Gabriel Machado [:goma] from comment #5)
> Hey guys. I've been working on this bug for a while. I have some question:
> A) In Job Type:
>        Should the jobgroup field be filled just with a string containing the
> id of the jobgroup?
>        I did something like:
>            'job_group': str(job_type['job_group'])
>        Is it correct?

No, you should retrieve the appropriate job group type from the database and reference it when creating the model.

i.e.

job_group = models.JobGroup.objects.get(...)
job_type = models.MachinePlatform.objects.get_or_create(
   job_group=job_group,
   ...
)

> B) In Repository:
>        The field repository_group asks for an integer(when I run the program
> using ./manage.py), however, when I get the repository from
> c.get_repositories, it gives me a dictionary containing 2 strings. What
> should I do on this situation? It seems I'm not understanding the whole
> structure of classes. :(

This is the same as for (A), you should retrieve an instance of the object that you want via the ORM and pass it in when getting/creating the object.

Referring to the Django DB/ORM documentation may also be useful here. https://docs.djangoproject.com/en/1.8/topics/db/

Comment 7

[Gabriel Machado [:goma]]

Attachment: Link to Github Pull Request

This is the link to the pull request

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8654304 [details] [review]
Link to Github Pull Request

This looks pretty good! A few small changes requested-- make those, amend the commit, and then I think we should be good to land.

Comment 9

[Gabriel Machado [:goma]]

Attachment: Link to Github Pull Request

ok. I've done all the modifications. I think it is fine now :)

Comment 10

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/e94f27f0c4f5eb4120baee31344d8815f48fd914
Bug 1193923 - Add support for importing reference data from a production instance

Comment 11

[William Lachance (:wlach)]

Comment on attachment 8654981 [details] [review]
Link to Github Pull Request

Works well except for one minor bug that I fixed. Thanks!