Add support for importing reference data from a production instance

RESOLVED FIXED

Status

Tree Management
Treeherder
P4
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: wlach, Assigned: wlach)

Tracking

Details

Attachments

(3 attachments)

For testing my performance schema, I started work on a script to allow importing reference data from an existing instance of treeherder. I'd like to get my WIP committed, after which maybe someone else can finish it?
Depends on: 1193925
Created attachment 8647128 [details] [review]
PR
Attachment #8647128 - Flags: review?(mdoglio)
Comment on attachment 8647128 [details] [review]
PR

Left a couple of comments more, but this is a r+
Attachment #8647128 - Flags: review?(mdoglio) → review+

Comment 3

3 years ago
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/b020dfc073114e4a4efcf8ee5cbe866b217e663e
Bug 1193923 - Initial ability to import reference data from another server

This an (incomplete) command for importing reference data from another server.
Needs more work before it's complete, but has already proven useful for my own
testing.
Ok, merged the initial script but we still need to fetch the other pieces of reference data to complete this. I think this is worth doing, I don't have time just at the moment but if someone wants to finish this, just go to this file:

https://github.com/mozilla/treeherder/blob/b020dfc073114e4a4efcf8ee5cbe866b217e663e/treeherder/model/management/commands/import_reference_data.py#L44

This lists out the set of things we still need to fetch and insert. So inside that file, fetch them with the treeherder client (you will probably need to add accessor methods for most/all of them to https://github.com/mozilla/treeherder/blob/master/treeherder/client/thclient/client.py), then create new database models for them following the pattern that I started for option collection and machine platform.

Updated

3 years ago
Priority: -- → P4
Hey guys. I've been working on this bug for a while. I have some question:
A) In Job Type:
       Should the jobgroup field be filled just with a string containing the id of the jobgroup?
       I did something like:
           'job_group': str(job_type['job_group'])
       Is it correct?
B) In Repository:
       The field repository_group asks for an integer(when I run the program using ./manage.py), however, when I get the repository from c.get_repositories, it gives me a dictionary containing 2 strings. What should I do on this situation? It seems I'm not understanding the whole structure of classes. :(
Flags: needinfo?(wlachance)
(In reply to Gabriel Machado [:goma] from comment #5)
> Hey guys. I've been working on this bug for a while. I have some question:
> A) In Job Type:
>        Should the jobgroup field be filled just with a string containing the
> id of the jobgroup?
>        I did something like:
>            'job_group': str(job_type['job_group'])
>        Is it correct?

No, you should retrieve the appropriate job group type from the database and reference it when creating the model.

i.e.

job_group = models.JobGroup.objects.get(...)
job_type = models.MachinePlatform.objects.get_or_create(
   job_group=job_group,
   ...
)

> B) In Repository:
>        The field repository_group asks for an integer(when I run the program
> using ./manage.py), however, when I get the repository from
> c.get_repositories, it gives me a dictionary containing 2 strings. What
> should I do on this situation? It seems I'm not understanding the whole
> structure of classes. :(

This is the same as for (A), you should retrieve an instance of the object that you want via the ORM and pass it in when getting/creating the object.

Referring to the Django DB/ORM documentation may also be useful here. https://docs.djangoproject.com/en/1.8/topics/db/
Flags: needinfo?(wlachance)
Created attachment 8654304 [details] [review]
Link to Github Pull Request

This is the link to the pull request
Attachment #8654304 - Flags: review?(wlachance)
Comment on attachment 8654304 [details] [review]
Link to Github Pull Request

This looks pretty good! A few small changes requested-- make those, amend the commit, and then I think we should be good to land.
Attachment #8654304 - Flags: review?(wlachance) → review-
Created attachment 8654981 [details] [review]
Link to Github Pull Request

ok. I've done all the modifications. I think it is fine now :)
Attachment #8654981 - Flags: review?(wlachance)

Comment 10

3 years ago
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/e94f27f0c4f5eb4120baee31344d8815f48fd914
Bug 1193923 - Add support for importing reference data from a production instance
Comment on attachment 8654981 [details] [review]
Link to Github Pull Request

Works well except for one minor bug that I fixed. Thanks!
Attachment #8654981 - Flags: review?(wlachance) → review+
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

3 years ago
Blocks: 1214618
You need to log in before you can comment on or make changes to this bug.