Improve speed of import_perf_data script using bulk_create

RESOLVED FIXED

Status

Tree Management
Perfherder
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: wlach, Assigned: vakila, Mentored)

Tracking

Details

(Whiteboard: perfherder-starter-bug)

User Story

Thank you for helping out with Treeherder!

You can find us on IRC at irc://irc.mozilla.org/treeherder

Here's some links to help get you started.

Project page:
https://wiki.mozilla.org/Auto-tools/Projects/Treeherder

Interacting with us, repo locations and links to set up a development version of the software:
https://wiki.mozilla.org/Auto-tools/Projects/Treeherder#Contributing
https://wiki.mozilla.org/Auto-tools/Projects/Treeherder#Source_and_Docs

A-Team general reference, coding style guides:
https://ateam-bootcamp.readthedocs.org

Attachments

(1 attachment)

We have a command in perfherder for importing performance data from an existing instance of treeherder/perfherder. The import_perf_data script in perfherder's speed could be improved if we inserted the performance datum objects in bulk.

You can see an example of where we used to do this here in a deprecated migration script:

https://github.com/mozilla/treeherder/blob/066f437ca52b8833b5178eb68a7f2d2480f97574/treeherder/perf/management/commands/migrate_perf_data.py#L92

The file you want to modify is `treeherder/perf/management/commands/import_perf_data.py`. Specifically around here:

https://github.com/mozilla/treeherder/blob/master/treeherder/perf/management/commands/import_perf_data.py#L66

To test your change, set up a development instance of treeherder per the user story, then from vagrant ssh, run:

./manage.py import_reference_data  # required to view performance data
./manage.py import_perf_data --interval 2592000 --filter-props suite:tcheck2 mozilla-central
I'd like to mentor this bug. If you're not new to treeherder/perfherder, please don't take this. :) It's meant to be a simple launching off point to doing further treeherder development.
Mentor: wlachance
Whiteboard: perfherder-starter-bug
Created attachment 8731799 [details] [review]
[treeherder] vakila:perf-data-bulk > mozilla:master

Updated

2 years ago
Assignee: nobody → anjanavakil
(Assignee)

Updated

2 years ago
Attachment #8731799 - Flags: review?(wlachance)
Keywords: autoland
Comment on attachment 8731799 [details] [review]
[treeherder] vakila:perf-data-bulk > mozilla:master

Looks great now, thanks!
Attachment #8731799 - Flags: review?(wlachance) → review+

Updated

2 years ago
Keywords: autoland

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.