1076774 - Consider segregating workers for more critical tasks (eg pushlog & job result ingestion)

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

Saw an IRC comment about us possibly having tasks like the pushlog ingestion in another VM to increase reliability.

I don't have a particular gut feeling either way - happy to defer to others on whether we should do this - but filing this bug so we don't forget about the idea.

As for how we define "critical tasks" - I would definitely call pushlog ingestion critical, and following closely behind, job result ingestion. Log parsing is definitely lower down the list.

Comment 1

[Mauro Doglio [:mdoglio]]

I implemented the pushlog-related change in bug 1076778. I will track the buildapi part here

Comment 2

[Mauro Doglio [:mdoglio]]

Attachment: Github PR #241 on treeherder-service

Comment 3

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/9555d428725a4eaf7cec24c65670a27b8ca3acd6
Bug 1076774 - use a separate worker for jobs ingestion

https://github.com/mozilla/treeherder-service/commit/7546d5d74a400542d653155377f125fe22e517b6
Merge pull request #241 from mozilla/bug-1076774-implement-buildapi-queue

Bug 1076774 - use a separate worker for jobs ingestion

Comment 4

[Mauro Doglio [:mdoglio]]

We have requested the VMs for the additional workers and changed the code where needed.
I will close this bug once everything is in place on production

Comment 5

[Ed Morley [:emorley]]

Great - thank you :-)