1656343 - Investigate improving performance of store-pulse-tasks

Status: NEW

(Tree Management :: Treeherder: Data Ingestion, enhancement)

Description

[Armen [:armenzg]]

Attachment: [screenshot] New Relic - 7 days of store-pulse-tasks

I've tried to look at historical data but I can't go far enough. I don't remember taking so much time of the overall pie (35% compared to 55% by log-parser).

aiohttp takes a good chunk of it.

I think I would like to add deeper NewRelic metrics & test out on prototype the idea of removing asynchronous usage of the Taskcluster library.

Comment 1

[Armen [:armenzg]]

Attachment: [screenshot] One single transaction - aiohttp takes 80% of the wall clock

Comment 2

[[e-mail] (:bhearsum)]

We have more metrics around this now in Grafana

Handling task messages is generally pretty quick these days, with some exceptions. We did find that the sole worker pod could only a certain number of events per minute, and we were backlogging because of that at certain times. We increased the number of pulse task workers to compensate for this.

It's probably not worth optimizing this further at this time. One thing we could do to speed things up if we need to is ensure that the pulse messages contain all the information needed for Treeherder to store the task. This would avoid the need to fetch the task definition and/or status as part of messaging handling.