1308528 - Serialize ingestion of job data

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[William Lachance (:wlach)]

To make implementing bug 1306674 easier, I'd like to experiment with serializing ingestion of job data. Trying to batch this operation makes it very difficult to understand and maintain (a concern when doing such a large refactor) and I'm pretty unconvinced we're actually gaining much performance by doing so.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1308528 > mozilla:master

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8798931 [details] [review]
[treeherder] wlach:1308528 > mozilla:master

Hey Cam, I was originally intending to put off landing this but I think it makes sense to go in sooner than later (since it's good to not change so much at once). Testing on treeherder-prototype, the perf differential seems pretty negligible. Extra bonus: we now handle several retry use cases properly where we didn't before (see the unit tests).

Probably some of the datasource logic could be made "nicer" but I didn't bother since all of that is going away soon.

Comment 3

[Cameron Dawson [:camd]]

This sounds like a great plan, yeah.  I'll review this on Monday.  It's been a long day and I want to be fresh to review this.  thanks for doing this!  :)

Comment 4

[Cameron Dawson [:camd]]

Comment on attachment 8798931 [details] [review]
[treeherder] wlach:1308528 > mozilla:master

Thanks for doing this!

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/61d89cd1a1df2b278d6a76581658a9f6010f410e
Bug 1308528 - Serialize ingestion of job data (#1906)

Comment 6

[William Lachance (:wlach)]

This has landed on stage, going to let it soak there for at least a day before attempting to deploy to production.