Closed Bug 1076769 Opened 10 years ago Closed 10 years ago
When an job with an unparsed log is accessed via the job details panel, parse it immediately
Broken out from bug 1074927. One of the problems with the current treeherder log parsing implementation compared to TBPL, is that there is no way to force log parsing for a job of interest, or otherwise raise the priority of the existing queued task. This would alleviate many of the use-case problems highlighted in bug 1074927's deps and dupes, since even if there is a backlog of queued log parser tasks, or if we didn't quite get the prioritisation algorithm right, the user can still access the log fairly rapidly. ie: 1) User clicks on job in job details panel 2) The job artefacts are requested, and either: a) The service recognises that the log isn't parsed and pokes the log parser. It then returns the artefact as before, since we can't block on it. b) The UI makes another call to a log parser endpoint once the returned artefact shows an unparsed log. 3) In response, the service either: a) Bumps the priority of the current queued parse job b) Adds a new queued task at the front of the queue (and the log parser knows to bail out when it reaches the original tasks) 4) If the UI received an artefact indicating an unparsed log, it knows to re-request the log every N seconds (with appropriate spinner/message in the UI) and/or has a retry button in the failure summary panel (next to the message about log parsing not complete).
Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/b03b3320994c0461892898cb1262c76cc1831eb7 Bug 1076769 - setup lock strategy for the log parser During a log parser run, the same log cannot be parsed by another worker. This is done using a memcached-based lock mechanism. https://github.com/mozilla/treeherder-service/commit/b6898b6f87e7189232a9e386307e6dd33b466164 Bug 1076769 - support parsing a log on demand https://github.com/mozilla/treeherder-service/commit/f4e979a2485dbb5cb0d2163d2f86c72d00faff77 Merge pull request #261 from mozilla/bug-1076769-parse-log-on-demand Bug 1076769 parse log on demand
Commits pushed to master at https://github.com/mozilla/treeherder-ui https://github.com/mozilla/treeherder-ui/commit/847dda8aeedfa128c8a859ddcb48d01e37cf83a8 Bug 1076769 - support parsing a log on demand https://github.com/mozilla/treeherder-ui/commit/19a695f29de0c6eaee81b86d6323148c86a0dc8c Merge pull request #245 from mozilla/bug-1076769-parsing-logs-on-demand Bug 1076769 - support parsing a log on demand
Attachment #8510309 - Flags: review?(cdawson) → review+
Attachment #8510310 - Flags: review?(cdawson) → review+
Partially reverted in: https://github.com/mozilla/treeherder-service/commit/0cdf0092a1e71ba49227e288747f2586f2dec04c desc: "revert latest changes to the parse-log task to reduce the load on memcached"
The code reverted was an optimization to avoid 2 log parsers to run in parallel on the same log. Given the performance consequences of using memcached delete, it's not worth it. This is in on production now
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/d548b279a85aa2c3bc43352f092686fba79e1905 Bug 1076769 - support parsing a log on demand https://github.com/mozilla/treeherder/commit/19eec1b5604b2a989210c6a4dca14ab88cfc7c68 Merge pull request #245 from mozilla/bug-1076769-parsing-logs-on-demand Bug 1076769 - support parsing a log on demand
You need to log in before you can comment on or make changes to this bug.