1076769 - When an job with an unparsed log is accessed via the job details panel, parse it immediately

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

Broken out from bug 1074927.

One of the problems with the current treeherder log parsing implementation compared to TBPL, is that there is no way to force log parsing for a job of interest, or otherwise raise the priority of the existing queued task.

This would alleviate many of the use-case problems highlighted in bug 1074927's deps and dupes, since even if there is a backlog of queued log parser tasks, or if we didn't quite get the prioritisation algorithm right, the user can still access the log fairly rapidly.

ie:
1) User clicks on job in job details panel
2) The job artefacts are requested, and either:
  a) The service recognises that the log isn't parsed and pokes the log parser. It then returns the artefact as before, since we can't block on it.
  b) The UI makes another call to a log parser endpoint once the returned artefact shows an unparsed log.
3) In response, the service either:
  a) Bumps the priority of the current queued parse job
  b) Adds a new queued task at the front of the queue (and the log parser knows to bail out when it reaches the original tasks)
4) If the UI received an artefact indicating an unparsed log, it knows to re-request the log every N seconds (with appropriate spinner/message in the UI) and/or has a retry button in the failure summary panel (next to the message about log parsing not complete).

Comment 1

[Mauro Doglio [:mdoglio]]

Attachment: Github PR #261 on treeherder-service

Comment 2

[Mauro Doglio [:mdoglio]]

Attachment: Github PR #245 on treeherder-ui

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/b03b3320994c0461892898cb1262c76cc1831eb7
Bug 1076769 - setup lock strategy for the log parser

During a log parser run, the same log cannot be parsed by another
worker. This is done using a memcached-based lock mechanism.

https://github.com/mozilla/treeherder-service/commit/b6898b6f87e7189232a9e386307e6dd33b466164
Bug 1076769 - support parsing a log on demand

https://github.com/mozilla/treeherder-service/commit/f4e979a2485dbb5cb0d2163d2f86c72d00faff77
Merge pull request #261 from mozilla/bug-1076769-parse-log-on-demand

Bug 1076769 parse log on demand

Comment 5

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-ui

https://github.com/mozilla/treeherder-ui/commit/847dda8aeedfa128c8a859ddcb48d01e37cf83a8
Bug 1076769 - support parsing a log on demand

https://github.com/mozilla/treeherder-ui/commit/19a695f29de0c6eaee81b86d6323148c86a0dc8c
Merge pull request #245 from mozilla/bug-1076769-parsing-logs-on-demand

Bug 1076769 - support parsing a log on demand

Comment 6

[Ed Morley [:emorley]]

Partially reverted in:
https://github.com/mozilla/treeherder-service/commit/0cdf0092a1e71ba49227e288747f2586f2dec04c
desc: "revert latest changes to the parse-log task to reduce the load on memcached"

Comment 7

[Mauro Doglio [:mdoglio]]

The code reverted was an optimization to avoid 2 log parsers to run in parallel on the same log. Given the performance consequences of using memcached delete, it's not worth it. This is in on production now

Comment 8

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d548b279a85aa2c3bc43352f092686fba79e1905
Bug 1076769 - support parsing a log on demand

https://github.com/mozilla/treeherder/commit/19eec1b5604b2a989210c6a4dca14ab88cfc7c68
Merge pull request #245 from mozilla/bug-1076769-parsing-logs-on-demand

Bug 1076769 - support parsing a log on demand