1310974 - Move autoclassification task to run after crossreference error lines

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[James Graham [:jgraham]]

In order to make autoclassification of unstructured lines work, and to make the API much simpler, it needs to run after we have matched structured and unstructured errors from the various logs. This unfortunately increases teh latency for the information being available, but the tradeoffs seem worthwhile (and if they are not we could consider solutions like having a single task for all log parsing that has internal parallelism rather than relying on the celery queue).

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:autoclassify_after_crossreference > mozilla:master

Comment 2

[Ed Morley [:emorley]]

Comment on attachment 8804624 [details] [review]
[treeherder] mozilla:autoclassify_after_crossreference > mozilla:master

r+ with the default added to the new param in `crossreference_error_lines()`, for backwards compat with already-queued jobs at deploy-time.

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3777efd8fce3bffd7e45b60d00161117de3e0b9d
Bug 1310974 - Perform autoclassification after crossreferencing error lines (#1955)

In the short term this change is a small negative as it will slow down
autoclassification tasks. However it's a necessary step to future
changes that will put the autoclassification results on the text log
error table and allow autoclassification of non-structured lines.

As part of this, we remove the use of callback tasks and just schedule
the crossreference task to run after the parser tasks. This is simpler
and reflects the fact that we only schedule one parser task at a time
now.