1080760 - Allow non-buildbot submitters to parse logs themselves and post the error summary to Treeherder directly

Status: RESOLVED FIXED

(Tree Management :: Treeherder: API, defect, P2)

Description

[Jonathan Griffin (:jgriffin)]

One of the factors that we're seeing at scale is logparsing falling behind due to job volume.  The number of nodes used for logparsing is currently fixed (sans intervention from IT), and doesn't deal with spikey load very well.

We'd like to be able to dynamically allocate parsers as needed.  One way to accomplish this would be to use TC-provisioned VM's.

Comment 1

[Ed Morley [:emorley]]

Morphing this based on the direction I believe we want to go.

Comment 2

[Bob Clary [:bc] (inactive)]

Per our conversion @mozlandia...

Attempting to get Autophone to meet the sheriffing requirements, I found that although I could parse my logs locally and obtain the test failure messages, there wasn't a public api that I could use to obtain the suggested matching bugs nor would Treeherder simply take my error messages and look up the suggested bugs for me.

Currently, I have taken code from Treeherder that will allow me to perform the look up locally and then submit the error messages and the suggested bugs using a 'Bug Suggestions' artifact. 

What I would find useful would to have Treeherder perform the lookup of the suggested matching bugs for me. I could submit a list of test failure messages and Treeherder would automatically perform the look up and population of the 'Bug Suggestions' artifact for my job.

Having the ability to perform the bug suggestions lookup via a public api would also be potentially useful for other tools but isn't a necessity for me if Treeherder would handle it for me.

If this is already available in some form, I would appreciate being educated about it. Thanks!

Comment 3

[Ed Morley [:emorley]]

Now that the volume of jobs being submitted from non-buildbot sources has increased significantly, fixing this bug would save us quite a bit of load on both the log processor nodes (we're struggling to catch up with a backlog at the moment).

Comment 4

[Ed Morley [:emorley]]

I think the importance of this is now higher than it was on the past, given the increasing number of TC jobs and the issues the current "create a fake buildbot log" workflow creates (eg bug 1149168).

Comment 5

[Cameron Dawson [:camd]]

Bumping this a bit higher as I'll be looking at this within the next week or so and task cluster submissions are continuing to grow.

Comment 6

[Ed Morley [:emorley]]

Re:

22:41 <camd> so, we have, in our settings some celery queues:https://github.com/mozilla/treeherder/blob/bug-suggestions-auto/treeherder/settings/base.py#L187-187
22:42 <camd> and apparently we setup "log_parser_hp" as a high priority queue.  
22:42 <camd> I'm just not sure how that works.  if it's maybe here: https://github.com/mozilla/treeherder/blob/bug-suggestions-auto/bin/run_celery_worker_log_parser#L25-25

We currently don't have queues that have a higher priority than others, we just have several queues with different names - with one of those names being log_parser_hp (where the hp stands for high priority). The celery worker then does a round-robbin on a number of log_parser queues. In effect jobs in the _hp queue do get a higher priority, since that queue is typically shorter, but it's not guaranteed. Bug 1076761 is filed for improving the behaviour here - see bug 1076761 comment 1.

For now though I'd just create another queue and put it alongside the others :-)

Comment 7

[Cameron Dawson [:camd]]

Attachment: auto generate bug suggestions artifact

Comment 8

[Cameron Dawson [:camd]]

Sorry to spam you all for reviews.  Feel free to bow out if you like.  :)

Ed: thanks for the clarification in comment 6.

Comment 9

[Ed Morley [:emorley]]

Comment on attachment 8596871 [details] [review]
auto generate bug suggestions artifact

Looks good at a first pass - have left some comments on the PR (view them on Reviewable to see them in situ). Guess the proof will be in the New Relic web transaction times :-)

Comment 10

[Cameron Dawson [:camd]]

I've gotten feedback from everybody that this is now in good shape.  Thanks so much for all your feedback!!  And for helping me track down those issues, mdoglio.

I'm waiting on Travis to come back clear and then I'll merge this.

Comment 11

[Ed Morley [:emorley]]

(In reply to Cameron Dawson [:camd] from comment #10)
> I've gotten feedback from everybody that this is now in good shape.  Thanks
> I'm waiting on Travis to come back clear and then I'll merge this.

Note: Travis has been having issues today - any failures at the "create the treeherder DB" step are http://www.traviscistatus.com/incidents/h91j8ft0qs2l

Comment 12

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3fa859e8d42f039c3f830f4e5719e8425bad89ed
Bug 1080760 - generate bug suggestions for text_log_summary artifacts

Whenever a ``text_log_summary`` with ``error_lines`` is submitted, generate an
additional ``Bug suggestions`` artifact for it.

https://github.com/mozilla/treeherder/commit/bfef91b71f4d30326e2a1bdbbb474ad9094f0666
Bug 1080760 - Tests for bug suggestion artifact changes

Comment 13

[Mauro Doglio [:mdoglio]]

Comment on attachment 8596871 [details] [review]
auto generate bug suggestions artifact

I forgot to r+ this :p

Comment 15

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/0530a8b0f712337efd93c321e588f1f08042268a
Revert bfef91b (Bug 1080760 - Tests for bug suggestion artifact changes)

Since we're about to revert the functionality it tests.

https://github.com/mozilla/treeherder/commit/6c7d48d50463c9a5768d48644ef621b2ba3de4de
Revert 3fa859e (bug 1080760) since the load on the web API is too high

Bug 1080760 moved bug suggestion generation, so that instead of
happening during log parsing, it occurred when the log artefacts were
then posted to the API - so that external submitters of logs could also
benefit from bug suggestions. Unfortunately the increase in load on the
web API is higher than expected, so I believe we should now try the
alternative approach, which is to have an additional async task that
generates the bug suggestions after the log parser has run and
submitted the artefacts to the API.

https://github.com/mozilla/treeherder/commit/4824072e39c78fd96ea4d668f536f2927d7473e0
Make mixins.py use the in-repo treeherder-client

The revert of bug 1080760 added back an import of the treeherder-client
package, which escaped conversion since it was not present when the
client was imported to the repo.

Comment 16

[Ed Morley [:emorley]]

Reverted for bug 1160856.

Comment 17

[Cameron Dawson [:camd]]

Attachment: PR

Comment 18

[Cameron Dawson [:camd]]

This latest PR sits on top of the refactor changes in Bug 1163182.  

This generates bug suggestions for text_log_summary artifacts submitted to the artifact REST endpoint. It schedules those asynchronously with celery.

If a job is submitted through the job REST endpoint and it contains a text_log_summary artifact with it, then the Bug suggestions artifact is created immediately (not asynchronously). I created Bug 1163253 to address possibly making that jobs endpoint asynchronous as well.

Comment 19

[Cameron Dawson [:camd]]

Attachment: ui PR

Comment 20

[Ed Morley [:emorley]]

Comment on attachment 8603674 [details] [review]
PR

r- for the perf, since this change should be a no-op perf-wise until submitters like taskcluster start using the new codepaths, right?

Comment 21

[Ed Morley [:emorley]]

Comment on attachment 8604351 [details] [review]
ui PR

Cancelling until I understand the service PR a bit better (https://github.com/mozilla/treeherder/pull/518#issuecomment-101073108), since implementation there affects this.

Comment 22

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #20)
> r- for the perf, since this change should be a no-op perf-wise until
> submitters like taskcluster start using the new codepaths, right?

Looking at this more, it may have just been noise from spikes in load. Let's get the refactor nits fixed and it landed and pushed to stage, and then we can have another look on stage now the load is more representative, being midweek.

Comment 23

[Cameron Dawson [:camd]]

Comment on attachment 8603674 [details] [review]
PR

I've redeployed this to stage and I think it looks good.

Comment 24

[Cameron Dawson [:camd]]

Comment on attachment 8604351 [details] [review]
ui PR

Minor UI change for this.

Comment 25

[Ed Morley [:emorley]]

Comment on attachment 8603674 [details] [review]
PR

Almost there :-)

Comment 26

[Ed Morley [:emorley]]

Comment on attachment 8604351 [details] [review]
ui PR

UI part looks good :-)

Comment 27

[Cameron Dawson [:camd]]

Comment on attachment 8603674 [details] [review]
PR

Ok, let's try this again.  :)

Comment 28

[Ed Morley [:emorley]]

Comment on attachment 8603674 [details] [review]
PR

r+ once the issues in https://github.com/mozilla/treeherder/pull/518#issuecomment-103478215 are resolved :-)

Comment 29

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/358e90f685e96759b66847e47db96601aeb874eb
Bug 1080760 - Auto-generate bug suggestions asynchronously

This introduces two new ways to generate ``Bug suggestions`` artifacts from
a ``text_log_summary`` artifact
1. POST a ``text_log_summary`` on the ``/artifact`` endpoint
2. POST a ``text_log_summary`` with a job on the ``/jobs`` endpoint.

Both of these cases will schedule an asynchronous task to generate the
``Bug suggestions`` artifact with ``celery``.

Artifact generation scenarios:

JobCollections
^^^^^^^^^^^^^^
Via the ``/jobs`` endpoint:

1. Submit a Log URL with no ``parse_status`` or ``parse_status`` set to "pending"
    * This will generate ``text_log_summary`` and ``Bug suggestions`` artifacts
    * Current *Buildbot* workflow

2. Submit a Log URL with ``parse_status`` set to "parsed" and a ``text_log_summary`` artifact
    * Will generate a ``Bug suggestions`` artifact only
    * Desired future state of *Task Cluster*

3. Submit a Log URL with ``parse_status`` of "parsed", with ``text_log_summary`` and ``Bug suggestions`` artifacts
    * Will generate nothing

ArtifactCollections
^^^^^^^^^^^^^^^^^^^
Via the ``/artifact`` endpoint:

1. Submit a ``text_log_summary`` artifact
    * Will generate a ``Bug suggestions`` artifact if it does not already exist for that job.

2. Submit ``text_log_summary`` and ``Bug suggestions`` artifacts
    * Will generate nothing
    * This is *Treeherder's* current internal log parser workflow

https://github.com/mozilla/treeherder/commit/aa16df1d485d17009a1707d699d9bdcca3a6db85
Bug 1080760 - UI Scan for bug-suggestions detached from log parsing

Comment 30

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/70ddc6a951c0cd316ca54edbf12dc5e56f03cd45
Bug 1080760 - Missing fixure added to fix tests