Closed Bug 1113873 Opened 5 years ago Closed 5 years ago

Ingest structured/json logs from blobber & make them available via the logslice and job artifacts API

Categories

(Tree Management :: Treeherder: Data Ingestion, defect, P4)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: camd, Assigned: camd)

References

(Blocks 1 open bug)

Details

Attachments

(7 files)

We need a new UI to view and interact with the new Structured logs provided by the work Ahal and Cmanchester have done.  These logs are already being generated for some jobs, but not all.  These are xpcshell, marionette-based tests and gaia unit tests.

The first step will just create a minimum viable product for viewing data from a structured log

1. Detect when a job has provided a structured log
2. Load a summary structured log to view just the error lines from the main log
3. Create a minimum UI to view these summary lines
4. Be able to load the lines surrounding the error to give context
Assignee: nobody → cdawson
Depends on: 1113868
Attached file structured_summary.log
Component: Treeherder → Treeherder: Log Viewer
Status: NEW → ASSIGNED
Priority: -- → P3
Summary: use structured logs in treeherder where they exist for tests → Use structured logs in treeherder where they exist for tests
Attached file plain-chunked_raw.log
Priority: P3 → P4
Attached file service PR
Attachment #8572294 - Flags: review?(emorley)
Attached file ui PR
Attachment #8572296 - Flags: review?(emorley)
Ed: I'd be happy to walk through this PR with you on Vidyo.  Mark Cote and I decided to make this feature only about generating the structured log summary and job association on the service side.  This opens up someone else (possibly me) creating a UI for it in the future.
Comment on attachment 8572294 [details] [review]
service PR

Nice work! Have commented on the PR :-)
Attachment #8572294 - Flags: review?(emorley)
Attachment #8572296 - Flags: review?(emorley) → review+
Making the bug summary reflect the PRs here. The existing bug 1043739 now seems like a good place to "make treeherder, particularly it's UI, use the new logs".
Component: Treeherder: Log Viewer → Treeherder: Data Ingestion
Summary: Use structured logs in treeherder where they exist for tests → Ingest structured/json logs from blobber & make them available via the logslice and job artifacts API
Comment on attachment 8572294 [details] [review]
service PR

I made the updates you requested.  Or I explained why I didn't.  Please have another look when you get a chance.  Thanks for the great feedback!
Attachment #8572294 - Flags: review?(emorley)
Comment on attachment 8572294 [details] [review]
service PR

Just one last change and this can land (no need for re-review):
https://github.com/mozilla/treeherder-service/pull/415#discussion_r25942579
(Think this block changed since my last review?)
Attachment #8572294 - Flags: review+
I'd also make sure it's rebased on master before pushing, due to the travis changes etc :-)
(In reply to Ed Morley [:edmorley] from comment #16)
> I'd also make sure it's rebased on master before pushing, due to the travis
> changes etc :-)

You're welcome to try my script for speeding this up :-)
https://pastebin.mozilla.org/8824604

Add it to your .bashrc (or wherever) and then just:
$ land-pr my-local-branch-name

From the root of the repo.

It will switch to that branch name, rebase it against master, force push to the PR (so github knows to auto-close once it's on master), push to master, switch to the local master & update that, then delete the now-merged branch both locally and on the remote.
Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/dd612f00059b1e8139d2c73464cc224854ec0157
Bug 1113873 - Add ``mozlog`` to requirements

This library is used to handle structured logs

https://github.com/mozilla/treeherder-service/commit/ca84e4f373b691c0637532974bb1ae45e85b7d27
Bug 1113873 - Change name of ``Structured Log`` artifact to ``text_log_summary``

This is to remove confusion between the REAL structured log and the generated
summary artifact for the textual logs.

https://github.com/mozilla/treeherder-service/commit/f63ac67a6a4b58dc503a93b5d55e55542db373a3
Bug 1113873 - Modify ``logslice`` endpoint to determine which log to slice by its name

It formerly just sliced the first log it came across for the job, which
was correct since we always had just one before.

https://github.com/mozilla/treeherder-service/commit/3ad96399b80469e1cb83477cac0d202603a964c1
Bug 1113873 - Abstract parsing a log and creating a log artifact from posting it

This allows other artifact creators to use the same posting code

https://github.com/mozilla/treeherder-service/commit/850e74be10e533f77c39a8608d702435162c5c40
Bug 1113873 - Add support for structured log parsing

This adds the structured log url as a ``job_log_url``
and adds the artifact builder to generate a summary of the structured log.
This does not yet enable to queue for parsing the structured logs.

https://github.com/mozilla/treeherder-service/commit/2bc1f0a017a507afcae8b371e673be5f78fcea1b
Bug 1113873 - Test for structured log parser

https://github.com/mozilla/treeherder-service/commit/d4bb357d5e5003ec5f6a1e6a5e72c8f61b461657
Bug 1113873 - Enable the structured log parser celery queue
Commit pushed to master at https://github.com/mozilla/treeherder-ui

https://github.com/mozilla/treeherder-ui/commit/b9a4d69dc529e119600e559827a9c90533c67ce0
Bug 1113873 - Support existence of struct log summary artifact

When the Service is generating the structured log summary artifact, the
UI needs to handle that it will come back in the
endpoint for log artifacts. This change will just ignore the new
artifact.
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Depends on: 1142559
Depends on: 1152742
Blocks: 1160117
Depends on: 1162664
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3c61a1cce4953390ca81d4ff9ae34ab6ab546718
Bug 1113873 - Support existence of struct log summary artifact

When the Service is generating the structured log summary artifact, the
UI needs to handle that it will come back in the
endpoint for log artifacts. This change will just ignore the new
artifact.
Depends on: 1162526
Depends on: 1187267
Depends on: 1187284
You need to log in before you can comment on or make changes to this bug.