774132 - Metlog support for mozbuild

Status: NEW

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

mach/mozbuild has a structured logging backend. All log messages are defined as a string action/name, a wall time, and a map of metadata. I purposefully implemented things this way because structured logging is awesome. Parsing is simple (no need to implement parsing of human-readable text) and thus machine consumption of the data is much easier to hook up.

I made sure that the format and logging API was compatible with Metlog (https://wiki.mozilla.org/Services/Sagrada/Metlog) so we could hook Metlog up to mach/mozbuild when the time was right.

For those not in the know, Metlog is a logging/streaming service built out by the Mozilla Services team (mostly the work of Rob Miller). It's pretty awesome. You hook your metlog-producing client up to an endpoint and log messages get streamed to it as they are written.

The benefits of using Metlog will likely only truly be realized by a distributed infrastructure. Once the buildbots build through mach, all the logs are structured at the point of generation. So, a lot of the complexity for parsing logs goes away. This means the human-readable format can be changed to suit humans without any concern about breaking systems relying on parsing a specific format.

Since Metlog can stream data, we could theoretically enable real-time log consumption for builds on TBPL. One example would be to stream build logs in real-time to a Node.js server and convert messages to interested websocket listeners on the other side. Easier real-time visibility into builds == win.

Anyway, the first step is hooking metlog up to mozbuild. This means checking metlog into mozilla-central and then a) converting the logger API to use metlog b) build a proxy to convert mozbuild's logger API to metlog messages. Once that is in place, magic can ensue.