1918118 - Reduce the number of times we parse Glean definitions yamls in tree

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P4)

Description

[Chris H-C :chutten]

In bug 1686265, Nika showed us how to get down to just two GeneratedFile directives in t/c/g/moz.build. Things have complicated somewhat in the two years since, and now we parse the same or similar sets of files many times a build (plus one more if it's artifact).

In the ideal case we shouldn't need to parse a given metrics.yaml, pings.yaml, or tags.yaml file more than once: we could parse the whole lot and then pass the subset of the object trees to the necessary code generators. Even following the existing pattern, there are only three unique sets of yamls in use (metrics+tags, pings+tags, metrics+pings+tags) suggesting any number of parses above three is likely especially unnecessary. (And for artifact builds (or any build without a compiler, honestly), we probably don't want to generate a bunch of compiled code that'll never be compiled.)

This bug is about reducing the number of times we call glean_parser. (For the bug reducing the amount of rebuilding necessary as a result of a change to a given yaml file, see bug 1879329.) Be sure to run any changes against a variety of build configurations on try because who knows, maybe there's a config that runs without a compiler but expects codegen to happen anyway so they can pass it on to a compile task later? <shrug emoji>

Comment 1

[[:sergesanspaille]]

It looks possible to cache the parsing of the yaml files parsing so that further invocation can just absorb the internal representation. i'll investigate if that's an easy win.

Comment 2

[[:sergesanspaille]]

Attachment: Bug 1918118 - Cache parsing of glean metrics r=glandium

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/085ebfdd51c9
Cache parsing of glean metrics r=chutten,TravisLong

Comment 4

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/085ebfdd51c9

Comment 5

[Magnus Melin [:mkmelin]]

Comment on attachment 9424372 [details]
Bug 1918118 - Cache parsing of glean metrics r=glandium

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Needed for Thunderbird Glean - bug 1954442.
  Other needed patches relies on this bug.
• User impact if declined:
• Fix Landed on Version: 132
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's build level improvement

Comment 6

[Magnus Melin [:mkmelin]]

The 128 patch I got crafting/fixing this is here: https://hg.mozilla.org/try/rev/516cd72b47f76df02f437a9034496a40031a24b1

Comment 7

[Magnus Melin [:mkmelin]]

Attachment: Bug 1918118 - Cache parsing of glean metrics

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Original Revision: https://phabricator.services.mozilla.com/D221944

Comment 8

[Travis Long [:travis_]]

Attachment: Bug 1918118 - Cache parsing of glean metrics

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Original Revision: https://phabricator.services.mozilla.com/D221944

Comment 9

[Travis Long [:travis_]]

Attachment: Bug 1918118 - Cache parsing of glean metrics

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Original Revision: https://phabricator.services.mozilla.com/D221944

Comment 10

[Travis Long [:travis_]]

Attachment: Bug 1918118 - Cache parsing of glean metrics

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Original Revision: https://phabricator.services.mozilla.com/D221944

Comment 11

[Travis Long [:travis_]]

Attachment: Bug 1918118 - Cache parsing of glean metrics

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Original Revision: https://phabricator.services.mozilla.com/D221944

Comment 12

[Travis Long [:travis_]]

Attachment: Bug 1918118 - Cache parsing of glean metrics

Glean metrics are described in yaml, which is notably slow to parse.
Instead of repeatedly parsing the same metrics again and again, cache
the parsing result.

Original Revision: https://phabricator.services.mozilla.com/D221944

Comment 13

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr128/rev/1679d5c000eb