1915388 - Design a buffered/local API for *Distributions

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, enhancement, P1)

Description

[Jan-Erik Rediger [:janerik]]

For tighter loops the overhead of start/stop or accumulateSingleSample might be too much.
stop/accumulateSingleSample causes allocations and a database write of the whole metric (and thus currently the whole database as well).
accumulateSamples offloads some of that overhead to the caller: the caller needs to measure and buffer on their end, before handing over all samples. That still has the memory allocation overhead to collect potentially thousands of samples.

As an alternative we could expose a wrapper around the underlying histogram and then merge that on commit.
This bug is about designing that API for the different target languages we need to support.

Comment 1

[Jan-Erik Rediger [:janerik]]

Attachment: Distribution Metrics - buffered API

Comment 2

[Jan-Erik Rediger [:janerik]]

This is similar to what's requested in https://github.com/mozilla/glean/pull/2945

Comment 3

[Max Inden]

Thank you Jan-Erik for the design document.

(In reply to Jan-Erik Rediger [:janerik] from comment #0)

As an alternative we could expose a wrapper around the underlying histogram and then merge that on commit.

Yes. That would solve https://github.com/mozilla/glean/pull/2945 and unblock https://bugzilla.mozilla.org/show_bug.cgi?id=1906853.

(In reply to Jan-Erik Rediger [:janerik] from comment #0)

This bug is about designing that API for the different target languages we need to support.

For the record, for https://bugzilla.mozilla.org/show_bug.cgi?id=1906853, I would only need a Rust API.

Do you have a rough timeline estimate? As in weeks, months, quarters, ...?

Let me know if I can be of any help.

Comment 4

[Jan-Erik Rediger [:janerik]]

(In reply to Max Inden from comment #3)

Thank you Jan-Erik for the design document.

(In reply to Jan-Erik Rediger [:janerik] from comment #0)

As an alternative we could expose a wrapper around the underlying histogram and then merge that on commit.

Yes. That would solve https://github.com/mozilla/glean/pull/2945 and unblock https://bugzilla.mozilla.org/show_bug.cgi?id=1906853.

(In reply to Jan-Erik Rediger [:janerik] from comment #0)

This bug is about designing that API for the different target languages we need to support.

For the record, for https://bugzilla.mozilla.org/show_bug.cgi?id=1906853, I would only need a Rust API.

Yes, but if we do move forward with this we (as in the Glean team) have to consider the other languages to avoid putting us into a corner where we can't support the API across languages.

Do you have a rough timeline estimate? As in weeks, months, quarters, ...?

Once we agree we want this and we agree that we ship only the Rust API for now (but with plans for the other APIs) we should get this going quickly. So week(s).
Rust implementation shouldn't be much, I have part of that in a WIP branch.

Comment 5

[Jan-Erik Rediger [:janerik]]

Seems the general consensus was a r+. For formality tagging both Travis and chutten once more.
I have a PoC implementation and I would say let's land this as experimental and have Max use it.

Comment 6

[Travis Long [:travis_]]

Sounds good to me!

Comment 7

[Chris H-C :chutten]

Sure, we can give it a go.

But if this is going to be experimental, maybe we think about metrics by which we might measure success or failure of the experiment?