1286951 - Update moztelemetry and mozaggregator for change in childPayloads

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Chris H-C :chutten]

Child histograms will now be aggregated in the client and sent in a place that isn't <payload-root>.childPayloads[0..N].{keyedH|h}istograms but will be, instead, (taking a cue from bug 1281795) in <payload-root>.processes.content.{keyedH|h}istograms.

This will require changes in the aggregator [1], which currently looks for these histograms in childPayloads (before aggregating them itself); and moztelemetry [2], which does so similarly, but also sometimes aggregates child histograms with parent histograms (see with_processes=True).

[1]: https://github.com/mozilla/python_mozaggregator
[2]: https://github.com/mozilla/python_moztelemetry/

Comment 1

[GitHub Autolander Bot]

Attachment: [python_moztelemetry] chutten:childtel > mozilla:master

Comment 2

[GitHub Autolander Bot]

Attachment: [python_moztelemetry] chutten:childtel > mozilla:master

Comment 3

[Chris H-C :chutten]

moztelemetry has been packaged and updated on pypi. New clusters are picking up the updated package. Telemetry Hello World works. Looks legit.

Comment 4

[Chris H-C :chutten]

I have spent a lot of time trying to quickly fix mozaggregator to work with child hgrams in processes.content, but as my attempts [1] show, I'm not having an easy or quick go of it.

I think it's time for me to stop wasting time and declare defeat.

Could someone actually able to run the tests (and thus able to have a build, run, debug cycle that isn't remote printf debugging by proxy) take over this bug and make it happen? The actual code change probably only needs to be

+    _extract_histograms(state, ping.get("payload", {}).get("processes", {}).get("content", {}), True)

Around aggregator.py line 233 (in _aggregate_ping).

Where it gets a little trickier is in updating the tests to:
1) Use a mix of old and new types of pings
2) Account for the expected difference in "count" as described on fhr-dev [2]

[1]: https://github.com/mozilla/python_mozaggregator/pull/16
[2]: https://mail.mozilla.org/pipermail/fhr-dev/2016-July/000997.html

Comment 5

[Roberto Agostino Vitillo (:rvitillo)]

Chris, the real issue is that the tests weren't running on your machine and we should have invested some time to understand why that is instead of using travis. I have pushed a commit yesterday which should allow you to run the test-suite on Windows. It would be great if you could confirm that it works and complete your patch since it's practically done.

Comment 6

[Chris H-C :chutten]

Well, that was quick. Getting the tests to work on my setup was indeed to key to realizing what was going wrong. Thank you for your assistance, :rvitillo

mdoglio r? https://github.com/mozilla/python_mozaggregator/pull/17

Comment 7

[Chris H-C :chutten]

Both PRs have merged. We're done here.