1364556 - All Histograms Failing in TMO Evolution and Measurement Views

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler]]

STR:

View the measurement dashboard for some telemetry value.
Attempt to look at data from release/52 or beta/53, etc.

Actual results:

uncaught exception: Could not obtain evolution: status 500 (https://aggregates.telemetry.mozilla.org/aggregates_by/build_id/channels/beta/?version=53&dates=20170504%2C20170421%2C20170418%2C20170417%2C20170416%2C20170414%2C20170413%2C20170412%2C20170411%2C20170410%2C20170409%2C20170408%2C20170407%2C20170406%2C20170405%2C20170404%2C20170403%2C20170402%2C20170401%2C20170331%2C20170330%2C20170329%2C20170328%2C20170327%2C20170326%2C20170324%2C20170323%2C20170322%2C20170321%2C20170320%2C20170319%2C20170317%2C20170316%2C20170315%2C20170314%2C20170313%2C20170310%2C20170309%2C20170308%2C20170307%2C20170306%2C20170304%2C20170301%2C20170227%2C20170222%2C20170219%2C20170218%2C20170217%2C20170215%2C20170210%2C20161216%2C20161122&metric=SSL_INITIAL_FAILED_CERT_VALIDATION_TIME_MOZILLAPKIX&application=Firefox)

and so on in the console

Comment 1

[Frank Bertsch [:frank]]

This is a known issue due to the recent addition of 'record_in_processes` to histogram definitions in histograms.json. We are waiting on some patches to moz-central which will fix the issue (I've set those as blockers for this bug).

In the meantime, nightly probes will work since their associated histogram definitions have `record_in_processes`.

Comment 3

[Frank Bertsch [:frank]]

This is fixed. We now have to backfill because this also broke the aggregation job. The most recent data will be from 5/10.

Comment 5

[(no longer active)]

Did this bug also affect Nightly, even though the summary talks about non-Nightly?

Comment 6

[Frank Bertsch [:frank]]

(In reply to :Ehsan Akhgari (super long backlog, slow to respond) from comment #5)
> Did this bug also affect Nightly, even though the summary talks about
> non-Nightly?

The job failing - and so the backfill - do affect nightly. However historical nightly aggregates were still available in TMO during the outage.

Comment 7

[Frank Bertsch [:frank]]

Backfill is complete. Closing this out.

Comment 8

[(no longer active)]

Thank you!

Comment 9

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to :Ehsan Akhgari (super long backlog, slow to respond) from comment #8)
> Thank you!

+1!