1347564 - Add the capability to manage several nightlies per day in the Aggregates dataset

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P3)

Description

[Sylvestre Ledru [:Sylvestre]]

We would like to move to two nightlies per day (mostly to accommodate Europeans)
As discussed here:
https://mail.mozilla.org/pipermail/fhr-dev/2017-March/001224.html
this will require some changes in the pipeline to get that done.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Is this "We'd like to do this because it'd be nice to have" or "This is needed to support Dawn" or something else? Trying to get a sense of priority here relative to the growing amount of work our team needs to do to support Quantum.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

In the dawn project, we believe that we should improve the quality of nightly to be able to keep the same level of quality.
Because nightly are live around 2pm Europe time, we cannot test the changes in our morning. Having a nightly ready for Europe in the morning would help in verify the bugs and having a shorter feedback loop.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

As mentioned in the fhr-dev thread, we're already dealing with respins in the current world, so this doesn't need to block moving to two nightlies a day. While aggregating by date isn't optimal, as far as I can tell, the main downside to this is lost granularity. Balancing that against the work currently underway to lower the latency of Telemetry data and the non-trivial amount of work this'll take to do properly, I'm putting it in the backlog for now. If we find ourselves in a situation where day-level resolution on Telemetry results isn't good enough, we can revisit the priority.

Comment 4

[Frank Bertsch [:frank]]

Now that we are really getting multiple nightlies a day, this seems to be relevant; but we are holding off on implementation until we have build-hub [0] integration, which should give push-log ranges for builds.

[0] https://github.com/mozilla-services/buildhub

Comment 7

[Andrew McCreight [:mccr8]]

Bug 1462391 comment 0 talks about a specific instance where Telemetry Evolution essentially gave the wrong regression date because of this build merging.