(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)
Also we should be careful as changing this to just go from 10->25 could affect code sheriffs and their work on other jobs.
As we move to running more jobs on 10 or 25 intervals, backfill should figure out how to backfill until the last job was seen. Possibly this would mean that treeherder itself would find the last time the desired job was seen (in the future the specific manifest) and use that as a parameter for how many revisions need backfill data.
These highlights show this isn't a trivial change. It's something that implies back & forth with the code sheriffs & assessment of the impact it may have (based on their current procedures). Also, it likely implies some more advanced, Treeherder-specific logic for dynamically figuring out the backfill range, which is out of perf team's scope & skillset.
Perf sheriffs agreed to sheriff more sparse data (given the new cost-effective measures). However, we expected that any changes related to reducing jobs per push would be taking care of, not to affect us.
In that it would be accompanied by an appropriate adjustment to the backfill ability.
In the past, perf sheriffs were used with one job every 5 pushes. The backfill ability could backfill 5 jobs.
At some point, we moved to having one job every 10 pushes. The backfill ability was adjusted to backfill 10 jobs.
Now, we have one job every 25 pushes. Yet, the backfill ability wasn't adjusted.
The task of reducing the job frequency should be an atomic one, involving reducing the job frequency and adjusting the backfill ability accordingly (at the same time). Otherwise, this task isn't 100% done & Perf sheriffs' resolution time is directly & negatively impacted.