1090441 - Only use the pushlog ingestion fall-back if a job was submitted more than N minutes ago

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

Until the analysis in bug 1085682 is complete we won't know for sure, but I think that the ingestion fall-back added in bug 1077136 might cause extra load, that we could avoid.

Imagine:
1) Someone pushes to a repo
2) Buildbot schedules a job on that revision, on that repo
3) Our job ingestion (or more, builds-pending.js) runs, tries to ingest the job, but doesn't find a matching revision in treeherder, since the pushlog ingestion hasn't yet run - since the revision is still extremely new.
4) The fall-back added in bug 1077136 is triggered, but now races with the pushlog import that just hadn't had time to run yet.
5) As a result, the load on json-pushes is higher, meaning we're more likely to timeout and possibly hit the very issues the fall-back was trying to wallpaper over.

To preserve the benefit of the fallback, whilst reducing the unnecessary fetches for new revisions, how about:

If a job's revision is unknown to treeherder:
   if the job's submit time is in the last N minutes (where N is perhaps 5?):
      skip it for now
   else:
      perform the fallback, like we do already


Thoughts?

Comment 1

[Cameron Dawson [:camd]]

Ed--  Yeah, this sounds reasonable.  It's worth doing to reduce load on json-pushes.  I'm not clear how much this is happening in production (grep'ing the logs is a bit of a hassle right now)  so, hard to gauge the impact.  However, this certainly won't hurt and will help at least some.  :)

Comment 3

[Cameron Dawson [:camd]]

Un-assigning from myself.  I'm not going to get to this soon.

Comment 4

[Ed Morley [:emorley]]

Rather than trying to filter out jobs that are newer than a certain age, it's going to be much easier to just skip fetch-missing-pushlog for pending/running jobs, and only do so for completed jobs --> bug 1234247.