1076792 - In the json-pushes "no cached revision" fall-back case, protect against >10 pushes having been made

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Mauro Doglio [:mdoglio]]

Treeherder is currently fetching 10 pushes as the default case (no cache available). We should try to increase it to 20 or so to avoid missing pushes on busy hours.

Comment 1

[Ed Morley [:emorley]]

As discussed on IRC, let's deprioritise this over bug 1077136 and bug 1076752, since it will require changes to the pushlog extension.

Comment 2

[Ed Morley [:emorley]]

Or we could use "startdate" in the fallback case - which supports a datetime:
https://hg.mozilla.org/hgcustom/version-control-tools/file/d8d096f81747/hgext/pushlog-legacy/pushlog-feed.py#l255
https://hg.mozilla.org/hgcustom/version-control-tools/file/d8d096f81747/hgext/pushlog-legacy/pushlog-feed.py#l190

...that way we can just set a startdate of "NOW - 1 HOUR" and so guarantee we're not missing pushes.

The only caveat to this, is that we wouldn't import older pushes for a repo that has only just been added to Treeherder, but perhaps we could handle that case separately too (and use no params, to get the 10 pushes fallback).

Comment 3

[Ed Morley [:emorley]]

Or perhaps:

if cached revision found:
  # As currently
else:
  fetch json-pushes with no params (last 10 pushes) # As currently
  #new:
  if date of oldest push retrieved above is newer than <NOW - 1 HOUR>, then also fetch json-pushes with starttime=<NOW - 1 HOUR> and ingest those too

Comment 4

[Ed Morley [:emorley]]

Or perhaps another approach would be to add the push id to the schema, and if memcached has been cleared, perform a query against the DB to find the highest pushid seen for that repo? This would guarantee zero gaps.

Comment 5

[Ed Morley [:emorley]]

(Everything else is basically a guess; how do we know that infra isn't going to go down for 4 hours etc)

Comment 6

[Cameron Dawson [:camd]]

json-pushes is only a fallback at this point due to Pulse ingestion.  And the cache would ALSO have to be invalid.  So the chances of this hitting us is very rare.