Closed Bug 1361872 Opened 7 years ago Closed 5 years ago

Pushlog ingestion: "Unable to get last push from cache for 'cedar', getting all pushes"

Categories

(Tree Management :: Treeherder: Infrastructure, enhancement, P3)

enhancement

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: emorley, Unassigned)

Details

Appears a lot in the logs:
https://papertrailapp.com/systems/treeherder-prod/events?q=worker_pushlog&focus=796534243898593287

I'm not sure why the last push isn't being cached?
No exceptions in New Relic or eg dyno memory errors in papertrail.

Cameron, do you have any ideas?
Flags: needinfo?(cdawson)
There hasn't been any new pushes to cedar since Dec 13th.  So a call to https://hg.mozilla.org/projects/cedar/json-pushes/?version=2 returns nothing.  The logic in that function just returns without updating the cache on line 105 if no pushes are returned.  It can't update the cache since it doesn't have a pushID to put in there.

So somewhere between last December and now, our cache got cleared (not surprising) and we've never had a new push to get a new pushID.  I think we're stuck that that log going off.  We can at least reduce the frequency now, though.
Flags: needinfo?(cdawson)
IT would seem to make more sense to figure out a way to get the last pushID that does not depend on there being recent pushes.

This just seems to be a broken design.
(In reply to Cameron Dawson [:camd] from comment #1)
> There hasn't been any new pushes to cedar since Dec 13th. 

Ah so I'd checked https://hg.mozilla.org/projects/cedar/ and seen there were recent commits, which is why I didn't think it could be that, but you are right there are no pushes recorded at https://hg.mozilla.org/projects/cedar/pushloghtml (or the json equivalent):

{
"lastpushid": "",
"pushes": {}
}

So it seems like cedar has been reset back to m-c, but that the script that does so leaves the pushlog empty. Perhaps a dummy push zero should be added as part of this script? Gregory, any thoughts? :-)

Alternatively if the repo is not used, we should mark it as onhold as Wes says.

Note that this isn't causing problems per se, just a lot of log noise (since the "cache last push" behaviour not working can actually be a sign of a problem, so it's correct to make noise).
Flags: needinfo?(gps)
The whole concept of repo resets is bonkers for multiple reasons, including any pushlog consumer getting of whack. I wish we didn't use project repos at all and relied on user repos for coordinating development and try for triggering automation outside of official landings. But people like their special snowflake sandboxes (and a unified treeherder view is nice), so we have project repos.

It is certainly possible to do anything during resets. Not having a pushlog is likely the result of an arbitrary decision made several years ago. File a hg.mozilla.org for what you want the pushlog to be like after a reset and it can be done. Either "empty pushlog" or "copy pushlog" are the easiest to implement. But inserting a dummy pushlog entry for the tip changeset isn't too much work.
Flags: needinfo?(gps)
(In reply to Gregory Szorc [:gps] from comment #12)
> File a hg.mozilla.org for what you want the pushlog to be like after a reset

Bug 1384329 :-)

Moving this to P3; no immediate plans to change it. If someone wants to pick it up, please feel free to assign yourself.

Priority: P2 → P3

We don't support twig repos anymore.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.