Closed
Bug 1361872
Opened 7 years ago
Closed 5 years ago
Pushlog ingestion: "Unable to get last push from cache for 'cedar', getting all pushes"
Categories
(Tree Management :: Treeherder: Infrastructure, enhancement, P3)
Tree Management
Treeherder: Infrastructure
Tracking
(Not tracked)
RESOLVED
INVALID
People
(Reporter: emorley, Unassigned)
Details
Appears a lot in the logs: https://papertrailapp.com/systems/treeherder-prod/events?q=worker_pushlog&focus=796534243898593287 I'm not sure why the last push isn't being cached? No exceptions in New Relic or eg dyno memory errors in papertrail. Cameron, do you have any ideas?
Flags: needinfo?(cdawson)
Comment 1•7 years ago
|
||
There hasn't been any new pushes to cedar since Dec 13th. So a call to https://hg.mozilla.org/projects/cedar/json-pushes/?version=2 returns nothing. The logic in that function just returns without updating the cache on line 105 if no pushes are returned. It can't update the cache since it doesn't have a pushID to put in there. So somewhere between last December and now, our cache got cleared (not surprising) and we've never had a new push to get a new pushID. I think we're stuck that that log going off. We can at least reduce the frequency now, though.
Flags: needinfo?(cdawson)
Would marking cedar as onhold be sufficient? https://github.com/mozilla/treeherder/blob/8061296172c80f3e4a186f3d959027f5406bbf4c/treeherder/model/fixtures/repository.json#L281
Comment 3•7 years ago
|
||
IT would seem to make more sense to figure out a way to get the last pushID that does not depend on there being recent pushes. This just seems to be a broken design.
Reporter | ||
Comment 4•7 years ago
|
||
(In reply to Cameron Dawson [:camd] from comment #1) > There hasn't been any new pushes to cedar since Dec 13th. Ah so I'd checked https://hg.mozilla.org/projects/cedar/ and seen there were recent commits, which is why I didn't think it could be that, but you are right there are no pushes recorded at https://hg.mozilla.org/projects/cedar/pushloghtml (or the json equivalent): { "lastpushid": "", "pushes": {} } So it seems like cedar has been reset back to m-c, but that the script that does so leaves the pushlog empty. Perhaps a dummy push zero should be added as part of this script? Gregory, any thoughts? :-) Alternatively if the repo is not used, we should mark it as onhold as Wes says. Note that this isn't causing problems per se, just a lot of log noise (since the "cache last push" behaviour not working can actually be a sign of a problem, so it's correct to make noise).
Flags: needinfo?(gps)
Comment hidden (off-topic) |
Comment hidden (off-topic) |
Comment hidden (off-topic) |
Comment hidden (off-topic) |
Comment hidden (off-topic) |
Comment hidden (off-topic) |
Comment hidden (off-topic) |
Comment 12•7 years ago
|
||
The whole concept of repo resets is bonkers for multiple reasons, including any pushlog consumer getting of whack. I wish we didn't use project repos at all and relied on user repos for coordinating development and try for triggering automation outside of official landings. But people like their special snowflake sandboxes (and a unified treeherder view is nice), so we have project repos. It is certainly possible to do anything during resets. Not having a pushlog is likely the result of an arbitrary decision made several years ago. File a hg.mozilla.org for what you want the pushlog to be like after a reset and it can be done. Either "empty pushlog" or "copy pushlog" are the easiest to implement. But inserting a dummy pushlog entry for the tip changeset isn't too much work.
Flags: needinfo?(gps)
Reporter | ||
Comment 13•7 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #12) > File a hg.mozilla.org for what you want the pushlog to be like after a reset Bug 1384329 :-)
Comment 14•5 years ago
|
||
Moving this to P3; no immediate plans to change it. If someone wants to pick it up, please feel free to assign yourself.
Priority: P2 → P3
Comment 15•5 years ago
|
||
We don't support twig repos anymore.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID
You need to log in
before you can comment on or make changes to this bug.
Description
•