Closed
Bug 1071577
Opened 10 years ago
Closed 10 years ago
Make pushlog ingestion more robust - round 1
Categories
(Tree Management :: Treeherder, defect, P1)
Tree Management
Treeherder
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: RyanVM, Assigned: jeads)
References
Details
Attachments
(1 file)
The below push shows on TBPL (with running jobs). Treeherder never picked it up.
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=f41dc09ee34b
The push after (https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=a70c83745785) appears on both as normal.
Comment 1•10 years ago
|
||
I noticed it shows on treeherder-dev, so there is nothing wrong with that push specifically. I guess for some reason the pushlog cache is striking again. I'm proposing to stop caching the last pushId fetched from each repo, since it's probably giving us more pain than benefits. :jeads :camd do you have an opinion on that?
Flags: needinfo?(jeads)
Updated•10 years ago
|
Flags: needinfo?(cdawson)
Comment 2•10 years ago
|
||
What makes you think it's the pushlog ID cache? I've just cleared the cache and nothing appeared at:
https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=f41dc09ee34b
Surely the issue is more "ingestion job fails/is raced -> we then never attempt to ingest that push ID again, even if we're using the full 24 hours of pushlog".
Updated•10 years ago
|
Assignee: nobody → mdoglio
Updated•10 years ago
|
Status: NEW → ASSIGNED
Comment 3•10 years ago
|
||
the reason why it's not ingesting that push even after the cache was cleared is that the maximum number of results returned by the pushlog service is 10 (I just realized it) and that push is at the moment 11th. I think the best solution is to remove the pushlog id cache and increase the number of elements fetched.
Updated•10 years ago
|
Flags: needinfo?(jeads)
Flags: needinfo?(cdawson)
Comment 4•10 years ago
|
||
Attachment #8493862 -
Flags: review?(jeads)
Comment 5•10 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder-service
https://github.com/mozilla/treeherder-service/commit/ba881e203092d6f016725ddf92eefb810db577cd
(Bug 1071577) - remove pushlog last fetched cache; also increase the pushlog time range to the last 24hr
https://github.com/mozilla/treeherder-service/commit/24c437d0d6d5950386489dd1852c1585f83adc93
Merge pull request #227 from mozilla/bug-1071577-remove-pushlog-cache
(Bug 1071577) - remove pushlog last fetched cache; also increase the pus...
Assignee | ||
Updated•10 years ago
|
Attachment #8493862 -
Flags: review?(jeads) → review+
Updated•10 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Comment 6•10 years ago
|
||
(In reply to Mauro Doglio [:mdoglio] from comment #3)
> the reason why it's not ingesting that push even after the cache was
> cleared is that the maximum number of results returned by the pushlog
> service is 10 (I just realized it) and that push is at the moment 11th. I
> think the best solution is to remove the pushlog id cache and increase the
> number of elements fetched.
The commit that landed here added &maxhours=24 to the json-pushes request URL, which is what I presume you meant by "increase the number of elements fetched"? If so, I don't think that will help, since "maxhours" isn't a valid json-pushes param - I know TBPL passes it, but there's no reference to it in the pushlog repo:
https://hg.mozilla.org/hgcustom/pushlog/file/86e6470a7c55/pushlog-feed.py#l255
As such, I believe the landing here actually makes things worse, since if there were > 10 pushes since the last request (I guess unlikely, but still), then we'll actually lose them forever, which we wouldn't have done when requesting by cached push ID directly.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 7•10 years ago
|
||
Commits pushed to master at https://github.com/mozilla/treeherder-service
https://github.com/mozilla/treeherder-service/commit/d7908793a59c410b0418c28cce712376af4e11d8
Bug 1071577 - Cache revision hashes to reduce pushlog load
https://github.com/mozilla/treeherder-service/commit/2c258f443d8c2f65838886614d5f24d5d78a11f9
Bug 1071577 - Moved result_set revision hash cache step to after associated revision storage
https://github.com/mozilla/treeherder-service/commit/4f7b069a34d7ef94bc7ae588a725846e8c2083db
Merge pull request #228 from mozilla/bug-1071577-revision-hash-cache
Bug 1071577 revision hash cache
Updated•10 years ago
|
Assignee: mdoglio → jeads
Comment 8•10 years ago
|
||
Changing summary to reflect what work was carried out; part 2 is occurring in bug 1072291.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Summary: Treeherder missing inbound push f41dc09ee34b → Make pushlog ingestion more robust - round 1
You need to log in
before you can comment on or make changes to this bug.
Description
•