Make pushlog ingestion more robust - round 1

RESOLVED FIXED

Status

Tree Management
Treeherder
P1
blocker
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: RyanVM, Assigned: jeads)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
The below push shows on TBPL (with running jobs). Treeherder never picked it up.
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=f41dc09ee34b

The push after (https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=a70c83745785) appears on both as normal.
I noticed it shows on treeherder-dev, so there is nothing wrong with that push specifically. I guess for some reason the pushlog cache is striking again. I'm proposing to stop caching the last pushId fetched from each repo, since it's probably giving us more pain than benefits. :jeads :camd do you have an opinion on that?
Flags: needinfo?(jeads)
Flags: needinfo?(cdawson)

Comment 2

4 years ago
What makes you think it's the pushlog ID cache? I've just cleared the cache and nothing appeared at:
https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=f41dc09ee34b

Surely the issue is more "ingestion job fails/is raced -> we then never attempt to ingest that push ID again, even if we're using the full 24 hours of pushlog".
Assignee: nobody → mdoglio
Status: NEW → ASSIGNED
the reason why it's not ingesting that push even after the cache was  cleared is that the maximum number of results returned by the pushlog service is 10 (I just realized it) and that push is at the moment 11th. I think the best solution is to remove the pushlog id cache and increase the number of elements fetched.
Flags: needinfo?(jeads)
Flags: needinfo?(cdawson)
Created attachment 8493862 [details] [review]
Github PR #227 on treeherder-service
Attachment #8493862 - Flags: review?(jeads)

Comment 5

4 years ago
Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/ba881e203092d6f016725ddf92eefb810db577cd
(Bug 1071577) - remove pushlog last fetched cache; also increase the pushlog time range to the last 24hr

https://github.com/mozilla/treeherder-service/commit/24c437d0d6d5950386489dd1852c1585f83adc93
Merge pull request #227 from mozilla/bug-1071577-remove-pushlog-cache

(Bug 1071577) - remove pushlog last fetched cache; also increase the pus...
(Assignee)

Updated

4 years ago
Attachment #8493862 - Flags: review?(jeads) → review+
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED

Updated

4 years ago
Depends on: 1072291

Comment 6

4 years ago
(In reply to Mauro Doglio [:mdoglio] from comment #3)
> the reason why it's not ingesting that push even after the cache was 
> cleared is that the maximum number of results returned by the pushlog
> service is 10 (I just realized it) and that push is at the moment 11th. I
> think the best solution is to remove the pushlog id cache and increase the
> number of elements fetched.

The commit that landed here added &maxhours=24 to the json-pushes request URL, which is what I presume you meant by "increase the number of elements fetched"? If so, I don't think that will help, since "maxhours" isn't a valid json-pushes param - I know TBPL passes it, but there's no reference to it in the pushlog repo:
https://hg.mozilla.org/hgcustom/pushlog/file/86e6470a7c55/pushlog-feed.py#l255

As such, I believe the landing here actually makes things worse, since if there were > 10 pushes since the last request (I guess unlikely, but still), then we'll actually lose them forever, which we wouldn't have done when requesting by cached push ID directly.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Updated

4 years ago
Depends on: 1072422

Comment 7

4 years ago
Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/d7908793a59c410b0418c28cce712376af4e11d8
Bug 1071577 - Cache revision hashes to reduce pushlog load

https://github.com/mozilla/treeherder-service/commit/2c258f443d8c2f65838886614d5f24d5d78a11f9
Bug 1071577 - Moved result_set revision hash cache step to after associated revision storage

https://github.com/mozilla/treeherder-service/commit/4f7b069a34d7ef94bc7ae588a725846e8c2083db
Merge pull request #228 from mozilla/bug-1071577-revision-hash-cache

Bug 1071577 revision hash cache

Updated

4 years ago
Assignee: mdoglio → jeads

Updated

4 years ago
Blocks: 1066402

Updated

4 years ago
Blocks: 1065775

Updated

4 years ago
Blocks: 1069267

Comment 8

4 years ago
Changing summary to reflect what work was carried out; part 2 is occurring in bug 1072291.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
Summary: Treeherder missing inbound push f41dc09ee34b → Make pushlog ingestion more robust - round 1

Updated

4 years ago
No longer blocks: 1065775

Updated

4 years ago
No longer depends on: 1072422

Updated

4 years ago
Blocks: 1076750
You need to log in before you can comment on or make changes to this bug.