Open Bug 1940016 Opened 1 month ago Updated 1 month ago

Accessing a treeherder link waits 10s

Categories

(Tree Management :: Treeherder: API, defect)

defect

Tracking

(Not tracked)

People

(Reporter: mayankleoboy1, Unassigned)

References

Details

I am trying to open this treeherde//perfcomparer link : https://treeherder.mozilla.org/api/perfcompare/results/?base_repository=try&base_revision=15193025d193d2b9d2f75173b35c8dd0a6ad1b8a&new_repository=try&new_revision=e9c6068c0cbfa6676d98e72bb4d0b5b78c51ec2c&framework=13&base_parent_signature=223447&new_parent_signature=223447
(Which is part of this perfcompare link)
It takes 10 seconds for the treeherder server to respond. The slow treeherder response has been going on for me since 1Nov2024.

See Also: → 1939786

HEre is a profile with networking preset logging: https://share.firefox.dev/4a8Vbxe

Alice, do you also see a delay if you open the two links on comment 0?

This started around 1Nov2024. Note my comment in https://chat.mozilla.org/#/room/#treeherder:mozilla.org on 1Nov.

The url from comment #0 loads here in 17s, both with production, and with Treeherder running locallly and the code rolled back to the end of September 2024 and the production database queried (there have been database changes, they might have been related to this).

Greg,
a) can you reproduce a duration difference for this request between recent and old code?
b) have a hunch what might be the issue?

Flags: needinfo?(gmierz2)

I wonder what the range of commits might have been that got pushed to prod around that time. I took a look at all the commits between oct 1st, and nov 1st, and I found this one for the perfcompare API (cc'ed beatrice about it) - it doesn't add much though: https://github.com/mozilla/treeherder/commit/34fb5344e4e4e09603092233a728e22bda1b4303

Regarding DB changes, are you referring to the fixes for the id sequence? I would be very surprised if that caused this, but it may be possible.

I'll check locally with the same data and see if I can do a bisection.

I'm unable to bisect it locally, Oct 1st has a similar response time to a commit on Dec 11th. I also had a local DB with the try runs in the given link ingested.

Maybe :wezhou has an idea of what could have changed? Something I wonder is if there is more data being queried now since PerfCompare always retrieves replicate data when it's available. PerfCompare has been in use much more in last few months so it's possible we've only started seeing the effects of large-scale usage. Could be useful to check what the resource usage is like.

Flags: needinfo?(gmierz2) → needinfo?(aryx.bugmail)

https://treeherder.mozilla.org/perfherder/alerts?id=2626
gets 616 kB of data from the /alertsummary/ endpoint in 7.4s

https://treeherder.mozilla.org/perfherder/alerts?id=2576
has also many alerts and gets a 52 kB response in 0.9s

Database performance data and HTTP logs have expired for Nov 1.

Do you get/generate more alerts now? The big request has 573 alerts.

Flags: needinfo?(aryx.bugmail)

We don't. Nothing big has changed with alert generation in a while now afaik but I'll poke around to see. I think the last big change could have come from the pdfpaint test. It added many additional subtests to alert on, but that was early last year.

You need to log in before you can comment on or make changes to this bug.