PWebrtcGlobal::Reply_GetStats is very visible in hang reports
Categories
(Core :: WebRTC: Audio/Video, defect)
Tracking
()
| Performance Impact | medium |
People
(Reporter: florian, Assigned: pehrsons, NeedInfo)
References
Details
(Keywords: perf:resource-use, perf:responsiveness, Whiteboard: [bhr:PWebrtcGlobal::Reply_GetStats])
I noticed that several stacks in the top hangs reported on BHR contain "Task PWebrtcGlobal::Reply_GetStats". They seem to be mostly free'ing memory (je_free).
| Reporter | ||
Comment 1•3 years ago
|
||
How much we see this hang seems to vary a lot from one day to another.
For the Nightly of 20231101, it's the #2 hang: https://fqueze.github.io/hang-stats/#date=20231101&row=1 (and #1 should be fixed by bug 1862712 that landed on November 3)
For the 20231101 nightly it's #6: https://fqueze.github.io/hang-stats/#date=20231030&row=5
For the 20231028 build it's #24: https://fqueze.github.io/hang-stats/#date=20231028&row=23
Updated•3 years ago
|
Comment 2•3 years ago
|
||
Is there a way to check locally how many background hangs you've had?
Comment 3•3 years ago
|
||
We think this is related to newer features we've added to about:webrtc and debugging in Nightly.
Florian, can you confirm this is getting reported in Nightly?
Updated•3 years ago
|
| Reporter | ||
Comment 4•3 years ago
|
||
(In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
Is there a way to check locally how many background hangs you've had?
I'm not sure. Doug, do you know?
(In reply to Jim Mathies [:jimm] from comment #3)
Florian, can you confirm this is getting reported in Nightly?
Yes, the reports are from Nightly. We only track hangs in Nightly, so I can't say if the same thing also happens on beta or release.
Comment 5•3 years ago
|
||
(In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
Is there a way to check locally how many background hangs you've had?
There's no easy way, no. The data is all visible if you go to about:telemetry, click "current data", select "Archived ping data", and filter to "bhr", but it's not symbolicated and there's no nice UI for exploring it.
Comment 6•3 years ago
|
||
Is there a way to get a sense of the number of users experiencing the hang? i.e. differentiating 14,000 users have the hang once vs. 1 user having it 14,000 times.
| Reporter | ||
Comment 7•3 years ago
|
||
(In reply to Jeff Muizelaar [:jrmuizel] from comment #6)
Is there a way to get a sense of the number of users experiencing the hang? i.e. differentiating 14,000 users have the hang once vs. 1 user having it 14,000 times.
There's no easy way. The raw data came from telemetry, so there must be a way, but I don't know how. Doug can probably give a sense of how much effort would be required. I assume a large part of the challenge will be that the data stored in telemetry is unsymbolicated, but given PWebrtcGlobal::Reply_GetStats is a label frame, maybe that can help.
Updated•2 years ago
|
Updated•2 years ago
|
Updated•2 years ago
|
Comment 8•2 years ago
|
||
@florian it's difficult to assess the performance impact here, are you able to use the impact calculator, or provide your thoughts on the impact?
Updated•2 years ago
|
| Reporter | ||
Comment 9•2 years ago
|
||
The Performance Impact Calculator has determined this bug's performance impact to be medium. If you'd like to request re-triage, you can reset the Performance Impact flag to "?" or needinfo the triage sheriff.
Platforms: [x] Windows [x] macOS [x] Linux
Impact on browser: Causes noticeable jank
Websites affected: Rare
Resource impact: Some
[x] Bug affects multiple sites
Updated•2 years ago
|
Comment 10•2 years ago
|
||
We looked at uses of this and could not see how this would impact everyday performance in the general population. Without more data this is pretty unactionable.
| Reporter | ||
Comment 11•2 years ago
|
||
(In reply to Jim Mathies [:jimm] from comment #10)
Without more data this is pretty unactionable.
Could this be related to bug 1724417?
Comment 12•2 years ago
|
||
I would expect this to show up in profiles during webrtc conferencing if it was a problem. I'll capture a profile today when using Meet and see what we can find.
Comment 13•2 years ago
|
||
Updated•2 years ago
|
| Reporter | ||
Comment 14•2 years ago
|
||
Here's a profile: https://share.firefox.dev/44xtWcV
It doesn't take long enough to be reported as BHR hangs, but "Task PWebrtcGlobal::Reply_GetStats" takes 13ms on the parent process main thread, 4 times per second: https://share.firefox.dev/3JAYhxl
I was initially going to put this profile in bug 1724417, as the content process taking plenty of CPU time is for panopto.com
I have only 2 panopto tabs in that content process:
- https://mozilla.hosted.panopto.com/Panopto/Pages/Home.aspx
- https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=9027bab9-beec-42fe-9d67-b14d010d1d90 (this video is paused in the middle)
Both tabs have been sitting in the background for about 24 hours. The content process priority in my profile is "background", so no video has been playing.
Comment 15•2 years ago
|
||
I'm seeing jank from this too: https://share.firefox.dev/3XIU5nj
Jim, can someone take a closer look?
Comment 16•2 years ago
|
||
Nico, ind taking a look at this profile?
Comment 17•1 year ago
|
||
(In reply to Florian Quèze [:florian] from comment #7)
(In reply to Jeff Muizelaar [:jrmuizel] from comment #6)
Is there a way to get a sense of the number of users experiencing the hang? i.e. differentiating 14,000 users have the hang once vs. 1 user having it 14,000 times.
There's no easy way. The raw data came from telemetry, so there must be a way, but I don't know how. Doug can probably give a sense of how much effort would be required. I assume a large part of the challenge will be that the data stored in telemetry is unsymbolicated, but given PWebrtcGlobal::Reply_GetStats is a label frame, maybe that can help.
This is a bit of a hack, but I think if you go to about:telemetry, select archived ping data, change the ping type to bhr, and click raw payload, you can run the following script in the console and it'll collect all of the matching hangs from all of your bhr pings.
async function iteratePings() {
const PSEUDOFRAME_TO_FIND = "PWebrtcGlobal::Reply_GetStats";
let allMatchingHangs = [];
let lastJSON = "";
do {
console.log(`Processing ${$("#ping-date").textContent}`);
let newJSON = $("#raw-payload-data").textContent;
if (newJSON == lastJSON) {
break;
}
lastJSON = newJSON;
let matching = JSON.parse(newJSON).hangs.filter(h => h.stack.find(f => f == PSEUDOFRAME_TO_FIND));
if (matching.length) {
allMatchingHangs.push(...matching);
console.log("Found one! (total: " + allMatchingHangs.length + ")");
}
$("#older-ping").click();
await new Promise(r => setTimeout(r, 100));
} while (true);
console.log("Done");
return allMatchingHangs;
}
iteratePings()
There's a good chance you have to tweak the script for it to actually work - I don't think I have any of these hangs locally so I couldn't verify.
Updated•1 year ago
|
| Assignee | ||
Comment 18•4 days ago
|
||
about:webrtc queries every peer connection in every content process for stats every 250ms. Even closed ones. The most expensive stats are string stats because they require allocations and buffer copies for IPC serialization/deserialization.
See this profile on bug 2042789.
Description
•