1980086 - glean pending_pings excessive number of files

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect)

Description

[Mark]

Attachment: troubleshooting_information.txt

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0

Steps to reproduce:

I applied Windows update "2025-07 Cumulative Update Preview for Windows 10 Version 22H2 for x64-based Systems (KB5062649)" and the usual Nightly updates to Firefox.

Actual results:

Firefox Nightly is doing an excessive number of disk I/Os and usually running the disk nearly at 100% most of the time and performance stinks. I even created a new profile and copied places.* over to the new profile, reinstalled my extensions (uBlock Origin, To Google Translate, IPvFoo, Hostname In Title) and theme (HORIZONTAL wood by bloochiz12), and I/O is still running near 100% almost all the time Firefox Nightly is up.

In addition to excessive disk I/O, I also discovered that the directory C:\Users\Mark\AppData\Roaming\Mozilla\Firefox\Profiles\qbmmte0u.Nightly1\datareporting\glean\pending_pings has 431,870 files in it for the profile having been created fresh just two days ago.

There have also been Firefox crashes lately, including https://crash-stats.mozilla.org/report/index/bp-6da1ce42-6c6f-4390-a6b8-6a7dd0250730

I have also done a "Quick Scan" and a "Microsoft Defender Offline Scan" and no viruses were reported.

Expected results:

I would expect disk I/O to be reasonably low after Firefox is fully started up and Firefox not crashing.

Comment 1

[Mark]

Performance profile uploaded: https://share.firefox.dev/4fhHHSc

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Toolkit::Telemetry' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Jan-Erik Rediger [:janerik]]

The generation of these files was fixed in bug 1979598, which landed in the nightly you're running.
However it still needs to deal with the files on startups, which causes problems.
The easiest way forward for you is to remove those files from datareporting\glean\pending_pings

I'll double-check that the underlying bug is fixed and I'm working on some mitigation to deal with the many files.

Comment 4

[Mark]

The easiest way forward for you is to remove those files from datareporting\glean\pending_pings

I found out that using Windows File Explorer takes a very long time in this situation. quickest way is to launch cmd.exe, cd to the directory, and then del *.

I also noticed a lot of files (169 for a couple days of using the profile I had created) in C:\Users\Mark\AppData\Roaming\Mozilla\Firefox\Profiles\qbmmte0u.Nightly1\datareporting\archived\2025-07.

After deleting the files in pending_pings, deleting the directory and recreating it (since it had over 400,000 files in it until I got them deleted) and updating Nightly this afternoon, I/O's seem normal.

Comment 5

[Jan-Erik Rediger [:janerik]]

(In reply to Mark from comment #4)

After deleting the files in pending_pings, deleting the directory and recreating it (since it had over 400,000 files in it until I got them deleted) and updating Nightly this afternoon, I/O's seem normal.

Thanks for reporting back. At least the fix preventing the issue then works.

(In reply to Mark from comment #4)

I also noticed a lot of files (169 for a couple days of using the profile I had created) in C:\Users\Mark\AppData\Roaming\Mozilla\Firefox\Profiles\qbmmte0u.Nightly1\datareporting\archived\2025-07.

Those are the archives legacy telemetry pings (the ones you can also view on about:telemetry). Deleting them is equally fine, Firefox also cleans them up on a rolling basis.

Comment 6

[Mark]

Thank you!

One final comment: there was still some lag since the directory of pending_pings was so large (even though it was empty). With Nightly down, I ended up deleting and recreating pending_pings, and that took care of the last bit of performance hit.