We have a small rate crash in mozilla::net::CacheIOThread::LoopOneLevel: https://crash-stats.mozilla.com/search/?signature=~mozilla%3A%3Anet%3A%3ACacheIOThread%3A%3ALoopOneLevel&date=%3C%3D2016-06-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature Which sometimes crashes on OOM. Might be worth checking how long the queue can get. I think a DIAGNOSTIC_ASSERT for 1000 operations could do as a start. Or should it be an unpliftable telemetry?
Summary: Diagnose how long cache IO thread queues can be long → Diagnose how long cache IO thread queues can be
Created attachment 8761654 [details] [diff] [review] v1 - we accumulate at the moment we dequeue a runnable on the IO thread, we do so separately for each level - there are 10 buckets for each IO thread level with 30 as a granularity - when there are e.g. 65 pending runnables at the moment we dequeue, we accumulate bucket #2 (60) and remember to only report (accumulate) when the queue length is more then 90 (60 + 30) next time - everything over 300 is reported in the last bucket - some types clean up
Attachment #8761654 - Flags: review?(michal.novotny)
Attachment #8761654 - Flags: review?(michal.novotny) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5038112b5f78 Cache I/O queue length telemetry, r=michal
sorry had to back this out for bustage like https://treeherder.mozilla.org/logviewer.html#?job_id=30066432&repo=mozilla-inbound
Backout by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/977d6bed6e0a Backed out changeset 5038112b5f78 for bustage
Created attachment 8765441 [details] [diff] [review] v1.1 https://treeherder.mozilla.org/#/jobs?repo=try&revision=cee75e4f673b14f8a714dfa368e8a71a795bbefb The two errors fixed, let's see.
Green, let's do it!
Pushed by email@example.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/4070b1ce5332 Cache I/O queue length telemetry, r=michal
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox50: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla50
You need to log in before you can comment on or make changes to this bug.