Hello Kim, Thanks for weighing in and pointing out the issues in 79. It helps to identify some regression. I take a look for the first 50 crashes at the stack of [@shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T19%3A49%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) in beta. I believe HTTP3 and nss are new crashes which is not covered in bug 1158189. This crashes could be eased to <20% once we have those fixed. Another 10% of crashes are from haning in poll. bug 1435962 comment 3 showes it could be a network driver bug. We could address the last mile once later. However, we don't have much things to do with haning Poll. I don't have strong feeling that it would be solved by not spinning main thread. HTTP3 are mostly owing to slow string operation and nss lives in non-main thread. If we don't spin the main thread loop, it still crashes or hangs elsewhere. I'll ni? dragana, who chases this crashes for a long time, in the next comment for more input. And file the responding bug to each components. I list the crashes reported which I triaged at the bottom of this comment. And I take another look on [the crashes in nightly](https://crash-stats.mozilla.org/signature/?product=Firefox&version=80.0a1&signature=shutdownhang%20%7C%20__pthread_cond_wait%20%7C%20%3Cname%20omitted%3E%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&date=%3E%3D2020-07-01T20%3A25%3A00.000Z&date=%3C2020-07-08T20%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports). All 7 crashes are H3 with different symbol. I also take a look at [the release crashes]() Around * fflush hangs, much more than beta https://crash-stats.mozilla.org/report/index/eab6ec5b-76c9-487f-b3ba-43dfa0200708#allthreads * deadlock ` nsNSSSocketInfo::CloseSocketAndDestroy()`, which disappears or is turned to other symbol in beta https://crash-stats.mozilla.org/report/index/a2ec85a3-383a-4925-993f-5f6910200708#allthreads * PR_POLL * other nss symboal * random To me, focusing on beta could be a good idea at this stage. As for other crash signature, [@ shutdownhang | __psynch_cvwait | <name omitted> | mozilla::TaskController::GetRunnableForMTTask ](https://crash-stats.mozilla.org/search/?release_channel=nightly&signature=%3Dshutdownhang%20%7C%20__psynch_cvwait%20%7C%20%3Cname%20omitted%3E%20%7C%20mozilla%3A%3ATaskController%3A%3AGetRunnableForMTTask&product=Firefox&version=80.0a1&date=%3E%3D2020-07-01T00%3A00%3A00.000Z&date=%3C2020-07-08T20%3A00%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) and other mozilla::TaskController::GetRunnableForMTTask are the mostly crashes in QuotaManager (some of them are even without socket thread). Only few of them are about H3 and nss, which are the known cases. [@ shutdownhang | mozilla::net::ShutdownEvent::PostAndWait ](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AShutdownEvent%3A%3APostAndWait&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T20%3A07%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) are about the `CacheFileIOManager::Shutdown`, which is a different issue. [shutdownhang | nsThread::Shutdown | mozilla::net::nsSocketTransportService::ShutdownThread ](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20mozilla%3A%3Anet%3A%3AnsSocketTransportService%3A%3AShutdownThread&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T20%3A08%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) is interesting. Half of them are known nss issues. Another half happens for deadlock in shutdown thread for ssl [here](https://hg.mozilla.org/releases/mozilla-beta/file/6e11fb81cbfb151d2c92ea050deca01d6c53babc/security/manager/ssl/SSLServerCertVerification.cpp#l187) Here's some reports: https://crash-stats.mozilla.org/report/index/a071e81d-0955-497f-bc3d-036210200708#allthreads https://crash-stats.mozilla.org/report/index/011da31f-1aeb-46cd-a81f-eac870200708#allthreads https://crash-stats.mozilla.org/report/index/3d063176-b728-437b-a4c9-c00410200708#allthreads This is the analysis for [@shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T19%3A49%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) in beta. * HTTP3 (23) * [neqo_glue::neqo_http3conn_fetch](https://searchfox.org/mozilla-central/rev/85ae3b911d5fcabd38ef315725df32e25edef83b/netwerk/socket/neqo_glue/src/lib.rs#267) https://crash-stats.mozilla.org/report/index/232d2d2e-e059-416c-84fc-4896e0200708#allthreads https://crash-stats.mozilla.org/report/index/27c83b2b-1977-45ca-8947-203520200708#allthreads https://crash-stats.mozilla.org/report/index/ce130f9f-2137-4d7f-b791-7b51d0200708#allthreads https://crash-stats.mozilla.org/report/index/4617e7fb-b701-40b0-871f-143f70200708#allthreads https://crash-stats.mozilla.org/report/index/ee901bb2-abc4-489e-9f7d-79f3f0200708#allthreads https://crash-stats.mozilla.org/report/index/76dc9ec3-abac-4490-8bea-5e55c0200708#allthreads https://crash-stats.mozilla.org/report/index/e369ceab-448b-4d14-8aba-899980200708#allthreads https://crash-stats.mozilla.org/report/index/cd15b360-f85c-44f4-a272-01a6d0200708#allthreads https://crash-stats.mozilla.org/report/index/a63fdd66-6a6d-4f9b-a658-6745b0200708#allthreads https://crash-stats.mozilla.org/report/index/69fdc1e0-912b-46ec-b278-7af360200708#allthreads https://crash-stats.mozilla.org/report/index/273cc629-48db-4f7f-b817-303970200708#allthreads https://crash-stats.mozilla.org/report/index/2be978fa-1e54-442d-ba98-4051f0200708#allthreads https://crash-stats.mozilla.org/report/index/eee87582-2bdb-4398-bcce-043c80200708#allthreads * slow nsTString<char>::Find in mozilla::net::Http3Stream::FindRequestContentLength() https://crash-stats.mozilla.org/report/index/e74302ff-a5bc-456b-9f62-7456b0200708#allthreads https://crash-stats.mozilla.org/report/index/f4d7beae-1758-4f7b-8ddf-cff990200708#allthreads https://crash-stats.mozilla.org/report/index/94ad5e0f-cdaa-4039-ae04-c680b0200708#allthreads https://crash-stats.mozilla.org/report/index/1546f888-d3c2-48b5-b1c5-c08b30200708#allthreads https://crash-stats.mozilla.org/report/index/5a584924-3e59-45f2-a637-0c6800200708#allthreads https://crash-stats.mozilla.org/report/index/b7159d2d-1570-4ffc-87f5-2b2a60200708#allthreads https://crash-stats.mozilla.org/report/index/05024233-a580-4315-af02-8580d0200708#allthreads * slow nsTString<char>::Find in mozilla::net::Http3Stream::GetHeadersString https://crash-stats.mozilla.org/report/index/2595aabb-e2d5-44c0-9479-bce420200708#allthreads https://crash-stats.mozilla.org/report/index/50b9aaf2-419d-4832-9dfd-368850200708#allthreads https://crash-stats.mozilla.org/report/index/f447bc10-2cf9-4ad2-9ff9-9fe760200708#allthreads * Deadlock with nss (17): Note to myself: search `RtlpWaitOnCriticalSection` for deadlock https://crash-stats.mozilla.org/report/index/0c967088-a97e-4dde-8bbe-e23570200708#allthreads https://crash-stats.mozilla.org/report/index/7bb239d5-c863-4ca8-9ad3-128ac0200708#allthreads https://crash-stats.mozilla.org/report/index/053f0a6b-b7d1-4ce4-bbc6-beb4c0200708#allthreads https://crash-stats.mozilla.org/report/index/763ae655-0a70-4422-97f7-0b86b0200708#allthreads https://crash-stats.mozilla.org/report/index/749aea7f-cfdc-4b49-8373-ed8db0200708#allthreads https://crash-stats.mozilla.org/report/index/6cb693ad-b88c-42e8-afef-5eceb0200708#allthreads https://crash-stats.mozilla.org/report/index/915564b0-e7b7-4ea1-bc55-1d7a20200708#allthreads https://crash-stats.mozilla.org/report/index/f2523d26-c4d4-4922-8231-5c7510200708#allthreads https://crash-stats.mozilla.org/report/index/88299646-ecf5-47e3-adb5-39b6b0200708#allthreads https://crash-stats.mozilla.org/report/index/b1286c47-7a56-4d38-ab7f-4c3530200708#allthreads https://crash-stats.mozilla.org/report/index/ec485865-dedb-4d43-b906-ac6320200708#allthreads https://crash-stats.mozilla.org/report/index/ba379840-e967-4e7c-bb37-962e90200708#allthreads https://crash-stats.mozilla.org/report/index/a1bf9f5c-ebe3-41e7-809b-a38ff0200708#allthreads https://crash-stats.mozilla.org/report/index/c6a3fff4-264e-4ffb-9b7a-2f55c0200708#allthreads https://crash-stats.mozilla.org/report/index/15cc0996-9cdd-47f3-a8cf-5c8500200708#allthreads https://crash-stats.mozilla.org/report/index/208140d4-60a6-403c-ba59-334a30200708#allthreads https://crash-stats.mozilla.org/report/index/62db5f81-1e2a-4eb9-bc8a-2b0780200708#allthreads * fflush hang in ssl https://crash-stats.mozilla.org/report/index/a524b967-5c38-46f9-885b-80b0b0200708#allthreads * Good old poll https://crash-stats.mozilla.org/report/index/dd4b29f7-99eb-4775-ad10-770fc0200708#allthreads https://crash-stats.mozilla.org/report/index/d9b215a1-bdab-4ed4-87df-04ceb0200708#allthreads https://crash-stats.mozilla.org/report/index/bbe33082-cc99-494c-b38c-930b40200708#allthreads https://crash-stats.mozilla.org/report/index/9e64769b-e785-4681-a319-e13320200708#allthreads https://crash-stats.mozilla.org/report/index/0c869f8c-f2e7-4754-8997-b6b9f0200708#allthreads https://crash-stats.mozilla.org/report/index/9198e493-305a-4eeb-86a0-c35020200708#allthreads * Random * Hang in PR_Write https://crash-stats.mozilla.org/report/index/ea70ecc5-30e1-442b-8743-deea60200708#allthreads * doing it's shutdown job. not hanging: https://crash-stats.mozilla.org/report/index/c8f012d0-5121-488a-a6f9-7d5070200708#allthreads * je_realloc https://crash-stats.mozilla.org/report/index/7e0cf42f-852c-4150-a2e0-0ec2d0200708#allthreads
Bug 1633342 Comment 16 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Hello Kim, Thanks for weighing in and pointing out the issues in 79. It helps to identify some regression. I take a look for the first 50 crashes at the stack of [@shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T19%3A49%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) in beta. I believe HTTP3 and nss are new crashes which is not covered in bug 1158189. This crashes could be eased to <20% once we have those fixed. Another 10% of crashes are from haning in poll. bug 1435962 comment 3 showes it could be a network driver bug. We could address the last mile once later. However, we don't have much things to do with haning Poll. I don't have strong feeling that it would be solved by not spinning main thread. HTTP3 are mostly owing to slow string operation and nss lives in non-main thread. If we don't spin the main thread loop, it still crashes or hangs elsewhere. I'll ni? dragana, who chases this crashes for a long time, in the next comment for more input. And file the responding bug to each components. I list the crashes reported which I triaged at the bottom of this comment. And I take another look on [the crashes in nightly](https://crash-stats.mozilla.org/signature/?product=Firefox&version=80.0a1&signature=shutdownhang%20%7C%20__pthread_cond_wait%20%7C%20%3Cname%20omitted%3E%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&date=%3E%3D2020-07-01T20%3A25%3A00.000Z&date=%3C2020-07-08T20%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports). All 7 crashes are H3 with different symbol. I also take a look at [the release crashes](https://crash-stats.mozilla.org/signature/?product=Firefox&version=78.0.1&signature=shutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&date=%3E%3D2020-07-01T20%3A18%3A00.000Z&date=%3C2020-07-08T20%3A18%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1) * fflush hangs, much more than beta https://crash-stats.mozilla.org/report/index/eab6ec5b-76c9-487f-b3ba-43dfa0200708#allthreads * deadlock ` nsNSSSocketInfo::CloseSocketAndDestroy()`, which disappears or is turned to other symbol in beta https://crash-stats.mozilla.org/report/index/a2ec85a3-383a-4925-993f-5f6910200708#allthreads * PR_POLL * other nss symboal * random To me, focusing on beta could be a good idea at this stage. As for other crash signature, [@ shutdownhang | __psynch_cvwait | <name omitted> | mozilla::TaskController::GetRunnableForMTTask ](https://crash-stats.mozilla.org/search/?release_channel=nightly&signature=%3Dshutdownhang%20%7C%20__psynch_cvwait%20%7C%20%3Cname%20omitted%3E%20%7C%20mozilla%3A%3ATaskController%3A%3AGetRunnableForMTTask&product=Firefox&version=80.0a1&date=%3E%3D2020-07-01T00%3A00%3A00.000Z&date=%3C2020-07-08T20%3A00%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) and other mozilla::TaskController::GetRunnableForMTTask are the mostly crashes in QuotaManager (some of them are even without socket thread). Only few of them are about H3 and nss, which are the known cases. [@ shutdownhang | mozilla::net::ShutdownEvent::PostAndWait ](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AShutdownEvent%3A%3APostAndWait&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T20%3A07%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) are about the `CacheFileIOManager::Shutdown`, which is a different issue. [shutdownhang | nsThread::Shutdown | mozilla::net::nsSocketTransportService::ShutdownThread ](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20nsThread%3A%3AShutdown%20%7C%20mozilla%3A%3Anet%3A%3AnsSocketTransportService%3A%3AShutdownThread&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T20%3A08%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) is interesting. Half of them are known nss issues. Another half happens for deadlock in shutdown thread for ssl [here](https://hg.mozilla.org/releases/mozilla-beta/file/6e11fb81cbfb151d2c92ea050deca01d6c53babc/security/manager/ssl/SSLServerCertVerification.cpp#l187) Here's some reports: https://crash-stats.mozilla.org/report/index/a071e81d-0955-497f-bc3d-036210200708#allthreads https://crash-stats.mozilla.org/report/index/011da31f-1aeb-46cd-a81f-eac870200708#allthreads https://crash-stats.mozilla.org/report/index/3d063176-b728-437b-a4c9-c00410200708#allthreads This is the analysis for [@shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown](https://crash-stats.mozilla.org/search/?release_channel=beta&release_channel=aurora&signature=%3Dshutdownhang%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnectionMgr%3A%3AShutdown&product=Firefox&version=78.0&version=78.0b7&version=78.0b8&version=78.0b9&version=79.0b1&version=79.0b2&version=79.0b3&version=79.0b4&version=79.0b5&date=%3E%3D2020-06-12T00%3A00%3A00.000Z&date=%3C2020-07-08T19%3A49%3A00.000Z&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports) in beta. * HTTP3 (23) * [neqo_glue::neqo_http3conn_fetch](https://searchfox.org/mozilla-central/rev/85ae3b911d5fcabd38ef315725df32e25edef83b/netwerk/socket/neqo_glue/src/lib.rs#267) https://crash-stats.mozilla.org/report/index/232d2d2e-e059-416c-84fc-4896e0200708#allthreads https://crash-stats.mozilla.org/report/index/27c83b2b-1977-45ca-8947-203520200708#allthreads https://crash-stats.mozilla.org/report/index/ce130f9f-2137-4d7f-b791-7b51d0200708#allthreads https://crash-stats.mozilla.org/report/index/4617e7fb-b701-40b0-871f-143f70200708#allthreads https://crash-stats.mozilla.org/report/index/ee901bb2-abc4-489e-9f7d-79f3f0200708#allthreads https://crash-stats.mozilla.org/report/index/76dc9ec3-abac-4490-8bea-5e55c0200708#allthreads https://crash-stats.mozilla.org/report/index/e369ceab-448b-4d14-8aba-899980200708#allthreads https://crash-stats.mozilla.org/report/index/cd15b360-f85c-44f4-a272-01a6d0200708#allthreads https://crash-stats.mozilla.org/report/index/a63fdd66-6a6d-4f9b-a658-6745b0200708#allthreads https://crash-stats.mozilla.org/report/index/69fdc1e0-912b-46ec-b278-7af360200708#allthreads https://crash-stats.mozilla.org/report/index/273cc629-48db-4f7f-b817-303970200708#allthreads https://crash-stats.mozilla.org/report/index/2be978fa-1e54-442d-ba98-4051f0200708#allthreads https://crash-stats.mozilla.org/report/index/eee87582-2bdb-4398-bcce-043c80200708#allthreads * slow nsTString<char>::Find in mozilla::net::Http3Stream::FindRequestContentLength() https://crash-stats.mozilla.org/report/index/e74302ff-a5bc-456b-9f62-7456b0200708#allthreads https://crash-stats.mozilla.org/report/index/f4d7beae-1758-4f7b-8ddf-cff990200708#allthreads https://crash-stats.mozilla.org/report/index/94ad5e0f-cdaa-4039-ae04-c680b0200708#allthreads https://crash-stats.mozilla.org/report/index/1546f888-d3c2-48b5-b1c5-c08b30200708#allthreads https://crash-stats.mozilla.org/report/index/5a584924-3e59-45f2-a637-0c6800200708#allthreads https://crash-stats.mozilla.org/report/index/b7159d2d-1570-4ffc-87f5-2b2a60200708#allthreads https://crash-stats.mozilla.org/report/index/05024233-a580-4315-af02-8580d0200708#allthreads * slow nsTString<char>::Find in mozilla::net::Http3Stream::GetHeadersString https://crash-stats.mozilla.org/report/index/2595aabb-e2d5-44c0-9479-bce420200708#allthreads https://crash-stats.mozilla.org/report/index/50b9aaf2-419d-4832-9dfd-368850200708#allthreads https://crash-stats.mozilla.org/report/index/f447bc10-2cf9-4ad2-9ff9-9fe760200708#allthreads * Deadlock with nss (17): Note to myself: search `RtlpWaitOnCriticalSection` for deadlock https://crash-stats.mozilla.org/report/index/0c967088-a97e-4dde-8bbe-e23570200708#allthreads https://crash-stats.mozilla.org/report/index/7bb239d5-c863-4ca8-9ad3-128ac0200708#allthreads https://crash-stats.mozilla.org/report/index/053f0a6b-b7d1-4ce4-bbc6-beb4c0200708#allthreads https://crash-stats.mozilla.org/report/index/763ae655-0a70-4422-97f7-0b86b0200708#allthreads https://crash-stats.mozilla.org/report/index/749aea7f-cfdc-4b49-8373-ed8db0200708#allthreads https://crash-stats.mozilla.org/report/index/6cb693ad-b88c-42e8-afef-5eceb0200708#allthreads https://crash-stats.mozilla.org/report/index/915564b0-e7b7-4ea1-bc55-1d7a20200708#allthreads https://crash-stats.mozilla.org/report/index/f2523d26-c4d4-4922-8231-5c7510200708#allthreads https://crash-stats.mozilla.org/report/index/88299646-ecf5-47e3-adb5-39b6b0200708#allthreads https://crash-stats.mozilla.org/report/index/b1286c47-7a56-4d38-ab7f-4c3530200708#allthreads https://crash-stats.mozilla.org/report/index/ec485865-dedb-4d43-b906-ac6320200708#allthreads https://crash-stats.mozilla.org/report/index/ba379840-e967-4e7c-bb37-962e90200708#allthreads https://crash-stats.mozilla.org/report/index/a1bf9f5c-ebe3-41e7-809b-a38ff0200708#allthreads https://crash-stats.mozilla.org/report/index/c6a3fff4-264e-4ffb-9b7a-2f55c0200708#allthreads https://crash-stats.mozilla.org/report/index/15cc0996-9cdd-47f3-a8cf-5c8500200708#allthreads https://crash-stats.mozilla.org/report/index/208140d4-60a6-403c-ba59-334a30200708#allthreads https://crash-stats.mozilla.org/report/index/62db5f81-1e2a-4eb9-bc8a-2b0780200708#allthreads * fflush hang in ssl https://crash-stats.mozilla.org/report/index/a524b967-5c38-46f9-885b-80b0b0200708#allthreads * Good old poll https://crash-stats.mozilla.org/report/index/dd4b29f7-99eb-4775-ad10-770fc0200708#allthreads https://crash-stats.mozilla.org/report/index/d9b215a1-bdab-4ed4-87df-04ceb0200708#allthreads https://crash-stats.mozilla.org/report/index/bbe33082-cc99-494c-b38c-930b40200708#allthreads https://crash-stats.mozilla.org/report/index/9e64769b-e785-4681-a319-e13320200708#allthreads https://crash-stats.mozilla.org/report/index/0c869f8c-f2e7-4754-8997-b6b9f0200708#allthreads https://crash-stats.mozilla.org/report/index/9198e493-305a-4eeb-86a0-c35020200708#allthreads * Random * Hang in PR_Write https://crash-stats.mozilla.org/report/index/ea70ecc5-30e1-442b-8743-deea60200708#allthreads * doing it's shutdown job. not hanging: https://crash-stats.mozilla.org/report/index/c8f012d0-5121-488a-a6f9-7d5070200708#allthreads * je_realloc https://crash-stats.mozilla.org/report/index/7e0cf42f-852c-4150-a2e0-0ec2d0200708#allthreads