Closed Bug 1253040 Opened 4 years ago Closed 4 years ago

54,200 instances of "NS_ENSURE_SUCCESS(rv, rv) failed with result 0xC1F30001" emitted from netwerk/cache2/CacheFileMetadata.cpp during linux64 debug testing

Categories

(Core :: Networking: Cache, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla48
Tracking Status
firefox47 --- fixed
firefox48 --- fixed

People

(Reporter: erahm, Assigned: michal)

References

(Blocks 1 open bug)

Details

(Whiteboard: [necko-active])

Attachments

(1 file)

This is by far the most prevalent warning during testing. The top offending tests seem to be fetch related.

> 54154 [NNNNN] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0xC1F30001: file netwerk/cache2/CacheFileMetadata.cpp, line 308

This warning [1] shows up in the following test suites:

> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-1-bm53-tests1-linux64-build3.txt:5008
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-1-bm52-tests1-linux64-build7.txt:4986
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-3-bm120-tests1-linux64-build5.txt:3773
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-6-bm115-tests1-linux64-build0.txt:3726
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-6-bm52-tests1-linux64-build2.txt:3722
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-3-bm117-tests1-linux64-build0.txt:3712
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-5-bm122-tests1-linux64-build7.txt:3540
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-5-bm123-tests1-linux64-build10.txt:3498
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-2-bm52-tests1-linux64-build5.txt:2416
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-8-bm67-tests1-linux64-build0.txt:2395
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-2-bm51-tests1-linux64-build2.txt:2283
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-8-bm125-tests1-linux64-build0.txt:2194
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-4-bm117-tests1-linux64-build0.txt:1317
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-4-bm125-tests1-linux64-build1.txt:1310
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-5-bm114-tests1-linux64-build1.txt:1184
> mozilla-central_ubuntu64_vm-debug_test-mochitest-5-bm116-tests1-linux64-build1.txt:1087
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-7-bm52-tests1-linux64-build3.txt:819
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-e10s-7-bm122-tests1-linux64-build0.txt:816
> mozilla-central_ubuntu64_vm-debug_test-mochitest-1-bm121-tests1-linux64-build2.txt:560
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-1-bm51-tests1-linux64-build0.txt:557
> mozilla-central_ubuntu64_vm-debug_test-mochitest-3-bm67-tests1-linux64-build1.txt:509
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-3-bm67-tests1-linux64-build1.txt:476
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-reftests-bm113-tests1-linux64-build0.txt:430
> mozilla-central_ubuntu64_vm-debug_test-web-platform-tests-reftests-e10s-bm114-tests1-linux64-build1.txt:429
> mozilla-central_ubuntu64_vm-debug_test-mochitest-4-bm116-tests1-linux64-build3.txt:425
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-4-bm116-tests1-linux64-build1.txt:239
> mozilla-central_ubuntu64_vm-debug_test-mochitest-8-bm117-tests1-linux64-build2.txt:218
> mozilla-central_ubuntu64_vm-debug_test-mochitest-2-bm125-tests1-linux64-build0.txt:206
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-8-bm122-tests1-linux64-build4.txt:199
> mozilla-central_ubuntu64_vm-debug_test-mochitest-other-bm67-tests1-linux64-build1.txt:193
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-2-bm51-tests1-linux64-build1.txt:171
> mozilla-central_ubuntu64_vm-debug_test-mochitest-7-bm124-tests1-linux64-build3.txt:153
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-4-bm67-tests1-linux64-build1.txt:136
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-7-bm123-tests1-linux64-build1.txt:131
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-7-bm122-tests1-linux64-build1.txt:123
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-1-bm67-tests1-linux64-build0.txt:120
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-3-bm121-tests1-linux64-build2.txt:119
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-6-bm51-tests1-linux64-build1.txt:97
> mozilla-central_ubuntu64_vm-debug_test-mochitest-6-bm54-tests1-linux64-build0.txt:95
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-6-bm118-tests1-linux64-build0.txt:84
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-5-bm114-tests1-linux64-build0.txt:66
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-5-bm116-tests1-linux64-build9.txt:64
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-6-bm67-tests1-linux64-build2.txt:63
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-4-bm120-tests1-linux64-build1.txt:63
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-1-bm52-tests1-linux64-build8.txt:54
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-5-bm52-tests1-linux64-build5.txt:47
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-6-bm114-tests1-linux64-build1.txt:43
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-3-bm67-tests1-linux64-build0.txt:43
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-7-bm54-tests1-linux64-build6.txt:42
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-4-bm115-tests1-linux64-build0.txt:42
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-8-bm68-tests1-linux64-build0.txt:36
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-3-bm54-tests1-linux64-build0.txt:33
> mozilla-central_ubuntu64_vm-debug_test-mochitest-e10s-browser-chrome-2-bm67-tests1-linux64-build0.txt:19
> mozilla-central_ubuntu64_vm-debug_test-mochitest-browser-chrome-2-bm51-tests1-linux64-build0.txt:19
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-2-bm67-tests1-linux64-build1.txt:11
> mozilla-central_ubuntu64_vm-debug_test-reftest-e10s-2-bm51-tests1-linux64-build0.txt:9
> mozilla-central_ubuntu64_vm-debug_test-mochitest-jetpack-bm114-tests1-linux64-build0.txt:7
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-7-bm51-tests1-linux64-build0.txt:7
> mozilla-central_ubuntu64_vm-debug_test-reftest-e10s-1-bm116-tests1-linux64-build1.txt:5
> mozilla-central_ubuntu64_vm-debug_test-reftest-3-bm52-tests1-linux64-build2.txt:5
> mozilla-central_ubuntu64_vm-debug_test-reftest-2-bm52-tests1-linux64-build1.txt:5
> mozilla-central_ubuntu64_vm-debug_test-mochitest-devtools-chrome-1-bm116-tests1-linux64-build1.txt:5
> mozilla-central_ubuntu64_vm-debug_test-crashtest-bm120-tests1-linux64-build1.txt:5
> mozilla-central_ubuntu64_vm-debug_test-mochitest-gl-bm114-tests1-linux64-build0.txt:3
> mozilla-central_ubuntu64_vm-debug_test-crashtest-e10s-bm51-tests1-linux64-build1.txt:3
> mozilla-central_ubuntu64_vm-debug_test-mochitest-push-bm67-tests1-linux64-build0.txt:2
> mozilla-central_ubuntu64_vm-debug_test-reftest-1-bm51-tests1-linux64-build0.txt:1

It shows up in 6756 tests. A few of the most prevalent:

> 636 - dom/tests/mochitest/fetch/test_fetch_cors_sw_empty_reroute.html
> 619 - dom/tests/mochitest/fetch/test_fetch_cors_sw_reroute.html
> 378 - dom/tests/mochitest/fetch/test_fetch_cors.html
> 349 - dom/security/test/cors/test_CrossSiteXHR_origin.html
> 282 - /_mozilla/service-workers/service-worker/fetch-event-redirect.https.html
> 272 - dom/base/test/test_bug682592.html
> 224 - /fetch/api/redirect/redirect-location.html
> 166 - /fetch/api/redirect/redirect-location-worker.html
> 153 - toolkit/mozapps/extensions/test/browser/browser_installssl.js
> 153 - /html/infrastructure/urls/resolving-urls/query-encoding/utf-16le.html

[1] https://hg.mozilla.org/mozilla-central/annotate/c59c022943f6/netwerk/cache2/CacheFileMetadata.cpp#l308
Blocks: logspam
Whiteboard: [necko-would-take]
Again, these tests are validating how we handle failure conditions.  I hesitate to remove warnings here since they are a valid signal under normal conditions.
I think these warnings started to appear due to changes in cache shutdown logic. I'll check it and eventually change the code so it won't bloat the console.
Assignee: nobody → michal.novotny
Whiteboard: [necko-would-take] → [necko-active]
This continues to be by far the most verbose warning during testing responsible for 30% of all warnings emitted. Michal did you get a chance to follow up on this?
Flags: needinfo?(michal.novotny)
Attached patch fixSplinter Review
Flags: needinfo?(michal.novotny)
Attachment #8739791 - Flags: review?(honzab.moz)
Comment on attachment 8739791 [details] [diff] [review]
fix

Review of attachment 8739791 [details] [diff] [review]:
-----------------------------------------------------------------

Yes!  Thanks.
Attachment #8739791 - Flags: review?(honzab.moz) → review+
https://hg.mozilla.org/mozilla-central/rev/c6f69694a1f9
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Comment on attachment 8739791 [details] [diff] [review]
fix

Approval Request Comment
[Feature/regressing bug #]: 1247432
[User impact if declined]: this may in some cases cause an unnecessary I/O on files that are no longer used and should be just removed, mainly visible during shutdown time
[Describe test coverage new/current, TreeHerder]: currently on m-a, heavily used code
[Risks and why]: zero, the patch is very simple
[String/UUID change made/needed]: none

Other reason I would like this be uplifted is to have it in the same version as the regression bug has landed.  That bug introduced a certain optimization feature and this small patch should have been part of it.

We are currently hunting HTTP cache shutdown hangs, and this small change could at least remove doubts it's causing issues when missing on 47.

Thanks.
Attachment #8739791 - Flags: approval-mozilla-beta?
Hello Jason, I need a second opinion on whether this fix needs to be uplifted to Beta47 or not. I am about to gtb 47.0b9 in an hour so a prompt reply is appreciated. At this point in Beta47 cycle, I am only taking fixes for critical recent regressions, severe stability and sec issues. I don't see this patch helping the stability situation. Or will it? Thanks!
Flags: needinfo?(jduell.mcbugs)
ritu - regressed on 47, should fix on 47.. that's a good fix and honza is the best suited engineer to make the call and he backs it.
(In reply to Patrick McManus [:mcmanus] from comment #10)
> ritu - regressed on 47, should fix on 47.. that's a good fix and honza is
> the best suited engineer to make the call and he backs it.

Ok Patrick. I trust your judgement call here. I'll take it. At best it should help, at worst hopefully it does no harm (fingers crossed).
Comment on attachment 8739791 [details] [diff] [review]
fix

This is a simple fix which should (at best) help with HTTP cache shutdown hangs we've seen on Fx47, Beta47+
Attachment #8739791 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Flags: needinfo?(jduell.mcbugs)
You need to log in before you can comment on or make changes to this bug.